Domain Discovery

The Domain Discovery feed is the largest feed of its kind, containing a daily list of all newly registered and newly observed domains from all TLDs, including those that do not publish zone files. On average, this feed contains nearly 350,000 new domains each day, making it a comprehensive source for any workflow that needs to track the creation of new domains.

Overview

Domains are apex-level (for example, example.com but not www.example.com), and the feed provides comprehensive coverage of the global domain landscape.

Use this feed when you need to:

• Comprehensive domain monitoring for a wide range of security and intelligence applications
• Track the global landscape of new domain registrations and observations
• Identify emerging threats and trends related to new domain creation
• Build comprehensive domain databases for research and analysis

Inclusion criteria: All newly registered and newly observed domains from all TLDs, including TLDs that do not publish zone files.

Requirements

You need the following to access Threat Feeds:

• An Enterprise Account with DomainTools, accessible at https://account.domaintools.com/my-account/
• Authentication credentials (API key for header authentication, or API username and key for HMAC or open key authentication)
• A way to interact with a REST API delivered through AWS

Obtain your API credentials from your group’s API administrator. API administrators can manage their API keys at https://research.domaintools.com, selecting the drop-down account menu and choosing API admin.

For assistance, contact enterprisesupport@domaintools.com.

Authentication

You can authenticate to the Domain Discovery APIs using three different methods. Choose the method that best fits your security requirements and technical environment.

API key (header) authentication

Authenticate your requests by including the API key in the header of each HTTP request. The API key serves as a unique identifier and authenticates your requests.

Required header:

X-Api-Key: $DOMAINTOOLS_API_KEY

Examples:

# Feed API request
curl -H "X-Api-Key: $DOMAINTOOLS_API_KEY" \
  'https://api.domaintools.com/v1/feed/domaindiscovery/?sessionID=myDomainMonitor'

# Download API request
curl -H "X-Api-Key: $DOMAINTOOLS_API_KEY" \
  'https://api.domaintools.com/v1/download/domaindiscovery/'

HMAC authentication

HMAC authentication is a secure alternative to API key-based methods. It requires signing each request with an HMAC digest derived from your API key, providing integrity and authenticity without exposing credentials directly in the request.

This method is recommended for systems where authentication credentials shouldn’t be stored in plain text or included directly in request URLs.

DomainTools supports MD5, SHA1, and SHA256 for the hashing algorithm. Use SHA256 — it’s the recommended choice and is more resistant to collision attacks than MD5 or SHA1.

Required query parameters:

• api_username: Your DomainTools API username
• signature: HMAC-SHA256 signature of api_username + timestamp + uri_path
• timestamp: Current UTC timestamp in ISO 8601 format (for example, 2025-06-01T15:30:00Z)

Constructing the HMAC signature:

signature = HMAC-SHA256(api_key, api_username + timestamp + uri_path)

URI path must include API version

The uri_path parameter must include the API version prefix. For example, use /v1/feed/nod/ not /feed/nod/.

Example Python signing function:

import hmac
import hashlib

def sign(api_username, api_key, timestamp, uri):
    params = f"{api_username}{timestamp}{uri}"
    return hmac.new(api_key.encode("utf-8"), params.encode("utf-8"), hashlib.sha256).hexdigest()

HMAC timestamp requirements

The timestamp parameter in HMAC authentication must be current (within a few minutes of the server time). The timestamps shown in these examples are static for demonstration purposes. In production, generate a fresh timestamp for each request using your system’s current time in ISO 8601 UTC format (e.g., 2025-01-06T15:30:00Z).

Examples:

# Feed API request with HMAC
curl 'https://api.domaintools.com/v1/feed/domaindiscovery/?api_username=YOUR_USERNAME&signature=HMAC_SIGNATURE&timestamp=2025-01-06T15:30:00Z&sessionID=myDomainMonitor'

# Download API request with HMAC
curl 'https://api.domaintools.com/v1/download/domaindiscovery/?api_username=YOUR_USERNAME&signature=HMAC_SIGNATURE&timestamp=2025-01-06T15:30:00Z'

Open key authentication

This is the easiest authentication scheme to implement, but also the least secure. Each request contains the full API key and API username as query parameters. We recommend using API key header authentication or HMAC authentication instead.

If you’re unsure about your authentication options, contact enterprisesupport@domaintools.com.

Required query parameters:

• api_username: Your API username
• api_key: Your API key

Examples:

# Feed API request
curl 'https://api.domaintools.com/v1/feed/domaindiscovery/?api_username=YOUR_USERNAME&api_key=YOUR_API_KEY&sessionID=myDomainMonitor'

# Download API request
curl 'https://api.domaintools.com/v1/download/domaindiscovery/?api_username=YOUR_USERNAME&api_key=YOUR_API_KEY'

Real-time Feed API

The Feed API provides real-time access to current Domain Discovery data. Use this API to poll for the latest feed updates at regular intervals, maintain a session to track your position in the feed, and filter results based on your specific needs. Due to the high volume of this feed (~350,000 domains per day), consider using appropriate filtering and polling strategies.

Base URL

https://api.domaintools.com/v1/feed/domaindiscovery/

Rate limits

Real-time feeds have the following rate limits:

• 2 queries per minute
• 120 queries per hour

If you exceed these limits, the API returns an error.

Response formats

The API supports two response formats:

NDJSON (Newline-Delimited JSON)

• Default format when no Accept header is specified
• Also known as JSON Lines (JSONL)
• One JSON object per line
• Efficient for streaming and processing large datasets
• Set Accept: application/x-ndjson to explicitly request this format

CSV (Comma-Separated Values)

• Set Accept: text/csv to request CSV format
• Add &headers=1 to the query parameters to include column headers as the first line
• Not available for all feeds; check the specific feed documentation for CSV support

Session management

Session management allows you to maintain your position in the feed data stream, ensuring you don’t miss or duplicate events when polling the API.

How sessions work:

• Start a new session: Provide a unique sessionID parameter of your choosing. By default, the API returns the past hour of results.
• Resume a session: Use the same sessionID in subsequent requests. The API returns all data since your last request.
• Handle large result sets: If a single request exceeds 10M results, the API returns an HTTP 206 response code. Repeat the same request with the same sessionID to receive the next batch of data until you receive an HTTP 200 response code.
• One request at a time: Do not send simultaneous requests with the same sessionID for the same feed. Wait for each request to complete before sending the next one. Concurrent requests with the same sessionID can produce errors or incomplete results.
• Delete a session: Use an HTTP DELETE request with your sessionID to clear the saved offset and start fresh.

Session ID requirements:

• 1 to 64 characters in length
• Alphanumeric characters and hyphens only ([a-zA-Z0-9-]+)
• Case-sensitive

Quick start

The standard access pattern is to periodically request the most recent feed data, as often as every 60 seconds.

curl -H "X-Api-Key: $DOMAINTOOLS_API_KEY" \
  'https://api.domaintools.com/v1/feed/domaindiscovery/?sessionID=myDomainMonitor'

This starts a new session and returns the last hour of data. Subsequent calls with the same sessionID return data since the last request.

Feed API parameters

sessionID

Type: String

Valid values: 1-64 alphanumeric characters and hyphens ([a-zA-Z0-9-]+)

Description: A unique identifier for the session, used for resuming data retrieval from the last point. Use a new sessionID to begin a new session, fetching the most recent hour by default. Reuse the same sessionID to return all feed data since your last request. If omitted, time window parameters (such as after/before) are required.

Example: sessionID=mySOC

Required: Yes, to continue where you left off (or use after/before instead)

after

Type: Integer or string

Valid values:

• Integer: -1 to -432,000 (relative seconds before current time)
• String: ISO 8601 datetime in UTC format (YYYY-MM-DDTHH:MM:SSZ)

Description: The start of the query window (inclusive). When using an integer, the value is in seconds relative to the current time. When using a string, provide an absolute timestamp. The query window covers the most recent 5 days. A value older than 5 days is capped to the start of that window, and the response begins 5 days back.

Example: after=-60 or after=2024-10-16T10:20:00Z

Required: Yes, if before or sessionID not provided

before

Type: Integer or string

Valid values:

• Integer: -1 to -432,000 (relative seconds before current time)
• String: ISO 8601 datetime in UTC format (YYYY-MM-DDTHH:MM:SSZ)

Description: The end of the query window (inclusive). When using an integer, the value is in seconds relative to the current time. When using a string, provide an absolute timestamp. The query window covers the most recent 5 days. A value older than 5 days places the window entirely outside it, and the response contains no records.

Example: before=-120 or before=2024-10-16T10:20:00Z

Required: Yes, if after or sessionID not provided

domain

Type: String

Valid values: Domain character set restricted by the DNS specification (letters, digits, hyphens). International characters should be specified in punycode. A trailing dot is acceptable.

Description: Filter for an exact domain or a domain substring by prefixing or suffixing your string with *. Multiple parameters are supported (for example, ?domain=*apple*&domain=*microsoft*). The URL-encoded version of * (%2A) may be required in some clients.

Example: domain=*bank* or domain=example.com

Required: No

fromBeginning

Type: Boolean

Valid values: true, false, 1, 0

Description: Requires a sessionID. Set fromBeginning=true on the first request with a new session ID to return the first hour of data in the time window rather than the last. false is the default. Using it with a session ID that already exists returns HTTP 406, so drop fromBeginning from subsequent requests. Using it without a sessionID, or passing a value that isn’t a boolean, returns HTTP 422.

Example: fromBeginning=true

Required: No

top

Type: Integer

Valid values: Positive integer, 1-1,000,000,000

Description: Limits the number of results in the response payload. Primarily intended for testing.

Example: top=10

Required: No

headers

Type: Boolean

Valid values: true, false, 1, 0

Description: Adds a header row as the first line of the response when text/csv is requested. Set headers=1 to enable; false is the default. Enabling it on a request that doesn’t ask for text/csv returns HTTP 422, as does a value that isn’t a boolean.

Example: headers=1

Required: No

Feed API response structure

The API returns responses in NDJSON (Newline-Delimited JSON), with each response containing one domain entry per line. Each entry contains a timestamp in ISO 8601 UTC format, and the domain.

Response fields:

timestamp (string): The observation timestamp in ISO 8601 UTC format.

• Example: "timestamp":"2024-11-15T16:14:39Z"

domain (string): The domain name without the trailing dot. Domain character set restricted by the DNS specification (letters, digits, hyphens).

• Example: "domain":"example.com"

Example NDJSON response:

{"timestamp":"2024-11-15T16:14:39Z","domain":"domiantools.com"}
{"timestamp":"2024-11-15T16:14:38Z","domain":"domsintools.com"}
{"timestamp":"2024-11-15T16:14:36Z","domain":"edomaintools.com"}
{"timestamp":"2024-11-15T16:14:35Z","domain":"omaintools.com"}
{"timestamp":"2024-11-15T16:14:35Z","domain":"v-domaintools.com"}

Example CSV response:

timestamp,domain
2024-11-15T16:14:39Z,domiantools.com
2024-11-15T16:14:38Z,domsintools.com
2024-11-15T16:14:36Z,edomaintools.com

Feed API response codes

Code	Status	Description
200	OK	The request was successful and all data has been delivered
206	Partial content	The request was successful, but only a portion of the data was returned. The request exceeded 10M results or the 1-hour evaluation window. Repeat the same request with the same sessionID to receive the next batch of data until you receive an HTTP 200 response
400	Bad request	The request is malformed
403	Forbidden	Missing or invalid API credentials
404	Not found	The requested resource (such as a sessionID) doesn’t exist
406	Not acceptable	Either the specified Accept header value isn’t supported (only application/x-ndjson and text/csv are accepted), or fromBeginning was used with a sessionID that already exists
422	Unprocessable entity	The request is syntactically valid but violates semantic or domain-specific rules (for example, invalid query parameter values)

Feed API examples

Basic session polling:

# Start a new session
curl -H "X-Api-Key: $DOMAINTOOLS_API_KEY" \
  'https://api.domaintools.com/v1/feed/domaindiscovery/?sessionID=myDomainMonitor'

# Resume the session (returns data since last request)
curl -H "X-Api-Key: $DOMAINTOOLS_API_KEY" \
  'https://api.domaintools.com/v1/feed/domaindiscovery/?sessionID=myDomainMonitor'

Time window filtering:

# Get data from a specific time range
curl -H "X-Api-Key: $DOMAINTOOLS_API_KEY" \
  'https://api.domaintools.com/v1/feed/domaindiscovery/?after=2025-01-06T10:00:00Z&before=2025-01-06T11:00:00Z'

Domain filtering:

# Filter for specific domain patterns (useful for monitoring specific TLDs or patterns)
curl -H "X-Api-Key: $DOMAINTOOLS_API_KEY" \
  'https://api.domaintools.com/v1/feed/domaindiscovery/?domain=*.io&sessionID=myDomainMonitor'

CSV format:

# Request CSV format with headers
curl -H 'Accept: text/csv' -H "X-Api-Key: $DOMAINTOOLS_API_KEY" \
  'https://api.domaintools.com/v1/feed/domaindiscovery/?headers=1&sessionID=myDomainMonitor'

# Request CSV format without headers
curl -H 'Accept: text/csv' -H "X-Api-Key: $DOMAINTOOLS_API_KEY" \
  'https://api.domaintools.com/v1/feed/domaindiscovery/?sessionID=myDomainMonitor'

Handling large result sets:

# If you receive HTTP 206, repeat the request to get the next batch
curl -H "X-Api-Key: $DOMAINTOOLS_API_KEY" \
  'https://api.domaintools.com/v1/feed/domaindiscovery/?sessionID=myDomainMonitor'

# Repeat until you receive HTTP 200
curl -H "X-Api-Key: $DOMAINTOOLS_API_KEY" \
  'https://api.domaintools.com/v1/feed/domaindiscovery/?sessionID=myDomainMonitor'

Delete a session:

# Clear the saved offset and start fresh
curl -X DELETE -H "X-Api-Key: $DOMAINTOOLS_API_KEY" \
  'https://api.domaintools.com/v1/feed/domaindiscovery/?sessionID=myDomainMonitor'

Real-time Download API

The Real-time Download API provides access to historical Domain Discovery data through temporary AWS S3 file links. Use this API to retrieve archived data you may have missed or to backfill your systems with historical information. Files are organized by hour and available for 90 days.

Base URL

https://api.domaintools.com/v1/download/domaindiscovery/

Download API parameters

limit

Type: Integer

Valid values: Positive integer

Description: Limits the number of files returned in the response, starting from the most recent. Use to control payload size or test specific cases.

Example: limit=10

Required: No

Download API response structure

The API returns a JSON response containing an array of downloadable files. Each file entry includes:

download_name (string): The feed identifier (domaindiscovery)

files (array): List of downloadable file entries

Each file object contains:

• name (string): Path and filename of the downloadable file
• last_modified (string): Timestamp of last modification in ISO 8601 UTC format
• etag (string): ETag (hash) used to verify file identity and versioning
• size (integer): File size in bytes
• url (string): Temporary signed URL to download the file from AWS

File naming convention:

• Data file: domaindiscovery/{YYYY-MM-DD}/domaindiscovery-{YYYYMMDD}.{HH00}-{HH00}.json.gz
• Checksum file: domaindiscovery/{YYYY-MM-DD}/domaindiscovery-{YYYYMMDD}.{HH00}-{HH00}.json.gz.sha256

Example response:

{
  "response": {
    "download_name": "domaindiscovery",
    "files": [
      {
        "name": "domaindiscovery/2024-11-19/domaindiscovery-20241119.1900-2000.json.gz.sha256",
        "last_modified": "2024-11-19T20:00:11+00:00",
        "etag": "\"67a6d9b0973b2d31ffb779dc8f7f8cfa\"",
        "size": 64,
        "url": "https://download.example.com/domaindiscovery/2024-11-19/domaindiscovery-20241119.1900-2000.json.gz.sha256?Expires=..."
      },
      {
        "name": "domaindiscovery/2024-11-19/domaindiscovery-20241119.1900-2000.json.gz",
        "last_modified": "2024-11-19T20:00:11+00:00",
        "etag": "\"67a6d9b0973b2d31ffb779dc8f7f8cfa\"",
        "size": 2450000,
        "url": "https://download.example.com/domaindiscovery/2024-11-19/domaindiscovery-20241119.1900-2000.json.gz?Expires=..."
      }
    ]
  }
}

Download API response codes

Code	Status	Description
200	OK	The request was successful
400	Bad request	The request is malformed
401	Unauthorized	Missing or invalid API credentials
403	Forbidden	Missing or invalid API credentials
404	Not found	No data to download
422	Unprocessable entity	The request is syntactically valid but violates semantic or domain-specific rules (for example, invalid query parameter values)

Download API file contents

The *.json.gz.sha256 file is a checksum containing a SHA-256 hash value used to verify the integrity of the downloaded file.

The *.json.gz file, when uncompressed, contains JSON data in the same format as the Feed API response (NDJSON with timestamp and domain fields).

Download API examples

List available files:

# Get the most recent files
curl -H "X-Api-Key: $DOMAINTOOLS_API_KEY" \
  'https://api.domaintools.com/v1/download/domaindiscovery/?limit=10'

Download and verify a file:

# Get the file list
curl -H "X-Api-Key: $DOMAINTOOLS_API_KEY" \
  'https://api.domaintools.com/v1/download/domaindiscovery/?limit=2' > files.json

# Extract the URL and download the data file
curl -o domaindiscovery-data.json.gz "$(jq -r '.response.files[1].url' files.json)"

# Download the checksum file
curl -o domaindiscovery-data.json.gz.sha256 "$(jq -r '.response.files[0].url' files.json)"

# Verify the integrity
sha256sum -c domaindiscovery-data.json.gz.sha256

Batch processing:

# Download multiple files in a loop
for url in $(curl -H "X-Api-Key: $DOMAINTOOLS_API_KEY" \
  'https://api.domaintools.com/v1/download/domaindiscovery/?limit=24' | \
  jq -r '.response.files[].url' | grep '\.json\.gz$'); do
  curl -O "$url"
done

Daily Download API

The Daily Download API provides daily batch summaries as an alternative to hourly real-time data. Use this when you need daily aggregated data rather than real-time updates.

Overview

Daily feed of newly registered and newly observed domains.

Inclusion threshold: Newly observed or registered

Format: Gzip-compressed CSV file; one domain name per line

Size: ~375,000 domains, ~2.5MB compressed

Base URL

https://api.domaintools.com/v1/download/daily_domain_discovery/

Daily Download parameters

The Daily Download API supports standard download parameters:

api_username

Type: string (required for HMAC and open key auth)

Your DomainTools API username

api_key

Type: string (required for open key auth)

Your DomainTools API key

signature

Type: string (required for HMAC auth)

HMAC signature of your request

timestamp

Type: string (required for HMAC auth)

Current timestamp for HMAC authentication in ISO 8601 format

limit

Type: integer (optional)

Limit the list of signed files. Ordering of files is always descending, so the latest files are first.

page

Type: integer (optional)

Select which page of results are returned. Pages begin at 0 with latest results.

prefix

Type: string (optional)

Filter results by date using the file prefix (format: NEW_DOMAINS_YYYYMMDD).

Daily Download response structure

The API returns a JSON response with signed URLs for downloadable files:

download_name (string): The feed identifier (daily_domain_discovery)

files (array): List of downloadable file entries

Each file object contains:

• name (string): File path (e.g., domain_discovery/NEW_DOMAINS_20240704.csv.gz)
• last_modified (string): Last modified date in ISO 8601 format
• etag (string): Entity tag (hash of the file)
• size (integer): Size in bytes
• url (string): Signed AWS download URL (valid for 12 hours)

Daily Download response codes

200: OK - The request was successful

400: Bad request

401: Unauthorized

403: Forbidden

404: No data to download

Daily Download file naming

Daily files follow this naming pattern:

domain_discovery/NEW_DOMAINS_YYYYMMDD.csv.gz

Example: domain_discovery/NEW_DOMAINS_20240704.csv.gz

File contents

The CSV file contains one domain name per line (no header, no timestamp field):

example.com
example.net
example.org

Daily Download examples

List available files:

curl -H "X-Api-Key: $DOMAINTOOLS_API_KEY" \
  'https://api.domaintools.com/v1/download/daily_domain_discovery/'

Filter by date prefix:

curl -H "X-Api-Key: $DOMAINTOOLS_API_KEY" \
  'https://api.domaintools.com/v1/download/daily_domain_discovery/?prefix=NEW_DOMAINS_20240704'

Download a specific file:

# Get the file list
curl -H "X-Api-Key: $DOMAINTOOLS_API_KEY" \
  'https://api.domaintools.com/v1/download/daily_domain_discovery/?limit=1' > files.json

# Download the file
curl -o daily-domains.csv.gz "$(jq -r '.response.files[0].url' files.json)"

# Decompress and view
gunzip daily-domains.csv.gz
head daily-domains.csv

Related resources

• Newly Observed Domains feed
• Newly Active Domains feed
• Domain Risk Score user guide