Threat Feeds data retrieval patterns

This page shows you how to:

• Choose between the Download API and Feed API for bulk data retrieval
• Retrieve a full day of feed data using either API
• Recover from ingestion gaps without losing your place in the stream

These patterns apply to any Threat Feed that supports the Real-time Feed API and Real-time Download API. Examples on this page use the Newly Observed Domains (NOD) feed.

Before you begin

• Obtain your API credentials from your DomainTools account dashboard.
• Confirm that your account has access to the feed you want to retrieve.
• Review session management to understand how sessionID (a persistent cursor that tracks your position in the feed), HTTP 206, and HTTP 200 responses work.

Which API to use

Scenario	Recommended API	Why
Backfill a full day or more	Download API	Files are pre-built hourly snapshots. No session state to manage, and you avoid loading the real-time stream.
Retrieve historical data with filters	Feed API	Supports query parameters like domain, risk score thresholds, and time windows that the Download API doesn't offer.
Catch up after a short outage	Feed API	Your existing sessionID already points to where you left off.

Retrieve a full day from the Download API

The Download API provides hourly snapshot files, already serialized and organized by date. This is the simplest way to retrieve a full day of data.

1. Request the file list for a specific day. Each hour produces a data file and a checksum file, so set limit=48 to cover 24 hours:

   curl -H 'X-Api-Key: YOUR_API_KEY' \
     'https://api.domaintools.com/v1/download/nod/?limit=48' > files.json
   

2. Download the hourly files. Each .json.gz file contains one hour of feed data in the same NDJSON format as the Feed API:

   for url in $(jq -r '.response.files[].url' files.json | grep '\.json\.gz$'); do
     curl -O "$url"
   done
   

3. Verify file integrity using the .sha256 checksum files:

   for url in $(jq -r '.response.files[].url' files.json | grep '\.sha256$'); do
     curl -O "$url"
   done
   sha256sum -c *.sha256
   

The Download API retains 90 days of hourly files. If a request fails with HTTP 403, verify your API credentials and feed access. For details on response structure and file naming, see the Download API section of your feed's documentation (for example, Domain Hotlist Download API).

Retrieve a full day from the Feed API

Use this approach when you need the Feed API's filtering capabilities or when you don't have access to the Download API for your feed.

1. Start a new session with after=-86400 (24 hours ago) and fromBeginning=true. This returns the first hour of data within that window:

   curl -H 'X-Api-Key: YOUR_API_KEY' \
     'https://api.domaintools.com/v1/feed/nod/?sessionID=backfill-2025-01-06&after=-86400&fromBeginning=true'
   

   The API responds with HTTP 206, indicating more data is available. The sessionID now points to the end of that first batch.

2. Drop the after and fromBeginning parameters. Continue calling the Feed API with only the sessionID:

   Warning

   Don't pass fromBeginning=true on subsequent calls. The API returns an error if fromBeginning is used with an existing session ID.

   curl -H 'X-Api-Key: YOUR_API_KEY' \
     'https://api.domaintools.com/v1/feed/nod/?sessionID=backfill-2025-01-06'
   

3. Repeat step 2 while the API returns HTTP 206. When you receive HTTP 200, you've retrieved all available data.

Note

Each request can return up to 10 million results. For high-volume feeds, a full day may require many iterations. Respect the rate limits of 2 queries per minute and 120 queries per hour.

Recover from ingestion lag

If your ingestion job goes down or falls behind, you don't need to start over. The Feed API retains your session position for up to 5 days. Your sessionID continues to point to the last record delivered to you.

To resume:

1. Restart your consumer with the same sessionID it was using before the outage.

2. Call the Feed API as you normally would — no extra parameters needed:

   curl -H 'X-Api-Key: YOUR_API_KEY' \
     'https://api.domaintools.com/v1/feed/nod/?sessionID=mySOC'
   

3. The API returns all data accumulated since your last successful request. Process the 206 responses as usual until you receive a 200.

If your outage exceeds 5 days, the session position expires. Create a new sessionID and use the full-day retrieval workflow or the Download API to backfill the gap.

Related resources

• Threat Feeds overview — feed descriptions and access method matrix
• Session management — how sessionID, HTTP 206, and session deletion work
• Individual feed documentation for API-specific parameters and response fields