Parsed Domain RDAP: Frequently Asked Questions¶

Is the RDAP Feed API still in a Beta testing phase?¶

...And if yes, should we expect it to affect our implementation in any way?

No, the Domain RDAP feed is already generally available (GA).

...How many records can we expect to fetch in a timeframe of 10 minutes?

Sampling a random 10-minute period, we see ~25K records / 225MB of data. This averages to ~9KB per Domain RDAP record. Any given 10-minute period could have more or less data, depending upon what’s happening in the wild (domain registrations, DNS resolution activity, etc).

If you fall behind in fetching real-time data, your integration complexity will increase slightly, requiring your job to paginate through results. Fetching Domain RDAP every 10 minutes gives a safe volume margin before requiring pagination.

If you do not need RDAP data in real-time, you can download hourly archive files using the Download API instead.

Is there a limit for how many requests per second we can send?¶

...Is there a quota of total requests we can issue in a particular timeframe (e.g. per month)?

At a steady-state, we expect a maximum query every 60 seconds (once per minute). Your quota will accommodate this, plus a little extra for testing and ingestion job recovery on your end.

Are the RDAP domain records overwritten on subsequent changes?¶

...For example, if we make a request to fetch the data with a lookback time of 10 minutes and a domain was updated 2 times in this timeframe, do we get 2 RDAP entries for that domain ? or only 1 that represents its last state?

Our Domain RDAP feed would contain multiple entries for the same domain.

Are the API Key and Secret the same for Feed APIs and Download API?¶

Yes.

Will you provide the API Header key and the API Key and Secret for the APIs?¶

...What about the license/access to the API?

Yes, we will provide the API Key.

Your current subscription to the Parsed Whois Domain Feed entitles you to access the Domain RDAP feed.

In the API it is specified we can send a session ID on the first query, is there an endpoint that provides this sessionID or where do we get it from?¶

No, there is no API endpoint to manage the sessionID directly. You control it from your end by passing the value.

You can delete the sessionID, by calling the delete method to the same API endpoint to delete the marker on our system.

Does the sessionID have an expiration time?¶

Currently, no, but in the future, we plan to expire unused sessionIDs after 5 days (the current data retention time in the Feed API).

What happens if I use the sessionID parameter and the after parameter in the same query and it’s not the first?¶

...Will it give me all results in the specified timeframe? Or will it give me the results of the specified timeframe minus the ones that were already fetched with the current sessionID.

We no longer recommend both the sessionID and the after parameter, including on the first request.

Now, you can supply just the sessionID, and it will “tail” the feed starting with the last hour of data, or 10M records, whichever is smaller. (In the case of the Domain RDAP feed, it should be 1 hour, since an hour should have \~150K records.)

The sessionID maintains a pointer/offset to the last record delivered. If you supply both the sessionID and after parameters, it will fetch only new results in that time window (i.e., not including results already delivered using that sessionID).

The intent of the sessionID parameter is to eliminate both duplicates and missing records at micro-batch boundaries.

Is there a possibility for the request to time out because the data volume is too large for the specified interval?¶

...E.g., there were too many domains updated/created in the last 10mins. Is there a pagination mechanism for this case?

Using the Feed API, timeouts should not occur, and you should not need to paginate, if you follow our recommendations of fetching at least every 10 minutes.

There is a small chance that the first fetch using sessionID alone (which will by-default fetch the most recent hour of data in the feed) could timeout. It is rare, and only happens if a particular hour has an unusually high volume of records. If you do get a timeout with the first fetch, you may add the after parameter to reduce the time window for the first tranche of data.

If you attempt to fetch >1 hour of data or >10M records (and assuming it does not timeout as cautioned), you will receive an HTTP 206 status code, indicating that you received partial results. To paginate through the remaining results, you can simply reissue the same query with the same sessionID parameter until you receive an HTTP 200 status code, at which point you will know you received all available data.

What does `first_request_timestamp` field (the one present in the raw_record) mean?¶

...What about each timestamp in the raw_record.requests array field?

first_request_timestamp: this value indicates when DomainTools made the request to the registry.

timestamp: this indicates when DomainTools actually got the data.

How long are the signed URLs from the Download API response available?¶

Hourly archive files are available for 90 days, and a signed URL for any given file is valid for 12 hours. In the event a signed URL for a given file expires, you can fetch a new signed URL from the Download API.

What is the average size we should expect for the DownloadAPI response archives?¶

Average file size is: 160MB for 1 hour of data.