title: DomainTools Farsight DNSDB API service_id: dnsdb auth_types: API Key (X-API-Key header) source_url: https://docs.domaintools.com/api/dnsdb/ api_specification: https://docs.domaintools.com/api/dnsdb/dnsdb-openapi.yml ## Key Terms Before diving in, understand these core concepts: - RRset (Resource Record Set): A collection of DNS records with the same owner name and record type - Rdata (Resource Data): The right-hand side of a DNS record containing the actual data (e.g., IP addresses, hostnames) - Bailiwick: The DNS zone context where an RRset was observed, representing the closest enclosing delegated zone - SAF (Streaming API Framing): The protocol used to wrap API responses with status indicators - JSONL: Newline-delimited JSON format used for API responses ## Document Structure This reference covers: 1. Getting Started: Authentication, rate limits, HTTP responses, quick start examples 2. Core Concepts: User guide, SAF protocol, summarize requests 3. Query Methods: RRset lookups (forward), Rdata lookups (inverse) 4. Query Parameters: Time-fencing, limits, offsets, and other options 5. Advanced Features: Flexible search with regular expressions and globs 6. CLI Tools: dnsdbq and dnsdbflex command-line interfaces 7. Search Syntax: Glob patterns and FCRE (Farsight Compatible Regular Expressions) 8. Reference: GitHub repositories and additional resources --- # Farsight DNSDB API Build our world-class domain intelligence into your own tools. This guide will help you get started with the DNSDB API v2. ## What is DNSDB? DNSDB is a database that stores and indexes both the passive DNS data available via Farsight Security's Security Information Exchange as well as the authoritative DNS data that various zone operators make available. DNSDB makes it easy to search for individual DNS RRsets and provides additional metadata for search results such as first seen and last seen timestamps as well as the DNS bailiwick associated with an RRset. ## API Versions There are two versions of the API: - API v1: Legacy version (see API v1 Reference) - API v2: Current version (documented here) ## Key Differences in API v2 - Results are encapsulated in Streaming API Framing (SAF) Protocol - Only "jsonl" output format is supported (specified as 'application/x-ndjson') - All API URLs now start with /dnsdb/v2 - The rate_limit request moved to /dnsdb/v2/rate_limit - Rdata values are always returned as an array - Improved error handling and status codes ## Quick Start ### 1. Set Up Authentication Set your API key as an environment variable: ```bash export DNSDB_API_KEY="your-api-key-here" ``` See Authentication for more details. ### 2. Check Your Quota Before making queries, check your available quota: ```bash curl -H "Accept: application/x-ndjson" -H "X-API-Key: $DNSDB_API_KEY" \ "https://api.dnsdb.info/dnsdb/v2/rate_limit" ``` See Rate Limits for more information. ### 3. Make Your First Query Lookup all RRsets for a domain: ```bash curl -H "Accept: application/x-ndjson" -H "X-API-Key: $DNSDB_API_KEY" \ "https://api.dnsdb.info/dnsdb/v2/lookup/rrset/name/example.com?limit=10" ``` ## Core Concepts ### Request Types DNSDB supports four types of requests: 1. Lookup - Primary query to search for individual DNS RRsets - RRset Lookups - Forward lookups based on owner name - Rdata Lookups - Inverse lookups based on Rdata values 2. Summarize - Returns a summary of RRsets that would be returned by a lookup query 3. Ping - End-to-end connectivity test (no API key required) 4. Rate Limit - Returns quota information ### Query Parameters Enhance your queries with optional parameters: - Time-Fencing - Filter results by time ranges - Other Parameters - Control result limits, aggregation, and more ### Common Query Patterns #### Name → Answers Get all historical answers for a domain: ```bash curl -H "Accept: application/x-ndjson" -H "X-API-Key: $DNSDB_API_KEY" \ "https://api.dnsdb.info/dnsdb/v2/lookup/rrset/name/www.example.com" ``` #### IP → Names Find all names pointing to an IP: ```bash curl -H "Accept: application/x-ndjson" -H "X-API-Key: $DNSDB_API_KEY" \ "https://api.dnsdb.info/dnsdb/v2/lookup/rdata/ip/104.244.13.104" ``` #### Wildcard Searches Enumerate subdomains: ```bash curl -H "Accept: application/x-ndjson" -H "X-API-Key: $DNSDB_API_KEY" \ "https://api.dnsdb.info/dnsdb/v2/lookup/rrset/name/*.example.com" ``` ## Understanding Results ### Streaming API Framing (SAF) All lookup and summarize results are wrapped in SAF protocol: ```json {"cond": "begin"} {"obj":{...result data...}} {"obj":{...result data...}} {"cond": "succeeded"} ``` The final cond value indicates success: - "succeeded" - Query completed successfully - "limited" - Result limit reached - Other values indicate errors or truncation See Streaming Protocol for details. ### Result Metadata Each result includes: - count: Number of times observed - time_first/time_last: First and last observation timestamps - rrname: Owner name - rrtype: Record type - rdata: Record data (always an array) - bailiwick: DNS zone context (rrset lookups only) ## Next Steps - Review the User Guide for detailed usage patterns - Check the FAQ for common questions - Explore CLI Tools for command-line access - Read about Flexible Search for advanced queries ## Getting Help - Review the FAQ - Contact DomainTools Support - Request a demo from the DomainTools sales team # DNSDB API authentication The DNSDB API is provided over an encrypted HTTPS transport over the Internet at the following URL: https://api.dnsdb.info/ ## API Key Authentication Authentication is performed by providing an API key (a long string of hexadecimal digits or dashes) in a special HTTP request header, X-API-Key. ### Example If your API key is d41d8cd98f00b204e9800998ecf8427e, then the following HTTP header should be added to the HTTP request: ```text X-API-Key: d41d8cd98f00b204e9800998ecf8427e ``` ### Authentication Errors If the X-API-Key header is not present, or the provided API key is not valid, a 403 Forbidden response will be returned to the HTTP client. ## Name Encoding All IDN DNS names used with the DNSDB API must be encoded in Punycode. Warning: Using non-basic ASCII characters may result in a 500 Internal Server error. ## Setting Up Your Environment For convenience in examples throughout this documentation, you can set your API key as an environment variable: ```bash export DNSDB_API_KEY="d41d8cd98f00b204e9800998ecf8427e" ``` Then use it in curl commands: ```bash curl -H "Accept: application/x-ndjson" -H "X-API-Key: $DNSDB_API_KEY" \ "https://api.dnsdb.info/dnsdb/v2/lookup/rrset/name/example.com" ``` # DNSDB service limits and quotas The DNSDB API implements quota management to control usage and ensure fair access to resources. ## Concurrent Connections The number of concurrent connections to a DNSDB API server may be limited. This limit is separate from the quota limit described below. Warning: If the concurrent connection limit is exceeded, the HTTP 503 "Service Unavailable" response code will be generated. ## Quota Types API keys have a primary quota, which limits the number of requests that can be made to the data-fetching endpoints (lookup, /dnsdb/v2/lookup, summarize, and /dnsdb/v2/summarize). There are three types of quotas: ### Time-Based Quotas Time-based quotas are usually applied on a daily basis and reset daily at 00:00 (midnight) in the UTC time zone. Time-based quotas can also be applied for arbitrary time-quantum, but this is less common. ### Block-Based Quotas For block quotas, there are a number of queries that are bought, with all, or a subset, of that number "split" off for a particular API key. Block quotas have: - A specific number of queries split with an expiration time (usually much longer than a day) - The DNSDB API server tracks the number of queries split, number used, and the expiration time - The number of queries bought is not visible to the DNSDB API server Warning: If a block quota is expired, a 401 "Unauthorized" response code will be generated with message 'Error: Quota is expired'. ### Unlimited Quotas Unlimited quotas do not limit the number of queries per day. ## Burst Rate Quota There may also be a secondary, burst rate, quota associated with an API key. The burst rate limits how many requests may be made in a short time window. Example: 5 requests in 360 seconds The parameters for burst rate are: - burst_size: Maximum number of requests in the window - burst_window: Time window in seconds ## Checking Your Quota You may query the /dnsdb/v2/rate_limit endpoint to obtain a JSON map containing quota information. Querying the /dnsdb/v2/rate_limit endpoint does not count against the quota limit. ### Rate Limit Response The /dnsdb/v2/rate_limit endpoint returns a JSON map named rate containing some or all of the following: | Key | Description | | --- | ----------- | | limit | The maximum number of API lookups that may be performed. This is the initial quota.| | remaining | For time-based quotas: the remaining number of API lookups that may be performed until the reset time. For block-based quotas: the remaining number of API lookups in the block quota.| | reset | For time-based quotas: UNIX epoch timestamp with second granularity indicating the next point in time when the quota limit will be reset. Usually this is at 00:00 (midnight) UTC. For block-based quotas: the value will be "n/a".| | expires | Only present for block-based quota: UNIX epoch timestamp with second granularity indicating when the quota will expire.| | results_max | Returns the maximum number of results that can be returned by these lookup methods. This overrides a "limit" query parameter if provided. For example, if "?limit=20000" is appended to the URL path but results_max=1000 then only up to 1000 results will be returned.| | offset_max | The maximum value that the offset query parameter can be. If it is higher then an HTTP 416 "Requested Range Not Satisfiable" response code will be returned with message "Error: offset value greater than maximum allowed." If the value is "n/a" then the offset parameter is not allowed for this API key, and similar 416 error will be generated. | | burst_size | The maximum number of API lookups that may be performed within this burst_window number of seconds. | | burst_window | The number of seconds over which a burst of queries is measured.| Note: If burst_size and burst_window are not returned then there is no burst rate quota applicable. ## Examples ### Time-Based Quota Example For a time-based quota, the following is an example of a /dnsdb/v2/rate_limit response performed on June 10, 2015 that indicates that the API key's quota limit will be reset at midnight UTC on June 11, 2015, and that 999 lookups are remaining out of a total quota of 1000 lookups: ```bash curl -H "Accept: application/x-ndjson" -H "X-API-Key: $DNSDB_API_KEY" \ "https://api.dnsdb.info/dnsdb/v2/rate_limit" ``` Response: ```json {"rate": {"reset": 1433980800, "limit": 1000, "remaining": 999}} ``` ### Block-Based Quota Example For a block-based quota, the following example indicates that the API key's quota limit will expire at Mon Apr 15 19:28:34 2019, 592 lookups were used out of a total quota of 600 lookups, the burst size is 10 queries in 5 minutes, the maximum number of results that can be requested from a single query are 256, and offset will fail if higher than 3000000: ```json {"rate": {"reset": "n/a", "burst_size": 10, "expires": 1555370914, "burst_window": 300, "offset_max": 3000000, "results_max": 256, "limit": 600, "remaining": 8}} ``` ### Unlimited Quota Example An unlimited quota API key has the corresponding /dnsdb/v2/rate_limit response: ```json {"rate": {"reset": "n/a", "limit": "unlimited", "remaining": "n/a"}} ``` ## X-RateLimit Headers Responses from the data-fetching endpoints contain information in the HTTP response headers that are a subset of that obtained from the /dnsdb/v2/rate_limit endpoint. These values are embedded as the X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Reset, and for block-based quotas, X-RateLimit-Expires headers. ### Time-Based Quota Headers For a time-based quota, responses to lookups will contain response headers like: ```text X-RateLimit-Limit: 1000 X-RateLimit-Remaining: 999 X-RateLimit-Reset: 1433980800 ``` ### Block-Based Quota Headers For a block-based quota, responses to lookups will contain response headers like: ```text X-RateLimit-Limit: 600 X-RateLimit-Remaining: 8 X-RateLimit-Reset: n/a X-RateLimit-Expires: 1555370914 ``` ### Unlimited Quota Headers For an unlimited API key, responses to lookups will contain response headers like: ```text X-RateLimit-Limit: unlimited X-RateLimit-Remaining: n/a X-RateLimit-Reset: n/a ``` ## Quota Exceeded Errors If you have exceeded your quota, the HTTP 429 "Too Many Requests" response code will be generated with message 'Error: Rate limit exceeded'. - For time-based quotas: The API key's daily quota limit is exceeded. The quota will automatically replenish, usually at the start of the next day. - For block-based quotas: The block quota is exhausted. You may need to purchase a larger quota. - For burst rate secondary quotas: There were too many queries within the burst window. The window will automatically reopen at its end. # DNSDB API HTTP response codes The DNSDB API uses standard HTTP response codes to indicate the success or failure of requests. ## Success Codes | Response Code | Description | | ------------- | ----------- | | 200 OK | The request was successfully processed. For requests that stream back data, the server interpreted the request correctly and began to reply, but look at the final "cond" for success or failure of returning the streamed data. | ## Client Error Codes | Response Code | Description | | ------------- | ----------- | | 400 Bad Request | The URL is formatted incorrectly, such as unrecognized lower-level components. | | 401 Unauthorized | Your API key may not query this API version, or the API key is not authorized (usually indicates the block quota is expired). | | 403 Forbidden | The X-API-Key header is not present, the provided API key is not valid, or the Client IP address is not authorized for this API key. | | 404 Not Found | The URL path had unrecognized or missing top-level components. Note: In APIv1 this error indicated that there were no results found. In APIv2, this is not an error - a 200 OK is returned with no "obj" lines in the SAF response. | | 415 Unsupported Media Type | The Accept: header does not specify a supported content type for this query. | | 416 Requested Range Not Satisfiable | The offset value is greater than the maximum allowed or if an offset value was provided when not permitted. | | 429 Too Many Requests | You have exceeded your quota and no new requests will be accepted at this time. For time-based quotas: The API key's daily quota limit is exceeded. The quota will automatically replenish, usually at the start of the next day. For block-based quotas: The block quota is exhausted. You may need to purchase a larger quota. For burst rate secondary quotas: There were too many queries within the burst window. The window will automatically reopen at its end. | ## Server Error Codes | Response Code | Description | | ------------- | ----------- | | 500 Internal Server Error | There is an error processing the request. Returns text/html content type regardless of what was requested. | | 503 Service Unavailable | The limit of number of concurrent connections is exceeded. | | 504 Gateway Timeout | The DNSDB Server timed-out before returning any results. Your query might be too complex, such as doing a wildcard search with restrictive time-fencing. You might simplify the query and retry it. | ## Content Types by Response Different response types return different content types: - Successful data responses (200 OK): Returns the content type specified in the Accept header (e.g., application/x-ndjson) - 500 Internal Server Error: Returns text/html (no matter what was requested) - Other API errors: Return text/plain (no matter what was requested) - nginx errors (e.g., 404 for unknown URL): Return text/html ## Examples ### Successful Query ```bash curl -i -H "Accept: application/x-ndjson" -H "X-API-Key: $DNSDB_API_KEY" \ "https://api.dnsdb.info/dnsdb/v2/lookup/rrset/name/example.com?limit=1" ``` Response: ```http HTTP/1.1 200 OK Content-Type: application/x-ndjson X-RateLimit-Limit: 1000 X-RateLimit-Remaining: 999 X-RateLimit-Reset: 1433980800 {"cond": "begin"} {"obj":{...}} {"cond": "succeeded"} ``` ### Bad Request ```bash curl -i -H "Accept: application/x-ndjson" -H "X-API-Key: $DNSDB_API_KEY" \ "https://api.dnsdb.info/dnsdb/v2/lookup/name/fsi.io" ``` Response: ```http HTTP/1.1 400 Bad Request Content-Type: text/plain Error: unable to parse request ``` ### Rate Limit Exceeded ```bash curl -i -H "Accept: application/x-ndjson" -H "X-API-Key: $DNSDB_API_KEY" \ "https://api.dnsdb.info/dnsdb/v2/lookup/rrset/name/example.com" ``` Response: ```http HTTP/1.1 429 Too Many Requests Content-Type: text/plain Error: Rate limit exceeded ``` # DNSDB API User Guide ## Introduction DNSDB is a database that stores and indexes both passive DNS data (from Farsight Security’s Security Information Exchange, SIE) and authoritative DNS data provided by various zone operators. It allows searching for individual DNS RRsets, with additional metadata for search results, such as first seen and last seen timestamps and the DNS bailiwick for each RRset. DNSDB also supports inverse (rdata) searches. ## DNSDB Capabilities and Limits Access to DNSDB is licensed in several ways, with different interfaces and tools providing various capabilities and limits. Understand your license and toolset for quota and feature differences. ### Trial Products | Product | Quota | Maximum Results | Duration | Data Available | Rate Limit | Query Privacy | | -------------------- | ------- | --------------- | -------- | -------------- | ---------- | ------------- | | Maltego Free Queries | 12/hour | 12 | N/A | 2010 to now | 12/hour | No | To request a DNSDB demonstration, contact the DomainTools sales team. ### Subscription Products | Product | Quota | Maximum Results | Duration | Data Available | Rate Limit | Query Privacy | | --------------- | -------------- | --------------- | -------- | -------------- | ---------- | ------------- | | Queries per Day | 1K – Unlimited | 10K – 1M | 1 Year | 2010 to now | None | Yes | --- ## User Guide Notes * In Curl examples, $APIKEY is an environment variable. Set it in the current shell with (example only): ```sh APIKEY="QmIodGqF12TKOf8bqBe6S6WxvZ4LTtzP1VlS09g0UApw28gedka545OcumVW4WHkB" ``` * API calls below use API Version 2. * Curl 7.42.0+ supports the --path-as-is option, preventing curl from merging/squashing /../ or /./ sequences. --- ## Primary Pivots When investigating historical DNS data, five primary pivots are useful: 1. Name → Answers (names and IPs) 2. Wildcard left hand side 3. Wildcard right hand side 4. IP → Names 5. Name → Names ### Name → Answers Specify a name to retrieve historical answers (A, AAAA, NS, MX, SOA, TXT, etc.). If no record type is specified, all are returned. #### Hostname Example * Hostname: www.fsi.io ```sh curl -s -H 'Accept: application/x-ndjson' -H "X-API-Key: $APIKEY" \ "https://api.dnsdb.info/dnsdb/v2/lookup/rrset/name/www.fsi.io?limit=10000" ``` #### Second Level Domain Example * Hostname: fsi.io ```sh curl -s -H 'Accept: application/x-ndjson' -H "X-API-Key: $APIKEY" \ "https://api.dnsdb.info/dnsdb/v2/lookup/rrset/name/fsi.io?limit=10000" ``` ### Wildcard Left Hand Side Enumerate all subdomains of a second-level domain. ```sh curl -s -H 'Accept: application/x-ndjson' -H "X-API-Key: $APIKEY" \ "https://api.dnsdb.info/dnsdb/v2/lookup/rrset/name/*.fsi.io?limit=10000" ``` ### Wildcard Right Hand Side Search for a base domain or TLD with a wildcard on the right. ```sh curl -s -H 'Accept: application/x-ndjson' -H "X-API-Key: $APIKEY" \ "https://api.dnsdb.info/dnsdb/v2/lookup/rrset/name/example.*?limit=10000" ``` ### IP → Names Return any names pointing to a specific IP. ```sh curl -s -H 'Accept: application/x-ndjson' -H "X-API-Key: $APIKEY" \ "https://api.dnsdb.info/dnsdb/v2/lookup/rdata/ip/104.244.13.0,24?limit=10000" ``` ### Name → Names Return any names pointing to a name (e.g., NS records). ```sh curl -s -H 'Accept: application/x-ndjson' -H "X-API-Key: $APIKEY" \ "https://api.dnsdb.info/dnsdb/v2/lookup/rdata/name/ns1.infocity.club/NS?limit=10000" ``` --- ## Considerations 1. If you plan to truncate displayed answers, make a larger query to the API (limit ≥10,000) to get recent results. 2. API data is not sorted. For investigations, sort by time_last to get the most recent records. 3. SOA records may contain email addresses—useful for pivoting into other data sources (e.g., historical WHOIS). 4. In a UI, allow hyperlink pivoting between names and IPs for better UX. 5. Enable filtering of DNS record types to parse results more easily. # DNSDB Streaming API Framing Protocol ## Introduction This document describes the Farsight Streaming API Framing (SAF) protocol, a data transfer protocol. It is initially used by DNSDB to transfer pre-standard IETF COF format JSON objects. SAF is basically a wrapper protocol around JSON objects. ## Audience This document is intended for programmers who want to write applications that can interact with the RESTful DNSDB API using JSON and HTTP. ## Format The data is transmitted as newline delimited JSON lines, aka JSONL or NDJSON. All string values are UTF-8. All attribute names are US-ASCII. Whitespace within the JSONL structure should be ignored. Each JSONL object contains the optional attributes: 1. "cond" is a condition. It is a scalar enum of string values: * "begin" * "ongoing" * "succeeded" * "limited" * "failed" 2. "msg" is a human readable message. It is a scalar string which ought to be reported to the end-user or in the system log. 3. "obj" contains data appropriate for the protocol using SAF encapsulation. It is a wrapped JSON object. For the DNSDB API, it contains a pre-standard IETF COF object. Other attributes, if present, would be the subject of some future protocol revision. The condition "begin" MUST be in the first, initiating JSONL object. This object MUST contain a "cond" and MAY contain a "msg". The conditions "succeeded", "limited", and "failed" are terminating conditions and MUST be in the last JSONL object. JSONL objects after the first and before the last MAY contain an "obj" attribute, MAY contain a "msg" attribute, and MAY either specify "cond" to be "ongoing" OR omit the "cond". The default value for "cond" is "ongoing". If a terminating condition is missing (i.e. the last object in the data stream does not contain one) then the client should interpret this as truncation and generate an appropriate error message to report to the end-user or in the system log. The previously received data objects may be discarded or used subject to the knowledge they may be incomplete. This is also the case if the data stream terminates with a TCP error. A client is free to retry a truncated operation. If any JSONL object cannot be parsed as valid JSON then both that JSONL object and all remaining JSONL objects must be discarded and some warning must be issued to the user or system log. If the terminating condition is "succeeded" or "limited" then the data objects are valid. If the terminating condition is "failed" then the data objects may be discarded or used subject to the knowledge they may be incomplete. If a JSONL object after the first and before the last, either having no "cond" attribute or having a "cond" value of "ongoing", contains a "msg" attribute then the "msg" describes an informational or warning condition which has no decisive bearing on the validity of the objects in the response stream. Other values of "msg" SHOULD be reported to the end-user or in the system log. If a JSONL object after the first and before the last, either having no "cond" attribute or having a "cond" value of "ongoing", contains neither a "obj" nor a "msg", then this is a "keep alive" message. A keep alive message is designed to signal to the client that the connection is still open. A legal representation of a keep alive message is the empty JSON object {}. Keep alives are useful for queries that take a long time to process before reporting anything to the user. Keep alives are also known as "heartbeats". ## State Machine The state machine grammar is: ``` initiating JSONL object := { "cond": "begin" [, "msg": "$message"]} ongoing JSONL object := { [ "cond": "ongoing",] ["obj": { $object }] [, "msg": "$message"]} terminating JSONL object := { "cond": "succeeded" [, "msg": "$message"]} | { "cond": "limited" [, "msg": "$message"]} | { "cond": "failed" [, "msg": "$message"]} START => initiating JSONL object => ongoing JSONL object OR terminating JSONL object ongoing JSONL object => ongoing JSONL object OR terminating JSONL object terminating JSONL object => END $message => scalar string $object => JSON object wrapped by the SAF protocol ``` ## Examples ### Simple successful example ``` {"cond": "begin"} {"obj":{"count":10392,"time_first":138126549...}} {"cond": "succeeded"} ``` The second JSONL object implicitly has the default condition of "ongoing". ### Long-running example with keep-alive messages This example shows a query that took a while to return the first response, and then had occasional delays in responding. Because "cond":"ongoing" is the default, it is normally omitted, which means that a "keep alive" message is transmitted as an empty JSON object, {}. The example also shows varying amounts of whitespace, as the whitespace should be ignored. ``` {"cond":"begin"} {} {} {"obj":{"count":10392,"time_first":138126549...}} ... {} {"obj": {"count": 1234,"time_first": 238126549...}} ... {} ... {"obj" : {"count" :456, "time_first":338126549...} } {"cond":"succeeded"} ``` ### Limited example ``` {"cond": "begin"} {"obj":{"count":10392,"time_first":138126549...}} {"obj":{"count":33,"time_first":19126549...}} {"cond":"limited", "msg":"Result limit reached"} ``` In this case, the results are valid and SHOULD be used. Note: The "Result limit reached" message does not imply that re-trying the query with a higher limit value will return more results, only that the limit was reached. ### Failure example ``` {"cond": "begin"} {"obj":{"count":33,"time_first":19126549...}} ... {"cond": "failed", "msg": "Processing timeout; results may be incomplete"} ``` In this case, the results MAY be used, but it is recommended to retry the request. # DNSDB summarize requests Summarize requests return a summary of RRsets that would be returned by a lookup query. This gives you an estimate of result size and provides at-a-glance information on when a given domain name, IP address, or other DNS asset was first-seen and last-seen by the global sensor network, as well as the total observation count. ## URL Path Scheme All DNSDB summarize requests are rooted in URL paths under the /dnsdb/v2/summarize hierarchy. Everything underneath /dnsdb/v2/summarize is the same as for /dnsdb/v2/lookup queries: ```text /dnsdb/v2/summarize/rrset/name/OWNER_NAME/RRTYPE/BAILIWICK /dnsdb/v2/summarize/rdata/TYPE/VALUE/RRTYPE ``` ## Query Parameters Summarize requests accept the same optional query parameters as lookup requests, with one additional parameter: | Parameter | Description | | --- | ----------- | | max_count | Controls stopping when the summary count reaches this value. The resulting total count can exceed max_count as it will include the entire count from the last rrset examined. The default is to not constrain the count. Example: appending "?max_count=100" to the URL path will stop the summary when its total count reaches 100.| See Query Parameters for all other available parameters. ## Result Format The DNSDB API only supports one Summarize result format, the "jsonl" format (though this should be specified in an HTTP ACCEPT header as 'application/x-ndjson'). ### RRset and Rdata Results | Key | Description | | --- | ----------- | | count | The number of times the RRset was observed via passive DNS replication. | | num_results | The number of results (RRsets) that would be returned from a Lookup. | | time_first, time_last | UNIX epoch timestamps with second granularity indicating the first and last times the RRset was observed via passive DNS replication. Only present if the RRsets were observed via passive DNS replication. | | zone_time_first, zone_time_last | UNIX epoch timestamps with second granularity indicating the first and last times the RRset was observed via zone file import. Only present if the RRsets were observed via zone file import. | ## Examples ### Example 1: Basic Summarize An example JSON object: ```json {"count":528,"num_results":4,"time_first":1557864746,"time_last":1560524861} ``` As encapsulated by the SAF, this would be expressed on the wire as: ```json {"cond":"begin"} {"obj":{"count":528,"num_results":4,"time_first":1557864746,"time_last":1560524861}} {"cond":"succeeded"} ``` ### Example 2: No Results Found If there are no RRsets found by the underlying query, the DNSDB API will return: ```json {"count":0, "num_results":0} ``` As encapsulated by the SAF: ```json {"cond":"begin"} {"obj":{"count":0,"num_results":0}} {"cond":"succeeded"} ``` ### Example 3: Summarize with limit and max_count ```bash curl -H "Accept: application/x-ndjson" -H "X-API-Key: $DNSDB_API_KEY" \ "https://api.dnsdb.info/dnsdb/v2/summarize/rrset/name/www.farsightsecurity.com\ ?limit=2&max_count=5000" ``` In this call, the summarize processing will limit itself to looking at two underlying results rows and will stop when the count value reaches max_count. Since the first row exceeds max_count, the summarize will only account for the first row. Response: ```json {"cond":"succeeded","obj":{"count":1127,"num_results":2,"time_first":1557859313, "time_last":1560537333}} ``` ### Example 4: Summarize with only limit ```bash curl -H "Accept: application/x-ndjson" -H "X-API-Key: $DNSDB_API_KEY" \ "https://api.dnsdb.info/dnsdb/v2/summarize/rrset/name/www.farsightsecurity.com?limit=2" ``` In this call, the summarize processing will limit itself to looking at two underlying results rows. The count is the sum of the counts from the two rows, and the time_first is from the first row while time_last is from the second row. Response: ```json {"cond":"succeeded","obj":{"count":1127,"num_results":2,"time_first":1557859313, "time_last":1560537333}} ``` ### Example 5: Summarize with only max_count ```bash curl -H "Accept: application/x-ndjson" -H "X-API-Key: $DNSDB_API_KEY" \ "https://api.dnsdb.info/dnsdb/v2/summarize/rrset/name/www.farsightsecurity.com\ ?max_count=50000" ``` Response: ```json {"cond":"succeeded","obj":{"count":1078,"num_results":2,"time_first":1573594176, "time_last":1576187607}} ``` ### Example 6: Summarize IP network ```bash curl -H "Accept: application/x-ndjson" -H "X-API-Key: $DNSDB_API_KEY" \ "https://api.dnsdb.info/dnsdb/v2/summarize/rdata/ip/104.244.13.104,29" ``` Response: ```json {"cond":"succeeded","obj":{"count":528,"num_results":4,"time_first":1557864746, "time_last":1560524861}} ``` # DNSDB RRset lookups RRset lookups query DNSDB's RRset index, which supports "forward" lookups based on the owner name of an RRset. ## URL Path Scheme ```text /dnsdb/v2/lookup/rrset/TYPE/VALUE/RRTYPE/BAILIWICK ``` Both TYPE and VALUE are required; RRTYPE and BAILIWICK are optional. If RRTYPE is not specified, the query functions as if "ANY" was specified. ## Type Parameter The TYPE parameter specifies how VALUE is interpreted: | Type | Description | | ---- | ----------- | | name | The VALUE is a DNS owner name in presentation format or wildcards as described below. We sometimes call this just an rrset search. | | raw | The VALUE is an even number of hexadecimal digits specifying a raw octet string. | ## Wildcards Wildcards come in two forms: - Left-hand wildcard (.example.com): Matches any RRsets whose owner names end with* the given domain name - Right-hand wildcard (www.example.): Matches any RRsets whose owner names start with* the given label(s) Note: Left-hand wildcard queries are somewhat more expensive and slower than right-hand wildcard queries. ## RRtype Parameter The OWNER_NAME and BAILIWICK are DNS names specified in DNS presentation format. RRTYPE is specified as a DNS RRtype mnemonic. The RRtype ANY is modified from its usual meaning: - A DNSDB lookup for RRtype ANY matches any RRtype except DNSSEC-related RRtypes: DS, RRSIG, NSEC, DNSKEY, NSEC3, NSEC3PARAM, DLV, CDS, CDNSKEY, and TA - The pseudo-mnemonic ANY-DNSSEC returns only those DNSSEC-related records - RRtype ANY may be specified for RRTYPE to perform bailiwick filtering without filtering on a particular RRtype ## Bailiwick Parameter The BAILIWICK may not be specified with raw queries. ## Wildcard Domain Consolidation Info: As of July 2022, DNSDB was changed to reduce junk wildcard domains. We are gradually rolling out a change to replace multiple wildcarded DNS rrnames with a single rrname starting with a WILDCARD. label. No other rrname labels contain uppercase letters, so records with this (all upper case) WILDCARD. were never in DNSDB before. Note that there are existing, real domain names that contain a wildcard. label (all lower case). ## Result Metadata The results of an rrset lookup return zero or more RRsets, along with the following metadata for each result: | Item | Description | | --- | ----------- | | count | The number of times the RRset was observed via passive DNS replication. | | bailiwick | The "bailiwick" of an RRset in DNSDB observed via passive DNS replication is the closest enclosing zone delegated to a nameserver which served the RRset. The "bailiwick" of an RRset in DNSDB observed in a zone file is simply the name of the zone containing the RRset. | | first seen | A UTC timestamp with seconds granularity indicating the first time an RRset was seen in the given bailiwick via passive DNS replication. | | last seen | A UTC timestamp with seconds granularity indicating the last time an RRset was seen in the given bailiwick via passive DNS replication. | | first seen in zone file | A UTC timestamp with seconds granularity indicating the first time an RRset was seen in the given bailiwick via zone file import. | | last seen in zone file | A UTC timestamp with seconds granularity indicating the last time an RRset was seen in the given bailiwick via zone file import. | Note: An rrset search result may be missing either the pair of (first seen, last seen) timestamps from passive DNS replication or from zone file import, but at least one pair of timestamps will always be present. This may change if a fundamentally new data source is introduced in the future, but there will always be at least one timestamp pair associated with an RRset. ## RRSIG Time Format Info: The DNSDB API returns RRSIG records with the expiration and inception times in Unix epoch time format (seconds since 1 January 1970 00:00:00 UTC). RFC 4034 also allows presentation of these times in YYYYMMDDHHmmSS format, which many DNSSEC tools and presentation libraries use. It is always possible to distinguish between these two formats because the YYYYMMDDHHmmSS format will always be exactly 14 digits, while the decimal representation of a 32-bit unsigned integer can never be longer than 10 digits. ## Result Format Results are returned in JSON Lines format with the following keys: | Key | Description | | --- | ----------- | | rrname | The owner name of the RRset in DNS presentation format. | | rrtype | The resource record type of the RRset, either using the standard DNS type mnemonic, or an RFC 3597 generic type, i.e. the string TYPE immediately followed by the decimal RRtype number. | | rdata | An array of one or more Rdata values. The Rdata values are converted to the standard presentation format based on the rrtype value. If the encoder lacks a type-specific presentation format for the RRset's rrtype, then the RFC 3597 generic Rdata encoding will be used. | | bailiwick | The "bailiwick" metadata value described above. | | count | The number of times the RRset was observed via passive DNS replication. | | time_first, time_last | UNIX epoch timestamps with second granularity indicating the first and last times the RRset was observed via passive DNS replication. Will not be present if the RRset was only observed via zone file import. | | zone_time_first, zone_time_last | UNIX epoch timestamps with second granularity indicating the first and last times the RRset was observed via zone file import. Will not be present if the RRset was only observed via passive DNS replication. | ## Examples ### Example 1: Lookup all RRsets for www.farsightsecurity.com ```bash curl -H "Accept: application/x-ndjson" -H "X-API-Key: $DNSDB_API_KEY" \ "https://api.dnsdb.info/dnsdb/v2/lookup/rrset/name/www.farsightsecurity.com?limit=2" ``` Response: ```json {"cond": "begin"} {"obj":{"count":5059,"time_first":1380139330,"time_last":1427881899, "rrname":"www.farsightsecurity.com.","rrtype":"A","bailiwick":"farsightsecurity.com.", "rdata":["66.160.140.81"]}} {"obj":{"count":17381,"time_first":1427893644,"time_last":1468329272, "rrname":"www.farsightsecurity.com.","rrtype":"A","bailiwick":"farsightsecurity.com.", "rdata":["104.244.13.104"]}} {"cond": "limited", "msg": "Result limit reached"} ``` ### Example 2: Lookup all NS RRsets ending in farsightsecurity.com ```bash curl -H "Accept: application/x-ndjson" -H "X-API-Key: $DNSDB_API_KEY" \ "https://api.dnsdb.info/dnsdb/v2/lookup/rrset/name/*.farsightsecurity.com/ns/farsightsecurity.com" ``` Response: ```json {"cond": "begin"} {"obj":{"count":51,"time_first":1372688083,"time_last":1374023864, "rrname":"farsightsecurity.com.","rrtype":"NS","bailiwick":"farsightsecurity.com.", "rdata":["ns.lah1.vix.com.","ns1.isc-sns.net.","ns2.isc-sns.com.", "ns3.isc-sns.info."]}} {"obj":{"count":495241,"time_first":1374096380,"time_last":1468324876, "rrname":"farsightsecurity.com.","rrtype":"NS","bailiwick":"farsightsecurity.com.", "rdata":["ns5.dnsmadeeasy.com.","ns6.dnsmadeeasy.com.","ns7.dnsmadeeasy.com."]}} {"cond": "succeeded"} ``` ### Example 3: Lookup DNSSEC records ```bash curl -H "Accept: application/x-ndjson" -H "X-API-Key: $DNSDB_API_KEY" \ "https://api.dnsdb.info/dnsdb/v2/lookup/rrset/name/*.farsightsecurity.com/ANY-DNSSEC?limit=2" ``` Response: ```json {"cond": "begin"} {"obj":{"count":1696,"zone_time_first":1374250920,"zone_time_last":1521734545, "rrname":"farsightsecurity.com.","rrtype":"DS","bailiwick":"com.", "rdata":["60454 5 2 3672C35CFA8FF14C9C223B84277BD645C0AF54BAD5790375FE797161E4801479"]}} {"obj":{"count":3,"zone_time_first":1374250920,"zone_time_last":1374423636, "rrname":"farsightsecurity.com.","rrtype":"RRSIG","bailiwick":"com.", "rdata":["DS 8 2 86400 1374774350 1374165350 8795 com. cuOdo+2G0yJpBN5ba2zxiljSzgtTzminrVc3CrsNxQPqc5YVQX4eBWMB +kpgSEXPT+DF2D9HwIsPpBDNdJekBpXIRW41Yl7IdZYHySqabn7hgt9M mk5KNy9gqCOK/JLRs07LPAm3wvfyYer8e0/7VCTEjF9/DMbMGsLLH3xr kBA="]}} {"cond": "limited", "msg": "Result limit reached"} ``` ### Example 4: Raw query for records under fsi.io ```bash curl -H "Accept: application/x-ndjson" -H "X-API-Key: $DNSDB_API_KEY" \ "https://api.dnsdb.info/dnsdb/v2/lookup/rrset/raw/0366736902696f00?limit=2" ``` Response: ```json {"cond": "begin"} {"obj":{"count":10392,"time_first":1381265499,"time_last":1428418529, "rrname":"fsi.io.","rrtype":"A","bailiwick":"fsi.io.","rdata":["66.160.140.76"]}} {"obj":{"count":69435,"time_first":1428433465,"time_last":1538014110, "rrname":"fsi.io.","rrtype":"A","bailiwick":"fsi.io.","rdata":["104.244.13.104"]}} {"cond": "limited", "msg": "Result limit reached"} ``` # DNSDB rdata lookups Rdata lookups query DNSDB's Rdata index, which supports "inverse" lookups based on Rdata record values. In contrast to rrset lookups, rdata lookups return only individual resource records (not full resource record sets) and lack bailiwick metadata. Tip: An rrset lookup on the owner name reported via an rdata lookup must be performed to retrieve the full RRset and bailiwick. ## URL Path Scheme ```text /dnsdb/v2/lookup/rdata/TYPE/VALUE/RRTYPE ``` Both TYPE and VALUE are required. The RRTYPE parameter is optional. ## Type Parameter The TYPE parameter specifies how VALUE is interpreted: | Type | Description | | ---- | ----------- | | name | The VALUE is a DNS domain name in presentation format, or a left-hand (".example.com") or right-hand ("www.example.") wildcard domain name. Note that left-hand wildcard queries are somewhat more expensive than right-hand wildcard queries. | | ip | The VALUE is one of an IPv4 or IPv6 single address, with a prefix length, or with an address range. If a prefix is provided, the delimiter between the network address and prefix length is a single comma (",") character rather than the usual slash ("/") character to avoid clashing with the HTTP URI path name separator. | | raw | The VALUE is an even number of hexadecimal digits specifying a raw octet string. | ## RRtype Parameter If RRTYPE is not specified, the query functions as if "ANY" was specified. - For name and raw rdata lookups, RRTYPE optionally filters the results by RRtype in the same manner as rrset lookups - For ip rdata lookups, the supported RRTYPE values are A, AAAA, and ANY, which can be used interchangeably: you can send an IPv4 or IPv6 address to either value and the data returned will be based on the IP address sent, not on the RRTYPE value - Any other RRTYPE value for ip lookups will return an "HTTP 400 Bad Request" response ## IP Address Examples Some examples of "ip" rdata URLs (noting that colon needs to be expressed as %3A): ```text https://api.dnsdb.info/dnsdb/v2/lookup/rdata/ip/10.0.0.1/ANY https://api.dnsdb.info/dnsdb/v2/lookup/rdata/ip/10.0.0.1-10.1.255.255 https://api.dnsdb.info/dnsdb/v2/lookup/rdata/ip/10.0.0.1,24/ANY https://api.dnsdb.info/dnsdb/v2/lookup/rdata/ip/2620%3A11c%3Af008%3A%3A,126 https://api.dnsdb.info/dnsdb/v2/lookup/rdata/ip/2620%3A11c%3Af008%3A%3A1-2620%3A11c%3Af008%3A%3Aff ``` ## Result Format Results are returned in JSON Lines format with the following keys: | Key | Description | | --- | ----------- | | rrname | The owner name of the resource record in DNS presentation format. | | rrtype | The resource record type of the resource record, either using the standard DNS type mnemonic, or an RFC 3597 generic type, i.e. the string TYPE immediately followed by the decimal RRtype number. | | rdata | The record data value. The Rdata value is converted to the standard presentation format based on the rrtype value. If the encoder lacks a type-specific presentation format for the resource record's type, then the RFC 3597 generic Rdata encoding will be used. | | count | The number of times the resource record was observed via passive DNS replication. | | time_first, time_last | UNIX epoch timestamps with second granularity indicating the first and last times the resource record was observed via passive DNS replication. | | zone_time_first, zone_time_last | UNIX epoch timestamps with second granularity indicating the first and last times the resource record was observed via zone file import. | ## Examples ### Example 1: Lookup records with IPv4 address 104.244.13.104 ```bash curl -H "Accept: application/x-ndjson" -H "X-API-Key: $DNSDB_API_KEY" \ "https://api.dnsdb.info/dnsdb/v2/lookup/rdata/ip/104.244.13.104" ``` Response: ```json {"cond": "begin"} {"obj":{"count":24,"time_first":1433550785,"time_last":1468312116, "rrname":"www.farsighsecurity.com.","rrtype":"A","rdata":"104.244.13.104"}} {"obj":{"count":9429,"time_first":1427897872,"time_last":1468333042, "rrname":"farsightsecurity.com.","rrtype":"A","rdata":"104.244.13.104"}} {"cond": "succeeded"} ``` ### Example 2: Lookup records in 104.244.13.104/29 network ```bash curl -H "Accept: application/x-ndjson" -H "X-API-Key: $DNSDB_API_KEY" \ "https://api.dnsdb.info/dnsdb/v2/lookup/rdata/ip/104.244.13.104,29" ``` Response (with keep-alive messages): ```json {"cond": "begin"} {} {"obj":{"count":24,"time_first":1433550785,"time_last":1468312116, "rrname":"www.farsighsecurity.com.","rrtype":"A","rdata":"104.244.13.104"}} {"obj":{"count":9429,"time_first":1427897872,"time_last":1468333042, "rrname":"farsightsecurity.com.","rrtype":"A","rdata":"104.244.13.104"}} {} {"cond": "succeeded"} ``` ### Example 3: Lookup records with IPv6 address ```bash curl -H "Accept: application/x-ndjson" -H "X-API-Key: $DNSDB_API_KEY" \ "https://api.dnsdb.info/dnsdb/v2/lookup/rdata/ip/2620:11c:f004::104" ``` Response: ```json {"cond": "begin"} {"obj":{"count":14,"time_first":1433845806,"time_last":1467828872, "rrname":"www.farsighsecurity.com.","rrtype":"AAAA","rdata":"2620:11c:f004::104"}} {"obj":{"count":5307,"time_first":1427897876,"time_last":1468333042, "rrname":"farsightsecurity.com.","rrtype":"AAAA","rdata":"2620:11c:f004::104"}} {"cond": "succeeded"} ``` ### Example 4: Lookup records in IPv6 network prefix ```bash curl -H "Accept: application/x-ndjson" -H "X-API-Key: $DNSDB_API_KEY" \ "https://api.dnsdb.info/dnsdb/v2/lookup/rdata/ip/2620:11c:f000::,126" ``` Response: ```json {"cond": "begin"} {"obj":{"count":2,"time_first":1574082633,"time_last":1574082633, "rrname":"gw.fmt1.fsi.io.","rrtype":"AAAA","rdata":["2620:11c:f000::1"]}} {"obj":{"count":261,"time_first":1573589461,"time_last":1576188661, "rrname":"r1.fmt1.fsi.io.","rrtype":"AAAA","rdata":["2620:11c:f000::2"]}} {"obj":{"count":241,"time_first":1573611061,"time_last":1576188661, "rrname":"r2.fmt1.fsi.io.","rrtype":"AAAA","rdata":["2620:11c:f000::3"]}} {"cond": "succeeded"} ``` ### Example 5: Lookup domains delegated to nameserver ```bash curl -H "Accept: application/x-ndjson" -H "X-API-Key: $DNSDB_API_KEY" \ "https://api.dnsdb.info/dnsdb/v2/lookup/rdata/name/ns5.dnsmadeeasy.com" ``` Response: ```json {"cond": "begin"} {"obj":{"count":1078,"zone_time_first":1374250920,"zone_time_last":1468253883, "rrname":"farsightsecurity.com.","rrtype":"NS","rdata":"ns5.dnsmadeeasy.com."}} {"obj":{"count":706617,"time_first":1374096380,"time_last":1468334926, "rrname":"farsightsecurity.com.","rrtype":"NS","rdata":"ns5.dnsmadeeasy.com."}} {"cond": "succeeded"} ``` ### Example 6: Lookup domains with mail exchanger ```bash curl -H "Accept: application/x-ndjson" -H "X-API-Key: $DNSDB_API_KEY" \ "https://api.dnsdb.info/dnsdb/v2/lookup/rdata/name/hq.fsi.io" ``` Response: ```json {"cond": "begin"} {"obj":{"count":45644,"time_first":1372706073,"time_last":1468330740, "rrname":"fsi.io.","rrtype":"MX","rdata":"10 hq.fsi.io."}} {"obj":{"count":19304,"time_first":1374098929,"time_last":1468333042, "rrname":"farsightsecurity.com.","rrtype":"MX","rdata":"10 hq.fsi.io."}} {"cond": "succeeded"} ``` ### Example 7: Raw query for records ```bash curl -H "Accept: application/x-ndjson" -H "X-API-Key: $DNSDB_API_KEY" \ "https://api.dnsdb.info/dnsdb/v2/lookup/rdata/raw/0366736902696f00?limit=2" ``` Response: ```json {"cond": "begin"} {"obj":{"count":6,"time_first":1413228451,"time_last":1413228451, "rrname":"local-data.fsi.io.","rrtype":"SOA", "rdata":"fsi.io. hostmaster.fsi.io. 2014050101 7200 3600 604800 3600"}} {"obj":{"count":25,"time_first":1412912798,"time_last":1412942807,"rrname":"dnstap.info.", "rrtype":"SOA","rdata":"fsi.io. hostmaster.fsi.io. 2014052824 7200 3600 25920000 3600"}} {"cond": "limited", "msg": "Result limit reached"} ``` # DNSDB query parameters DNSDB API queries use two types of parameters to control searches: 1. Path Parameters - Required components that form the core query structure 2. Query String Parameters - Optional filters that refine results ## Path Parameters Path parameters are embedded in the URL path and define what you're searching for. They vary by lookup method. ### RRset Lookup Path Parameters RRset lookups follow this URL structure: ```text /dnsdb/v2/lookup/rrset/TYPE/VALUE/RRTYPE/BAILIWICK ``` | Parameter | Required | Description | |-----------|----------|-------------| | TYPE | Yes | How VALUE is interpreted: name (DNS owner name or wildcard) or raw (hex octet string) | | VALUE | Yes | The search value - interpretation depends on TYPE | | RRTYPE | No | DNS record type (A, NS, MX, etc.). If omitted, functions as "ANY" | | BAILIWICK | No | Zone context for filtering results. Cannot be used with raw queries | #### TYPE Parameter Values | Type | Description | |------|-------------| | name | VALUE is a DNS owner name in presentation format or wildcards (.example.com or www.example.). Left-hand wildcard queries are more expensive than right-hand. | | raw | VALUE is an even number of hexadecimal digits specifying a raw octet string | #### RRTYPE Special Values - ANY - Matches any RRtype except DNSSEC-related types (DS, RRSIG, NSEC, DNSKEY, NSEC3, NSEC3PARAM, DLV, CDS, CDNSKEY, TA) - ANY-DNSSEC - Returns only DNSSEC-related record types - If omitted, functions as if "ANY" was specified #### Examples ```bash # Lookup all RRsets for a domain /dnsdb/v2/lookup/rrset/name/example.com # Lookup only A records /dnsdb/v2/lookup/rrset/name/example.com/A # Wildcard search for subdomains /dnsdb/v2/lookup/rrset/name/*.example.com/NS/example.com # Raw query (hex for "fsi.io") /dnsdb/v2/lookup/rrset/raw/0366736902696f00 ``` ### Rdata Lookup Path Parameters Rdata lookups follow this URL structure: ```text /dnsdb/v2/lookup/rdata/TYPE/VALUE/RRTYPE ``` | Parameter | Required | Description | |-----------|----------|-------------| | TYPE | Yes | How VALUE is interpreted: name, ip, or raw | | VALUE | Yes | The search value - interpretation depends on TYPE | | RRTYPE | No | DNS record type filter. If omitted, functions as "ANY" | #### TYPE Parameter Values | Type | Description | |------|-------------| | name | VALUE is a DNS domain name in presentation format, or left-hand (.example.com) or right-hand (www.example.) wildcard | | ip | VALUE is IPv4/IPv6 address, with prefix length, or address range. Use comma (,) instead of slash (/) for prefix delimiter | | raw | VALUE is an even number of hexadecimal digits specifying a raw octet string | #### IP Address Formats For ip type queries, VALUE can be: - Single address: 10.0.0.1 - Network prefix: 10.0.0.1,24 (note comma, not slash) - Address range: 10.0.0.1-10.1.255.255 - IPv6 (URL-encoded): 2620%3A11c%3Af008%3A%3A,126 Note: For ip lookups, RRTYPE values A, AAAA, and ANY are interchangeable - results are based on the IP address provided, not the RRTYPE value. #### Examples ```bash # Find all names pointing to an IP /dnsdb/v2/lookup/rdata/ip/104.244.13.104 # Find names in a network /dnsdb/v2/lookup/rdata/ip/104.244.13.104,29 # Find domains using a nameserver /dnsdb/v2/lookup/rdata/name/ns5.dnsmadeeasy.com # Find domains with specific mail server /dnsdb/v2/lookup/rdata/name/mail.example.com ``` ## Query String Parameters Query string parameters are appended to the URL with ? and & to filter and control results. These are optional and work with both rrset and rdata lookups. ### Available Parameter Types - Time-Fencing Parameters - Filter results by first seen/last seen timestamps - Other Parameters - Control result limits, aggregation, formatting, and more ### Example with Query Parameters ```bash # Combine path and query string parameters /dnsdb/v2/lookup/rrset/name/example.com/A?limit=100&time_last_after=1468281600 ``` This query: - Uses path parameters: name (TYPE), example.com (VALUE), A (RRTYPE) - Uses query parameters: limit=100, time_last_after=1468281600 ## Summarize Queries Summarize queries use the same path parameter structure but under /dnsdb/v2/summarize: ```text /dnsdb/v2/summarize/rrset/TYPE/VALUE/RRTYPE/BAILIWICK /dnsdb/v2/summarize/rdata/TYPE/VALUE/RRTYPE ``` See Summarize for details on summarize-specific behavior. ## Next Steps - Learn about Time-Fencing Parameters for temporal filtering - Explore Other Parameters for result control - Review RRset Lookups for detailed examples - Review Rdata Lookups for inverse search examples # DNSDB time-fencing query parameters DNSDB records when a specific DNS record was first and last observed. You can filter results by time using time-fencing query parameters. ## Supported Parameters | Parameter | Description | | --- | ----------- | | time_first_before | Provide results before the defined timestamp for when the DNS record was first observed. | | time_first_after | Provide results after the defined timestamp for when the DNS record was first observed. | | time_last_before | Provide results before the defined timestamp for when the DNS record was last observed. | | time_last_after | Provide results after the defined timestamp for when the DNS record was last observed. | ## Time Format Time values must be specified as: - Unix/Epoch timestamp: Integer with seconds granularity (e.g., 1420070400) - Relative time: Negative integer representing seconds in the past (e.g., -31536000 for one year ago) ## Examples ### Absolute Time (Unix Timestamps) Filter for records first observed before January 1, 2015: ```bash curl -H "Accept: application/x-ndjson" -H "X-API-Key: $DNSDB_API_KEY" \ "https://api.dnsdb.info/dnsdb/v2/lookup/rrset/name/example.com?time_first_before=1420070400" ``` Filter for records last observed before 2013: ```bash curl -H "Accept: application/x-ndjson" -H "X-API-Key: $DNSDB_API_KEY" \ "https://api.dnsdb.info/dnsdb/v2/lookup/rrset/name/example.com?time_last_before=1356998400" ``` ### Relative Time (Seconds in the Past) Filter for records first observed within the last year: ```bash curl -H "Accept: application/x-ndjson" -H "X-API-Key: $DNSDB_API_KEY" \ "https://api.dnsdb.info/dnsdb/v2/lookup/rrset/name/example.com?time_first_after=-31536000" ``` Filter for records last observed within the last 31 days: ```bash curl -H "Accept: application/x-ndjson" -H "X-API-Key: $DNSDB_API_KEY" \ "https://api.dnsdb.info/dnsdb/v2/lookup/rrset/name/example.com?time_last_after=-2678400" ``` ## Combining Time Parameters You can combine multiple time parameters to create specific time ranges. ### Records Only Observed in 2015 Combine time_first_after and time_last_before to get records where both first and last observation occurred in 2015: ```bash curl -H "Accept: application/x-ndjson" -H "X-API-Key: $DNSDB_API_KEY" \ "https://api.dnsdb.info/dnsdb/v2/lookup/rrset/name/example.com?time_first_after=1420070399&time_last_before=1451606400" ``` ### Old Records Recently Observed Get records first observed before 2012 but last observed within the last month (long-lived records that are still active): ```bash curl -H "Accept: application/x-ndjson" -H "X-API-Key: $DNSDB_API_KEY" \ "https://api.dnsdb.info/dnsdb/v2/lookup/rrset/name/example.com?time_first_before=1325376000&time_last_after=-2678400" ``` ## Important Notes - The DNSDB API server may have API key-dependent time-fencing restrictions - These restrictions cannot be expanded with query parameters but can be used to narrow the time ranges - Query parameters can completely override default time ranges if permitted by your API key - All timestamps use seconds granularity - Relative times must be preceded by a minus sign (-) ## Common Time Values For reference, here are some common relative time values: | Duration | Seconds | Parameter Value | | --- | --- | --- | | 1 hour | 3,600 | -3600 | | 1 day | 86,400 | -86400 | | 1 week | 604,800 | -604800 | | 31 days | 2,678,400 | -2678400 | | 1 year | 31,536,000 | -31536000 | # DNSDB other query parameters In addition to time-fencing parameters, DNSDB supports several other optional query parameters that control result behavior and metadata. ## Common Parameters The following optional parameters may be present in lookup and summarize query URLs: | Parameter | Description | | --- | ----------- | | limit | Limit for the number of results returned via these lookup methods. There is a built-in limit to the number of results that are returned via these lookup methods. The default limit is set at 10,000. This limit can be raised or lowered by setting the "limit" query parameter. There is also a maximum number of results allowed; requesting a limit greater than the maximum will only return the maximum. See results_max below for information on that maximum. If "?limit=0" is used then DNSDB will return the maximum number of results allowed. Obviously, if there are less results for the query than the requested limit, only the actual amount can be returned. Example: appending "?limit=20000" to the URL path will set the response limit to 20,000 results.| | swclient | Name of the API client software generating the DNSDB query. Limited to twenty alphanumeric characters. This may be logged by the DNSDB API server. Farsight support can help you debug a new API client using this and the following parameter. There is no default. Example: "?swclient=dnsdbq". | | version | Version number of the API client software generating the DNSDB query. Limited to twenty alphanumeric characters plus dash, underscore, and period. This may be logged by the DNSDB API server. There is no default. Example: "?version=1.1a". | | id | Client software specific identity of the user of the API client. Comprised of an alphanumeric string, a colon, and an alphanumeric string, limited to thirty characters. This may be logged by the DNSDB API server. There is no default. Example: "?id=dnsq:91e6245ad31387".| | aggr | Aggregated results group identical rrsets across all time periods and is the classic behavior from querying the DNSDB. This means you could get the total number of times an rrset has been observed, but not when it was observed. Unaggregated results ungroup identical rrsets, allowing you to see how the domain name was resolved in the DNS across the full-time range covered in DNSDB (subject to time fencing). This can give a more accurate impression of record request volume across time because it will reveal the distinct timestamps of records whose values are repeated. You can answer questions like, "Was a domain parked for a long time, mostly unused, until it was repurposed for serving malware or relaying spam, but then was abandoned again?" It allows you to see if a record was observed heavily in the last week vs. having been observed constantly for years. This is a boolean value. Use True, the default, for the aggregated results or False for unaggregated results. The value is case insensitive and can be abbreviated. Example: "?aggr=f".| | humantime | A boolean value that is True if time values (in time_first, time_last, zone_time_first, zone_time_last) should be returned in human readable (RFC3339 compliant) format or False if Unix-style time values in seconds since the epoch should be returned. False is the classic behavior from querying the DNSDB and is the default value for this option. The value is case insensitive and can be abbreviated. Example: "?humantime=t".| ## Lookup-Specific Parameter The following optional parameter may be present in a lookup query URL: | Parameter | Description | | --- | ----------- | | offset | How many rows to offset (e.g. skip) in the results. This implements an incremental result transfer feature, allowing you to view more of the available results for a single query. The rows are offset prior to the limit parameter being applied, therefore offset allows seeing additional results past a limit that matches the maximum number of results. Note that DNSDB recalculates the results for each query and the order of results might not be preserved. Therefore, this capability is not a valid way to walk all results over multiple queries -- some results might be missing and some might be duplicated. The actual offset that can be used is limited or for certain API keys, offset is not allowed -- see the offset_max rate_limit key below. The offset value must be a positive integer. The default is 0, which means do not offset the rows. Example to return the second million results (assuming results_max is 100000): "?limit=100000&offset=1000000".| ## Summarize-Specific Parameter The following optional parameter may be present in a summarize query URL: | Parameter | Description | | --- | ----------- | | max_count | max_count controls stopping when we reach that summary count. The resulting total count can exceed max_count as it will include the entire count from the last rrset examined. The default is to not constrain the count. Example: appending "?max_count=100" to the URL path will stop the summary when its total count reaches 100.| ## Usage Examples ### Limit results to 100 ```bash curl -H "Accept: application/x-ndjson" -H "X-API-Key: $DNSDB_API_KEY" \ "https://api.dnsdb.info/dnsdb/v2/lookup/rrset/name/example.com?limit=100" ``` ### Use unaggregated results ```bash curl -H "Accept: application/x-ndjson" -H "X-API-Key: $DNSDB_API_KEY" \ "https://api.dnsdb.info/dnsdb/v2/lookup/rrset/name/example.com?aggr=false" ``` ### Return human-readable timestamps ```bash curl -H "Accept: application/x-ndjson" -H "X-API-Key: $DNSDB_API_KEY" \ "https://api.dnsdb.info/dnsdb/v2/lookup/rrset/name/example.com?humantime=true" ``` ### Combine multiple parameters ```bash curl -H "Accept: application/x-ndjson" -H "X-API-Key: $DNSDB_API_KEY" \ "https://api.dnsdb.info/dnsdb/v2/lookup/rrset/name/example.com\ ?limit=1000&aggr=f&humantime=t&swclient=myclient&version=1.0" ``` ### Use offset for pagination ```bash # First page curl -H "Accept: application/x-ndjson" -H "X-API-Key: $DNSDB_API_KEY" \ "https://api.dnsdb.info/dnsdb/v2/lookup/rrset/name/example.com?limit=1000" # Second page curl -H "Accept: application/x-ndjson" -H "X-API-Key: $DNSDB_API_KEY" \ "https://api.dnsdb.info/dnsdb/v2/lookup/rrset/name/example.com?limit=1000&offset=1000" ``` Warning: The offset parameter is not a reliable way to walk all results over multiple queries because DNSDB recalculates results for each query and the order might not be preserved. Some results might be missing and some might be duplicated. # DNSDB Flex Search API Guide ## Introduction This page documents the Flex Search API extensions to DNSDB APIv2. DNSDB APIv2 has two components: an enhanced Standard Search capability and Flexible Search. Flexible search adds ways to search DNSDB by regular expressions and globs (aka wildcarding). The Farsight Flexible Search Reference Guide should be read first, before reading this API programmer's guide. In conjunction with the DNSDB APIv2 page, this page describes how to make DNSDB Flexible search queries via the RESTful API. ## Audience This document is intended for programmers who want to write applications that can interact with the Flex API flexible search extensions to the RESTful DNSDB APIv2 using JSON and HTTP. ## High level features Flexible Search and its Flex API: * Flex provides a flavor of "regular expressions" known as "extended regular expressions," as used by the Unix egrep command. It supports most egrep-style regular expressions (except for capturing groups and backreferences), including simple string searches and partial label search. The particulars are known as "Farsight Compatible Regular Expressions" (FCRE) and are documented at FCRE Reference Guide. * Flex provides advanced wildcard search features, known in the Unix world as globs. They are documented at Glob Reference Guide. * Flex returns JSONL (newline delimited JSON) data, similar to what Standard Search returns, but the data are not the IETF Passive DNS - Common Output Format (COF) format. ### URL encoding When you're building URIs, some characters can't be used as normally written. These characters need to be written in their percent encoded form instead. Those special characters used in Flex regular expressions and globs patterns are: | special character | use in URLs | | ----- | ----- | | \! | %21 | | " | %22 | | / | %2F | | ? | %3F | | % | %25 | | \[ | %5B | | \\ | %5C | | \] | %5D | See the Example URLs section below for some examples. ### curl issue In this document we show many curl examples. When using the Flex API with curl and you have use square brackets or curly brackets in the URL, you are supposed to encode them as mentioned above. curl usually works anyway without them encoded, but by default curl will interpret them itself. For example, if you ran a curl command with URL .../dnsdb/v2/regex/rrnames/example[1-2] it would actually fetch two separate URLs: .../dnsdb/v2/regex/rrnames/example1 .../dnsdb/v2/regex/rrnames/example2 to stop this, ideally use URL encoding; second best, use the --globoff (or -g) parameter to curl. All our curl examples show this parameter in use, even when not strictly necessary. ## Summary of differences between APIv2 and the Flex API APIv1 and APIv2 only allow whole-label left-hand-side and right-hand-side wildcards, like .fsi.io or www.fsi.*. Flexible search allows searches using regular expressions and globs (aka wildcarding) on rrnames and rdata. - DNSDB calls searches on the owner name (i.e. left-hand-side of a DNS query-response) an "rrset" query. Flexible search calls that an "rrnames" query, which is more consistent with DNS standards and reflects that the results from the search contain rrnames, not full RRsets. - The URL structure below /dnsdb/v2/ is completely different. * The returned data stream, while still encapsulated in Farsight's Streaming API Framing protocol, are different JSON objects. - The Flex API does not provide count, time_first, and time_last in returned results. In the Flex API, the aggr, humantime, and max_count query parameters are not* supported; the other APIv2 parameters are supported. - The limit parameter behaves slightly differently. In the Flex API, limit applies to the number of unique RRnames returned and not the number of JSON response objects returned, as it does in APIv2. That is, in the Flex API, limit=1 might return 1 unique RRname, but multiple JSON response objects, one per RRType for that name. * There are new error messages within the SAF "msg" object. - The X-RateLimit- headers are not returned in Flex API responses. Use the Standard Search API rate_limit request to get your quota information. ### Summary of commonalities between APIv2 and the Flex API * Both are encapsulated using the Farsight's Streaming API Framing protocol. * Both use the same API keys. * Both share the same quotas. * Both use the same HTTP Content-Types. * Both return rrtype. * Both return the same HTTP Response codes. ## Streaming API Framing The Flex API returns streamed data encapsulated in Farsight's Streaming API Framing protocol. The SAF protocol is only applicable for HTTP responses that had an HTTP status of 200. Here is an example Flex API query and response. Line breaks were added for readability, but the actual results were on one line: ``` https://api.dnsdb.info/dnsdb/v2/regex/rrnames/example\..*/ANY?limit=1 {"cond": "begin"} {"obj":{"rrname":"example.s.ac.","rrtype":"A"}} {"cond": "succeeded"} ``` ## Authentication The Flex API, as part of the DNSDB API 2.0, is provided over an encrypted HTTPS transport over the Internet at https://api.dnsdb.info/. Authentication is performed by providing an API key (a long string of hexadecimal digits or dashes) in a special HTTP request header, X-API-Key. For example, if your API key is d41d….8427e, then the following HTTP header should be added to the HTTP request: ``` X-API-Key: d41d....8427e ``` If the X-API-Key header is not present, or the provided API key is not valid, a 403 Forbidden response will be returned to the HTTP client. ## Getting an API Key To request a demonstration or a trial API key please contact the DomainTools sales team. ## Query URL structure The query URL structure is: api.dnsdb.info/dnsdb/v2/SEARCH_METHOD/WHAT_TO_SEARCH/VALUE/RRTYPE?query-parameters RRTYPE is optional, so the following is also valid: api.dnsdb.info/dnsdb/v2/SEARCH_METHOD/WHAT_TO_SEARCH/VALUE?query-parameters * /dnsdb/v2 - The first two path components. * SEARCH_METHOD \= Third-level path component: * regex - regular expression search * glob - full wildcarding * WHAT_TO_SEARCH \= Fourth-level path component: * rrnames - search in rrnames, supports "forward" searches based on the owner name of an RRset. * rdata - search in rdata, supports "inverse" searches based on RData record values * VALUE \= Fifth-level path component: * For rrnames queries, "LHS" DNS owner name regular expression or glob * For rdata queries, "RHS" normalized resource record value regular expression or glob * RRTYPE \= Sixth-level path component - Optional: * The official DNS RRType string value or TYPE\# where \# is the numeric value of the RRType. * If an undefined RRType (but not TYPE\# form) is specified (i.e. not in the DNS RFCs), then an HTTP 400 (Bad Request) error will be returned with message "Error: RRTYPE has an unsupported value". * The RRtype ANY is modified somewhat from its usual meaning. A search for RRtype ANY will match any supported RRtype. * ANY is the default if RRTYPE is not specified. * DNSSEC-related RRtypes are not permitted: CDS, CDNSKEY, DNSKEY, DS, DLV, NSEC, NSEC3, NSEC3PARAM, RRSIG, TA, nor our own ANY-DNSSEC. * If one of those DNSSEC RRTypes is specified, then an HTTP 400 (Bad Request) error will be returned with message "Error: RRTYPE has an unsupported DNSSEC value". * Only SOA, CNAME, HINFO, MX, NS, NAPTR, PTR, RP, SRV, SPF, or TXT (plus their TYPE\# equivalents) are indexed for rdata queries. If another RRType is provided, then an HTTP 400 (Bad Request) error will be returned with message "Error: rdata searches are not allowed for the specified RRType". * If any of the path component values are not recognized, then an HTTP 400 (Bad Request) error will be returned with message " has an unsupported value". * The maximum length of the VALUE path component and the exclude parameter value (which is described below) are 4096 characters each before URL encoding. ## Optional query parameters ### Time-fencing query parameters The DNSDB records when a specific DNS record was first and last observed. You may filter results by time using the "time_first_before," "time_first_after," "time_last_before," and "time_last_after" query parameters. These parameters expect a integer (Unix/Epoch time) with seconds granularity or a relative time in negative seconds, as relative times must be in the past from now. Note that the DNSDB API server may have API key-dependent time-fencing restrictions that cannot be expanded with these parameters but can reduce the results (narrow the time ranges, or completely override them). The following time-fencing optional parameters may be present in a query URL: | Parameter | Description | | ----- | ----- | | time_first_before | provide results before the defined timestamp for when the DNS record was first observed. For example, the URL parameter "time_first_before=1420070400" will only provide matching DNS records that were first observed before (or older than) January 1, 2015\. | | time_first_after | provide results after the defined timestamp for when the DNS record was first observed. For example, the URL parameter "time_first_after=-31536000" will only provide results that were first observed within the last year. (365 days \ 24 hours/day \ 60 minutes/hr \* 60 minutes/sec \= 31536000\) | | time_last_before | provide results before the defined timestamp for when the DNS record was last observed. For example, the URL parameter "time_last_before=1356998400" will only provide results for DNS records that were last observed before 2013\. | | time_last_after | provide results after the defined timestamp for when the DNS record was last observed. For example, the URL parameter "time_last_after=-2678400" will only provide results that were last observed after 31 days ago. | Combinations of the time parameters may be used to strictly provide or exclude results for specific time-ranges. For example, to only have results when the first observed date and the last observed date are both only in 2015, you can use "time_first_after=1420070399" combined with "time_last_before=1451606400". As another time combination example, to get DNS records that were first observed before 2012 and last observed within the last month (recently-observed records which have not changed in a very long time), use "time_first_before=1325376000" and relative "time_last_after=-2678400". #### Other query parameters The following other optional parameters may be present in a query URL: | Parameter | Description | | ----- | ----- | | exclude | The "exclude" parameter is used to exclude (i.e. filter-out) results that match it. It is described below this table. | | limit | Limit for the number of unique rrnames or rdata value results returned via these search methods. The default limit is set at 10,000. This limit can be raised or lowered by setting the "limit" query parameter. There is also a maximum number of results allowed; requesting a limit greater than the maximum will only return the maximum. See results_max in a rate_limit response for that maximum. If "?limit=0" is used then DNSDB will return the maximum number of results allowed. Obviously, if there are less results for the query than the requested limit, only the actual amount can be returned. Example: appending "?limit=20000" to the URL path will set the response limit to 20,000 results. | | offset | How many rows to offset (e.g. skip) in the results. This implements an incremental result transfer feature, allowing you to view more of the available results for a single query. The rows are offset prior to the limit parameter being applied, therefore offset allows seeing additional results past a limit that matches the maximum number of results. Note that DNSDB recalculates the results for each query and the order of results might not be preserved. Therefore, this capability is not a valid way to walk all results over multiple queries \-- some results might be missing and some might be duplicated. The actual offset that can be used is limited or for certain API keys, offset is not allowed \-- see the Farsight DNSDB API Version 2 section on offset_max. The offset value must be a positive integer. The default is 0, which means do not offset the rows. Example to return the second million results (assuming results_max is 100000): "?limit=100000\&offset=1000000". | | id | Client software specific identity of the user of the API client. Comprised of an alphanumeric string, a colon, and an alphanumeric string, limited to thirty characters. This may be logged by the DNSDB API server. There is no default. Example: "?id=dnsq:91e6245ad31387". | | swclient | Name of the API client software generating the DNSDB query. Limited to twenty alphanumeric characters. This may be logged by the DNSDB API server. Farsight support can help you debug a new API client using this and the following parameter. There is no default. Example: "?swclient=dnsdbflex". | | version | Version number of the API client software generating the query. Limited to twenty alphanumeric characters plus dash, underscore, and period. This may be logged by the DNSDB API server. There is no default. Example: "?version=1.1a". | #### The exclude parameter As mentioned above, the exclude parameter is used to exclude (i.e. filter-out) results that match it. Conceptually, this is like the shell pipeline: egrep $VALUE batch-output.json $ head -1 batch-output.json | jq . { "count": 2411, "zone_time_first": 1275401003, "zone_time_last": 1484841664, "rrname": "wikipedia.org.", "rrtype": "NS", "bailiwick": "org.", "rdata": [ "ns0.wikimedia.org.", "ns1.wikimedia.org.", "ns2.wikimedia.org." ] } ``` ## ASINFO/CIDR Lookups When the \-a option is used, every address seen in a response will cause a DNS lookup under the domain specified by the \-D option. This stream of DNS queries might be an intolerable information leak depending on the nature of the underlying research, and it could also lead to unusably bad performance depending on the placement of your configured recursive DNS service. For best results, always use an on-server or on-LAN recursive DNS service, and consider whether to configure that recursive DNS service to be a "stealth secondary" of the zone denoted by the \-D option. For the default \-D value, more information can be found online at http://archive.routeviews.org/dnszones/. Use of DNS lookups to retrieve ASINFO/CIDR metadata can be extremely fast and surveillance-free, but some attention must be paid in order to obtain that outcome. For occasional low-volume use, your current recursive DNS placement and configuration is probably good enough. Note that while Passive DNS information is historical, the ASINFO/CIDR annotations made possi-ble using the \-a and \-D options are based on current information. Internet routing system information may have changed since the DNS data was recorded. More information about this can be found online at https://github.com/dnsdb/dnsdbq/blob/master/README. ## Files ~/.isc-dnsdb-query.conf ~/.dnsdb-query.conf /etc/isc-dnsdb-query.conf or /etc/dnsdb-query.conf: configuration file which can specify the API key, etc. variables. The first of these which is readable will be used, alone, in its entirety. See the DNSDBQ\_CONFIG\_FILE environment variable which can specify a different configuration file to use. The variables which can be set in the configuration file are as follows: * DNSDBQ\_SYSTEM: contains the default value for the \-u option described above. * DNSDB\_API\_KEY, APIKEY contains the user's DNSDB apikey (no default). * DNSDB\_SERVER: contains the URL of the DNSDB API server (default is ), and optionally the URI prefix for the database. The default URI prefix for system "dnsdb2" is "/dnsdb/v2/lookup"; the default for "dnsdb1" is "/lookup". * CIRCL\_AUTH, CIRCL\_SERVER: enable access to a passive DNS system compatible with the CIRCL.LU system. ## Environment * DNSDBQ\_CONFIG\_FILE: specifies the configuration file to use, overriding the internal search list. * DNSDB\_API\_KEY: contains the user's apikey. The older APIKEY environment variable has been retired, though it can still be used in the configuration file. * DNSD\_SERVER: contains the URL of the DNSDB API server, and optionally a URI prefix to be used (default is "/lookup"). If not set, the configuration file is consulted. * DNSDBQ\_TIME\_FORMAT: controls how human readable date times are displayed from the \-p csv output format. If "iso" then ISO8601 (RFC3339) format is used, for example; "2018-09-06T22:48:00Z". If "csv" then an Excel CSV compatible format is used; for example, "2018-09-06 22:48:00". ## Exit Status Success (exit status zero) occurs if a connection could be established to the back end database server, even if no records matched the search criteria. Failure (exit status nonzero) occurs if no connection could be established, perhaps due to a network or service failure, or a configuration error such as specifying the wrong server hostname. # DNSDB Flexible Search Query Tool Reference ## Synopsis dnsdbflex \[\-cdFjhqTUv46\] \[\--exclude glob|regular\_expression\] \[\--force\] \[\--glob glob\] \[\--mode terse\] \[\--regex regular\_expression\] \[\-A timestamp\] \[\-B timestamp\] \[\-l query\_limit\] \[\-O offset\] \[\-s search\_what\] \[\-t rrtype\] \[\-u server\_sys\] ## DESCRIPTION dnsdbflex constructs and issues flexible search queries to Farsight Security's DNSDB system. The flexible searches include regular expressions and globs (i.e. full wildcarding). The results from dnsdbflex can be displayed directly as JSON or emitted in the dnsdbq batch file input format. Using the batch file format allows "pivoting" from an flexible search into complete DNSDB results. Values to \--glob, \--regex, or \--exclude are called search expressions. Search expressions must match the the data indexed in DNSDB's flexible search database. All DNS rrnames end in a dot. All rdata indexed in the database will end in a dot if it's a host name or will end in a double quote for other data. A search expression that does not conform with that may be a wasted query. For example, dnsdbflex \--glob '\.fsi.io' will not match anything because globs are right-anchored and that search expression does not end in a dot. In glob search expressions, dnsdbflex normally detects such violations and disallows them. Add a trailing dot to match something, dnsdbflex \--glob '\.fsi.io.' dnsdbflex doesn't detect such violations in regex search expressions. For example, dnsdbflex \--regex '.\.fsi.io$' will not match anything but dnsdbflex \--regex '.\.fsi.io.$' does (note the dot before the dollar sign). Search expressions must contain 7 bit clean, printable ASCII characters. Use Punycode IDN encoding to search for IDN domain names. Use \\DDD (where DDD is the decimal value of the character) to match non-printable characters in rdata strings. See the Farsight Compatible Regular Expression Reference Guide (FCRE) for a description of the wildcarding syntax, as well as more examples. You'll need to get an API key from Farsight to use dnsdbflex with DNSDB. Note that certain types of API keys may not be allowed to use this API, in which case, dnsdbflex will fail with error message "The type of API key is not allowed to use the DNSDB Flex API". ## OPTIONS Either \--glob or \--regex must be specified. Both cannot be specified at the same time. \--exclude glob|regular\_expression: Filters out results selected by a glob or regular expression. If \--glob was specified, then \--exclude takes a glob. If \--regex was specified, then \--exclude takes a regular expression. \--force: Issue search queries even if rejected by dnsdbflex's pattern checks. \--glob glob: Specify that dnsdbflex should do a glob search. Only the \* and \[\] glob operators are supported. Can abbreviate as \--g. \--mode terse: Specify mode of information to return in results. * terse: Can abbreviate as t. This is the only value currently supported and is the default, so the \--mode option need not be specified. * For rrnames queries, returns the rrname and rrtype. * For rdata queries, returns normalized rdata, rrtype, and raw\_rdata. \--regex regular\_expression: Specify that dnsdbflex should do a regular expression search in the FCRE syntax. Can abbreviate as \--r. \-A timestamp: Specify a backward time fence. Only results seen by the passive DNS sensor network on or after this time will be selected. See also \-c. See the TIME FENCING section for more information. \-B timestamp: Specify a forward time fence. Only results seen by the passive DNS sensor network on or before this time will be selected. See also \-c. See the TIME FENCING section for more information. \-c: By default, \-A and \-B (separately or together) will select partial overlaps of database tuples and time search criteria. To match only complete overlaps, add the \-c ("completeness") command line option (this is also known as "strict" mode). See the TIME FENCING section for more information. \-d: enable debug mode. Repeat for more debug output. \-F: specify batch output mode, outputting results in the batch format that dnsdbq \-f can read. \-F includes the rrtype in the batch file format queries \-- in contrast to \-T described below. * If searching for rdata, if an rdata value is not printable or contains whitespace, it will emit it using the raw\_rdata hex value and output a comment line giving the non-raw format. * See dnsdbq(1) for documentation of the batch format. \-h: emit usage and quit. \-j: output in JSON format, which is the default. \-j and \-F are mutually exclusive. \-l query\_limit: query for that limit's number of responses. If specified as 0 then the DNSDB API server will return the maximum limit of results allowed. If \-l is not specified, then the query will not specify a limit, and the DNSDB API server may use its default limit. \-O offset: to offset by \#offset the results returned by the query. This gives you approximate incremental results transfers. Results can be reordered between queries, so using progressively higher offsets is not guaranteed to return all results without duplicates or gaps. Offset cannot be negative and the default is 0\. \-q: (quiet) suppresses sending warning messages to stderr. \-s rrnames|rdata: what data to search. * rrnames: Search in the rrnames part of DNS records, aka the left-hand side. Can abbreviate as n. This is the default. * rdata: Search in the rdata part of DNS records, aka the right-hand side. Can abbreviate as d. \-t rrtype: specify the resource record type desired. Default is ANY. * For rrnames queries, valid rrtypes include those defined in DNS RFCs, including ANY, except DNSSEC types are not allowed (ANY-DNSSEC, CDNSKEY, CDS, DLV, DNSKEY, DS, NSEC, NSEC3, NSEC3PARAM, RRSIG, and TA resource record types). Also valid are TYPE\# values. * For rdata queries, only the following rrtypes are valid: CNAME, HINFO, MX, NAPTR, NS, PTR, RP, SOA, SPF, SRV, and TXT. Also valid are their TYPE\# values. \-T: Like \-F but does not include the rrtype in the batch file queries. This allows pivots to match against all available rrtypes. The batch output will also include a comment for each line including the rrtype. \-u server\_sys: specifies the syntax of the RESTful URL. The only system currently supported is "dnsdb2", which is the default. \-U: turns off TLS certificate verification (unsafe). \-v: report the version of dnsdbflex and exit. \-4: force connecting to the DNSDB server via IPv4. \-6: force connecting to the DNSDB server via IPv6. ## EXAMPLES ``` # Regular expression search of all rrnames that contain a coke label, # for all rrtypes, limit of 10 results. $ dnsdbflex --regex '.*.coke..*' -l 10 ``` ``` # Same query without using default values $ dnsdbflex --regex '.*.coke..*' -l 10 -s rrnames --mode terse ``` ``` # Glob search of all names that contain a coke label and have an 'A' RRType. $ dnsdbflex --glob '*.coke.*' -l 10 -t A ``` ``` # Pivot those results into dnsdbq for full DNSDB API results in json # form. Note that up to 11 DNSDB query quota units will be consumed, # 1 by dnsdbflex and 10 by dnsdbq. If we did not specify the RRType # to dnsdbflex, then it might return more than 10 results (one for # each RRType for each name) and we'd use more than 11 DNSDB query # quota units. $ dnsdbflex --glob '*.coke.*' -l 10 -t A -F | dnsdbq -f -j ``` ``` # Get names containing "coke" but then exclude all those containing "diet". $ dnsdbflex --glob '*.coke.*' --exclude '.*diet.*' -l 10 ``` ``` # Same query, but using regular expressions $ dnsdbflex --regex '.*.coke..*' --exclude '.*.diet..*' -l 10 ``` ## TIME FENCING Farsight's DNSDB flexible search provides time fencing options for searches. The \-A and \-B options take a timestamp as an argument. The timestamps may be one of following forms. * positive unsigned integer : in Unix epoch format. * negative unsigned integer : negative offset in seconds from now. * YYYY-MM-DD \[HH:MM:SS\] : in absolute form, in UTC time, as DNSDB does its fencing using UTC time. * %uw%ud%uh%um%us : the relative form with explicit labels (w=weeks, d=days, h=hours, m=minutes, s=seconds). Calculates offset from UTC time, as DNSDB does its fencing using UTC time. A few examples of how to use time fencing options: ``` # Responses after Aug 22, 2015 (midnight), # excluding records ALSO seen before that time. $ dnsdbflex... -c -A 2015-08-22 # Responses from 2015 (midnight to midnight), # but not excluding records ALSO seen outside that time range. $ dnsdbflex... -B 2016-01-01 -A 2015-01-01 ``` Certain settings for time fences may be used to accelerate queries for rrnames and rdata values which have been recently observed or which were first observed in the distant past. Time fencing may accelerate the query if either \-A or \-B (but not both) are supplied without \-c. A few examples of how to use time fencing options where the query may be accelerated: ``` # Responses after 2015-08-22 14:36:10, # but not excluding records ALSO seen before that time. $ dnsdbflex... -A "2015-08-22 14:36:10" # Responses from the last 60 minutes, # but not excluding records ALSO seen before that time. $ dnsdbflex... -A "-3600" # Responses after Aug 22, 2015 (midnight), # but not excluding records ALSO seen before that time. $ dnsdbflex... -A 2015-08-22 # Responses before Jan 22, 2013 (midnight), # but not excluding records ALSO seen after that time. $ dnsdbflex... -B 2013-01-22 ``` ## FILES ~/.dnsdb-query.conf or /etc/dnsdb-query.conf: configuration file which can specify the API key, etc. variables. The first of these which is readable will be used, alone, in its entirety. See the DNSDBQ\_CONFIG\_FILE environment variable which can specify a different con- figuration file to use. For backwards compability, ~/.isc-dnsdb-query.conf and /etc/isc-dnsdb-query.conf are also valid, but deprecated. The variables which can be set in the configuration file are as follows: * DNSDB\_API\_KEY, APIKEY: contains the user's API key (no default). * DNSDB\_SERVER: contains the URL of the DNSDB API server (default is https://api.dnsdb.info/), and optionally the URI prefix for the database. DNSDBQ\_SYSTEM: contains the default value for the u* option described above. Can only be "dnsdb2". If unset, dnsdbflex will probe for any configured system. ## ENVIRONMENT The following environment variables affect the execution of dnsdbflex: * DNSDBQ\_CONFIG\_FILE: specifies the configuration file to use, overriding the internal search list. * DNSDB\_API\_KEY: contains the user's apikey. The older APIKEY environment variable has been retired, though it can still be used in the configuration file. * DNSDB\_SERVER contains the URL of the DNSDB API server, and optionally a URI prefix to be used. If not set, the configuration file is consulted. * DNSDBQ\_SYSTEM See DNSDBQ\_SYSTEM in the FILES section above. ## EXIT STATUS Success (exit status zero) occurs if a connection could be established to the back end database server, even if no records matched the search criteria. Failure (exit status nonzero) occurs if no connection could be established, perhaps due to a network or service failure, or a configuration error such as specifying the wrong server hostname. ### SEE ALSO * dnsdbq(1), jq(1), libcurl(3) # DNSDB Search Time Fencing ## Introduction DNSDB® 2.0 introduces a host of powerful new features and improvements - the biggest of which is Flexible Search. Both the DNSDB Standard Search API and the Flexible Search API support searching within specific time constaints, which we call "time fencing". Time-fencing parameters are used for matching results based on before or after the first or last observation times for the DNS records. By default, DNSDB Standard Search and Flexible API clients search for results throughout all our data, from 2010 on. Sometimes, you only care about recent data. An example is "what did the DNS look like in the past day?", or "what new records started being used in the past day?". Or maybe you want to know about a particular historical period, i.e. "what did the DNS look like between 12/1/2024 to 1/5/2025". Or "what records were only used in that timeframe?" ## DNSDB Time Fencing Here is how to do those searches using our command-line tools dnsdbq and dnsdbflex: "what did the DNS look like in the past day?" is a loose time fence. Using dnsdbq, use a relative timestamp of \-86400, computed as 60 seconds \ 60 minutes \* 24 hours with the \-A option: dnsdbq ... -A -86400. * "what new records started being used in the past day?" is a complete (aka strict) time fence. Add the \-c option: dnsdbq ... -A -86400 -c. * "what did the DNS look like between 12/1/2015 to 1/5/2016" is a loose time fence. Using dnsdbq, use both \-A and \-B with full time stamps: dnsdbq ... -A 2015-12-01 -B 2016-01-05. * "what records were only used in that timeframe?" is a complete (aka strict) time fence. Add the \-c option: dnsdbq ... -A 2015-12-01 -B 2016-01-05 -c. The same options work for dnsdbflex. Note that there is a slight difference in behavior for the time-fencing queries when using DNSDB Standard API versus the Flexible Search API. We will illustrate this using dnsdbq and dnsdbflex, and add some corresponding screen shots from DNSDB Scout®. Using dnsdbq, we search for all the farsightsecurity.com A records, sorting by time_last in csv form: ``` $ dnsdbq -r farsightsecurity.com -p csv -t A -s -k last time_first,time_last,zone_first,zone_last,count,bailiwick,rrname,rrtype,rdata "2013-07-17 22:08:50","2013-09-25 15:47:47",,,628,"farsightsecurity.com.", "farsightsecurity.com.","A","149.20.4.207" "2013-09-25 15:37:03","2015-04-01 06:17:25",,,6350,"farsightsecurity.com.", "farsightsecurity.com.","A","66.160.140.81" "2015-04-01 14:17:52","2018-09-27 00:29:43",,,36770,"farsightsecurity.com.", "farsightsecurity.com.","A","104.244.13.104" "2018-09-27 11:18:14","2020-08-11 19:47:37",,,7503,"farsightsecurity.com.", "farsightsecurity.com.","A","104.244.14.108" "2020-08-11 21:02:15","2020-11-06 12:30:55",,,1577,"farsightsecurity.com.", "farsightsecurity.com.","A","104.244.14.95" ``` Converting that csv output to a table form for readability: Let's look around the time stamp "2020-08-11 19:47:37" in that data. (Note that DNSDB uses UTC time, as our DNS sensors are located around the entire world.) The last two rows were seen before and after that time. Only the last row was seen only after that time. To illustrate: A Standard Search for those within or overlapping that time range (not completely in the time range, aka loosely) - this includes those also seen before that time: ``` $ dnsdbq -r farsightsecurity.com -p csv -t A -s -k last -A "2020-08-11 19:47:37" time_first,time_last,zone_first,zone_last,count,bailiwick,rrname,rrtype,rdata "2018-09-27 11:18:14","2020-08-11 19:47:37",,,7503,"farsightsecurity.com.", "farsightsecurity.com.","A","104.244.14.108" "2020-08-11 21:02:15","2020-11-06 12:30:55",,,1577,"farsightsecurity.com.", "farsightsecurity.com.","A","104.244.14.95" ``` The dnsdbq and dnsdbflex \-A option is expressed in DNSDB Scout as a Time Fencing "Seen After" parameter. Their \-B option is expressed in DNSDB Scout as a Time Fencing "Seen Before" parameter. The \-c option, for complete or strict, as the "Strict Mode" checkbox. Using DNSDB Scout, you should enter "2020-08-11 19:47:37" in the "Seen After" box with "Strict Mode" unchecked for a similar answer. Then searching results completely within the time range (with \-c) - this excludes those rows also seen before that time: ``` $ dnsdbq -r farsightsecurity.com -p csv -t A -s -k last -A "2020-08-11 19:47:37" -c time_first,time_last,zone_first,zone_last,count,bailiwick,rrname,rrtype,rdata "2020-08-11 21:02:15","2020-11-06 12:30:55",,,1577,"farsightsecurity.com.", "farsightsecurity.com.","A","104.244.14.95" ``` Using DNSDB Scout, check the "Strict Mode" box for a similar answer. A big difference between Standard Search and Flexible Search is that Flexible Search RRNames searches only the Left-Hand Side (LHS) data, shown in the green box: The Flex data does not have the RData values, i.e. the Right-Hand Side (RHS), in the red box, nor the other values struck-out above. Flex also has the time values, which are neither LHS nor RHS, in the blue box. Effectively, then, Flex Search has the subset of the above data, in the blue and green boxes, which effectively collapses to a single record using the earliest time_first and latest time_last: Going back to that time stamp above, if we make a loose Flex search we'll get a result: ``` $ dnsdbflex --g farsightsecurity.com. -t A -A "2020-08-11 19:47:37" {"rrname":"farsightsecurity.com.","rrtype":"A"} ``` Even without the \-A option, we'll get the same result: ``` $ dnsdbflex --g farsightsecurity.com. -t A {"rrname":"farsightsecurity.com.","rrtype":"A"} ``` And with DNSDB Scout: But if we Flex search completely within the time range, we get "no results found for query": ``` $ dnsdbflex --g farsightsecurity.com. -t A -A "2020-08-11 19:47:37" -c Query status: NOERROR (no results found for query.) ``` Using DNSDB Scout, check the "Strict Mode" box for a similar answer. This should make sense now, as the row existed before and after the timestamp. In the DNSDB data available to Standard Search, there is an A record with a new IP (104.244.14.95) at 2020-08-11 21:02:15 so including the right-hand-side we see completely a new row at that time. To repeat what we showed above: ``` $ dnsdbq -r farsightsecurity.com -p csv -t A -s -k last -A "2020-08-11 19:47:37" -c time_first,time_last,zone_first,zone_last,count,bailiwick,rrname,rrtype,rdata "2020-08-11 21:02:15","2020-11-06 12:30:55",,,1523,"farsightsecurity.com.", "farsightsecurity.com.","A","104.244.14.95" ``` In summary: there are cases where time fencing with Standard Search will return more rows of data than a similar Flex Search. # DNSDB search glob guide Globbing is an advanced form of wildcard searches, more powerful than DNSDB's Standard Search left-hand or right-hand wildcards, but not as advanced as Farsight Compatible Regular Expressions (FCRE). They can be simpler to write, especially for API users who are not familiar with regular expressions. In general, Farsight's glob implementation follows standard Unix glob(7) semantics, but not what's sometimes referred to as "extended globbing." Glob searches are evaluated against the DNS master file form of the hostnames (aka rrnames) and rdata values, which by design contains only printable ASCII characters. All non-printable characters, including octets outside the ASCII range, are converted to escape sequences in the form \DDD (backslash followed by three decimal digits) per RFC 1035. This is only applicable to RData (RHS) queries. ## Glob Syntax A glob is a string of printable characters with the following characters given special meaning: ` \-- Match any zero or more characters. * ? \-- Match exactly any one character. * [ \-- Begin a character class. Any of the contained characters or ranges will match. * ] \-- End a character class. * \ \-- Escape the next character (but not within a character class) Any other characters in globbing pattern get matched exactly as written, except that characters are not case sensitive. ### Character Class Syntax A character class is a set of characters enclosed between an opening [ and a closing ]. A simple example is [m-z1-3] to match characters m through z and 1 to 3\. Within the character class, the following characters are handled specially: * ! \-- If the first character after the opening [, denotes a negated character class, i.e. a class which matches any character not listed in the remainder of the class. * ] \-- If the first character after the opening [ or [!, encodes a literal ] as a member of the class. A ] after the first character after the opening [ or [! ends the character class. * - \-- If the first character after the opening [ or [! or the last character before the closing ], encodes a literal - as a member of the character class. * If between two characters A and B, encodes the range of characters between A and B, inclusive, as members of the character class. The character A must occur before B in ASCII encoding. The sequences [. and [= are not allowed between the opening [or [! and the closing ], to prevent confusion with unsupported POSIX collation sequences and collation classes. If the sequence [: appears in a character class, it must be the beginning of one of the following POSIX character classes: * [:alnum:] (POSIX character class) \-- Alphanumeric characters 0-9, A-Z, and a-z * [:alpha:] (POSIX character class) \-- Alphabetic characters A-Z, a-z * [:blank:] (POSIX character class) \-- Blank characters (space and tab) * Only printable characters occur in searchable strings and space is the only printable whitespace character, thus use of [:blank:] is equivalent to a space character. * Tabs in data appear as the escape sequence \009 and can be matched with the glob pattern \\009. * [:cntrl:] (POSIX character class) \-- Control characters * Only printable characters occur in searchable strings, so [:cntrl:] will not match any characters. * Control characters in data will appear as escape sequences in the form \DDD (backslash followed by three digits). To match one of those, you need to escape the backslash with another backslash. For example, to match the literal string \004, use the glob pattern \\004. * [:digit:] (POSIX character class) \-- Decimal digits 0-9 * [:graph:] (POSIX character class) \-- Any printable character other than space. * Only printable characters occur in searchable strings, thus a character class containing [:graph:] is equivalent to [! ] (negated character class containing only a space). * [:lower:] (POSIX character class) \-- Lower case alphabetic characters a-z * Hostnames will be folded to lower case, thus use of [:lower:] is equivalent to [:alpha:]. * [:print:] (POSIX character class) \-- Any printable character * Only printable characters occur in searchable strings, so [:print:] will match any character. * [:punct:] (POSIX character class) \-- Punctuation characters (printable characters other than space and [:alnum:]) * [:space:] (POSIX character class) \-- Any whitespace character * The space character is the only printable whitespace character, thus use of [:space:] is equivalent to a space character. * [:upper:] (POSIX character class) \-- Upper case alphabetic characters A-Z * Since all of our data is indexed as lower-case, this is not useful as it is equivalent to [:lower:]. * [:xdigit:] (POSIX character class) \-- Hexadecimal digits 0-9, a-f, A-F The above named character classes must appear inside an enclosing [ and ], e.g. [[:digit:][:punct:]] to match a digit or punctuation character. Without the enclosing braces, [:digit:] will match the characters :, d, i, g, or t. Neither the above character classes nor a character range may begin or end a character range. For example, the character class expressions [0-[:alpha:]] and [a-n-z] are invalid. All other characters between the opening [ or [! and the closing ] are added to the character class, including the backslash \ character. There is no way to express a character class containing a single ! character. ### Important notes * Glob searches are not case sensitive. * Globbing patterns are "anchored" front and back by default. (This is a major difference from FCRE.) * All hostnames (rrnames) in the DNS dataset end in a ., which must be accounted for in globs. Therefore, a search for .com will not match any hostnames. A glob that searches in rrnames must end in something that matches a ., so *.com. would match what was intended. * All well-formed rdata we currently index in the DNS dataset ends in a . or a ", which should be accounted for in globs. Therefore, a glob that searches in rdata should* end in something that matches a . or a ". * There must be at least two consecutive non-wildcard characters in the pattern. The implicit front and back anchor counts as a non-wildcard character. ## Examples * To match hostnames with a label containing the word "smoke": Glob pattern: smoke* * Search type: rrnames search * Example results: * smokeping.pdf.ac. * smoke.tesla.ac. * To match hostnames with a label containing the word "cider" but not containing "hard": Glob pattern: cider* Search type: rrnames search with exclude filter hard* * Example results: * ciderpress.ca. * colombus.citycider2018.eventbrite.ca. * To match hostnames with a label ending in "www." and a later label starting with ".com": Glob pattern: www..com * Search type: rrnames search * Example results: * www.example.com. * dev-www.subdomain.example.com. * www.example.com.cdn.net. * stage-www.dev.community.org. * To match hostnames starting with "www." and ending in ".com.": Glob pattern: www..com. * Search type: rrnames search * Example results: * www.example.com. * www.subdomain.example.com. * To match hostnames starting with "www." and ending with ".com" with no other dots in between: This cannot* be done in a general way using globs; use regular expression instead. * To match hostnames starting with "www" optionally preceded by a "dev-" or "stage-" prefix in a .net or .edu domain: This cannot* be done in a general way using globs; use regular expression instead. * To match TXT records encoding an SPF policy with a ~all default: Glob pattern: "v=spf1 ~all" * Search type: rdata search * Example results: * "v=spf1 a mx ~all" * "v=spf1 a 10.2.0.0/16 ~all" * To match single character domain names (which are really two character domain names when you add the implicit trailing '.'): * Glob pattern: ?. * Search type: rrnames or rdata search * Example results: * a. * 0\. * To match "bri" followed by exactly any three characters followed by "morning" followed by anything (or nothing): Glob pattern: bri???morning * Search type: rrnames search * Note: A question mark matches exactly one character * Example results: * brightmorning.com * brightmorningtoday.com * To match "ns" followed by any single digit followed by anything (or nothing) and ending in ".net.": Glob pattern: ns[0-9].net.` * Search type: rrnames search * Example results: * ns0.fsi.net * ns0abc.fsi.net # DNSDB Search with Compatible Regular Expressions DNSDB Farsight Compatible Regular Expressions (FCRE) provides regular expression (regexp) functionality for searching Domain Name System (DNS) hostnames and rdata values in DNSDB. The system evaluates regexp searches against the DNS master file form of the hostnames and rdata values, which by design contains only printable ASCII characters. The system converts all non-printable characters, including octets outside the ASCII range, to escape sequences in the form \DDD (backslash followed by three decimal digits) per RFC 1035. This is only applicable to RData (RHS) queries. For this limited use case, DNSDB FCRE provides a simplified subset of the Portable Operating System Interface (POSIX) Extended Regular Expression syntax, with the most notable restrictions being: 1. Only printable characters are allowed in a regexp. 2. Hexadecimal or octal escape sequences aren't allowed in a regexp. 3. Only special characters may be escaped with '\\'. Note that '\]' and '}' aren't considered special characters, but '\[' and '{' are. 4. POSIX collating elements (e.g., \[=ch=\], \[.a.\]) in character classes aren't supported. The sequences \[= and \[. aren't allowed in character classes. 5. As in POSIX regexps, the character '\\' has no special meaning within a character class, so the class \[\\w\] matches the characters '\\' or 'w'. 6. Capturing groups and backreferences aren't supported. Note that restriction (3) means that Perl Compatible Regular Expressions (PCRE) extensions such as \w (word characters) and \d (digits) aren't allowed in FCRE regexps. ## Regexp syntax A regular expression is a string of printable characters. The following characters have special meaning: * \ \-- Escape the next character, which must be a special character. A regexp may not end with an unescaped backslash, or contain an unescaped backslash followed by a character other than another backslash or the special characters listed below, except inside of a character class. * ^ \-- Matches the beginning of the subject string. * $ \-- Matches the end of the subject string. * [ \-- Begin a character class * . \-- A special character class matching any character. * ( \-- Begin a sub-pattern. Sub-patterns may occur within other sub-patterns. * ) \-- End a sub-pattern. * | \-- Specify an alternative match. A pattern or subpattern matches if the pattern before or after the '|' matches. ` \-- Match the previous character, character class, or subpattern zero or more times. * ? \-- Match the previous character, character class, or subpattern at most once. * + \-- Match the previous character, character class, or subpattern at least once. * { \-- If followed by a character other than a decimal digit, is treated as a literal '{' character. Such a '{' may be escaped with a backslash even though it isn't technically a special character in this context. If followed by a decimal digit, begins a bounded match specification. "{n}" matches exactly n repetitions of the previous character, character class, or subpattern. "{n,m}" with m >=n matches at least n but at most m repetitions. ## Character class syntax A character class is a set of characters enclosed between an opening [ and a closing ]. Within the character class, the following characters have special handling: * ^ \-- If the first character after the opening [, denotes a negated character class, i.e. a class which matches any character not listed in the remainder of the class * ] \-- If the first character after the opening [ or [^, encodes a literal ] as a member of the class. A ] after the first character after the opening [ or [^ ends the character class. * - \-- If the first character after the opening [ or [^ or the last character before the closing ], encodes a literal - as a member of the character class. If between two characters A and B, encodes the range of characters between A and B, inclusive, as members of the character class. The character A must occur before B in ASCII encoding. The sequences [. and [= aren't allowed between the opening [or [^ and the closing ], to prevent confusion with unsupported POSIX collation sequences and collation classes. If the sequence [: appears in a character class, it must be the beginning of one of the following POSIX character classes: * [:alnum:] (POSIX character class) \-- Alphanumeric characters 0-9, A-Z, and a-z * [:alpha:] (POSIX character class) \-- Alphabetic characters A-Z, a-z * [:blank:] (POSIX character class) \-- Blank characters (space and tab) * Only printable characters occur in searchable strings and space is the only printable whitespace character, thus use of [:blank:] is equivalent to a space character. * Tabs in data appear as the escape sequence \009 and can be matched with the pattern \009. * [:cntrl:] (POSIX character class) \-- Control characters * Only printable characters occur in searchable strings, so [:cntrl:] won't match any characters. * Control characters in data will appear as escape sequences in the form \DDD (backslash followed by three digits). To match one of those, you need to escape the backslash with another backslash. Use the pattern \[:digit:]{3} in a regular expression. * [:digit:] (POSIX character class) \-- Decimal digits 0-9 * [:graph:] (POSIX character class) \-- Any printable character other than space. * Only printable characters occur in searchable strings, thus a character class containing [:graph:] is equivalent to [^ ] (negated character class containing only a space). * [:lower:] (POSIX character class) \-- Lower case alphabetic characters a-z * Hostnames will be folded to lower case, thus use of [:lower:] is equivalent to [:alpha:]. * [:print:] (POSIX character class) \-- Any printable character * Only printable characters occur in searchable strings, so [:print:] matches any character. * [:punct:] (POSIX character class) \-- Punctuation characters (printable characters other than space and [:alnum:]) * [:space:] (POSIX character class) \-- Any whitespace character (tab, newline, vertical tab, form feed, carriage return, and space) * The space character is the only printable whitespace character, thus use of [:space:] is equivalent to a space character. * Tabs in data appear as the escape sequence \009 and can be matched with the pattern \009. The other characters can also be matched by searching for their decimal equivalent. * [:upper:] (POSIX character class) \-- Upper case alphabetic characters A-Z * Since all of our data is indexed as lower-case, this isn't useful as it is equivalent to [:lower:]. * [:xdigit:] (POSIX character class) \-- Hexadecimal digits 0-9, a-f, A-F The above named character classes must appear inside an enclosing [ and ], e.g. [[:digit:][:punct:]] to match a digit or punctuation character. Without the enclosing braces, [:digit:] will match the characters :, d, i, g, or t. Neither the above character classes nor a character range may begin or end a character range. For example, the character class expressions [0-[:alpha:]] and [a-n-z] are invalid. All other characters between the opening [ or [^ and the closing ] are added to the character class, including the backslash \ character. There is no way to express a character class containing a single ^ character: an escaped \^ should be used instead of a character class. ## Important notes * Regular expression searches aren't case sensitive. Regular expression patterns are not* "anchored" front and back by default. (This is a major difference from glob searches.) * To exactly match a literal . (such as between labels in a DNS name), you need to escape it with a backslash. Example pattern: google\.com. This isn't necessary if the . is inside a character class, for example foo[.-_]bar. If you don't escape the ., the pattern google.com will match 'googlexcom', 'google_com', etc. * All rrnames (i.e. hostnames) in the DNS dataset end in a ., which must be accounted for in regular expressions. * All well-formed rdata we currently index in the DNS dataset ends in a . or a ", which should be accounted for in regular expressions. Wildcard operators (., , +, ?, and character classes like [a-z]) must have at least two non-wildcard characters immediately before or after them. Anchors (^ and $) count as non-wildcard characters for this purpose. ### Valid wildcard patterns Pattern: ^vpn-[a-z]+\..$ - Valid: The $ anchor provides a second non-wildcard character after .* * Pattern: ^vpn-[a-z]+\..{2,} - Valid: Explicit 2+ character match after the dot Pattern: example\.com. - Valid: "com" provides 3 characters before the wildcard Pattern: ^www\..\.com\.$ - Valid: Anchors and literal characters satisfy the requirement ### Invalid wildcard patterns Pattern: ^vpn-[a-z]+\.. - Invalid: Only one character (the escaped dot) before the final .* wildcard Pattern: x. - Invalid: Only one character before the wildcard Pattern: .y - Invalid: Only one character after the wildcard * Pattern: .+example - Invalid: Only one character (the start anchor is implicit but not present) ## Examples The following examples show regular expression patterns and some of their matching values: Pattern: www\..\.com * Matches: Hostnames with a label ending in "www." and a later label starting with ".com" * Example results: * www.example.com. * dev-www.subdomain.example.com. * www.example.com.cdn.net. * stage-www.dev.community.org. Pattern: ^www\..\.com * Matches: Hostnames starting with "www." and ending in ".com" * Example results: No results * Note: Hostnames in the DNS dataset contain a trailing ".", which must be accounted for in regexps. This pattern is missing the trailing dot. Pattern: ^www\..\.com\.$ * Matches: Hostnames starting with "www." and ending in ".com." * Example results: * www.example.com. * www.subdomain.example.com. * Pattern: ^www\.[^.]+\.com\.$ * Matches: Hostnames starting with "www." and ending with ".com" with no other dots in between * Example results: * www.example.com. * www.other-domain.com. * Pattern: ^((dev|stage)-)?www\.[^.]+\.(net|edu)\.$ * Matches: Hostnames starting with "www" optionally preceded by a "dev-" or "stage-" prefix in a .net or .edu domain * Example results: * www.college.edu * dev-www.isp.net Pattern: ^"v=spf1 . ~all"$ * Matches: TXT records encoding an SPF policy with a ~all default * Example results: * "v=spf1 a mx ~all" * "v=spf1" " a " "10.2.0.0/16" " ~all" * Pattern: (^|[-.])star([-]?)z[-._] * Matches: Hostnames that start with "star", or have "star" as a label or otherwise separate from other letters/digits, followed by an optional dash or underscore, then a z, then a period, dash or underscore * Use case: Looking for a visibly embedded trademark * Example results: * star-z.at. * edge-star-z-mini-shv-02-mia3.goldmansachs.de. * starz.webex.com. * shooting-starz.tv.` # DNSDB GitHub repositories Farsight Security maintains a collection of open-source tools and libraries for working with DNSDB and passive DNS data. These resources are designed for developers building applications that integrate with or process DNSDB data. ## Core Libraries - nmsg - Network message encapsulation library (C) - mtbl - Immutable sorted string table library (C) - go-nmsg - Golang implementation of the nmsg network message encapsulation library - golang-framestream - Frame Streams implementation in Go ## Python Extensions - pydnstable - Python extension module for libdnstable - pywdns - Python extension module for libwdns - pymtbl - Python extension module for libmtbl - pynmsg - Python extension module for nmsg ## Utilities - dnstable-convert - Conversion utility for passive DNS data - nmsg-relay - Utility for uploading NMSG data using the sielink protocol - sielink - Protocol library for uploading data to the Farsight Security Information Exchange - dnstap-sensor - Dnstap-based SIE DNS sensor - go-config - Useful golang types for JSON and YAML config files ## All Repositories For the complete list of repositories and additional tools, visit: https://github.com/orgs/farsightsec/repositories ## Licensing Most repositories are licensed under the Apache License 2.0 or Mozilla Public License 2.0. Check individual repository licenses for specific terms. # DNSDB FAQ ## General questions ### What is passive DNS? Passive DNS uses observed cache miss traffic collected from above recursive resolvers to build a database detailing relationships between domain names, IP addresses, and nameservers. That historical database can then be queried to get a report of: * Domains that have been seen associated with a particular IP or IP range * IPs that have been seen associated with a particular domain name * Domain names that are known to be using a particular authoritative nameserver, etc. * The date and time range associated with associations and changes One example of the value of using passive DNS can be seen when you compare what passive DNS finds vs what you may get when you just request a PTR record for that same IP address. For instance, let’s request the PTR record for 128.223.142.89: ```sh $ dig -x 128.223.142.89 +short www.uoregon.edu. ``` This PTR makes it appear as if 128.223.142.89 is home to www.uoregon.edu, and at one time perhaps it was. However, at the time this example was prepared, www.uoregon.edu was actually at: ```sh $ dig www.uoregon.edu +short drupal-cluster5.uoregon.edu. 128.223.142.244 ``` If we check dnsdb and ask to just see passive DNS records for 128.223.142.89 from the last 6 months (24 weeks), we can see more current results that look like: ```sh $ dnsdb_query.py -i 128.223.142.89 --after=24w | sort cfc.uoregon.edu. IN A 128.223.142.89 culjp.org. IN A 128.223.142.89 oregon-ix.com. IN A 128.223.142.89 oregon-ix.net. IN A 128.223.142.89 oregonix.net. IN A 128.223.142.89 oregonix.org. IN A 128.223.142.89 virt-www.uoregon.edu. IN A 128.223.142.89 www.culjp.org. IN A 128.223.142.89 ``` The equivalent command using the dnsdbq CLI command is: ```sh dnsdbq -i 128.223.142.89 -A 24w | fgrep -v ';' | grep . | sort ``` ### How does passive DNS data differ from WHOIS data? WHOIS is an online distributed database that documents control over particular Internet resources such as domain names, blocks of IP addresses, and autonomous system numbers (ASNs). WHOIS normally contains manually-maintained point of contact information, as well as information about the dates when resources were received or modified, plus additional details associated with resources (these details may vary depending on the type of resource or the specific WHOIS operator). Passive DNS is a database that contains automatically collected information gleaned from DNS queries and responses, and consists of observed and imputed relationships between domain names, IP addresses, and nameservers. Passive DNS also captures other types of data delivered via DNS, such as DKIM/DMARC records, SPF records, etc. ### How much data is in the DNSDB? The DNSDB database currently has over 100 billion unique DNS records. We currently see over 200,000 new raw observations/second totaling over 2TB of DNS data collected daily. ### How far back does DNSDB data go? When did you begin collecting and saving data for DNSDB? While DNSDB’s data collection began in 2007, various improvements were made over time. The currently utilized NMSG-based passive DNS architecture was put into production on June 24, 2010, and that is the earliest date you will see for passive DNS data. For example: ```sh $ dnsdb_query.py -r www.google.com/cname -s time_first ;; bailiwick: google.com. ;; count: 83,954,084 ;; first seen: 2010-06-24 04:22:00 -0000 ;; last seen: 2012-09-06 10:49:14 -0000 www.google.com. IN CNAME www.l.google.com. [etc] ``` The roughly equivalent command using the dnsdbq CLI command is: ```sh $ dnsdbq -r www.google.com/cname -s ``` Some data obtained from ICANN Zone File Access (ZFA) programs may go back slightly further. For example: ```sh $ dnsdb_query.py -r google.com/NS/com ;; bailiwick: com. ;; count: 2,157 ;; first seen in zone file: 2010-04-24 16:12:21 -0000 ;; last seen in zone file: 2016-03-30 16:14:20 -0000 google.com. IN NS ns1.google.com. google.com. IN NS ns2.google.com. google.com. IN NS ns3.google.com. google.com. IN NS ns4.google.com. ``` The equivalent command using the dnsdbq CLI command is: ```sh dnsdbq -r google.com/NS/com ``` ### I understand that DomainTools collects passive DNS from sensors located all around the Internet Who specifically contributes data to DomainTools? Do you have any sensors in country X or country Y? DomainTools has more than 400 sensors deployed Internet-wide, but we do not disclose either the identities of our sensor operators or their locations. ### I’m not seeing some domains that I think I should be seeing in DNSDB. Are you filtering anything out? Yes, some content is intentionally filtered from DNSDB for operational reasons. As of July, 2022 DNSDB was changed to reduce the amount of junk wildcard domains in its database. We are gradually rolling out a change to replace multiple wildcarded DNS rrnames with a single rrname that starts with a WILDCARD. label. No other rrname labels contain uppercase letters, so records with this (all upper case) WILDCARD. were never in DNSDB before. Note that there are existing, real, domain names that contain a wildcard. label (all lower case). While we do not disclose proprietary details of what we filter, broad categories of content filtered from DNSDB include (but aren’t limited to): * Randomized subdomain attack (garbage) DNS traffic * Inverse address records for dynamic domain names that match dynamic domain name patterns * DNS block list query and response traffic * Busy sites with wildcarded CNAMES (such as blogspot.com) ### I understand that DNSDB adds data to the database as it’s seen. What about domains that get created, but never get used? Do you somehow include them, too? Yes, DomainTools has visibility into domains that have been created but which are not yet used from its participation in the ICANN Zone File Access (ZFA) data sharing program. While we routinely add all domains learned via that ZFA process, the vast majority of DNSDB’s data comes from passively observed DNS traffic. ### Can I use DNSDB as a basis for making quantitative estimating about a domain’s “popularity” or “importance?” Because DomainTools collects above the recursive resolver, DomainTools only sees cache miss traffic. The volume of cache miss traffic is largely based on a domain’s popularity. Thus, you can at least get a rough sense of a domain’s relative popularity, e.g.: ```sh $ dnsdb_query.py -r www.google.com/a | grep count | awk '{print $3}' | \ sed 's/,//g' | paste -sd+ - | bc 1964126517 ``` Vs. ```sh $ dnsdb_query.py -r 7o8xg9qm0c.com/a | grep count | awk '{print $3}' | \ sed 's/,//g' | paste -sd+ - | bc 2830 ``` Here, www.google.com` has been seen far more often than the other relatively-obscure/seemingly-randomly-named domain however an analyst should avoid making hard quantified comparisons (avoid: “domain X is n times more popular than domain Y”). ### Can DNSDB help me ascertain information relevant to DKIM/DMARC and/or SPF? Yes. DKIM/DMARC and SPF both use the DNS to store the records they need. For example: ```sh $ dnsdb_query.py -r \*._domainkey.dmarc.org ;; bailiwick: dmarc.org. ;; count: 974 ;; first seen: 2012-02-02 07:30:08 -0000 ;; last seen: 2018-10-06 15:19:36 -0000 clochette._domainkey.dmarc.org. IN TXT "v=DKIM1; k=rsa; p=MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQCvWWQyy4vbyeNt8YN0KEHfPb5j/ BZHcOD7xu8rPbUoMFD6tskk9kpJOBOlWvei3hx6HWAqa7Q8EzlQc0ijqsRxSgMhvFnYUAKM2yewGF6+ QVsCPrLal0XvqOF+uAtScBj0BRYvTI9algsH+1DK8VzZ/bvOdoCM3rj8DJ/D8D3ugQIDAQAB" $ dnsdb_query.py -r _dmarc.dmarc.org ;; bailiwick: dmarc.org. ;; count: 181 ;; first seen: 2012-03-17 19:02:34 -0000 ;; last seen: 2018-09-30 08:04:47 -0000 _dmarc.dmarc.org. IN TXT "v=DMARC1; p=none; pct=100; rua=mailto:reports@dmarc.org; ruf=mailto:reports@dmarc.org" $ dnsdb_query.py -r dmarc.org/txt | grep spf | sort -u dmarc.org. IN TXT "v=spf1 a mx -all" ``` ### Does DNSDB include AAAA (IPv6) records? DNSSEC records? All IETF-defined record-types? Unlike some other passive DNS services that may offer only A or AAAA records, DNSDB includes all IETF-defined records types, including all IPv6- and all DNSSEC-related record types. For example, you can look for Google’s AAAA (IPv6) records by specifying: ```sh $ dnsdb_query.py -r www.google.com/aaaa ``` You can also easily find reverse IPv6 records in the ip6.arpa domain with PTR records in DNSDB. When it comes to DNSSEC, you can see the DS records for a sample domain such as internet2.edu by querying: ```sh $ dnsdb_query.py -r internet2.edu/ds ``` You can also see DNSKEY records, RRSIG records, and NSEC records in the data. ## Pricing-Related Questions ### How much does DNSDB cost for a “typical” user? To request a demonstration of DNSDB or to inquire about a trial API key please contact the DomainTools sales team. Some general pricing principles for your background: * There are three types of quotas: time-based, block-based, and unlimited. * Time-based quotas are usually applied on a daily basis and resets daily at 00:00 (midnight) in the UTC time zone. Time-based quotas can also be applied for arbitrary time-quanta, but this is unusual. These quotas are tiered/priced according to maximum daily usage with tiers starting at 1,000 queries/day. * Unlimited quotas do not limit the number of queries/day. * Discounts are available for multi-year prepaid purchases. * DNSDB Export is also available for “on-premises” deployment for use in offline environments, latency sensitive environments, or circumstances where special access to the DNSDB data is required. * Because DNSDB has security-sensitive information, all customers must be pre-approved for access. DomainTools reserves the right to decline any potential customer at its sole discretion. ### Do unused DNSDB queries “carry forward” or “roll over” for use in subsequent days? You mentioned that for time-based quotas, users contract for a maximum number of DNSDB queries per day. No. The query volume tier you purchase is essentially a “reservation of capacity” on our infrastructure. ### For block-based quotas, what happens if there are still unused queries left when the quota expires? Unless renewed prior to expiration, when a block-based quota expires, unused queries are lost. If the Block Quota Subscription is renewed PRIOR to expiration, unused queries will be added to the new subscription quota. ### If I purchase an “unlimited” license for DNSDB, does that mean I can open hundreds of parallel (concurrent) query streams? No, all customers are limited to a maximum of 10 parallel query streams at one time. If more parallel (concurrent) query streams are required for your use case, let’s discuss DNSDB Export as an option. ### I run busy recursive resolvers. If I contribute data from them, can I get a discount from DomainTools? Yes. Discount levels are based on the value of the contributions, measured by volume and uniqueness of the data shared. In a few cases, partners who have shared substantial volumes of unique data (such as large ISPs) have been eligible for 100% discounts. ### I’d like to contribute data. How do I install and run a sensor? If you would like to contribute data, please contact us and let us know. The entire DomainTools community would be delighted. DomainTools Passive DNS collects DNS response data received by caching, recursive DNS servers distributed around the Internet. This data is aggregated and made available via the DomainTools Security Information Exchange platform where it is imported in an anonymized form into DomainTools DNSDB. Operating a DomainTools Passive DNS sensor improves the quality of data available from DomainTools DNSDB and aids anti-abuse research. The passive DNS sensor only collects the DNS data received by a caching server as the result of recursion. The queries sent by individual clients are never logged. The sensor also offers the ability to zero out the IP address of the resolver. ### I run a farm of busy authoritative nameservers. Can I contribute data to DNSDB and get a discount? We do participate in ICANN’s Zone File Access (ZFA) program, but don’t currently collect data from authoritative nameserver operators. If you operate authoritative nameservers and would like to discuss data sharing opportunities, please get in touch with DomainTools. ### I’m a university faculty member or graduate student. Do you have discounted or free access to DNSDB for academic researchers? DomainTools enthusiastically supports academic research, and is happy to consider requests for discounted or free access to DNSDB. Please contact us with your request and we will evaluate it to see what’s possible. ### I fight cyber crime on a volunteer basis (no pay or other compensation, just working for the good of the Internet). Can I obtain discounted or free access to DNSDB? DomainTools is pleased to support bona-fide “do-gooders” working to better the Internet by offering deeply discounted or free access to DNSDB. Please contact us with your request and we will evaluate it to see what’s possible. ## Privacy-Related Questions ### Where are DomainTools' facilities physically located? What about DomainTools sensor operators and customers? DomainTools is headquartered in San Mateo, California, USA. Our data centers are located in the states of Virginia and California of the USA. We have customers and sensor operators distributed internationally, in addition to numerous domestic customers and sensor operators. ### We’d like to contribute passive DNS data, but need to understand how you protect end user privacy and sensor operator identities. What steps do you take? Because DomainTools collects cache miss traffic from above large recursive resolvers, query traffic appears to come from the recursive resolvers themselves rather than any individual user. This architecture provides substantial privacy protection for end users at sites that contribute data to DNSDB. As a security policy matter, DomainTools does not disclose the identity of DomainTools' sensor operators. ### Can DomainTools tell what queries I’m issuing? What if I need assured query privacy as a matter of policy, regulation or operational security? DomainTools logs all queries made to DNSDB for accounting- and troubleshooting-related purposes. If you need assured query privacy, DNSDB Export (which leverages an on-premises copy of DNSDB) will allow you to have that. ## Technical Jargon-Related Questions ### What is the difference between “recursive resolvers” and “authoritative nameservers” Recursive resolvers are used by users to resolve the names of the sites they’re interacting with, whatever and wherever those might be. For example, if you visit www.cnn.com, a recursive resolver will translate that domain name to the IP address your computer needs. Recursive resolvers are most commonly run by ISPs, enterprises, colleges or universities, etc., for the benefit of their local users, although some recursive resolvers may be intentionally open to anyone, like Google’s 8.8.8.8. Authoritative nameservers are different. They get designated by the domain owner when the domain owner registers a new domain name, and are used to describe the relationship between domain names and the IP addresses used by that specific domain. Authoritative nameservers may be run by the domain owner directly, or by a third party such as a domain name registrar or hosting company. Authoritative nameservers only know about/answer for the specific domain names assigned to them. ### What is a “bailiwick?” The bailiwick of a content DNS server is quite a simple notion. It is the domain that was used in the referral that directed a resolving proxy DNS server to that content DNS server in the first place. When a superdomain’s content DNS servers issue a referral saying “Ask those servers over there about that particular domain.”, then the domain in the referral is the bailiwick of the content DNS servers when they come to be queried. For example, if the net. content DNS servers respond to an enquiry for the name an.example.net. with a referral to content DNS servers at 10.0.0.1 and 10.0.0.2 for the domain example.net., then the bailiwick of the latter servers, when they come to be queried, is example.net.. Bailiwick is the scope of authority of any particular content DNS server, determined by following a chain of referrals from the root of the DNS namespace. A content DNS server may only be trusted where the information it provides is about names within its own bailiwick. ### What’s a “base domain?” A “base domain” is what registrants purchase from a registrar when they buy a new domain name. For example, nytimes.com is a base domain name. ### What’s a “fully qualified domain name?” A “fully qualified domain name” is any hostname, and usually includes a base domain name. For example, www.cnn.com is a fully qualified domain name. printer23 is an example of a local domain name that is not fully qualified. ### What’s an “RRset?” All DNS resource records of the same name, class, and type from a DNS response. For example, a server that is doing load balancing via DNS might have two, three, or even more A records for a given fully qualified domain name. (see RFC2136 just above section 1.1) For example: ```sh www.google.com. 300 IN A 74.125.227.145 www.google.com. 300 IN A 74.125.227.148 www.google.com. 300 IN A 74.125.227.146 www.google.com. 300 IN A 74.125.227.144 www.google.com. 300 IN A 74.125.227.147 ``` ### What are the different RRTypes and what do they show? Examples of each? DNS Resource Record types are described at RFC6895 section 3.1. Some DNS record types are very common, including (but not limited to): Record Type - Function A Maps a domain name to IPv4 address AAAA - Maps a domain name to IPv6 address CNAME - Maps one domain name to another NS - Defines a domain’s nameserver PTR - Maps an IP address to a domain name MX - Defines a domain’s mail exchanger TXT - Returns some specified text content Another DNS record type that’s less-common is the SRV record. SRV records are defined in RFC 2782. SRV records allow a site to instantiate services on non-standard port numbers, and to easily load balance services across multiple servers of varying size. Some sites, including DomainTools, find this quite useful and rely heavily on SRV records. Examples of the various record types include: ```sh www.princeton.edu. 67 IN A 140.180.223.42 mx.smtp.ucla.edu. 3600 IN AAAA 2607:f010:3fe:102::ff:fe01:ac www.uoregon.edu. 60 IN CNAME drupal-cluster5.uoregon.edu. caltech.edu. 43200 IN NS tepid.ni.caltech.edu. 112.4.193.128.in-addr.arpa. 3600 IN PTR www.orst.edu. columbia.edu. 3600 IN MX 10 mail-in.cc.columbia.edu. ucdavis.edu. 14400 IN TXT "v=spf1 ip4:198.17.84.4/32 ip4:198.17.84.15/32 ip4:128.120.0.0/16 ip4:169.237.0.0/16 ip4:152.79.0.0/16 include:stspg-customer.com include:sendgrid.net include:spf.boardbooks.com ~all" ``` ### What’s “RRname” and what is its significance? An RRname is a Resource Record Name or DNS label. This is the left hand side of a DNS record: ```sh www.princeton.edu. 67 IN A 140.180.223.42 ``` ### What’s “Rdata”? Rdata is the value (or right hand side) of the DNS record. ```sh www.princeton.edu. 67 IN A 140.180.223.42 ``` ## Usage Questions ### I’d like to get more than the 10,000 results in the web interface (or more than the 1,000,000 results in the API/CLI interface). What are my options? You can do this if you have an API key subscription to access the data. If you routinely need to do queries resulting in more than 1,000,000 results, the best path forward is DNSDB Export. When you have an on-premises copy of the database, you can tailor how you access the data and potentially retrieve an unlimited number of matching records. ### What output formats are available? DNSDB Scout™, displays results in the browser window and is available for Google Chrome and Firefox, as well as in a web site most browsers can access.. You can download results as a JSON or CSV file, or copy and paste that data. See DNSDB Scout for more information The web GUI interface at https://www.dnsdb.info/ is intended solely for occasional use, displays results in the browser window. You can copy and paste that data into a report or plain text file if you want to save those results. The Python CLI dnsdb_query.py client offers more flexibility. Output formats for the dnsdb_query.py client include plain text (the default) and JSON format via the \-j or \--json options. The C-language CLI dnsdbq client supports plain text, JSON, CSV and DNS formatted output, as well as querying for your remaining quota. It is open source on github at . See also the “Example scripts” section of the DNSDB API documentation. ### Can you provide some examples of how I might write time-constrained DNSDB queries? dnsdb_query.py –help states: ```txt Usage: dnsdb_query.py [options] Options: [...] --before=BEFORE only output results seen before this time --after=AFTER only output results seen after this time Time formats are: "%Y-%m-%d", "%Y-%m-%d %H:%M:%S", "%d" (UNIX timestamp), "-%d" (Relative time in seconds), BIND format (e.g. 1w1h, (w)eek, (d)ay, (h)our, (m)inute, (s)econd) ``` Some examples include: Show entries from the last hour: ```sh $ dnsdb_query.py -r www.google.com --after=1h ``` Show entries from after 2015-6-18: ```sh $ dnsdb_query.py -i 216.170.114.3 --after=2015-6-18 ``` Show entries from between the dates shown: ```sh $ dnsdb_query.py -i 216.170.114.3 --after=2015-6-18 --before=2016-1-1 ``` The DNSDB API documentation states that you may filter results by time using the time_first_before, time_first_after, time_last_before, and time_last_after query parameters. These parameters expect a UTC timestamp with seconds granularity or a relative time in seconds (preceded by \-). ### Can I do time-constrained (i.e. time fencing) queries in the web interfaces or DNSDB Scout™? Yes, we support time-fencing in DNSDB Scout. ### What’s the difference between \-r, \-n, and \-i queries in the DomainTools-provided dnsdb_query.py and dnsdbq DNSDB clients? The different options are: \-r: queries the RRset \-n: queries the Rdata by name \-i: queries the Rdata by IP address or by CIDR prefix For more, see RRset and Rdata Demystified. If you are starting with an IP address, you must use \-i because that’s the only option that allows you to query by IP address: ```sh $ dnsdb_query.py -i 104.244.13.104 fsi.io. IN A 104.244.13.104 www.fsi.io. IN A 104.244.13.104 olddocs.fsi.io. IN A 104.244.13.104 farsighsecurity.com. IN A 104.244.13.104 www.farsighsecurity.com. IN A 104.244.13.104 farsightsecurity.com. IN A 104.244.13.104 www.farsightsecurity.com. IN A 104.244.13.104 archive.farsightsecurity.com. IN A 104.244.13.104 ``` If you are starting with a domain name, you will normally want to use the \-r option: ```sh $ dnsdb_query.py -r www.fsi.io ;; bailiwick: fsi.io. ;; count: 57 ;; first seen: 2013-10-08 21:20:49 -0000 ;; last seen: 2014-12-04 21:38:28 -0000 www.fsi.io. IN A 66.160.140.76 ;; bailiwick: fsi.io. ;; count: 36 ;; first seen: 2015-06-07 06:13:14 -0000 ;; last seen: 2016-03-09 02:40:20 -0000 www.fsi.io. IN A 104.244.13.104 ;; bailiwick: fsi.io. ;; count: 5 ;; first seen: 2013-12-19 14:36:40 -0000 ;; last seen: 2014-10-17 18:57:45 -0000 www.fsi.io. IN AAAA 2001:470:b0::76 ;; bailiwick: fsi.io. ;; count: 11 ;; first seen: 2015-06-09 10:30:06 -0000 ;; last seen: 2015-07-29 08:45:55 -0000 www.fsi.io. IN AAAA 2620:11c:f004::104 ``` The most common time when you will use \-n, is when you’re searching for domains associated with a nameserver. ```sh $ dnsdb_query.py -n ns7.dnsmadeeasy.com 3dg.biz. IN NS ns7.dnsmadeeasy.com. chal.biz. IN NS ns7.dnsmadeeasy.com. cpcl.biz. IN NS ns7.dnsmadeeasy.com. g3ms.biz. IN NS ns7.dnsmadeeasy.com. icti.biz. IN NS ns7.dnsmadeeasy.com. u3o8.biz. IN NS ns7.dnsmadeeasy.com. 90501.biz. IN NS ns7.dnsmadeeasy.com. cpynl.biz. IN NS ns7.dnsmadeeasy.com. cvent.biz. IN NS ns7.dnsmadeeasy.com. [etc] ``` The equivalent commands using the dnsdbq cli client to the above commands are: ```sh $ dnsdbq -i 104.244.13.104 $ dnsdbq -r www.fsi.io $ dnsdbq -n ns7.dnsmadeeasy.com ``` ### How can I check my quota? You can use a command such as: ```sh $ dnsdbq -I ``` or ```sh $ curl --header "X-API-Key: my-api-key-here" https://api.dnsdb.info/lookup/rate_limit ``` Your API Key will normally be in .dnsdb-query.conf in your home directory. For more information, please refer to Service Limits section of the DNSDB API documentation. ### Do you support mid-string wildcard searches, or searches using regular expressions? Yes via DNSDB Flex. ### Do you support wildcards in the form of CIDR prefix notation or IP ranges? Yes, using Flex, or when using the DomainTools supplied dnsdb_query.py client you can search by CIDR prefix or by IP range. Example of a permitted query: As of July, 2022 DNSDB was changed to reduce the amount of junk wildcard domains in its database. We are gradually rolling out a change to replace multiple wildcarded DNS rrnames with a single rrname that starts with a WILDCARD. label. No other rrname labels contain uppercase letters, so records with this (all upper case) WILDCARD. were never in DNSDB before. Note that there are existing, real, domain names that contain a wildcard. label (all lower case). ```sh $ dnsdb_query.py -i 128.223.0.0/16 ``` or equivalently ```sh $ dnsdb_query.py -i 128.223.0.0-128.223.255.255 ``` Some seemingly-equivalent-looking queries, however, will fail: ```sh $ dnsdb_query.py -i 128.223.* HTTP Error 400: Bad Request $ dnsdb_query.py -i 128.223.*.* HTTP Error 400: Bad Request ``` CIDR prefix queries are also supported for IPv6: ```sh $ dnsdb_query.py -i 2001:48A8::/32 ``` or equivalently ```sh $ dnsdb_query.py -i 2001:48A8::-2001:48A8:FFFF:FFFF:FFFF:FFFF:FFFF:FFFF ``` The equivalent command using the dnsdbq CLI command is: ```sh $ dnsdbq -i 2001:48A8::-2001:48A8:FFFF:FFFF:FFFF:FFFF:FFFF:FFFF | fgrep -v ';' | grep . ``` ### How many parallel (or “simultaneous”) queries can I have outstanding simultaneously via the API? Ten (10). ### I want to be able to guarantee that my queries to the database are not observed. Is there any way for me to obtain a complete copy of the database for use “on premises” within a secure enclave that’s not connected to the Internet? DNSDB Export (an on-premises installation of DNSDB) provides total query privacy. ### I’d like to have real time access to the full stream of DNS data as it’s added to DNSDB. Is there some way to get that? If you have a process that can utilize the real-time stream of observations that goes into DNSDB, DomainTools makes several different streams of data available to participants on the Security Information Exchange (SIE). The streams are created as byproducts of deduplication and filtering during processing before the data is put into the database. To request a demonstration or to inquire about a trial API key please contact the DomainTools sales team. ### I’d like to be able to routinely watch for just select terms of interest (keywords, brands, etc.). Do you have some way to support that? Yes, you can do this using SIE’s Newly Observed Domains and Newly Observed Hostnames. To request a demonstration or to inquire about a trial API key please contact the DomainTools sales team. ### How do I search for subdomains in DNSDB? In CLI/API and WebUI? Please see Lookup methods section of the DNSDB API documentation. As an example of searching for all domains in cs.uoregon.edu, you’d enter: $ dnsdb_query.py -r \*.cs.uoregon.edu When using the CLI, shell escape the- by preceding it with a backslash. When using the WebUI, omit the backslash. ### For some reverse-IP lookups, there’s lots of results, often associated with cheap web hosts experiencing high client turnover. Is there a way to limit the results to only current results? Or, at a minimum, only receive the result from the most recent date that the dns name resolved to that IP You can use time fencing to limit the results returned. See above in question VI-3. For example, to get results from just the last hour: $ dnsdb_query.py -r www.google.com --after=1h You can also sort results by time last seen: $ dnsdb_query.py -s time_last -r www.google.com ### Is it possible to find DNS responses from only a particular nameserver IP? While this can be done by monitoring Channel 202 in Security Information Exchange (SIE), it is not currently possible in DNSDB. ### I forgot my password? How do I reset my password? Request a password reset by contacting EnterpriseSupport@domaintools.com. ### Do you “lock” accounts? Generally, we only lock accounts if we see evidence of compromise, use inconsistent with DomainTools' terms of service, or we are unable to reach a user at their email address of record. If you believe your account may have been locked, please contact EnterpriseSupport@domaintools.com for assistance. ### How do I request credentials for a new teammate in my organization, or what if a team mate has left and I’d like to retire their credentials? Your registered point of contact for your contract should contact EnterpriseSupport@domaintools.com. ### How do I learn about maintenance notifications? Contact EnterpriseSupport@domaintools.com and ask to be added to the notification list.