Newly Observed Domains (NOD) on SIE: User Guide¶
Getting Started¶
When choosing a Newly Observed Domains (NOD) subscription you will determine the format and transport you want to use to retrieve NOD data sets. At the beginning of the provisioning process for the NOD subscription you will be asked to provide one of the following incremental zone transfer methods, depending on what delivery methods you have chosen.
For general information on DNS Response Policy zones, consult https://dnsrpz.info/.
For more information on the Security Information Exchange (SIE) platform, consult our technical documentation or the SIE User Guide.
Incremental Zone Transfers (IXFR)¶
If you choose to use Incremental Zone Transfers (IXFR) you will need to give DomainTools a list of IPv4 and IPv6 addresses to which you would like DNS NOTIFY messages to be sent, as well as IPv4 and IPv6 addresses or small address blocks that will be allowed access (the two lists are typically the same).
Rsync¶
If you choose to use rsync you will need to create an SSH key pair and share the public key with DomainTools. DomainTools has a strong preference for 4096 bit RSA keys. Using SSH in an automated script can be done with an SSH key generated with an empty passphrase.
You can create such a key pair with the following command:
$ ssh-keygen -t rsa -b 4096 -N '' -C farsight_security -f /path/to/filename
Generating public/private rsa key pair.
Your identification has been saved in /path/to/filename.
Your public key has been saved in /path/to/filename.pub.
The key fingerprint is:
SHA256:r1IKQPxpp9erHZ9tAaAnllp5mIrai6N1441p0NMueOg farsight_security
You will need to share the file created by ssh-keygen with the file extension .pub.
Retrieve DNS Zones via Incremental Zone Transfers (IXFR)¶
A Zone Transfer is a term used to refer to the process by which the contents of a DNS Zone file is copied from a primary DNS Server to a secondary DNS server. IXFR is a term used to refer to a incremental zone transfer vs a full zone transfer (AXFR).
DomainTools recommends using Incremental Zone Transfers to consume NOD as it provides a near real time mechanism to retrieve NOD updates.
To configure your DNS infrastructure to use Incremental DNS Zone Transfers as the transport for NOD, DomainTools will need a list of IPv4 and IPv6 addresses to which you would like DNS NOTIFY messages to be sent, as well as IPv4 and IPv6 addresses or small address blocks that will be allowed access (the two lists are typically the same).
You will receive a DNS TSIG key, which will look like the following:
The zones are served by the following masters. This list is subject to change as DomainTools grows the service over time and you will be given reasonable notice to reconfigure your name server.
masters “fsi-ixfr-masters” {
104.244.13.88 key “FSI-####-#-key”;
104.244.14.88 key “FSI-####-#-key”;
2620:11c:f004::88 key “FSI-####-#-key”;
2620:11c:f008::88 key “FSI-####-#-key”;
};
These settings are attached as named.fsi-####-#.conf where ####-# is your account number. DomainTools recommends that you use this file with BIND's include statement in your configuration to simplify future updates. Also included will be an example configuration that you may incorporate into your existing name server's configuration.
You will need to add rules to your firewall's access control list(s) for DomainTools hosts to send UDP packets to port 53 of your DNS server so that it can receive the DNS NOTIFY packets for updates. This will allow your DNS server to receive incremental updates every few seconds. If you do not add these firewall rules your zones will only update every ten minutes as per the refresh field in the zone's Start of Authority record.
Using TSIGs for authentication requires reasonably synchronized system clocks. Ensure that your server is enabled to use NTP for clock synchronization.
NOD Response Policy Zones (RPZ)¶
DomainTools Security makes NOD available via Domain Name Service Response Policy Zones (DNS RPZ). DNS RPZ is a method that allows a name server administrator to overlay custom information on top of the global DNS to provide alternate responses to queries. You can read more about DNS RPZ at https://dnsrpz.info/.
Response Policy Zones are delivered as one of seven DNS zones suitable for deployment as a DNS Response Policy Zone. Each zone file name contains a time which corresponds to how old the domains in the zone are believed to be. DomainTools recommends the use of the 3h.rpz.dns-nod.net zone as a starting point. You will receive DNS NOTIFY messages for all seven zones but the preference is that you only download the zones that you will actively use, to avoid duplication:
5m.rpz.dns-nod.net10m.rpz.dns-nod.net30m.rpz.dns-nod.net1h.rpz.dns-nod.net3h.rpz.dns-nod.net12h.rpz.dns-nod.net24h.rpz.dns-nod.net
The domains included in the seven DNS zones correspond to the age of the domain as the names were first observed by our sensor network. Think of RPZ as various rolling window from five minutes to 24 hours. In other words, newly observed domains (NOD) are bucketed by age and formed into seven different zone files.
When a NOD is first observed, it is inserted into all seven zone files and as the domain ages, our systems remove the name from the appropriate RPZ file starting with the 5m file first rolling up through the files to the last 24h file. A once newly observed domain will ultimately age out and it will no longer be included/found in any of the RPZ zone files.
DomainTools does not support RPZ files with domains older than 24 hours old.
In the meantime, here is a link that includes additional information about DNS Response Policy Zones that you might find helpful:
Further technical information may be found in the NOD Technical Data Sheet and NOD User Documentation.
Example config files below, see the welcome email for personalized examples.
File named.fsi-####-#.conf: Ensure the name of this file matches the name of the file you get in the welcome email.
key “FSI-####-#-key” {
algorithm HMAC-SHA512
secret “SECRET
};
masters “fsi-ixfr-masters” {
104.244.13.88 key “FSI-####-#-key”;
104.244.14.88 key “FSI-####-#-key”;
2620:11c:f004::88 key “FSI-####-#-key”;
2620:11c:f008::88 key “FSI-####-#-key”;
};
File named.fsi-nod.conf:
// add this to your options clause
options {
response-policy {
zone “3h.rpz.dns-nod.net” policy given;
// zone “3h.rpz.dns-nod.net” policy passthru; # audit with logging
};
};
// optionally, add something like this to your logging clause and send
// to your SIEM
logging {
channel named-rpz {
file “/var/log/rpz.log” versions 3 size 250k;
print-time yes;
print-category yes;
print-severity yes;
severity info;
};
category rpz {
named-rpz;
};
};
// include the keyfile we created above
include “/etc/bind/named.fsi-####-#.conf”;
zone “3h.rpz.dns-nod.net” {
type slave;
file “3h.rpz.dns-nod.net.zone”;
masters { fsi-ixfr-masters; };
allow-query {localhost;};
allow-transfer {none;};
};
NOD DNS Blackhole List Zone (DNSBL)¶
DomainTools Security makes NOD available in DNS-based Blackhole List (DNSBL) Zone file format. DNSBLs convey information over DNS and allow subsequent processes make decisions based on the provided answers. DNSBLs are most commonly used to assist in the scoring of SPAM email with applications like Spamassassin and Postfix. (Examples are below.)
Using the named.fsi-####-#.conf configuration file as described above; you can use the following snip-it in your name server configuration file to consume the NOD DNSBL file via IXFR.
include “/etc/bind/named.fsi-####-#.conf”;
zone “v1.bl.dns-nod.net” {
type slave;
file “v1.bl.dns-nod.net.zone”;
masters { fsi-ixfr-masters; };
allow-query {localhost;};
allow-transfer {none;};
};
Using dig for troubleshooting IXFR¶
IXFR depends on retrieving the SOA (Start Of Authority) record for the zone: this record contains a serial number, and the first step in the IXFR process is to compare the serial number with what the recipient server has locally. In practice, many connectivity issues can be understood by looking at what happens when a request is made to retrieve the SOA record for the zone.
The basic command looks like this:
dig -y '<hashing algorithm>:<your key name>:<your secret>' @<one of our servers> \
<one of our zones> SOA
Using the example data given above:
dig -y 'HMAC-SHA512:FSI-####-#-key:NOT=REALLY=YOUR=SECRET' @104.244.13.88 v1.bl.dns-nod.net SOA
; <<>> DiG 9.8.3-P1 <<>> -y HMAC-SHA512 @104.244.13.88 v1.bl.dns-nod.net SOA
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 14186
;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 2, ADDITIONAL: 1
;; WARNING: recursion requested but not available
;; QUESTION SECTION:
;v1.bl.dns-nod.net. IN SOA
;; ANSWER SECTION:
v1.bl.dns-nod.net. 86400 IN SOA a.ns.dns-nod.net.v1.bl.dns-nod.net.
nod-admin.fsi.io. 1479754872 600 300 86400 300
;; AUTHORITY SECTION:
v1.bl.dns-nod.net. 86400 IN NS a.ns.dns-nod.net.
v1.bl.dns-nod.net. 86400 IN NS b.ns.dns-nod.net.
;; TSIG PSEUDOSECTION:
fsi-####-#-key. 0 ANY TSIG hmac-sha512. 1479754882 300 64
NOT=REALLY=YOUR=SECRET 14186 NOERROR 0
;; Query time: 71 msec
;; SERVER: 104.244.13.88#53(104.244.13.88)
;; WHEN: Mon Nov 21 11:01:22 2016
;; MSG SIZE rcvd: 258
That shows a successful response. The two most important pieces of information are:
- the query status shown on the HEADER line
- the TSIG section which will show any key-related issues
and of course, that you got a response at all. (If you're sniffing packets, it's a lot easier to look for the SOA query than to try to recognize a zone transfer which probably spans multiple packets.)
Retrieve DNS Zones via Rsync¶
NOD RPZ files can be retrieved via Rsync when DNS Zone Transfers are not desired. Rsync is a file copying tool, it is known for its delta-transfer algorithm, which reduces the amount of data sent over the network by sending only the differences between the source files and the existing files at the destination.
NOD Response Policy Zones (RPZ) Starting Recommendation¶
DomainTools recommends the use of the 3h.rpz.dns-nod.net zone as a starting
point. There are separate files based on the age of the domain names.
Please only download the file for the zone that you will actively use.
path - filename:
nod/rpz/5m.rpz.dns-nod.net.zonenod/rpz/10m.rpz.dns-nod.net.zonenod/rpz/30m.rpz.dns-nod.net.zonenod/rpz/1h.rpz.dns-nod.net.zonenod/rpz/3h.rpz.dns-nod.net.zonenod/rpz/12h.rpz.dns-nod.net.zonenod/rpz/24h.rpz.dns-nod.net.zone
Example shell command to download the three hour NOD RPZ Zone file to the directory /srv/nod:
rsync -az -e "ssh -p 49222 –i/path/to/sshkey" \
USERNAME@rsync.dns-nod.net:nod/rpz/3h.rpz.dns-nod.net.zone /srv/nod/
The following is an example BIND configuration for RPZ:
options {
response-policy { zone "3h.rpz.dns-nod.net"; };
};
zone "3h.rpz.dns-nod.net" {
type master;
file "/srv/nod/3h.rpz.dns-nod.net.zone";
allow-query {localhost;};
allow-transfer {none;};
allow-update {none;};
};
Example shell script to be run by cron to download and load the three hour NOD RPZ zone file using BIND utility rndc. If your operating system distribution includes the flock - manage locks from shell scripts command, using it will prevent multiple cronned instances from overrunning each other (its semantics are similar to time):
#!/bin/sh
rsync -az -e "ssh -p 49222 –i/path/to/sshkey" \
USERNAME@rsync.dns-nod.net:nod/rpz/3h.rpz.dns-nod.net.zone /srv/nod/
rndc reload 3h.rpz.dns-nod.net
Using NOD RPZ this way will cause DNS queries for any domains in the selected database to return an NXDOMAIN response, should those domains be queried by an end-user.
NOD DNS Blackhole List (DNSBL) File Availability (for rbldnsd)¶
DomainTools also makes NOD available in a DNS-based Blackhole List (DNSBL) file format for rbldnsd. Note this is a different format than the v1.bl.dns-nod.net file, but rbldnsd may be used to serve it as the same DNSBL zone.
Your user name on the DomainTools server will be "USERNAME".
Example shell command to download the one hour NOD DNSBL Zone file to the directory /srv/nod:
Use rbldnsd to serve the NOD DNSBL Zone¶
DomainTools NOD DNSBL Zones can be loaded and served by rbldnsd which is a small authoritative-only nameserver designed to serve DNS-based blocklists (DNSBLs). For more information see https://rbldnsd.io/.
Configure rbldnsd to load the NOD DNSBL Zone¶
The following is an example of the rbldnsd command-line arguments.
Starting rbldnsd:
Configuring BIND to forward DNSBL queries to rbldnsd¶
Sample BIND configuration forwarding to local port 5053 being served by rbldnsd:
Using the NOD DNSBL¶
Many applications can be configured to use a DNSBL. This user guide gives guidance on two open source applications that can be configured to consume a DNSBL (SpamAssassin and Postfix).
When using NOD in this manner, DNS queries for A type (IP address) resource records of the form domain.v1.bl.dns-nod.net will return an address that indicates the age of domain if it is in the NOD database. The response should be interpreted as follows:
| Period | Response |
|---|---|
| 0-5 minute | 127.0.0.2 |
| 5-10 minute | 127.0.0.3 |
| 10-30 minute | 127.0.0.4 |
| 30-60 minute | 127.0.0.5 |
| 1-3 hours | 127.0.0.6 |
| 3-12 hours | 127.0.0.7 |
| 12-24 hours | 127.0.0.8 |
“NXDOMAIN” (no such domain) response will be returned if the domain is not in the NOD database.
TXT DNS queries will return more detailed information about the domain if it is in the database. For example:
The first_seen date is in Unix seconds. That value can be converted to "human time" with a command such as:
You may test your setup using your favorite DNS query tool and the special test domain test.dns-nod.net and invalid.dns-nod.net (with your resolver using the DNSBL zone).
$ host invalid.dns-nod.net.v1.bl.dns-nod.net
Host invalid.dns-nod.net.v1.bl.dns-nod.net not found: 3(NXDOMAIN)
Configure SpamAssassin to use NOD DNSBL¶
To configure SpamAssassin to use NOD DNSBL, append the following to /etc/spamassassin/local.cf or ~/.spamassassin/user_prefs, adjusting scores to taste:
urirhssub URIBL_NOD_5M v1.bl.dns-nod.net. A 127.0.0.2
body URIBL_NOD_5M eval:check_uridnsbl('URIBL_NOD_5M')
describe URIBL_NOD_5M Contains an URL that is only 5 minutes old in
Farsight Passive DNS
tflags URIBL_NOD_5M net domains_only
score URIBL_NOD_5M 5
urirhssub URIBL_NOD_10M v1.bl.dns-nod.net. A 127.0.0.3
body URIBL_NOD_10M eval:check_uridnsbl('URIBL_NOD_10M')
describe URIBL_NOD_10M Contains an URL that is only 10 minutes old in
Farsight Passive DNS
tflags URIBL_NOD_10M net domains_only
score URIBL_NOD_10M 5
urirhssub URIBL_NOD_30M v1.bl.dns-nod.net. A 127.0.0.4
body URIBL_NOD_30M eval:check_uridnsbl('URIBL_NOD_30M')
describe URIBL_NOD_30M Contains an URL that is only 30 minutes old in
Farsight Passive DNS
tflags URIBL_NOD_30M net domains_only
score URIBL_NOD_30M 5
urirhssub URIBL_NOD_1H v1.bl.dns-nod.net. A 127.0.0.5
body URIBL_NOD_1H eval:check_uridnsbl('URIBL_NOD_1H')
describe URIBL_NOD_1H Contains an URL that is only one hour old in
Farsight Passive DNS
tflags URIBL_NOD_1H net domains_only
score URIBL_NOD_1H 5
urirhssub URIBL_NOD_3H v1.bl.dns-nod.net. A 127.0.0.6
body URIBL_NOD_3H eval:check_uridnsbl('URIBL_NOD_3H')
describe URIBL_NOD_3H Contains an URL that is only three hours old in
Farsight Passive DNS
tflags URIBL_NOD_3H net domains_only
score URIBL_NOD_3H 5
urirhssub URIBL_NOD_12H v1.bl.dns-nod.net. A 127.0.0.7
body URIBL_NOD_12H eval:check_uridnsbl('URIBL_NOD_12H')
describe URIBL_NOD_12H Contains an URL that is only twelve hours old in
Farsight Passive DNS
tflags URIBL_NOD_12H net domains_only
score URIBL_NOD_12H 5
urirhssub URIBL_NOD_24H v1.bl.dns-nod.net. A 127.0.0.8
body URIBL_NOD_24H eval:check_uridnsbl('URIBL_NOD_24H')
describe URIBL_NOD_24H Contains an URL that is only 24 hours old in
Farsight Passive DNS
tflags URIBL_NOD_24H net domains_only
score URIBL_NOD_24H 5
Configure Postfix to use NOD DNSBL¶
For postfix to reject everything:
To cut off at a particular age threshold using postfix version 2.8 or higher:
NOD and Local Firewalls¶
If you need to configure your firewalls to allow access to the NOD systems at DomainTools, please be aware that servers may be rotated in and out of service without notice, so we recommend you do not configure access to specific IP addresses. You should configure to allow access through 104.244.12.0/22 to be able to access NOD services without interruption.