title: DomainTools Farsight SIE API service_id: farsight-sie auth_types: API Key source_url: https://docs.domaintools.com/api/sie/ api_specification: https://app.swaggerhub.com/apis/DomainToolsLLC/SIE_Batch/ ## Overview The Security Information Exchange (SIE) is a real-time security data platform that collects and distributes over 200,000 passive DNS observations per second from a global sensor network. SIE provides access to passive DNS data, newly observed domains, spam and phishing URLs, darknet telescope data, and other security intelligence feeds. SIE data is accessible through three delivery methods: - SIE Batch: REST API and web interface for downloading the last 12–18 hours of batched channel data. Best for periodic updates. - SIE Remote Access (AXA): Real-time streaming of SIE channels using the AXA (Advanced Exchange Access) toolkit. Supports server-side filtering by domain or IP. - SIE Direct Connect: Dedicated co-located blade server with direct SIE network access for the highest-volume use cases. Access is provisioned by DomainTools Enterprise Support. Contact enterprisesupport@domaintools.com to get started. ## Key Capabilities - Real-time and batch access to passive DNS (pDNS) observations - Newly Observed Domains (NOD), Newly Active Domains (NAD), and Newly Observed Hostnames (NOH) channels - Spam, phishing, IDS/firewall, and darknet telescope data channels - NMSG binary format with JSON conversion via nmsgtool - Server-side filtering in Remote Access mode (IP watches, DNS watches, channel watches) - AXA C API and Python bindings for custom integrations ## Related Resources - SIE Batch OpenAPI specification: https://app.swaggerhub.com/apis/DomainToolsLLC/SIE_Batch/ - AXA C Programming API reference: https://docs.domaintools.com/api/sie/axa-c-api-reference.html - GitHub repositories: https://github.com/orgs/farsightsec/repositories ## This Documentation This reference covers SIE access methods, channel overview, SIE Batch (web interface and REST API), the NOD channel guide, SIE Remote Access (AXA toolkit: overview, user guide, missing manual, accounting, and tutorials), and the NMSG protocol user guide. --- Please select a subcategory in the navigation column. ## Introduction SIE gives you real-time access to data from our global sensor network. That data includes over half a million passive DNS (pDNS) observations per second, as well as other key security data points. DomainTools processes this data into usable formats, stream it over real-time channels, and provides you with tools for your use case. Access SIE Channels through the following mechanisms: * SIE Batch is a web interface and REST API with access to the last 12-18 hours of data from your subscribed feeds. * SIE Remote Access creates a tunnel from SIE to the analyst’s system, and supports a REST API (see AXAMD, below). * SIE Direct Connect is a leased blade server with pre-installed SIE tools that provides direct access to the SIE network. You are responsible for ongoing server maintenance. Data formats are detailed in the File and Data Formats section. Feeds are typically available in NMSG and JSON formats, depending on the access method (some JSON availability is through conversion with nmsgtool). * NMSG is a streaming binary format, based on Google Protocol Buffers, that handles high volume, real-time traffic. SIE delivers most channels with NMSG. The included nmsgtool (detailed below) can convert NMSG to JSON. * JSON, typically in JSONL/NDJSON format, is provided directly by certain feeds. * Packet Capture (libpcap) is used for channels that preserve packet structure. ### Access Access to SIE is provisioned by DomainTools Enterprise Support at . Batch and SIE Remote Access is provisioned with an API key. SIE Direct Connect requires the customer's public key and originating IP address(es) for SSH access. ### System Requirements A server configured by DomainTools for SIE Direct Connect has the following operating system (OS) and hardware specifications. | Component | Specification | | :---- | :---- | | Compute | One (1) x Intel Quad Core Xeon E3 3.40Ghz | | Memory | 16GB RAM | | Storage | Two (2) x 2TB 7200RPM drives (configured for RAID 1, 2TB available for customer use) | | Network (Internet connection) | 100Mbps | | Network (SIE connection) | 10gigabit | These hardware specifications are adequate for acquiring, processing, compressing, and buffering data from high throughput SIE channels. Intensive processing and analysis on data from SIE channels with the highest data volumes may require additional memory (RAM). ## Channels Overview The following table provides the channel, name, description, bitrate, payload rate, and notes on each channel. All channels are available in NMSG and JSON formats unless otherwise noted. | Number | Name | Description | Bitrate | Payloads | Notes | | :----- | :--- | :---------- | :------ | :------- | :---- | | 24 | Spam-Full | Spam-Full shares full copies of spam emails sent to email spamtrap systems. The honeypots have been configured to collect and store all email messages for analysis and they use email addresses that have never been used to receive email, or are no longer in use and should not be receiving email. | 2.4kbps (104kbps) | 2/sec (14/sec) | | | 25 | Spam-Select | Spam-Select shares key fields from spam emails sent to email honeypot/spamtrap systems. The spamtraps have been configured to collect and store all email messages for analysis and they use email addresses that have never been used to receive email, or are no longer in use and should not be receiving email. | 2kbps (105kbps) | 1/sec (3/sec) | | | 27 | Phishing URLs | Includes URLs, the brand target, and other information related to phishing campaigns. | <1kbps (2kbps) | <1/sec (2/sec) | | | 42 | IDS and Firewall Log Data | Information about network traffic that is blocked by Intrusion Detection Systems (IDS) and firewall devices. The data is anonymized and batched every five minutes. | 7mbps (35mbps) | 390/sec (2kbps) | Not available over AXAMD. | | 80 | Conficker Sinkhole | HTTP connection information from Conficker-infected clients to ‘sinkholed’ command and control servers. | 385kbps (520kbps) | 210/sec (280/sec) | Not available over AXAMD. | | 115 | DDos Events | Evidence of DDoS and DRDoS (Distributed Reflection Denial of Service) attacks based on analysis of data from captured network packets destined from unused network space. | <1kbps (<1kbps) | <1/sec (1.5/sec) | | | 207 | DNSDB De-Duplicated Data | Passive DNS observations after the deduplication processing phase and immediately prior to the verification phase. | 144mbps (170mbps) | | Not available over Remote Access or AXAMD; Batch files available as NMSG. | | 208 | DNSDB Verified Data | Deduplicated passive DNS data with low value entries removed, and verified for balliwick-appropriate data (misleading resource record information is removed). | 88mbps (117mbps) | | Not available over Remote Access or AXAMD; Batch files available as NMSG. | | 211 | Newly Active Domains (NAD) | Previously observed domains (via Channel 204\) becoming active after 10 or more days of inactivity. | 62kbps (900kbps) | 53/sec (760/sec) | | | 212 | Newly Observed Domains (NOD) | Domains not previously observed by SIE. It can also be accessed by: NOD RPZ; See also: NOD FAQ | 3kbps (18kbps) | 2/sec (13/sec) | | | 213 | Newly Observed Hostnames (NOH) | Fully Qualified Domain Names (FQDNs) not previously seen when compared to the DNSDB historical database. | Under review | TBA | | | 214 | DNS Changes | Domains, hostnames, or record data that is unknown to DNSDB, either because the data is for a new domain or hostname or because the record data for a domain or hostname has changed. These changes may include new RR types, new or changed IP addresses, or a change in the authoritative name servers for a domain. More information: The DNS Changes Channel | 4.9mbps (8mbps) | TBA | Not available over AXAMD. | | 220 | DNS Errors | Non-deduplicated DNS responses that returned a non-zero Response Code (RCODE). This includes NXDomain, ServFail, Refused, FormErr, NotImp, and other RCODEs. | 570mbps (860mbps) | TBA | Only available over Direct Connect. | | 221 | NX Domains | Derived from Channel 220 DNS Errors, but only includes NXDomain RCODEs. More information: Introducing NXD | 52mbps(73mbps) | TBA | Not available through AXAMD. | | 255 | Heartbeat (Testing) | For monitoring purposes. | 1kbps (1kbps) | TBA | | ## Installation ### Debian GNU/Linux 11 Bullseye Debian GNU/Linux is supported through an apt package repository. Ubuntu and other Debian-based systems may be able to use this repository as well. Enable the bullseye-farsightsec package repository by copying the GPG key in place and adding a sources.list entry: ```bash $ wget -O debian-farsightsec.gpg https://dl.farsightsecurity.com/debian/debian-farsightsec.gpg $ echo deb [arch=amd64] http://dl.farsightsecurity.com/debian bullseye-farsightsec main | sudo tee -a /etc/apt/sources.list.d/debian-farsightsec.list $ sudo cp debian-farsightsec.gpg /etc/apt/trusted.gpg.d/ $ sudo apt-get update ``` #### Package Signing Key Our open source Debian packages are hosted at https://dl.farsightsecurity.com/debian/. The GPG keyid is 9B4F9753 (https://dl.farsightsecurity.com/debian/debian-farsightsec.gpg). Its GPG key details are: ```bash pub rsa4096 2025-05-05 [SC] [expires: 2030-05-04] 6BE3C9395BF00DFF2B1A7A45AEEA86469B4F9753 uid DomainTools, LLC (Public Package Signing Key) ``` The previous package signing key was A511AE06. For "apt-get update" use, this key file can be copied to the /etc/apt/sources.list.d/ directory or imported using apt-key add. #### Package Installation Install the core nmsg packages and SIE plugins: ```bash $ sudo apt-get install nmsgtool nmsg-msg-module-sie ``` Install packages for developing C application: ```bash $ sudo apt-get install libnmsg-dev nmsg-msg-module-sie-dev libwdns-dev ``` Install packages for developing Python applications: ```bash $ sudo apt-get install python-nmsg python-wdns ``` Install packages for SIE remote access (sratool, sratunnel): ```bash $ sudo apt-get install axa-tools ``` ### From Source on Rocky 9 (RHEL-compatible) Rocky Linux is compatible with Red Hat Enterprise Linux (RHEL). This installation was tested on Rocky Linux 9.2 with default install options for the Server with GUI. These instructions should work for all RHEL-compatible distributions which have access to the EPEL and CRB repositories or their equivalents. It results in these changes: * A local user was created with sudo rights * VM name was set to example-vm.local * A password was set for root to enable su All updates and patches were applied: $ sudo dnf update #### Setup, Dependencies, and Configuration The following dependencies were installed. Note that both the “Extra Packages for Enterprise Linux: (a.k.a. EPEL) and "Code Ready Builder" (a.k.a crb) repositories containing extra libraries and developer tools are required to compile required packages. ```bash $ sudo dnf install epel-release $ sudo /usr/bin/crb enable # OR sudo dnf config-manager --set-enabled crb $ sudo dnf group install "Development Tools" $ sudo dnf install protobuf-c-devel libedit-devel yajl-devel json-c-devel $ sudo dnf install libpcap-devel zeromq-devel libbsd-devel lmdb-devel ``` Note: you should not need the following, as the group install should cover it: ```bash $ sudo dnf install autoconf automake libtool pkg-config git zlib ``` Ensure paths are set correctly: ```bash $ su # echo "/usr/local/lib" > /etc/ld.so.conf.d/local.conf # exit $ export PKG_CONFIG_PATH=/usr/local/lib/pkgconfig ``` Create a space to work in: ``` $ mkdir ~/fsi ``` You are now ready to install axa and its dependencies. #### Install wdns Clone the repository and install: ```bash $ cd ~/fsi/ $ git clone https://github.com/farsightsec/wdns.git $ cd wdns $ ./autogen.sh $ ./configure $ make $ sudo make install ``` Validate the install was successful. You should now see at least the following files. Note that other files could also be in this directory. ```bash $ ls /usr/local/lib libwdns.a libwdns.la libwdns.so libwdns.so.1 libwdns.so.1.3.1 pkgconfig ``` Ensure the libraries are available from /usr/local/lib: ```bash $ su # ldconfig -v | grep libwdns libwdns.so.1 -> libwdns.so.1.3.1 # exit ``` #### Install nmsg Clone the repository and install: ```bash $ cd ~/fsi/ $ git clone https://github.com/farsightsec/nmsg.git $ cd nmsg $ ./autogen.sh $ ./configure $ make $ sudo make install ``` Validate the install was successful. You should now see at least the following files. Note that other files could also be in these directories: ```bash $ ls /usr/local/lib libnmsg.a libnmsg.so libnmsg.so.8.1.1 libwdns.la libwdns.so.1 nmsg` `libnmsg.la libnmsg.so.8 libwdns.a libwdns.so libwdns.so.1.3.1 pkgconfig $ ls /usr/local/lib/nmsg nmsg_flt1_sample.la nmsg_flt1_sample.so nmsg_msg9_base.la nmsg_msg9_base.so ``` Ensure the libraries are available from /usr/local/lib: ```bash $ su # ldconfig -v | grep libnmsg libnmsg.so.8 -> libnmsg.so.8.1.1 # exit ``` #### Install sie-nmsg Clone the repository and install: ```bash $ cd ~/fsi/ $ git clone https://github.com/farsightsec/sie-nmsg.git $ cd sie-nmsg $ ./autogen.sh $ ./configure $ make $ sudo make install ``` Validate the install was successful. You should now see at least the following files. Note that other files could also be in these dirs. ```bash $ ls /usr/local/lib libnmsg.a libnmsg.so libnmsg.so.8.1.1 libwdns.la libwdns.so.1 nmsg libnmsg.la libnmsg.so.8 libwdns.a libwdns.so libwdns.so.1.3.1 pkgconfig $ ls /usr/local/lib/nmsg nmsg_flt1_sample.la nmsg_msg9_base.la nmsg_msg9_sie.a nmsg_msg9_sie.so nmsg_flt1_sample.so nmsg_msg9_base.so nmsg_msg9_sie.la ``` #### Install axa client Clone the repository and install: ```bash $ cd ~/fsi/ $ git clone https://github.com/farsightsec/axa.git $ cd axa $ ./autogen.sh $ ./configure $ make $ sudo make install ``` Validate the install was successful. You should now see at least the following files. Note that other files could also be in this directory. ```bash $ ls /usr/local/bin axa_link_certs axa_server_cert nmsgtool sratunnel` `axa_make_cert sratool ``` Ensure the libraries are available from /usr/local/lib: ```bash $ su # ldconfig -v | grep libaxa libaxa.so.3 -> libaxa.so.3.0.0 # exit ``` Validate the tool can successfully execute. Note that running sratool like this only proves all libraries and files are installed correctly; it is still necessary to configure access to stream data successfully. ```bash $ sratool -V sratool built using AXA library 3.0.1, supporting AXA protocols v1 to v2; currently using v2 client HELLO: {"hostname":"example-vm.local","uname_sysname":"Linux", "uname_release":"5.14.0-284.30.1.el9_2.x86_64", "uname_version":"#1 SMP PREEMPT_DYNAMIC Sat Sep 16 09:55:41 UTC 2023", "uname_machine":"x86_64","origin":"sratool","libaxa":"3.0.1","libnmsg":"1.1.2", "libwdns":"0.12.0","libyajl":20100,"OpenSSL":"OpenSSL 3.0.7 1 Nov 2022", "AXA protocol":2} ``` ### FreeBSD #### Binary Package Installation Install the core nmsg packages and SIE plugins: ```bash pkg install nmsg sie-nmsg ``` Optionally: Install packages for developing Python applications: ```bash pkg install py-pynmsg py-pywdns ``` Optionally: Install packages for SIE remote access (sratool, sratunnel): ```bash pkg install axa ``` #### Installation from Source using FreeBSD Ports If you do not require the Doxygen generated API documentation, you may wish to disable the DOXYGEN option when building these ports to avoid building Doxygen and its many dependencies. ```bash (cd /usr/ports/net/nmsg && make install clean) (cd /usr/ports/net/sie-nmsg && make install clean) (cd /usr/ports/net/py-pynmsg && make install clean) (cd /usr/ports/dns/py-pywdns && make install clean) (cd /usr/ports/net/axa && make install clean) ``` #### Uninstallation ```bash pkg delete wdns nmsg sie-nmsg py-pynmsg py-pywdns axa ``` ### MacOS 14 Sonoma These instructions are for installing AXA Tools (to enable SIE Remote Access) on Mac computers running MacOS 14.x Sonoma. See additional notes at the end for older Intel-based Macs. #### Setup, Dependencies, and Configuration The following tools and dependencies were installed. This configuration depends on the third-party brew package manager to install all required dependent libraries. Install Xcode command line tools. This shell command opens up a new UX window to confirm the install of the XCode tools. If you have the full version of XCode installed, you can skip this step. ```bash $ xcode-select --install ``` Install Homebrew from brew.sh: ```bash $ /bin/bash -c "$(curl -fsSL` `https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)" ``` Homebrew may require you to make modifications to your shell’s profile. Make these changes before proceeding to ensure brew is set up and configured correctly. Use Homebrew to install all required tools and libraries for AXA and its dependencies: ```bash $ brew install wget openssl@1.1 $ brew install autoconf automake libtool pkgconfig $ brew install protobuf protobuf-c libpcap yajl zmq lmdb json-c ``` Create a space to work in: ```bash $ mkdir ~/fsi ``` #### OpenSSL Configuration OpenSSL requires some extra configuration to ensure we use the brew installed version instead of the version pre-installed by macOS. Alter your ~/.profile or ~/.zprofile file with the following or set these exports directly in the terminal: ```bash export LDFLAGS="-L/opt/homebrew/opt/openssl@1.1/lib" export CPPFLAGS="-I/opt/homebrew/opt/openssl@1.1/include" export PKG_CONFIG_PATH="/opt/homebrew/opt/openssl@1.1/lib/pkgconfig" ``` After setting these OpenSSL config changes, you may need to source your profile changes or restart your shell. You are now ready to install axa and its dependencies. #### Install wdns Clone the repository and run the install: ```bash $ cd ~/fsi $ git clone https://github.com/farsightsec/wdns.git $ cd wdns $ ./autogen.sh $ ./configure $ make $ sudo make install ``` Validate the install was successful. You should now see at least the following files. Note that other files could also be in this dir. ```bash $ ls /usr/local/lib libwdns.1.dylib libwdns.a libwdns.dylib libwdns.la pkgconfig ``` #### Install nmsg Clone the repository and run the install. Note the change to configure: ```bash $ cd ~/fsi $ git clone https://github.com/farsightsec/nmsg.git $ cd nmsg $ ./autogen.sh $ ./configure --with-libpcap=/opt/homebrew/opt/libpcap $ make $ sudo make install ``` Validate the install was successful. You should now see at least the following files. Note that other files could also be in these directories: ```bash $ ls /usr/local/lib libnmsg.8.dylib libnmsg.dylib libwdns.1.dylib libwdns.dylib nmsg libnmsg.a libnmsg.la libwdns.a libwdns.la pkgconfig $ ls /usr/local/lib/nmsg nmsg_flt1_sample.la nmsg_flt1_sample.so nmsg_msg9_base.la nmsg_msg9_base.so ``` #### Install sie-nmsg Clone the repository and run the install. ```bash $ cd ~/fsi $ git clone https://github.com/farsightsec/sie-nmsg.git $ cd sie-nmsg $ ./autogen.sh $ ./configure $ make $ sudo make install ``` Validate the install was successful. You should now see at least the following files. Note that other files could also be in these dirs. ```bash $ ls /usr/local/lib libnmsg.8.dylib libnmsg.dylib libwdns.1.dylib libwdns.dylib nmsg libnmsg.a libnmsg.la libwdns.a libwdns.la pkgconfig $ ls /usr/local/lib/nmsg nmsg_flt1_sample.la nmsg_msg9_base.la nmsg_msg9_sie.a nmsg_msg9_sie.so nmsg_flt1_sample.so nmsg_msg9_base.so nmsg_msg9_sie.la ``` #### Install axa client Clone the repository and run the install. Note the change to configure: ```bash $ cd ~/fsi $ git clone https://github.com/farsightsec/axa.git $ cd axa $ ./autogen.sh $ ./configure --with-libpcap=/opt/homebrew/opt/libpcap $ make $ sudo make install ``` Validate the install was successful. You should now see at least the following files. Note that other files could also be in this directory. ```bash $ ls /usr/local/bin axa_link_certs axa_server_cert nmsgtool sratunnel` `axa_make_cert sratool ``` Validate the tool can successfully execute. Note that running sratool like this only proves all libraries and files are installed correctly, you will still need to set up access to stream data successfully. ```bash $ sratool -V sratool built using AXA library 3.0.2, supporting AXA protocols v1 to v2; currently using v2 client HELLO:` `{"hostname":"example-mac.local","uname_sysname":"Darwin","uname_release":"23.2.0", "uname_version":"Darwin Kernel Version 23.2.0: Wed Nov 15 21:53:34 PST 2023; root:xnu-10002.61.3~2/RELEASE_ARM64_T8103","uname_machine":"arm64","origin":"sratool", "libaxa":"3.0.2","libnmsg":"1.1.2","libwdns":"0.12.0","libyajl":20100, "OpenSSL":"OpenSSL 3.1.4 24 Oct 2023","AXA protocol":2} ``` #### Intel-based Mac Differences Apple Silicon (M1) Macs and older Intel-based Macs have some differences to be aware of since brew operates differently on the two platforms. brew installs all installed packages to /opt/homebrew on Apple Silicon Macs, it installs into /usr/local for Intel-based Macs. All references to /opt/homebrew above will need to be changed. Sudo is not required for make install on Intel Macs as brew changes /usr/local permissions. Sudo is required for Apple Silicon Macs. Since brew operates inside /usr/local on Intel-based Macs and is more permissive with directory permissions, other users of the Mac may have access to these tools by default Intel-based Mac sratool output for V3.x: ```bash $ sratool -V # on an Intel Mac sratool built using AXA library 3.0.1, supporting AXA protocols v1 to v2; currently using v2 client HELLO: {"hostname":"example-mac.local","uname_sysname":"Darwin","uname_release":"23.1.0", "uname_version":"Darwin Kernel Version 23.1.0: Mon Oct 9 21:27:27 PDT 2023; root:xnu-10002.41.9~6/RELEASE_X86_64","uname_machine":"x86_64","origin":"sratool", "libaxa":"3.0.1","libnmsg":"1.1.2","libwdns":"0.12.0","libyajl":20100, "OpenSSL":"OpenSSL 1.1.1w 11 Sep 2023","AXA protocol":2} ``` ## Access Methods This section outlines SIE access methods and links to detailed documentation for each. Information on the file and data formats is available in the Data Formats section. ### SIE Batch Web Interface - Connect: https://batch.sie-remote.net/ - More information: - SIE Batch User Guide - What’s SIE Batch The Batch web interface is a frontend for the REST API that provides full access to the last 12-18 hours of batched SIE data. This batch method is designed for periodic updates that can (with short delay periods) approach real-time. For real-time feeds, consult Remote Access and Direct Connect options, below. Log in with your API key, which you obtain from enterprisesupport@domaintools.com. The interface will present each subscribed channel, the channel’s data format, and default download windows. To use the web interface, select the channel and date range. The SIE web interface will confirm the channel, display its average traffic, and set the default date range. To download current data, set the end date-time to your current time, minus ~10 seconds. SIE Batch does not de-duplicate overlapping downloads. ### SIE Batch REST API - Connect: https://batch.sie-remote.net/siebatchd/v1/ - More information: - SIE Batch API Reference - SIE Batch OpenAPI Specification at SwaggerHub - SIE Batch API: A libcurl Example in C - Batch REST API Reference The Batch REST API is the backend to the Batch Web Interface. Like the web interface, it provides full access to the last 12-18 hours of batched SIE data. This batch method is designed for periodic downloads, rather than as a source of real-time data. For real-time feeds, see Remote Access and Direct Connect options, below. #### Validating an API Key Obtain your API key from enterprisesupport@domaintools.com. Use the key to issue a POST command. For example, with curl: ```bash APIKEY=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx curl -d '{"apikey":"'$APIKEY'"}' https://batch.sie-remote.net/siebatchd/v1/validate ``` In response, SIE will issue a response similar to: ```bash HTTP/1.1 200 OK { "profile": { "username": "siebatch-customer", "siebatch": { "ch212": { "description": "Newly Observed Domains" }, "ch213": { "description": "Newly Observed Fully Qualified Domain Names" }, } }, "_status": "OK", "_message": "" } ``` ### SIE Remote Access (AXA Toolkit) - Connect: Depending on the channel format, you may need to perform further processing after implementing Remote Access. See the Data Formats section below. - More information: - Remote Access Toolkit GitHub repository \+ manpages - AXA User Guide - AXA Manual - AXA Technical Overview SIE Remote Access (SRA) provides real-time streams of all but the highest bandwidth SIE channels. It uses a suite of tools referred to collectively as the AXA (Advanced Exchange Access) toolkit, which implement the Remote Access transport layer (also known as the AXA Transport Layer). In order to reduce bandwidth, SRA provides subscribers with the ability to invoke a server-side filtering capability across a set of channels, selecting only that subset of records that match specific domain name / IP address search criteria. Use Remote Access (AXA) tools sratool and sratunnel to select and define search or filtering criteria for your SIE channels, control rate limiting, and receive accounting messages. #### SIE Remote Access Tool (sratool) sratool is used to test, debug, inspect, or stream SIE Remote Access connections. In its most common invocation, it connects to a Remote Access server, issues AXA protocol messages, and displays the responses. sratool can be automated, but is typically used for interactive user-supplied commands. - More information: - Remote Access Toolkit GitHub repository \+ manpages - AXA User Guide - AXA Manual - AXA Technical Overview #### SIE Remote Access Tunnel (sratunnel) sratunnel is a production command-line tool that streams SIE data to the local network. It automates Remote Access server connections and updates. - More information: - Remote Access Toolkit GitHub repository + manpages - AXA User Guide - AXA Manual - AXA Technical Overview ##### Core Features ###### Watches There are four kinds of watches you can set with Remote Access: * IP Watches: used to express interest in SIE messages containing a specified IP address or CIDR block. AXA understands both IPv4 and IPv6 address types. * DNS Watches: used to express interest in SIE messages containing a specified hostname, domain, or wildcard. * Channel Watches: used to express interest in SIE messages from an entire channel. This is useful to enable “the firehose” for a given channel and ask SRA to send everything from the specified channel rather than matching IP address information or DNS names. * Error watches: used to express interest in SIE messages that cannot be decoded by the server. ###### Rate Limiting Rate limiting is used to limit the rate of incoming AXA messages as emitted from the server to the client. ###### Accounting - More information: - Understanding Accounting Accounting tracks traffic totals. Server-side, SIE maintains a series of per-client packet counters, and emits periodic accounting messages. The command is available from sratool and sratunnel via the \-A command line option. ### SIE Remote Access REST API (AXAMD) - More Information: - axamd Client GitHub repository - Using AXAMD with Newly Observed Domains The AXA Middleware Daemon (AXAMD) provides a REST API interface for the Remote Access service. A Python module, axamd_client, is also available in the axamd GitHub repository. ### SIE Direct Connect Note: You are responsible for the server's ongoing health and maintenance. - More information: - Introduction to nmsgtool Direct Connect provides the fastest connection to the SIE network, with a dedicated, co-located blade server. DomainTools provisions and pre-configures the server and provides root access. Contact enterprisesupport@domaintools.com to discuss your options. With Direct Connect, you use nmsgtool to access and process SIE data on your blade server before transferring it to your network. ## Data Formats SIE data can be available in NMSG, JSON (and NDJSON), and packet capture (pcaplib) formats. The NMSG format, introduced next, is unique to DomainTools and includes libraries and reference implementations for processing SIE data. ### NMSG (Network Message) - More information: - NMSG GitHub repository - NMSG User Guide - Introduction to NMSG - NMSG C API - NMSG headers and encoding - NMSG C API Reference - NMSG Python SDK - NMSG loss tracking - Introduction to nmsgtool NMSG is a file and transmission format that encapsulates typed, structured data into payloads which are packed into containers. The NMSG format relies on Google Protocol Buffers to encode the payload header. Each payload is associated with a specific message schema. nmsg can be extended at runtime with new message types, and can be converted to JSON with the included nmsgtool. #### NMSG Libraries and Tools libnmsg is the reference implementation of this format and provides an extensible interface for creating and parsing messages in NMSG format. Individual NMSG payloads are distinguished by assigned vendor ID and message type values and libnmsg provides a modular interface for registering handlers for specific message types. libnmsg makes it easy to build new message types using a Protocol Buffers compiler. #### C Library: libnmsg NMSG is delivered to the application programmer as a C library called libnmsg. The library presents a rich API for the programmer to build NMSG-capable applications and configure, tune, and/or tweak its many options and features. The reference implementation of libnmsg is the included nmsgtool, a thin wrapper around libnmsg that provides powerful NMSG functionality at the Unix command-line. #### Python Library: pynmsg Also available is a Python extension module, pynmsg, that enables NMSG development using the Python programming language. See the pynmsg distribution and the pynmsg Python extension source. ### JSON and NDJSON/JSONL To see which channels and methods are available in JSON or Newline Delimited JSON (also known as JSON Lines), review the Channels and Formats table (above). nmsgtool can natively convert from NSMG to JSON; consult the NMSG section above. ### Packet Capture Format (PCAP) A small number of channels arrive in packet capture format. ## Detailed Channel Information ### Channels 204, 208, 209: Value-Added Passive DNS These channels consist of: - Channel 207: DNSDB Deduplicated Data - Channel 208, DNSDB Verified Data - Channel 204, Processed DNS Data (used by DNSDB) #### Using Passive DNS Channel Data Data acquired from Channel 204, 207, or 208 is returned in NMSG format for all access methods. NMSG is an adaptable container format that allows for consistent or variable message types. The nmsgtool program is a tool for acquiring a variety of different inputs, like data streams from the network, capturing data from network interfaces, reading data from files, or even standard input and making NMSG payloads available to one or more outputs. The nmsgtool program can acquire data from SIE Channel 220 and convert it to a ND-JSON (newline-delimited JSON) text format for display or additional processing and analysis. nmsgtool is a program written by Farsight and released as open source. After data for Channel 220 has been acquired, written, and saved to a file, you need to decode it to ND-JSON using nmsgtool. The [-r pdns-data.nmsg] option tells nmsgtool to read binary NMSG data from a file, [-c 1] limits the output to single NMSG payload, and [-J -] displays the record in ND-JSON format to stdout, which is typically the screen. ```sh $ nmsgtool -r pdns-data.nmsg -c 1 -J - (returned ND-JSON record) ``` Once the data has been formatted to ND-JSON, a record from the DNS Changes channel will look similar to the following. The following output can be sent to another tool for additional processing. ```json {"time":"2020-04-06 21:48:59.039279480","vname":"SIE","mname":"dnsdedupe", "message":{"type":"EXPIRATION", "count":2,"time_first":"2020-04-06 18:47:22", "time_last":"2020-04-06 18:47:22","bailiwick":"example.com.", "rrname":"www.example.com.", "rrclass":"IN","rrtype":"CNAME","rrttl":3600,"rdata":["dns.example.com."]}} ``` The open source software package is available on Debian and can be installed using $ sudo apt-get install jq. The output from nmsgtool in JSON format [-J -] can be piped to jq using the following: ```sh $ nmsgtool -r pdns-data.nmsg -c 1 -J - | jq -r '.' { "time": "2020-04-06 21:48:59.039279480", "vname": "SIE", "mname": "dnsdedupe", "message": { "type": "EXPIRATION", "count": 2, "time_first": "2020-04-06 18:47:22", "time_last": "2020-04-06 18:47:22", "bailiwick": "example.com.", "rrname": "www.example.com.", "rrclass": "IN", "rrtype": "CNAME", "rrttl": 3600, "rdata": [ "dns.example.com." ] } } ``` #### Data Format for SIE Passive DNS Channels 204, 207, 208 The SIE NMSG dnsdedupe schema is a DNS Query and Response resource record (RR) schema that observes and collects data returned from a query. The data available from these channels contain NMSG SIE:dnsdedup type messages that include the following fields: | KEY | VALUE | | :-- | :-----| | time | Time when hostname was first observed in Channel 204. | | vname | Vendor Name, SIE. | | mname | Message type, dnsdedupe. | | group | Reason DNS message was rejected. | | message | Embedded JSON record describing the observed DNS Query and Response RR. | The embedded NMSG message payload is JSON formatted and includes the following fields: | KEY | VALUE | | :-- | :---- | | type| Types are INSERTION or EXPIRATION.| | count | Number of times an RRset was observed since the last message was sent to the channel. | | time_first | Indicates first time the RRset was observed by pDNS. Unix epoch timestamps with second granularity in UTC. Field is not present if the RRset was only observed from a zone file import.| | time_last | Indicates last time the RRset was observed by pDNS. Unix epoch timestamps with second granularity in UTC. Field is not present if the RRset was only observed from a zone file import. | | response_ip | IP address of the name server replying to the query. Field always exists in Channel 207 and optional in Channels 204 and 208. | | rrname | Domain name or hostname of the query observed by pDNS or extracted from a zone file import. | | rrclass | RR CLASS is always "Internet (IN)", which is decimal value "1". | | rrtype | RR TYPE describes the type of RR, e.g., A(1), NS(2), CNAME(5). | | rrttl | Time to live (TTL) of the RR. | | rdata | Data that describes the RR type (may repeat). | | bailiwick | The domain under which the RRset answer was given. Field always exists in channels 204 and 208. It is not returned in Channel 207. For example, an authoritative generic TLD (gTLD) name server for "com." may respond with different answers for the same query than the authoritative name servers for "farsightsecurity.com." would respond with. | #### Understanding Passive DNS INSERTION and EXPIRATION Mesages DNS data sent to channel DNSDB Deduplicated Data (207), DNSDB Verified Data (208), or Processed DNS Data (204) will be either INSERTION or EXPIRATION type messages. To understand what INSERTION and EXPIRATION mean, we need to discuss how deduplication is implemented in SIE. During processing, the waterfall model maintains a cache table of observed RRsets as a large ring buffer in memory. When DNS data is received by SIE, the cache table is checked to see if the RRset already exists. If the RRset exists in the cache table, the cache entry's count is incremented and the time_last field is updated, and the record discarded as a duplicate. If the RRset does not exist in the cache table, the record is inserted into the cache, and an "INSERTION" record is sent from the deduplicator to the next phase of processing. This causes the oldest record in the ring buffer to be expired from the cache table, and an "EXPIRATION" record is sent from the deduplicator to document the removal. These records are broadcast to the DNSDB Deduplicated Data (207) channel and for the verification phase of the waterfall mode. If you are primarily interested when an RRset is first observed, you can focus on "INSERTION" records and if you are interested in how often an RRset is observed, you should monitor "EXPIRATION" records. #### Example Message -- INSERTION Record The time_first and time_last fields for INSERTION records are always the same and the count is always 1. In the following example, the query was received from IP address 10.10.10.10 (which is acting as the authoritative name server for com.) and the message indicates an SOA record was observed in the response. rrttl displays Time to Live (TTL) value for the record that would be used when caching the data, and rdata is the data returned for the query. ```json { "time": "2020-04-06 22:39:55.429865036", "vname": "SIE", "mname": "dnsdedupe", "message": { "type": "INSERTION", "count": 1, "time_first": "2020-04-06 22:38:48", "time_last": "2020-04-06 22:38:48", "response_ip": "10.10.10.10", "bailiwick": "com.", "rrname": "com.", "rrclass": "IN", "rrtype": "SOA", "rrttl": 86400, "rdata": [ "dns.example.com. dns2.example.com. 1586212699 1800 900 604800 86400" ] } } ``` #### Example Message -- EXPIRATION Record The following example message is for a AAAA resource record. The data returned in the rdata field are the IPv6 addresses for the domain in the rrname field, which is www.example.com.. With the site acting as its own authoritative name server in the bailiwick. Since count is 1, time_first will match time_last, indicating only one query was seen before record expired from the hash table. If value of count was more than 1, time_last may or may not be the same as time_first. ```json { "time": "2020-04-06 22:39:57.420893762", "vname": "SIE", "mname": "dnsdedupe", "message": { "type": "EXPIRATION", "count": 1, "time_first": "2020-04-06 19:32:42", "time_last": "2020-04-06 19:32:42", "bailiwick": "www.example.com.", "rrname": "www.example.com.", "rrclass": "IN", "rrtype": "AAAA", "rrttl": 120, "rdata": [ "2001:db8::1" "2001:db8::2" "2001:db8::3" "2001:db8::a" "2001:db8::b" "2001:db8::c" ] } } ``` SIE Batch provides access to the last 12-18 hours of batched SIE data through both a web interface and REST API. ## Access Methods - Web Interface Guide: Use the browser-based interface to download SIE data. - REST API Reference: Programmatically access SIE Batch data. ## Getting Started To access SIE Batch, you need an API key from DomainTools Enterprise Support at enterprisesupport@domaintools.com. ## More Information - SIE Batch OpenAPI Specification - Main SIE User Guide The Security Information Exchange (SIE), from Farsight Security® Inc. (now a part of DomainTools), is a highly scalable security information sharing platform. Think of SIE as "radar for the Internet"—a way to study what is happening online. Farsight collects and redistributes more than 200,000 raw observations per second from its global network of sensors. Farsight also applies unique proprietary methods to improve the usability of that data, sharing refined intelligence with SIE customers directly and via DNSDB, one of the world's largest passive DNS databases. SIE distributes a variety of data types for security professionals, including: * Raw and processed passive DNS data * Darknet/darkspace telescope data * SPAM sources and URLs * Phishing URLs * Connections from malware-infected systems (as seen by a sinkhole) * Intrusion detection system (IDS) and firewall connection block data SIE Batch is a delivery method that gives you access to a RESTful API to download data as needed. It also has a web-based interface to define your data sets and download them. With SIE Batch you can select the data sets and time periods of interest to you, download that data and have it available for your analysis. SIE Batch allows you to access data two ways: * Via the SIE Batch API: The API allows you to write programs to pull down data for processing automatically. * Interactively: There is a web-based interface that acts as a front end to the API and allows you to select and download sets of data on demand. SIE Batch gives you access to the most recent data distributed via the SIE system. How much data is available depends on the channel you are pulling data from, but is typically the most recent 12-18 hours. SIE Batch is not intended for near-real-time data access. Use SIE Batch for periodic downloads, such as hourly updates. If your use case requires timely access to data (for example, real-time or near-real-time), use SIE Remote Access (SRA) instead. ## Accessing SIE Data Interactively via SIE Batch The SIE Batch system requires a subscription to the SIE data. When you set up the subscription, you receive an API key that gives you access to the system. If you do not have an active subscription, contact the DomainTools sales team. After you log in to the browser API, you see the SIE Batch dashboard. SIE data is returned in one of two formats: Newline Delimited JSON (ND-JSON) and NMSG. ND-JSON formatted files have a .ndjson suffix, while NMSG formatted files have a .nmsg suffix. Most channels return data in ND-JSON format, with the highest volume channels using NMSG because it is more compact. ```json { "time": "2020-01-13 17:53:00.097326040", "vname": "SIE", "mname": "newdomain", "source": "a1ba02cf", "message": { "domain": "clienttons.com.", "time_seen": "2020-01-13 16:16:04", "bailiwick": "ipv4-only.cname.clienttons.com.", "rrname": "jdkyqftipq6rwxq4s7ca-pw7etn-d8f0af301.ipv4-only.cname.clienttons.com.", "rrclass": "IN", "rrtype": "CNAME", "rdata": [ "a248.b.akamai.net." ], "keys": [], "new_rr": [] } } ``` { .img-full } On this page, select the channel and date range for the data you want. The system confirms the channel, sets a default date range for the records to download, and shows you the average hourly data volume for the channel. You can accept this date range or specify your own. You can set any date range as long as the data is available. Channel data expires from the system between 12 and 20 hours, depending on the data rate for each channel. { .img-full } Click Start to generate a data file for download. { .img-full } Click Download to download the file through the browser. For channels with high data rates, this can take some time. Click Copy to copy the URL to your clipboard so you can pass it to a processing program or other system. To get the most current data, set the time to approximately ten seconds before the current time. The generated URL downloads the same data set if used again later, as long as the data remains available. The chfetch API call is equivalent to the Download button in the browser. Note: If you download multiple batches of data with overlapping time periods, the system does not deduplicate the data. Either avoid combining data sets with overlapping time ranges, or deduplicate the data during the merge. Below the Direct Download section is a Quick Download section that provides commonly used downloads for your subscribed channels, giving you quick access to the most recent data available. { .img-full } Click the appropriate time segment for the channel you want to download. To see an estimate of the file size, hover over the download button. After you download the files, you can process and evaluate the data with your own programs. ## Newline Delimited JSON (ND-JSON) formatted files ND-JSON files are formatted text files. The specific fields vary by channel. The following example shows data from Channel 213 (Newly Observed Domains): ```json {"time":"2020-01-13 17:53:00.097143888","vname":"SIE","mname":"newdomain", "source":"a1ba02cf","message":{"domain":"alibaba.com.","time_seen":"2020-01-13 17:51:47", "bailiwick":"alibaba.com.","rrname":"fuz8fk.tdum.alibaba.com.", "rrclass":"IN","rrtype":"CNAME","rdata":["tdumproxy.alibaba.com."], "keys":[],"new_rr":[]}} {"time":"2020-01-13 17:53:00.097326040","vname":"SIE","mname":"newdomain", "source":"a1ba02cf","message":{"domain":"clienttons.com.","time_seen":"2020-01-13 16:16:04", "bailiwick":"ipv4-only.cname.clienttons.com.", "rrname":"jdkyqftipq6rwxq4s7ca-pw7etn-d8f0af301.ipv4-only.cname.clienttons.com.", "rrclass":"IN","rrtype":"CNAME","rdata":["a248.b.akamai.net."],"keys":[],"new_rr":[]}} {"time":"2020-01-13 17:53:00.097453117","vname":"SIE","mname":"newdomain", "source":"a1ba02cf","message":{"domain":"yandex.ru.","time_seen":"2020-01-13 17:51:52", "bailiwick":"yandex.ru.", "rrname":"203859815.verify.yandex.ru.", "rrclass":"IN","rrtype":"CNAME","rdata":["an.yandex.ru."],"keys":[],"new_rr":[]}} ``` You can view ND-JSON files directly or use any tool that supports the ND-JSON format. ## NMSG formatted files NMSG files use a binary format and cannot be viewed directly. Farsight provides tools that decode and display NMSG formatted content. The NMSG tool is available on GitHub at https://github.com/farsightsec/nmsg. To view NMSG data, run nmsgtool, which formats an NMSG file as readable text. The following example shows output from Channel 221 (NXDomains) using the command nmsgtool -r: ```text [43] [2020-01-13 17:46:47.996798921] [2:6 SIE dnsnx] [a1ba02cf] [] [] qname: xvu.co.ls. qclass: IN (1) qtype: AAAA (28) response_ip: 196.216.168.70 soa_rrname: co.ls. [70] [2020-01-13 17:46:47.996805233] [2:6 SIE dnsnx] [a1ba02cf] [] [] qname: 246.25.155.49.in-addr.arpa. qclass: IN (1) qtype: PTR (12) response_ip: 194.146.106.106 soa_rrname: 49.in-addr.arpa. [68] [2020-01-13 17:46:47.996816452] [2:6 SIE dnsnx] [a1ba02cf] [] [] qname: vla1-3s19.yndx.net.yandex.net. qclass: IN (1) qtype: AAAA (28) response_ip: 93.158.134.1 soa_rrname: yandex.net. [64] [2020-01-13 17:46:47.996831866] [2:6 SIE dnsnx] [a1ba02cf] [] [] qname: USTPE2LJ6XDVZ1.jacobs.com. qclass: IN (1) qtype: SOA (6) response_ip: 13.107.24.8 soa_rrname: jacobs.com. [63] [2020-01-13 17:46:47.996873084] [2:6 SIE dnsnx] [a1ba02cf] [] [] qname: _ldap._tcp.pdc._msdcs.sg.com. qclass: IN (1) qtype: SRV (33) response_ip: 207.204.40.129 soa_rrname: sg.com. ``` ## SIE Batch API See the SIE Batch API Reference for API documentation. ## Getting Started When choosing a Newly Observed Domains (NOD) subscription you will determine the format and transport you want to use to retrieve NOD data sets. At the beginning of the provisioning process for the NOD subscription you will be asked to provide one of the following incremental zone transfer methods, depending on what delivery methods you have chosen. For general information on DNS Response Policy zones, consult https://dnsrpz.info/. For more information on the Security Information Exchange (SIE) platform, consult our technical documentation or the SIE User Guide. ## Incremental Zone Transfers (IXFR) If you choose to use Incremental Zone Transfers (IXFR) you will need to give DomainTools a list of IPv4 and IPv6 addresses to which you would like DNS NOTIFY messages to be sent, as well as IPv4 and IPv6 addresses or small address blocks that will be allowed access (the two lists are typically the same). ## Rsync Create an SSH key pair and share the public key with DomainTools. This system currently accepts 4096 bit RSA keys. We know this is not ideal. We will update this doc when this system accepts current encryption standards (talk to your DomainTools representative for more information about the update). OpenSSH no longer supports RSA 4096 by default. To enable it for a single rsync connection, use the following: ```sh rsync -az -e 'ssh -p 49222 -i/myfolder/mysshkey -o PubkeyAcceptedKeyTypes=+ssh-rsa -o HostKeyAlgorithms=+ssh-rsa' your-login@rsync.dns-nod.net:nod/ /destination_folder/ ``` ## Retrieve DNS Zones via Incremental Zone Transfers (IXFR) A Zone Transfer is a term used to refer to the process by which the contents of a DNS Zone file is copied from a primary DNS Server to a secondary DNS server. IXFR is a term used to refer to a incremental zone transfer vs a full zone transfer (AXFR). DomainTools recommends using Incremental Zone Transfers to consume NOD as it provides a near real time mechanism to retrieve NOD updates. To configure your DNS infrastructure to use Incremental DNS Zone Transfers as the transport for NOD, DomainTools will need a list of IPv4 and IPv6 addresses to which you would like DNS NOTIFY messages to be sent, as well as IPv4 and IPv6 addresses or small address blocks that will be allowed access (the two lists are typically the same). You will receive a DNS TSIG key, which will look like the following: ```sh key "FSI-####-#-key" { algorithm HMAC-SHA512; secret “SECRET”; } ``` The zones are served by the following masters. This list is subject to change as DomainTools grows the service over time and you will be given reasonable notice to reconfigure your name server. ```sh masters “fsi-ixfr-masters” { 104.244.13.88 key “FSI-####-#-key”; 104.244.14.88 key “FSI-####-#-key”; 2620:11c:f004::88 key “FSI-####-#-key”; 2620:11c:f008::88 key “FSI-####-#-key”; }; ``` These settings are attached as named.fsi-####-#.conf where ####-# is your account number. DomainTools recommends that you use this file with BIND's include statement in your configuration to simplify future updates. Also included will be an example configuration that you may incorporate into your existing name server's configuration. You will need to add rules to your firewall's access control list(s) for DomainTools hosts to send UDP packets to port 53 of your DNS server so that it can receive the DNS NOTIFY packets for updates. This will allow your DNS server to receive incremental updates every few seconds. If you do not add these firewall rules your zones will only update every ten minutes as per the refresh field in the zone's Start of Authority record. Using TSIGs for authentication requires reasonably synchronized system clocks. Ensure that your server is enabled to use NTP for clock synchronization. ## NOD Response Policy Zones (RPZ) DomainTools Security makes NOD available via Domain Name Service Response Policy Zones (DNS RPZ). DNS RPZ is a method that allows a name server administrator to overlay custom information on top of the global DNS to provide alternate responses to queries. You can read more about DNS RPZ at https://dnsrpz.info/. Response Policy Zones are delivered as one of seven DNS zones suitable for deployment as a DNS Response Policy Zone. Each zone file name contains a time which corresponds to how old the domains in the zone are believed to be. DomainTools recommends the use of the 3h.rpz.dns-nod.net zone as a starting point. You will receive DNS NOTIFY messages for all seven zones but the preference is that you only download the zones that you will actively use, to avoid duplication: - 5m.rpz.dns-nod.net - 10m.rpz.dns-nod.net - 30m.rpz.dns-nod.net - 1h.rpz.dns-nod.net - 3h.rpz.dns-nod.net - 12h.rpz.dns-nod.net - 24h.rpz.dns-nod.net The domains included in the seven DNS zones correspond to the age of the domain as the names were first observed by our sensor network. Think of RPZ as various rolling window from five minutes to 24 hours. In other words, newly observed domains (NOD) are bucketed by age and formed into seven different zone files. When a NOD is first observed, it is inserted into all seven zone files and as the domain ages, our systems remove the name from the appropriate RPZ file starting with the 5m file first rolling up through the files to the last 24h file. A once newly observed domain will ultimately age out and it will no longer be included/found in any of the RPZ zone files. DomainTools does not support RPZ files with domains older than 24 hours old. Example config files below, see the welcome email for personalized examples. File named.fsi-####-#.conf: Ensure the name of this file matches the name of the file you get in the welcome email. ```sh key “FSI-####-#-key” { algorithm HMAC-SHA512 secret “SECRET }; masters “fsi-ixfr-masters” { 104.244.13.88 key “FSI-####-#-key”; 104.244.14.88 key “FSI-####-#-key”; 2620:11c:f004::88 key “FSI-####-#-key”; 2620:11c:f008::88 key “FSI-####-#-key”; }; ``` File named.fsi-nod.conf: ```sh // add this to your options clause options { response-policy { zone “3h.rpz.dns-nod.net” policy given; // zone “3h.rpz.dns-nod.net” policy passthru; # audit with logging }; }; // optionally, add something like this to your logging clause and send // to your SIEM logging { channel named-rpz { file “/var/log/rpz.log” versions 3 size 250k; print-time yes; print-category yes; print-severity yes; severity info; }; category rpz { named-rpz; }; }; // include the keyfile we created above include “/etc/bind/named.fsi-####-#.conf”; zone “3h.rpz.dns-nod.net” { type slave; file “3h.rpz.dns-nod.net.zone”; masters { fsi-ixfr-masters; }; allow-query {localhost;}; allow-transfer {none;}; }; ``` ## NOD DNS Blackhole List Zone (DNSBL) DomainTools Security makes NOD available in DNS-based Blackhole List (DNSBL) Zone file format. DNSBLs convey information over DNS and allow subsequent processes make decisions based on the provided answers. DNSBLs are most commonly used to assist in the scoring of SPAM email with applications like Spamassassin and Postfix. (Examples are below.) Using the named.fsi-####-#.conf configuration file as described above; you can use the following snip-it in your name server configuration file to consume the NOD DNSBL file via IXFR. ```sh include “/etc/bind/named.fsi-####-#.conf”; zone “v1.bl.dns-nod.net” { type slave; file “v1.bl.dns-nod.net.zone”; masters { fsi-ixfr-masters; }; allow-query {localhost;}; allow-transfer {none;}; }; ``` ## Using dig for troubleshooting IXFR IXFR depends on retrieving the SOA (Start Of Authority) record for the zone: this record contains a serial number, and the first step in the IXFR process is to compare the serial number with what the recipient server has locally. In practice, many connectivity issues can be understood by looking at what happens when a request is made to retrieve the SOA record for the zone. The basic command looks like this: ```sh dig -y '::' @ \ SOA ``` Using the example data given above: ```sh dig -y 'HMAC-SHA512:FSI-####-#-key:NOT=REALLY=YOUR=SECRET' @104.244.13.88 v1.bl.dns-nod.net SOA ; <<>> DiG 9.8.3-P1 <<>> -y HMAC-SHA512 @104.244.13.88 v1.bl.dns-nod.net SOA ; (1 server found) ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 14186 ;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 2, ADDITIONAL: 1 ;; WARNING: recursion requested but not available ;; QUESTION SECTION: ;v1.bl.dns-nod.net. IN SOA ;; ANSWER SECTION: v1.bl.dns-nod.net. 86400 IN SOA a.ns.dns-nod.net.v1.bl.dns-nod.net. nod-admin.fsi.io. 1479754872 600 300 86400 300 ;; AUTHORITY SECTION: v1.bl.dns-nod.net. 86400 IN NS a.ns.dns-nod.net. v1.bl.dns-nod.net. 86400 IN NS b.ns.dns-nod.net. ;; TSIG PSEUDOSECTION: fsi-####-#-key. 0 ANY TSIG hmac-sha512. 1479754882 300 64 NOT=REALLY=YOUR=SECRET 14186 NOERROR 0 ;; Query time: 71 msec ;; SERVER: 104.244.13.88#53(104.244.13.88) ;; WHEN: Mon Nov 21 11:01:22 2016 ;; MSG SIZE rcvd: 258 ``` That shows a successful response. The two most important pieces of information are: - the query status shown on the HEADER line - the TSIG section which will show any key-related issues and of course, that you got a response at all. (If you're sniffing packets, it's a lot easier to look for the SOA query than to try to recognize a zone transfer which probably spans multiple packets.) ## Retrieve DNS Zones via Rsync NOD RPZ files can be retrieved via Rsync when DNS Zone Transfers are not desired. Rsync is a file copying tool, it is known for its delta-transfer algorithm, which reduces the amount of data sent over the network by sending only the differences between the source files and the existing files at the destination. ## NOD Response Policy Zones (RPZ) Starting Recommendation DomainTools recommends the use of the 3h.rpz.dns-nod.net zone as a starting point. There are separate files based on the age of the domain names. Please only download the file for the zone that you will actively use. path - filename: - nod/rpz/5m.rpz.dns-nod.net.zone - nod/rpz/10m.rpz.dns-nod.net.zone - nod/rpz/30m.rpz.dns-nod.net.zone - nod/rpz/1h.rpz.dns-nod.net.zone - nod/rpz/3h.rpz.dns-nod.net.zone - nod/rpz/12h.rpz.dns-nod.net.zone - nod/rpz/24h.rpz.dns-nod.net.zone Example shell command to download the three hour NOD RPZ Zone file to the directory /srv/nod: ```sh rsync -az -e "ssh -p 49222 –i/path/to/sshkey" \ USERNAME@rsync.dns-nod.net:nod/rpz/3h.rpz.dns-nod.net.zone /srv/nod/ ``` The following is an example BIND configuration for RPZ: ```sh options { response-policy { zone "3h.rpz.dns-nod.net"; }; }; zone "3h.rpz.dns-nod.net" { type master; file "/srv/nod/3h.rpz.dns-nod.net.zone"; allow-query {localhost;}; allow-transfer {none;}; allow-update {none;}; }; ``` Example shell script to be run by cron to download and load the three hour NOD RPZ zone file using BIND utility rndc. If your operating system distribution includes the flock - manage locks from shell scripts command, using it will prevent multiple cronned instances from overrunning each other (its semantics are similar to time): ```sh #!/bin/sh rsync -az -e "ssh -p 49222 –i/path/to/sshkey" \ USERNAME@rsync.dns-nod.net:nod/rpz/3h.rpz.dns-nod.net.zone /srv/nod/ rndc reload 3h.rpz.dns-nod.net ``` Using NOD RPZ this way will cause DNS queries for any domains in the selected database to return an NXDOMAIN response, should those domains be queried by an end-user. ## NOD DNS Blackhole List (DNSBL) File Availability (for rbldnsd) DomainTools also makes NOD available in a DNS-based Blackhole List (DNSBL) file format for rbldnsd. Note this is a different format than the v1.bl.dns-nod.net file, but rbldnsd may be used to serve it as the same DNSBL zone. Your user name on the DomainTools server will be "USERNAME". Example shell command to download the one hour NOD DNSBL Zone file to the directory /srv/nod: ```sh rsync -az -e "ssh -p 49222 –i/path/to/sshkey" USERNAME@rsync.dns-nod.net:v1/nod.rbldnsd /srv/nod/ ``` ## Use rbldnsd to serve the NOD DNSBL Zone DomainTools NOD DNSBL Zones can be loaded and served by rbldnsd which is a small authoritative-only nameserver designed to serve DNS-based blocklists (DNSBLs). For more information see https://rbldnsd.io/. ## Configure rbldnsd to load the NOD DNSBL Zone The following is an example of the rbldnsd command-line arguments. Starting rbldnsd: ```sh rbldnsd -b 127.0.0.1/5053 -r /srv/nod/ v1.bl.dns-nod.net:dnset:nod.rbldnsd -p /var/run/rbldnsd.pid ``` ## Configuring BIND to forward DNSBL queries to rbldnsd Sample BIND configuration forwarding to local port 5053 being served by rbldnsd: ```sh zone "v1.bl.dns-nod.net" IN { type forward; forward first; forwarders { 127.0.0.1 port 5053; }; }; ``` ## Using the NOD DNSBL Many applications can be configured to use a DNSBL. This user guide gives guidance on two open source applications that can be configured to consume a DNSBL (SpamAssassin and Postfix). When using NOD in this manner, DNS queries for A type (IP address) resource records of the form domain.v1.bl.dns-nod.net will return an address that indicates the age of domain if it is in the NOD database. The response should be interpreted as follows: | Period | Response | | ------------ | --------- | | 0-5 minute | 127.0.0.2 | | 5-10 minute | 127.0.0.3 | | 10-30 minute | 127.0.0.4 | | 30-60 minute | 127.0.0.5 | | 1-3 hours | 127.0.0.6 | | 3-12 hours | 127.0.0.7 | | 12-24 hours | 127.0.0.8 | “NXDOMAIN” (no such domain) response will be returned if the domain is not in the NOD database. TXT DNS queries will return more detailed information about the domain if it is in the database. For example: ```sh $ dig +short svetlanovskiy.accountant.v1.bl.dns-nod.net 127.0.0.2 ``` ```sh $ dig +short svetlanovskiy.accountant.v1.bl.dns-nod.net txt "first_seen=1461953815" ``` The first_seen date is in Unix seconds. That value can be converted to "human time" with a command such as: ```sh $ date -d @1461953815 Fri Apr 29 11:16:55 PDT 2016 ``` You may test your setup using your favorite DNS query tool and the special test domain test.dns-nod.net and invalid.dns-nod.net (with your resolver using the DNSBL zone). ```sh $ host test.dns-nod.net.v1.bl.dns-nod.net test.dns-nod.net.v1.bl.dns-nod.net has address 127.0.0.2 ``` ```sh $ host invalid.dns-nod.net.v1.bl.dns-nod.net Host invalid.dns-nod.net.v1.bl.dns-nod.net not found: 3(NXDOMAIN) ``` ## Configure SpamAssassin to use NOD DNSBL To configure SpamAssassin to use NOD DNSBL, append the following to /etc/spamassassin/local.cf or ~/.spamassassin/user_prefs, adjusting scores to taste: ```sh urirhssub URIBL_NOD_5M v1.bl.dns-nod.net. A 127.0.0.2 body URIBL_NOD_5M eval:check_uridnsbl('URIBL_NOD_5M') describe URIBL_NOD_5M Contains an URL that is only 5 minutes old in Farsight Passive DNS tflags URIBL_NOD_5M net domains_only score URIBL_NOD_5M 5 urirhssub URIBL_NOD_10M v1.bl.dns-nod.net. A 127.0.0.3 body URIBL_NOD_10M eval:check_uridnsbl('URIBL_NOD_10M') describe URIBL_NOD_10M Contains an URL that is only 10 minutes old in Farsight Passive DNS tflags URIBL_NOD_10M net domains_only score URIBL_NOD_10M 5 urirhssub URIBL_NOD_30M v1.bl.dns-nod.net. A 127.0.0.4 body URIBL_NOD_30M eval:check_uridnsbl('URIBL_NOD_30M') describe URIBL_NOD_30M Contains an URL that is only 30 minutes old in Farsight Passive DNS tflags URIBL_NOD_30M net domains_only score URIBL_NOD_30M 5 urirhssub URIBL_NOD_1H v1.bl.dns-nod.net. A 127.0.0.5 body URIBL_NOD_1H eval:check_uridnsbl('URIBL_NOD_1H') describe URIBL_NOD_1H Contains an URL that is only one hour old in Farsight Passive DNS tflags URIBL_NOD_1H net domains_only score URIBL_NOD_1H 5 urirhssub URIBL_NOD_3H v1.bl.dns-nod.net. A 127.0.0.6 body URIBL_NOD_3H eval:check_uridnsbl('URIBL_NOD_3H') describe URIBL_NOD_3H Contains an URL that is only three hours old in Farsight Passive DNS tflags URIBL_NOD_3H net domains_only score URIBL_NOD_3H 5 urirhssub URIBL_NOD_12H v1.bl.dns-nod.net. A 127.0.0.7 body URIBL_NOD_12H eval:check_uridnsbl('URIBL_NOD_12H') describe URIBL_NOD_12H Contains an URL that is only twelve hours old in Farsight Passive DNS tflags URIBL_NOD_12H net domains_only score URIBL_NOD_12H 5 urirhssub URIBL_NOD_24H v1.bl.dns-nod.net. A 127.0.0.8 body URIBL_NOD_24H eval:check_uridnsbl('URIBL_NOD_24H') describe URIBL_NOD_24H Contains an URL that is only 24 hours old in Farsight Passive DNS tflags URIBL_NOD_24H net domains_only score URIBL_NOD_24H 5 ``` ## Configure Postfix to use NOD DNSBL For postfix to reject everything: ```sh reject_rhsbl_sender = v1.bl.dns-nod.net ``` To cut off at a particular age threshold using postfix version 2.8 or higher: ```sh reject_rhsbl_sender = v1.bl.dns-nod.net=127.0.0.[2-8] ``` ## NOD and Local Firewalls If you need to configure your firewalls to allow access to the NOD systems at DomainTools, please be aware that servers may be rotated in and out ## Introduction Farsight Security’s (now a part of DomainTools) Advanced Exchange Access (AXA) is a suite of tools and library code that brings the Security Information Exchange (SIE) directly to the end-user's network. SIE is a scalable and adaptable real-time data streaming and information sharing platform. SIE collects and provides access to more than 200,000 observations per-second of raw data from its global sensor network. Farsight also applies unique and proprietary methods for improving usability of the data, directly sharing the refined intelligence with SIE customers and DNSDB®, one of the world's largest passive DNS (pDNS) databases. The diverse set of data available from SIE includes the following and is relevant and useful for practitioners in various technology roles: - Raw and processed passive DNS data - Darknet/darkspace telescope data - SPAM sources and URLs - Phishing URLs and associated targeted brands - Connection attempts from malware-infected systems (as seen by a sinkhole) - Network traffic blocked by Intrusion Detection Systems (IDS) and firewall devices Each unique set of data in SIE is known as a channel and the data acquired from a specific channel can be customized to meet the needs of each customer, enabling you to subscribe to and access only the channels needed to solve your problem. A channel in SIE may be the result of raw data analysis, or a subset of data from other channels. The data available from SIE channel subscription packages includes: Raw Passive DNS: Real-time observations of DNS cache-miss traffic sent from DNS recursive resolvers on the Internet to authoritative name servers. The DNS information includes authoritative DNS data that various zone operators make available and responses from authoritative name servers to recursive resolver queries Value-Added Passive DNS: Real-time de-duplicated, filtered, and verified passive DNS (pDNS) data observed on the Internet Newly Observed Domains (NOD) and Newly Observed Hostnames (NOH): Real-time actionable insights for domains and hostnames, fully qualified domain names (FQDNs), when they are first successfully resolved on the Internet Base Channels: A collection of threat intelligence channels that provide access to honeypot data (darknet and spam) and botnet (e.g., Conficker) sinkhole data. The data also includes threat intelligence for phishing campaigns and log data for network traffic blocked by Intrusion Detection Systems (IDS) and firewall devices Premium Channels: A range of premium security-related feeds including malware metadata, IOCs and other telemetry. Subscribers consume the intelligence as real-time event flows rather than traditional batch transfers - which are inherently delayed The SIE Channel Guide provides an overview of the channels available from SIE. ## Advanced Exchange Access Toolkit The Advanced Exchange Access (AXA) toolkit contains tools and a C library to bring Farsight's real-time data and services directly from the Farsight Security Information Exchange (SIE) to the subscriber's network edge. AXA enables subscribers to connect to Farsight's subscription-based SRA (SIE Remote Access) servers. These servers provide access to data and services built from Farsight's SIE. ### Contents of the Advanced Exchange Access Toolkit Farsight freely provides several end-user tools to stream AXA data in the Advanced Exchange Access Toolkit. This distribution contains the following: - sratool: A command-line tool used to connect to an SRA server, set watches, enable SIE channels, and stream data. - sratunnel: A command-line tool that streams SIE data to the local network. - libaxa: A C library providing an API for the AXA protocol including: - connection instantiation/teardown - message encapsulation/decapsulation - watch parsing/loading - control packet rate limits, sampling rates, window sizes, and many other AXA-specific functions ## Farsight SIE Remote Access SIE Remote Access (SRA) is Farsight's software solution to make SIE content available to remote users. SRA enables SIE channel traffic to be delivered through a TCP stream across the Internet. In order to reduce bandwidth, SRA provides subscribers with the ability to invoke a server-side filtering capability across a set of channels, selecting only that subset of records that match specific domain name / IP address search criteria. The Advanced Exchange Access suite of tools and library code is the software that implements Farsight's SRA service. The AXA suite consists of two Unix command line tools and one C library developers can use to build custom SRA applications. ### sratool sratool is the AXA Swiss Army Knife. It is a versatile tool used to test, debug, or stream AXA connections. It connects to an SRA server, sends protocol messages and displays the responses. It can also tunnel SIE data like sratunnel does. ### sratunnel sratunnel transfers selected SIE data from the remote server to the local network. The connection to the server is created and restored, with binary exponential delays between retries. sratunnel is the workhorse of the AXA family. It is used to transfer SIE data from the remote server to the local network. It is what Farsight uses for production deployment of SIE data to the customer. sratunnel can be thought of as a fast, efficient, and smart conduit for SIE data. Data goes in one end and sratunnel can emit the data into different output formats, including: - NMSGs to a UDP port - NMSGs to a TCP port - NMSGs to a file - pcap to a file - pcap to a network interface ### libaxa libaxa is the C programming library that exposes the AXA API to the application programmer. ## The AXA Protocol The AXA protocol is documented in the section titled AXA Protocol in the README file found on the GitHub repository for the Farsight Advanced Exchange Access Toolkit.  ### AXA Limits Some of the channels offered by the SIE network burst to an extremely high bitrate. AXA has two ways to deal with such network-hungry situations: optional filtering and loss-tolerance built into the protocol. Filtering can take one of the following forms: - Via the rate limit option to reduce the flow of ingress data to a certain number of packets per second. - Via one or more IP-based or DNS-based "watches" to limit the flow of data to specific assets the subscriber wishes to observe. Finally, AXA is a deliberately lossy protocol. If a subscriber requests more data than the network can carry, data overruns will occur. When this happens, loss markers are transmitted reliably within the AXA stream to inform the subscriber via the AXA accounting subsystem (see below). At this point, the subscriber's possible mitigation strategies include: - ask for less data via rate limiting - increase network capacity and/or other host resources - treat the SRA stream as a chunky and non-representative sample of the total SIE data - pursue Direct Access to SIE ### AXA Watches In the AXA world, the way an end-user registers interest in an asset is through what's known as a watch. These are fundamental building blocks of AXA sessions. In order to get anything done, an end-user must specify one or more watches to inform the AXA server what she is interested in seeing. There are four different types of AXA watches, each is described below: - IP Watches: used to express interest in SIE messages containing a specified IP address or CIDR block. AXA supports both IPv4 and IPv6 address types. - DNS Watches: used to express interest in SIE messages containing a specified hostname, domain, or wildcard hostname. - Channel Watches: used to express interest in all SIE messages from a given channel. This is useful to enable "the firehose" for a given channel and ask SRA to send everything from the specified channel rather than matching IP address information or DNS names. Only valid for SRA connections. - Error watches: used to express interest in SIE messages that cannot be decoded by the server. Only valid for SRA connections and intended only for debugging. Watches are referenced by tags. A tag is simply an arbitrary 16-bit integer label used to keep track of individual watches. When connected to an SRA server, tags must be unique. Each watch will use a different tag. Depending on the tool, tagging may be done explicitly by the end-user or implicitly, by the tool. ### SRA and AXA RESTful Interface The Farsight Security Advanced Exchange Access (AXA) RESTful Interface adds a streaming HTTP interface on top of the AXA toolkit to enable developers of web-based applications to interface with Farsight Security's SIE Remote Access (SRA) servers. The SRA module facilitates the real time streaming of data from the Security Information Exchange (SIE) over HTTP using a RESTful API. Access is controlled via an API key that is passed as the X-API-Key HTTP header. The Farsight Security AXA RESTful Interface does not have any specific operating system requirements as it is delivered over a RESTful API. Farsight Security provides a convenience command line interface (CLI) tool that doubles as a Python extension module which is compatible with various modern operating systems. AXA REST requires HTTPS permitted outbound to axa-sie.domaintools.com. Subscribers must have purchased a service entitlement from Farsight Security and have been provisioned an API key.Accounting Messages By default, axamd will return AXA accounting messages containing current counter statistics relevant to the current session. For more details on these packet counts, see the Accounting section below ## Rate Limiting Rate limiting is used to limit the rate of incoming AXA messages transmitted from the server to the client. While it works for SRA, it is primarily useful for high bitrate SIE channels such as the DNS Errors channels. ## Sampling Sampling is a way to statistically sample a percentage of messages from the server. For example, if a value of 50 is chosen, AXA will return a uniformly distributed random sample of 50% of the messages caught by watches.  ## TCP Buffer Sizing AXA allows the end-user to get or set the TCP send buffer size used by the server. These buffers serve to accumulate outgoing data that the network stack has not yet been able to put on the wire and data that has been received from the wire but not yet read by applications. In simpler terms, if you know what you're doing, these options allow the end-user to adjust the size of the in-kernel TCP buffers to optimize network performance. As an aside, these options do not directly adjust the TCP window size (see TCP Windows and Window Scaling for more information). Note that internally, TCP actually allocates twice the size of the buffer requested (using the extra space for internal purposes. This is why AXA returns a buffer size that is twice the size requested. The minimum size is 1024 and the maximum is 262142. Only sratool support this option for TCP-based connections. ## Accounting AXA has a series of server-side packet counters used to track traffic totals. This is the mechanism by which AXA tracks, logs, and communicates server-side packet information. This information is intended for users of SIE Remote Access (SRA). ### What is Accounting? Accounting is AXA's way of keeping track of traffic totals. Server-side, AXA maintains a series of per-client packet counters (a full list is below). The AXA protocol message AXA_P_OP_ACCT sent from client to server is used to query this data. The feature is available from sratool via the acct command. It is also available from sratunnel via the -A command line option. Accounting messages are also logged server-side. ### Accounting Glossary The AXA accounting counters are described below. - total-congested: Packets that were unable to be forwarded to the client due to connection issues, although it could also be client load (i.e., the connection isn't removing data from the server fast enough, or the client isn't reading and processing the data fast enough). Note that: high server load might increase total-missed, but it would generally decrease total-congested by throttling the input to the congested pipe / process. - total-filtered: This is an SRA input counter. It measures packets/nmsgs that have been read on the input side (usually directly from the SIE network) and will be submitted to the filtering logic (i.e.: an AXA watch). - total-missed: This is an SRA loss counter. It measures packets that the SRA server failed to receive on the input side either because the daemon was too busy or because they were lost during transit. This counter tracks NMSG and pcap-based loss. - total-ratelimited: This measures packets that were dropped instead of being forwarded to the SRA client to comply with the rate limits specified for the client. - total-sent: This measures packets that have been sent to an SRA client. ## Introduction Security Information Exchange (SIE) is the world's largest real-time threat intelligence platform — it aggregates, filters and broadcasts diverse Internet-security related information so security professionals can more accurately and quickly identify, map, and protect from cybercriminal activity. There are multiple delivery mechanisms to consume data on SIE--consult the SIE User Guide for more information. This user guide will discuss and illustrate the tools found in the Farsight Advanced Exchange Access Toolkit to connect and consume data from SIE. ### Advanced Exchange Access Toolkit The Advanced Exchange Access (AXA) toolkit contains tools and a C library to bring Farsight's real-time data and services directly from the Farsight Security Information Exchange (SIE) to the subscriber's network edge. AXA enables subscribers to connect to Farsight's subscription-based SRA (SIE Remote Access) and RAD (Real-time Anomaly Detector) servers. These servers provide access to data and services built from Farsight's SIE. SRA streams real-time SIE data while RAD streams real-time anomaly detection data (from services such as Brand Sentry and Domain Sentry). ## Requirements ### Operating System Linux, FreeBSD or other POSIX compliant operating systems. ### Hardware The minimum hardware requirements to get started with tools in the Advanced Exchange Access Toolkit are listed below. Depending on the amount of data being processed, the resources may need to be increased accordingly. - 1 CPU - 1 GB Memory - 1 GB Disk ### Network Tools in the Advanced Exchange Access Toolkit require permitted outbound to sra.sie-remote.net and rad.sie-remote.net over TCP using port 22. ### Service Entitlement Subscribers must have purchased a SIE service entitlement from Farsight Security and have been provisioned access using a SSH key. ## Contents of the Advanced Exchange Access Toolkit The Advanced Exchange Access Toolkit distribution contains the following: - sratool: A test/debug/instructional command-line tool used to connect to an SRA server, set watches, enable SIE channels, and stream data. - radtool: A test/debug/instructional command-line tool used to connect to a RAD server, set watches, enable anomaly detection modules, and stream data. - sratunnel: A production command-line tool that streams SIE data to the local network. - radtunnel: A production command-line tool that streams anomaly data to the local network. - libaxa: A C library providing an API for the AXA protocol including: - connection instantiation/teardown, - message encapsulation/decapsulation, - watch parsing/loading, - trie storage and lookup, - control packet rate limits, sampling rates, window sizes, and many other AXA-specific functions. For usage details on sratool, radtool, sratunnel, and radtunnel, please see their respective man pages (included in the distribution). ## Installing Advanced Exchange Access Toolkit (axa-tools) ### Debian 8 and Ubuntu 14.04/16.04 These instructions use Debian packages created, maintained and hosted by DomainTools. 1. Download the Farsight Apt signing key. ```sh $ sudo wget -O /etc/apt/trusted.gpg.d/debian-farsightsec.gpg \ https://dl.farsightsecurity.com/debian/archive.pubkey ``` 2. Add the Farsight Debian repository. ```sh $ echo "deb http://dl.farsightsecurity.com/debian wheezy-farsightsec main" \ | sudo tee -a /etc/apt/sources.list.d/debian-farsightsec.list ``` 3. Resynchronize the package index files. ```sh $ sudo apt update ``` 4. Install the Advanced Exchange Access Toolkit (axa-tools). ```sh $ sudo apt install axa-tools ``` ### Build from Source See the section titled Building manually https://github.com/farsightsec/axa/blob/master/README.md#building-manually in the README file found on the GitHub repository for the Farsight Advanced Exchange Access Toolkit https://github.com/farsightsec/axa ### Configuring Advanced Exchange Access with SSH 1. At the time of provisioning you would have been asked to generate a SSH key pair used for authentication. The following steps will reference this key, make sure you reference the correct directory path when configuring the key. ```sh $ ssh-keygen -t rsa -b 4096 -C farsight_security -f ~/.ssh/farsight_security ``` 2. Create or edit the SSH config file with the following: ```sh $ vim ~/.ssh/config ``` Add the following: ```sh Host sra.sie-remote.net rad.sie-remote.net IdentityFile ~/.ssh/farsight_security ``` ## Usage Examples ### Prerequisites - A host with Linux, FreeBSD or other POSIX compliant operating system installed - A SIE Remote Access entitlement from Farsight Security - Having already exchanged a SSH keypair with Farsight Security - Having installed and configured the Advanced Exchange Access Toolkit ### sratool sratool is a test/debug/instructional command-line tool used to connect to an SRA server, set watches, enable SIE channels, and stream data. ### Stream SIE traffic with sratool An example using sratool to emit five messages seen on SIE Channel 255 (SIE Heartbeat Channel): 1. $ sratool 2. sra> connect ssh:sra-service@sra.sie-remote.net: connect to an SRA server using the SSH transport. SSH used its keyring to prove the user's identity, so there was no 'password:' prompt. The HELLO response from the remote end displays its version number and the protocol level. 3. sra> count 5: instruct the sratool client to stop after five messages are output. 4. sra> channel 255 on: instruct the remote end to listen to SIE channel 255 which was OK'd by the server indicating that it is provisioned for this channel according to the authentication and authorization level. 5. sra> 1 watch ch=255: watch all content on channel 255 (with no rate limiting or filtering). ```sh $ sratool sra> connect ssh:sra-service@sra.sie-remote.net HELLO srad version 1.2.1 sra AXA protocol 1 sra> count 5 sra> channel 255 on OK CHANNEL ON/OFF channel ch255 on sra> 1 watch ch=255 1 OK WATCH started 1 ch255 base encode TEXT 1 ch255 base encode TEXT 1 ch255 base encode TEXT 1 ch255 base encode TEXT 1 ch255 base encode TEXT packet count limit exceeded sra> exit ``` ### sratunnel sratunnel is a production command-line tool that streams SIE data to the local network. ### Create a persistent connection to SIE An example using sratunnel as a background process to stream nmsg messages from SIE Channel 255 (SIE Heartbeat Channel) to the loopback interface on port 8000. 1. Invoke sratunnel with the following arguments. ```sh $ sratunnel -s 'ssh:sra-service@sra.sie-remote.net' -c 255 \ -w ch=255 -o nmsg:udp:127.0.0.1,8000 & ``` 2. Use ```tcpdump``` to confirm messages are being streamed. ```sh $ sudo tcpdump -i lo -c 5 -nn port 8000 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on lo, link-type EN10MB (Ethernet), capture size 262144 bytes 11:18:41.204425 IP 127.0.0.1.36707 > 127.0.0.1.8000: UDP, length 941 11:18:58.672776 IP 127.0.0.1.36707 > 127.0.0.1.8000: UDP, length 941 11:19:16.312962 IP 127.0.0.1.36707 > 127.0.0.1.8000: UDP, length 941 11:19:33.833821 IP 127.0.0.1.36707 > 127.0.0.1.8000: UDP, length 941 11:19:51.277784 IP 127.0.0.1.36707 > 127.0.0.1.8000: UDP, length 941 5 packets captured 10 packets received by filter 0 packets dropped by kernel ``` 3. Bring the background process to the foreground. ```sh $ fg ``` 4. Kill the sratunnel process by pressing Control-C. ### Process messages with nsmgtool The nmsgtool program is a single tool for taking inputs from a variety of different inputs like data streams from the network, capturing data from network interfaces, reading data from files or even standard input and making NMSG payloads available to one or more outputs. ### Installing nmsgtool 1. Install https://www.farsightsecurity.com/technical/SIE-user-guide/sie-debian/ nmsgtool: ```sh $ sudo apt install nmsgtool nmsg-msg-module-sie ``` ### Viewing a stream of nmsg's from sratunnel with nmsgtool An example using sratunnel as a background process to stream nmsg messages from SIE Channel 255 (SIE Heartbeat Channel) to the loopback interface on port 8000; using nmsgtool to connect to the loopback interface and print the nmsg to the terminal in presentation format. 1. Invoke sratunnel with the following arguments: ```sh $ sratunnel -s 'ssh:sra-service@sra.sie-remote.net' -c 255 -w ch=255 -o nmsg:udp:127.0.0.1,8000 & ``` 2. Invoke nmsgtool to connect to the loopback interface on port 8000, process three payloads and print the output to the terminal using the presentation format. ```sh $ nmsgtool -l 127.0.0.1/8000 -c 3 -o - [23] [2017-06-28 19:53:51.844574928] [1:11 base encode] [1ba02cfd] [] [] type: TEXT payload: [23] [2017-06-28 19:53:52.345241069] [1:11 base encode] [1ba02cfd] [] [] type: TEXT payload: [23] [2017-06-28 19:53:52.845875978] [1:11 base encode] [1ba02cfd] [] [] type: TEXT payload: ``` 3. Bring the background process to the foreground. ```sh $ fg ``` 4. Kill the sratunnel process by pressing Control-C. ### Saving a stream of nmsg's from sratunnel with nmsgtool An example using sratunnel as a background process to stream nmsg messages from SIE Channel 255 (SIE Heartbeat Channel) to the loopback interface on port 8000; using nmsgtool to connect to the loopback interface and saving the output to a rotating set of files using the nmsgtool kicker function. 1. Invoke sratunnel with the following arguments: ```sh $ sratunnel -s 'ssh:sra-service@sra.sie-remote.net' -c 255 \ -w ch=255 -o nmsg:udp:127.0.0.1,8000 & ``` 2. Invoke nmsgtool to connect to the loopback interface on port 8000, save nmsg files to disk every sixty seconds as a background process. ```sh $ nmsgtool -l 127.0.0.1/8000 -t 60 -k '/bin/true' -w ch255 & ``` 3. List the saved files using ls. ```sh $ ls -l total 16 -rw-r--r-- 1 demo demo 5518 Jun 28 16:03 ch255.20170628.2002.1498698127.548592412.nmsg -rw-r--r-- 1 demo demo 6436 Jun 28 16:04 ch255.20170628.2003.1498698180.574404303.nmsg ``` 4. Read one of the files using nmsgtool and outputting the results to the terminal in JSON: ```sh $ nmsgtool -r ch255.20170628.2003.1498698180.574404303.nmsg -J - {"time":"2017-06-28 20:03:02.061745882","vname":"base", "mname":"encode","source":"1ba02cfd", "message":{"type":"TEXT","payload":"IkZTSSBTSUUgaGVhcnRiZWF0Ig=="}} {"time":"2017-06-28 20:03:02.562045097","vname":"base", "mname":"encode","source":"1ba02cfd", "message":{"type":"TEXT","payload":"IkZTSSBTSUUgaGVhcnRiZWF0Ig=="}} {"time":"2017-06-28 20:03:03.062705039","vname":"base", "mname":"encode","source":"1ba02cfd", "message":{"type":"TEXT","payload":"IkZTSSBTSUUgaGVhcnRiZWF0Ig=="}} ``` 5. Bring the nmsgtool background process to the foreground. ```sh $ fg ``` 6. Kill the nmsgtool process by pressing Control-C. 7. Bring the sratunnel background process to the foreground. ```sh $ fg ``` 8. Kill the sratunnel process by pressing Control-C. ## AXA Protocol The AXA protocol is documented in the section titled AXA Protocol https://github.com/farsightsec/axa/blob/master/README.md#axa-protocol in the README file found on the GitHub repository for the Farsight Advanced Exchange Access Toolkit https://github.com/farsightsec/axa. ## Limits Some of the channels offered by the SIE network burst to an extremely high bitrate (some over 500Mbps). AXA has two ways to deal with such network-hungry situations: optional filtering and loss-tolerance built into the protocol. Filtering can take one of the following forms: - Via the rate limit option to reduce the flow of ingress data to a certain number of packets per second. - Via one or more IP-based or DNS-based "watches" to limit the flow of data to specific assets the subscriber wishes to observe. Finally, AXA is a deliberately lossy protocol. If a subscriber requests more data than the network can carry, data overruns will occur. When this happens, loss markers are transmitted reliably within the AXA stream to inform the subscriber via the AXA accounting subsystem https://www.farsightsecurity.com/2015/09/24/mschiffm-axa-accounting/. At this point, the subscriber's possible mitigation strategies include: - ask for less data via rate limiting - increase network capacity and/or other host resources - treat the SRA stream as a chunky and non-representative sample of the total SIE data - pursue Direct Access to SIE Farsight's Advanced Exchange Access (AXA) is a suite of tools and library code that brings the Security Information Exchange (SIE) directly to the end-user's network. This article explains some of the less documented AXA features and options. Before reading this document, we recommend you become familiar with AXA basics and related technologies by reading the AXA Advanced Exchange document. ### AXA Subscriber Tools Farsight freely provides several end-user tools to stream AXA data. They are: sratool/radtool: The AXA Swiss Army knives, used to test / debug SRA or RAD connections and stream presentation format AXA watch hits (sratool) or AXA anomaly hits (radtool). sratunnel/radtunnel: The AXA workhorses, used to tunnel binary SIE data. axamd_client: Convenience tool that is a RESTful client and doubles as a Python extension library, used to stream AXA hits formatted as JSON blobs. Many of the features and options we'll be discussing are isomorphic amongst all of the tools and examples will be given where appropriate. ### AXA Watches In the AXA world, the way an end-user registers interest in an asset is through what's known as a watch. These are fundamental building blocks of AXA sessions. In order to get anything done, an end-user must specify one or more watches to inform the AXA server what she is interested in seeing. There are four different types of AXA watches, each is described below: - IP Watches: used to express interest in SIE messages containing a specified IP address or CIDR block. AXA understands both IPv4 and IPv6 address types. - DNS Watches: used to express interest in SIE messages containing a specified hostname, domain, or wildcard. - Channel Watches: used to express interest in SIE messages from an entire channel. This is useful to enable "the firehose" for a given channel and ask SRA to send everything from the specified channel rather than matching IP address information or DNS names. Only valid for SRA connections. - Error watches: used to express interest in SIE messages that cannot be decoded by the server. Only valid for SRA connections and intended only for debugging. Watches are referenced by tags. A tag is simply an arbitrary 16-bit integer label used to keep track of individual watches. When connected to an SRA server, tags must be unique. Each watch will use a different tag. When connected to a RAD server, watch tags can be reused. This is because anomaly modules are designed to be able to use multiple watches per invocation. This is accomplished by "bundling" together watches under a single tag. Depending on the tool, tagging may be done explicitly by the end-user or implicitly, by the tool. Examples follow. ### AXA Watch Examples: SRA The following examples show how to watch for SIE messages on channel 204 containing the IP address 10.0.0.1, are in the CIDR block 192.168.0/24, or are in the *.farsightsecurity.com domain. #### sratool The end-user specifies a unique tag per watch: ``` $ sratool sra> connect ... sra> 1 watch ip=10.0.0.1 sra> 2 watch ip=192.168.0/24 sra> 3 watch dns=*.farsightsecurity.com sra> channel 204 on ... ``` #### sratunnel With sratunnel, tags are generated internally so the end-user only needs to specify the -w option and the assets above: ``` $ sratunnel -w ip=10.0.0.1 -w ip=192.168.0/24 \ -w dns=\*.farsightsecurity.com -s ... -o ... -c 204 ``` #### axamd_client With axamd_client, tags are also internally generated so the invocation is similar to sratunnel's: ``` $ axamd_client --server https://axamd.sie-remote.net/ --watches ip=10.0.0.1 \ ip=192.168.0/24 dns=\*.farsightsecurity.com --channels 204 ``` ### AXA Watch Examples: RAD The end-user wishes to invoke the domain_sentry service (which is delivered through RAD) and ensure the DNS name www.farsightsecurity.com always has the A record 104.244.13.104 and the AAAA record 2620:11c:f004:0:0:0:0:104. #### radtool The end-user specifies the watches as in the sratool example above, but notice this time the watches are bundled using the same tag. This is to let the RAD server know that "all of these watches are to be used with the domain_sentry service invocation with the same tag". ``` $ radtool
...
rad> connect ... rad> 1 watch ip=104.244.13.104 rad> 1 watch ip=2620:11c:f004:0:0:0:0:104 rad> 1 watch dns=www.farsightsecurity.com rad> 1 anomaly domain_sentry types=a,aaaa ``` #### radtunnel With radtunnel, tags are generated internally so the end-user only needs to specify the -w option and the IP asset as above. ``` $ radtunnel -w ip=104.244.13.104 -w ip=2620:11c:f004:0:0:0:0:104 \ -w dns=www.farsightsecurity.com -s ... -o ... ``` #### axamd_client With axamd_client, as above, tags are also generated internally: ``` $ axamd_client --server https://axamd.sie-remote.net/ --watches ip=10.0.0.1 \ ip=192.168.0/24 dns=www.farsightsecurity.com \ --anomalies domain_sentry types=a,aaaa ``` ### Rate Limiting Rate limiting is used to limit the rate of incoming AXA messages as emitted from the server to the client. While it works for both SRA and RAD, it is primarily useful for high bitrate SIE channels such as the Passive DNS Channel Package channels and the DNS Errors channels. #### RATE LIMITING EXAMPLES The end-user wishes to watch for all SIE messages on channel 204 (a channel with an average bandwidth of 21Mbps) but only wishes to receive (at most) 10 messages per second. ##### sratool With sratool, the end-user has the option to receive rate limit loss reports. As such, she specifies a rate limit value of 10 and a 5 second interval between server rate limit reports: ``` $ sratool sra> connect ... sra> rate 10 5 * OPTION RATE LIMIT 10 per second; current value=0 5 seconds between reports sra> 1 watch ch=204 sra> channel 204 on ... * MISSED missed 0 input packets, dropped 0 for congestion, dropped 57911 for rate limit, filtered 72417 since 2016/10/11 14:25:32 ``` ##### sratunnel With sratunnel (or radtunnel), rate limiting is specified as a simple command line option: ``` $ sratunnel -r 10 -w ch=204 -s ... -o ... -c 204 ``` ##### axamd_client With axamd_client, like sratunnel, rate limiting is specified as a command line option: ``` $ axamd_client --server https://axamd.sie-remote.net/ --rate-limit 10 \ --watches ch=204 --channels 204 ``` Loss reports are not available with sratunnel, radtunnel, or axamd_client. ### Sampling Sampling is a way to statistically sample a percentage of messages from the server. For example, if a value of 50 is chosen, AXA will return a uniformly distributed random 50% sample of the messages its watches catch. Enabling sampling with any of the tools is quite straightforward. #### sratool ``` $ sratool
sra> connect ...
sra> sample 50 * OPTION sample 50.00% ... ``` #### sratunnel ``` $ sratunnel -m 50 -s ... ``` #### axamd_client ``` $ axamd_client --server https://axamd.sie-remote.net/ --sample-rate 50 ... ``` ### TCP Buffer Sizing AXA allows the end-user to get or set the TCP send buffer size used by the server. As described in Increasing TCP Window Size on Stack Overflow https://stackoverflow.com/questions/14381303/increasing-tcp-window-size, these buffers serve to accumulate outgoing data that the stack has not yet been able to put on the wire and data that has been received from the wire but not yet read by your application respectively. In simpler terms, if you know what you're doing, these options allow the end-user to adjust the size of the in-kernel TCP buffers presumably with the goal of optimizing network performance. As an aside, these options do not directly adjust the TCP window size (see *TCP Windows and Window Scaling* http://packetlife.net/blog/2010/aug/4/tcp-windows-and-window-scaling/ for more information). Note that internally TCP actually allocates twice the size of the buffer requested (using the extra space for administrative purposes and internal kernel stuff. This is why when you set a value, AXA returns to you a buffer size that is twice the size you requested. The minimum size is 1024 and the maximum is 262142. Only sratool and radtool support this option for TCP-based connections. #### sratool With sratool, getting and setting the window size is easy: ``` $ sratool sra> connect ... sra> window * OPTION bufsize=262142 sra> window 8192 * OPTION bufsize=16384 ``` ### Accounting AXA has a series of server-side packet counters used to track traffic totals. They are fully described in *Farsight's Advanced Exchange Access Internals: Understanding Accounting* https://www.farsightsecurity.com/txt-record/2015/09/24/mschiffm-axa-accounting/. #### sratool Requesting accounting totals with sratool: ``` $ sratool sra> connect ... sra> acct * OK ACCOUNTING total-filtered=113564 total-missed=0 total-collected=0 total-sent=1 total-ratelimited=0 total-congested=0 ``` #### sratunnel With sratunnel, accounting totals can be synchronously emitted to stdout every five seconds with the following: ``` $ sratunnel -d -A 5 -s ... ACCOUNTING total-filtered=4247 total-missed=0 total-collected=0 total-sent=1356 total-ratelimited=0 total-congested=817 ``` (The -d option enables debug mode which is required to see accounting output.) #### axamd_client With axamd_client, accounting totals are sent by default every 60 seconds. This value is tunable using the --report-interval option: ``` $ axamd_client --report-interval 10 ... {"tag":"*","op":"OK","orig_op":"ACCOUNTING","str":"total-filtered=2912 total-missed=0 total-collected=2912 total-sent=0 total-ratelimited=0 total-congested=0"} ``` ### Commands File Both sratool and radtool provide a convenience option that enables the end-user to specify commands to run prior to command invocation. This feature provides the end-user with a simple mechanism to script commonly used commands. A commands file is a text file with valid newline-delimited sratool or radtool commands. A common example is to script the connection procedure. In the following example, we setup a commands file that will connect to a RAD server and list the anomaly modules we are provisioned for: ``` $ cat > radtool.cmd connect tls:user@axa-server,1022 list anomalies ^D $ radtool -c radtool.cmd * HELLO radd version 1.4.1 axa-server AXA protocol 1 * domain_sentry [10/10] * brand_sentry [5/5] rad> ``` The output of the list anomalies command presents the end-user with the list of currently provisioned anomaly modules as well as the number of available domains and IP prefixes for domain_sentry and the number of available brands for brand_sentry. ### Additional Information - *SIE Channel Guide https://www.farsightsecurity.com/assets/media/download/fsi-sie-channel-guide.pdf - Security Information Exchange (SIE) protects from cybercrime https://www.farsightsecurity.com/solutions/security-information-exchange/ - Software - sratool/radtool/sratunnel/radtunnel https://github.com/farsightsec/axa - AXAmd client https://github.com/farsightsec/axamd_client ### About Farsight Security Farsight Security, Inc. is the world’s largest provider of historical and real-time DNS intelligence solutions. We enable security teams to qualify, enrich and correlate all sources of threat data and ultimately save time when it is most critical - during an attack or investigation. Our solutions provide enterprise, government and security industry personnel and platforms with unmatched global visibility, context and response. Farsight Security is headquartered in San Mateo, California, USA. Learn more about how we can empower your threat platform and security team with Farsight Security passive DNS solutions at www.farsightsecurity.com or follow us on Twitter: @FarsightSecInc. This document was originally published on Farsight's blog as *Advanced Exchange Access: The Missing Manual* by Mike Schiffman https://www.farsightsecurity.com/txt-record/2016/10/20/mschiffm-axa-missing-manual/ This article explains Farsight Security® Inc.'s (now a part of DomainTools) Advanced Exchange Access (AXA) accounting subsystem. This is the mechanism by which AXA tracks, logs, and communicates server-side packet information. This information is intended for users of SIE Remote Access (SRA). To get the most from this article, it is recommended that you be comfortable with the material in the following Farsight Security Blog articles: ### What is Accounting? Accounting is AXA's way of keeping track of traffic totals. Server-side, AXA maintains a series of per-client packet counters (a full list is below). The AXA protocol message AXA_P_OP_ACCT sent from client to server is used query this data. The command is available from sratool and radtool as acct. It is also available from sratunnel and radtunnel via the -A command line option. Accounting messages are also logged server-side. ### Accounting Glossary The AXA accounting counters are described below. - total-collected: On the RAD side, packets that have been successfully run through an anomaly detector. - total-congested: Packets that were unable to be forwarded to the client due to connection issues, although it could also be client load (i.e., the the connection isn't removing data from the server fast enough, or the client isn't reading and processing the data fast enough). Note that: high server load might increase total-missed, but it would generally decrease total-congested by throttling the input to the congested pipe / process. - total-filtered: This is an SRA input counter. It measures packets/nmsgs that have been read on the input side (usually directly from the SIE network) and will be submitted to the filtering logic (i.e.: an AXA watch). - total-missed: This is an SRA loss counter. It measures packets that the SRA server failed to receive on the input side either because the daemon was too busy or because they were lost during transit. This counter tracks NMSG- and pcap-based loss. - total-ratelimited: This measures packets that were dropped instead of being forwarded to the SRA or RAD client to comply with the rate limits specified for the client. - total-sent: This measures packets that have been sent to an SRA or RAD client. ### Accounting with sratool Let's have a look at a common sratool-based example. sratool is used to connect to an SRA server and a "fire hose" watch is set for our popular DNS Changes channel. ``` $ sratool sra> connect tls:mschiffm@sra-server,1021 * HELLO srad version 1.2.1 sra-server AXA protocol 1 sra> ch 214 on ; 1 wa ch=214 [watch hits omitted] ``` This is left to run for approximately three minutes of wall clock time. Next, the output is paused and the accounting command is run. ``` sra> pause * OK PAUSE output paused sra> acct * OK ACCOUNTING total-filtered=66360 total-missed=0 total-collected=0 total-sent=64912 total-ratelimited=0 total-congested=0 ``` - We see that 66,360 nmsgs were read from the SIE channel. For the watch we set this should actually be a good approximation of the overall packet rate. And indeed channel 214 has an average payload rate of approximately 340 payloads per second. Over this very small three minute (180 second) window we observed ~369 packets. - There were no missed packets. In this case, this is the everything's OK alarm. We expect there to be no packet loss on the SIE input side. If there were for such a low bandwidth channel, this would be indicative of server-side network fault or server overload. - We see that 64,912 packets were sent from the SRA server to our sratool client. This is good and what we expect. - Because there was no congestion or rate limiting, you may wonder why the number of sent packets is lower than the number of filtered packets. This is because when we stopped sending packets to the client, we didn't stop the server from filtering them. SRA was still reading from the input channel but wasn't forwarding any packets. ### Accounting with sratunnel Let's have a look at another example, this time using sratunnel. Using the timeout utility, sratunnel is invoked to run for three minutes. It connects to the same SRA server and sets a fire hose watch for the DNS Errors channel. Finally, the -A 180 -d option string instructs sratunnel to emit accounting statistics every 180 seconds and the results are written to a file. ``` $ timeout 180 sratunnel -s tls:mschiffm@sra-server,1021 -w "ch=220" -c 220 -A 180 -d -o nmsg:file:test-220.nmsg connecting to tls:mschiffm@sra-server,1021 ACCOUNTING total-filtered=4183437 total-missed=44 total-collected=0 total-sent=84371 total-ratelimited=0 total-congested=4096266 ``` - This time, 4,183,437 packets were read from the SIE channel. Using the same logic as above, we find that this equates to approximately 23,241 payloads per second which is commensurate to Channel 220's searing average per second rate of ~24,000 payloads (it is sourced from our Passive DNS feed which has a per second payload rate of ~120,000). - 44 packets were missed on the input side. This is a .001% rate of loss and well within acceptable limits. - Only 84,371 packets were sent to the client. To understand why, we look at the congestion number. - 4,096,266 packets were lost due to congestion. In this case, this high number is likely due to the fact that the client is located on the other side of the country from the SIE data center where the SRA server is located. Additionally, the client's downstream Internet connection is just plain too slow to keep up with the high rate of channel 220. - Farsight Network Message (NMSG) - Intro to nmsgtool - Headers and Encoding - Python Programming API - Loss Tracking Explained ## Tutorial 1: Watch Newly Observed Domains This tutorial assumes you've signed up to receive Farsight's Newly Observed Domains (NOD) datafeed to watch newly active domains and ensure your users don't visit any newly minted – often malicious – domains. It shows you how to examine the Newly Observed Domains (NOD) feed in real time. Commands and their output are listed with discussion below. ``` 1 $ sratool 2 > connect ssh:sra-service@sra-eft.sie-remote.net 3 * HELLO srad version 0.2.5 sra-eft AXA protocol 1 4 > 1 watch ch=211 5 1 OK WATCH started 6 > count 5 7 > channel 211 on 8 * OK CHANNEL ON/OFF channel ch211 on 9 1 ch211 SIE newdomain 10 flyinghorse-colorado.com/A: flyinghorse-colorado.com 11 1 ch211 SIE newdomain 12 treatmentforboils.com/NS: treatmentforboils.com 13 1 ch211 SIE newdomain 14 servicedeck.com/NS: servicedeck.com 15 1 ch211 SIE newdomain 16 www.markenmacher.eu/A: markenmacher.eu 17 1 ch211 SIE newdomain 18 recruitniks.com/NS: recruitniks.com 19 packet count limit exceeded 20 > count 21 packet printing stopped by count 1990 packets ago ``` Lines 1-3: Invoke sratool, and use the connect command to establish a connection to the SRA server. The connection is managed via SSH, meaning all of the benefits conferred by the SSH protocol are available to sratool. Upon success, the client emits the hello string from the AXA_P_OP_HELLO message which was sent by the server and contains the server's software version, name, and AXA protocol verison. Lines 4-5: Inform the server we want to watch SIE channel 211 traffic (this is the NOD channel). The server responds with the current watch status. The watch is the most fundamental sratool command. This is how sratool "signs up" to receive data from the SRA server. As its name implies, watch sets up a watch which is a low-level primitive that tells the SRA server that the client is interested in nmsg messages or IP packets that meet one of the following criteria: * is to, from, or contains the specified address * contains the specified domain name * arrived on the specified SIE channel * are SIE messages that could not be decoded A watch is given a tag that is an integer label used to refer to the watch. An SRA server connection or session can have zero or more watches at a time and the user can add or delete watches as needed. Note that sratool allows only a single SRA connection at a time. Line 6: Using the count command, we inform sratool we only want to see 5 packets. After this number is met, sratool will stop emitting packets to the screen (though traffic may still be flowing from server to client). Lines 7-8: With the channel command, enable channel 211 (NOD). The current channel status is printed. Another fundamental command to sratool is channel. Issued alone on the command-line, it will emit the entire list of available SIE channels for which the user is provisioned. Lines 9-19: sratool emits 5 NOD packets as it receives then from the server. Once the packet count limit is reached, emission stops. Lines 20-21: Issuing the count command with no arguments prints the current count status. In this case, we find 1990 NOD packets have been streamed to the client, but since we exceeded our limit, they were not emitted to the screen. ## Tutorial 2: Counts And Limits Continuing in the session above, let's tweak a few knobs and press a few buttons. ``` 22 > list watches 23 1 ch=ch211 24 > 1 delete 25 1 OK STOP watch deleted 26 > rate 27 RATE LIMITS 28 unlimited per second; current value=307 29 10 seconds between reports 30 > rate 1 31 RATE LIMITS 32 1 per second; current value=2 33 10 seconds between reports ``` Lines 22-23: The list watches command prints all of the active watches. We've still got one going, we're just not emitting any packets to the screen. Lines 24-25: We delete the watch by referencing its tag with the delete command. Lines 26-29: Another handy command, rate allows us to query the rate limiter and control it. Currently, there is no rate limiting in play – packets will be emitted as quickly as they appear. For lower bandwidth channels, like NOD, this is might not be a problem. For the DNSDB channels, which are much higher bandwidth, we'll want to limit the rate at which those packets are sent by the server to sratool. Lines 30-33: Using the rate command, we set a rate limit of 1 packet per second. This will come in handy in the last part of the tutorial where we'll examine DNSDB. ## Tutorial 3: Watch For A Specific Domain In Farsight's Passive Dns Feed As a bonus, let's peek at SIE channel 202 traffic, Farsight's raw passive DNS feed. ``` 34 > 2 watch dns=*.github.com 35 2 OK WATCH started 36 > channel 202 on 37 * OK CHANNEL ON/OFF channel ch202 on 38 2 ch202 base dnsqr response UDP_QUERY_RESPONSE 39 204.13.250.16.53 > 68.105.29.142.17296 IP TTL=58 UDP 86 bytes 40 DNS: raw.github.com IN A qr aa NOERROR 1 ans, 0 auth, 0 add RRs 41 2 ch202 base dnsqr response UDP_QUERY_RESPONSE 42 208.78.71.16.53 > 208.106.17.39.64372 IP TTL=56 UDP 153 bytes 43 DNS: api.github.com IN A qr aa cd NOERROR 1 ans, 4 auth, 0 add RRs 44 2 ch202 base dnsqr response UDP_QUERY_RESPONSE 45 204.13.250.16.53 > 68.105.28.174.52707 IP TTL=58 UDP 89 bytes 46 DNS: malsup.github.com IN A qr aa NOERROR 1 ans, 0 auth, 0 add RRs 47 * MISSED 48 lost 0 input packets, dropped 0 for congestion, 49 121 for per sec limit 50 since 2014/12/08 17:29:38 51 2 ch202 base dnsqr response UDP_QUERY_RESPONSE 52 204.13.250.16.53 > 68.105.28.174.47116 IP TTL=58 UDP 149 bytes 53 DNS: github.com IN A qr aa NOERROR 1 ans, 4 auth, 0 add RRs ``` Lines 34-35: We set another watch, this time we want to watch for the wild card domain "\.github.com". Anything matching this domain will be emitted, such as www.github.com and github.com* itself. Lines 36-37: We turn on channel 202, Farsight's raw passive DNS channel. Lines 38-53: All domains matching the watch are emitted. ## Sratunnel At the end of our last tutorial, you probably wondered aloud: "…this is cute but I need real-time bulk transfer of SIE data back to my network. Does this technology even exist in the modern world?" Yes, yes it does. sratunnel is the workhorse of the AXA family. It is used to transfer SIE data from the remote server to the local network. It is what Farsight uses for production deployment of SIE data to the customer. sratunnel can be thought of as a fast, efficient, and smart conduit for SIE data. Data goes in one end and sratunnel has a variety of nozzles the user can custom fit on the other end to emit the data into different output formats, including: * NMSGs to a UDP port * NMSGs to a TCP port * NMSGs to a file * pcap to a file * pcap to a network interface ### Tunnel Newly Observed Domains Now that you know how to use the Newly Observed Domains (NOD) datafeed to watch newly active domains and ensure your users don't visit any newly minted – often malicious – domains, we are going to show you how to to tunnel the data to your local network for bulk analysis. #### NMSG Primer To consume the data in this tutorial, you'll need another Farsight implement called nmsgtool. It is a deeply useful all-purpose tool for working with NMSGs (network messages). NMSG is the format Farsight uses to type, structure and package arbirtary data for transit. Much of Farsight's data is packaged and delivered as NMSG. A detailed discussion of the NMSG suite will be covered in future blog series. For now it's just important to understand that you can work with NMSGs using nmsgtool. Onward\! This tutorial will show you, gallant Farsight datafeed customer, how to plumb the Newly Observed Domains (NOD) feed from SIE to your local network, in real time. Commands and their output are listed with discussion below. ``` 1 $ sratunnel -s 'ssh:sra-service@sra-eft.sie-remote.net' \ 2 > -c 211 \ 3 > -w ch=211 \ 4 > -o nmsg:udp:127.0.0.1,8430 ``` Line 1: Invoke sratunnel. The \-s option instructs the tool where and how to connect. The option string should look familiar, it's the same one used with sratool with the same intent and results. Securely connect via SSH as sra-service to sra-eft.sie-remote.net. Line 2: The \-c option sets the channel you want to stream. We want NOD which is channel 211\. Line 3: The \-w option sets the watch. You learned last week that a watch is how to inform the tool what to look for. In this case, everything on channel 211\. Line 4: Finally, we specify \-o, which tells sratunnel where to put the data it streams. In the case above, we've snapped on a shiney new "NMSGs to localhost on port 8430" nozzle and that's where we'll find our output. Well done\! You've plumbed your first SRA session. Data is aflowin'. Let's build a small corpus and have a look… ``` 5 $ nmsgtool -l 127.0.0.1/8430 \ 6 > -c 20000 \ 7 > -o channel-211.txt 8 $ head -8 channel-211.txt 9 [98] [2014-12-16 23:31:06.438992023] [2:5 SIE newdomain] [a1ba02cf] [] [] 10 domain: befrenshee.com. 11 time_seen: 2014-12-16 23:28:19 12 rrname: befrenshee.com. 13 rrclass: IN (1) 14 rrtype: NS (2) 15 rdata: ns67.domaincontrol.com. 16 rdata: ns68.domaincontrol.com. 17 $ grep ^domain: channel-211.txt | awk '{print $2}' > NOD.txt ``` Line 5: We use nmsgtool to connect to the loopback address on port 8430. Line 6: The \-c option specifies a maximum count of payloads to capture. Line 7: The \-o option tells nmsgtool to write presentation output to a file. Line 8: Let's examine one entry… Line 9: Each NMSG datagram contains a fixed-length header containing the message size, a UTC timestamp, the message type, a 32-bit source identifier and optional SIE operator and group codes (both empty in this case). Line 10: The fresh young domain, hot off the press\! Line 11: How fresh? At the time of this writing, that timestamp of when the domain was observed was just over two minutes old. Lines 12-16: The DNS meta-data associated with the domain. Line 17: You're free to manipulate the data however you see fit. As per the above, you can take the list of 20,000 young domains and feed the file into the young domain crunching automation of your choice… :) This document explains the NMSG protocol and provides an introduction to some example tools that can be used to acquire and process NMSG packets. Common use cases for the tools will also be reviewed. NMSG is an adaptable container format that allows for consistent or variable message types. NMSG container data may be streamed to a file or transmitted as UDP datagrams. NMSG containers can contain multiple NMSG messages or a fragment of a message too large to fit in a single container. The data in an NMSG container can also be compressed. Additional capabilities include sequencing and rate-limiting. This includes: * Adaptable: NMSG functionality can be modified to meet the needs of new data formats using its adaptable message module interface. As new data feeds are added to SIE, related message modules can be developed for nmsg that do not necessitate library compilation or API changes. * Container-Based: NMSG data is serialized inside containers that can contain one payload, many payloads, or the fragment of payload too large to fit in a single container. * Wire Format: NMSG specifies a wire-format optimized for transmitting UDP datagrams over jumbo Ethernet. * File Format: NMSG also specifies an on-disk file-format for storage of NMSG data. * Container Data: A core principle of NMSG is data compatibility. Some of the data Farsight consumes, transmits, and stores is inadequately represented in its native format (such as frames, packets, datagrams, segments, or other data formats). As such, NMSG was designed to be ignorant about the data it transports. NMSG offloads the details of encoding to external message modules and can also work with opaque containers. * Dynamic Message Types: NMSG provides an adaptable message module interface that can be modified at run-time for message types it understands. This ensures the library is generic and offloads the more unique message handling to external modules that can be loaded as-needed. * Compression: NMSG supports per-payload compression. This is implemented in nmsg using zlib. * Fragmentation: Payloads that are too large to fit in a single container for the underlying transport, NMSG provides a fragmentation capability that is seamless to the user or application programmer. * Sequencing: NMSG can optionally be configured to assign sequentially increasing numbers to the containers it emits. This can be used by the receiving end to detect potential container loss. * Rate-Limiting: NMSG can optionally be configured to rate-limit its emission of containers to ensure receivers on slower networks are not overwhelmed. NMSG is available for the application programmer as a C library called libnmsg. The library offers a complete API for the programmer to build NMSG-capable applications and configure, tune, and/or tweak its many options and features. The reference implementation of libnmsg is nmsgtool, which is a thin wrapper around the libnmsg C library. nmsgtool provides comprehensive NMSG functionality at the command line interface (CLI) for Unix-like systems. To ensure development is easy in Python and Perl, modules also exist for each programming language. Note: Using NMSG requires that Farsight Security has provisioned at least one of the following Access Methods to acquire data from SIE: 1. SIE Remote Access (SRA) (Also known as Advanced Exchange Access (AXA)) 2. SIE Direct Connect 3. SIE Blade Server leased from Farsight 4. SIE Batch ## Introduction nmsgtool is a CLI tool for the libnmsg library and is a wrapper around libnmsg's input/output (I/O) engine. libnmsg controls the transmission, storage, creation, and conversion of NMSG payloads. nmsgtool's primary purpose is for prototyping and debugging access to NSMG container format data. While it provides comprehensive NMSG functionality and can acquire and store data from SIE, it also lacks some advanced features of libnmsg. ## NMSG Inputs and Outputs The nmsgtool program is a single tool for acquiring a variety of different inputs, like data streams from the network, capturing data from network interfaces, reading data from files, or even standard input and making NMSG payloads available to one or more outputs. The outputs are files in binary or human-readable (ASCII presentation) form, or binary payloads to network sockets for transport. Without having to create a program for each function, nmsgtool handles all types of data processing which includes serialization, fragmentation, compression, striping or mirroring, rolling file outputs, and executing data processing programs on file outputs. The nmsgtool program is a single tool for acquiring a variety of different inputs, like data streams from the network, capturing data from network interfaces, reading data from files, or even standard input and making NMSG payloads available to one or more outputs. The outputs are files in binary or human-readable (ASCII presentation) form, or binary payloads to network sockets for transport. Without having to create a program for each function, nmsgtool handles all types of data processing which includes serialization, fragmentation, compression, striping or mirroring, rolling file outputs, and executing data processing programs on file outputs. nmsgtool inputs can take the following forms: * File that contains binary NMSG data, which could have been created nmsgtool * A socket that is configured to transport binary NMSG data * IP packets in packet capture library (libpcap) file (also known as "PCAP") * IP packets acquired from a network interface * A file containing ASCII presentation data nmsgtool outputs can take the following forms: * Binary NMSG data stored in a file * Binary NMSG data sent to a network socket * ASCII presentation data in a file and including standard output (stdout) * You can specify more than one of each ### Features nmsgtool is multithreaded libnmsg has a feature that enables you to create user-specified Dynamic Shared Objects (DSOs) for inline filtering of network messages, which may be quicker than post filtering the data. nmsgtool can load these filtering DSOs using the \-F or \--filter CLI argument. Documentation about this feature is available at: https://github.com/farsightsec/nmsg/blob/master/nmsg/io.h\#L139-L206 https://github.com/farsightsec/nmsg/blob/master/fltmod/nmsg_flt_sample.c ## Limitations nmsgtool does not provide easy access to seqsrc NMSG container data. ## Considerations Performance and potential container loss may be dependent on, but is not limited to, the following. All may be potential reasons for nmsgtool to throttle and drop containers. * Transmitting NMSG data over the Internet * Network bandwidth * Congestion between the transmitting instance of nmsgtool and program on the system receiving the NMSG data * Host system or receiving system * Disk * Memory * CPU * Network interface ## Examples The following examples and use-cases demonstrate the functionality of nmsgtool. ### Display Data when Physically Connected to SIE A common nmsgtool use-case for SIE customers is to acquire real-time data from an SIE channel and display the NMSG data on the screen. The following invocation of nmsgtool reads one (1) NMSG payload [-c 1] from SIE Channel 212, Newly Observed Domains (NOD), [-C ch212] and emits it to stdout as ASCII presentation data [-o -]. ``` $ nmsgtool -C ch212 -c 1 -o - [72] [2015-02-03 13:35:29.678474903] [2:5 SIE newdomain] [a1ba02cf] [] [] domain: s47rbh.xyz. time_seen: 2015-02-03 13:33:20 rrname: s47rbh.xyz. rrclass: IN (1) rrtype: NS (2) rdata: ns1.51dns.com. rdata: ns2.51dns.com. ``` Note: If no outputs are specified, ASCII presentation to stdout [-o -] is the default behavior of nmsgtool. In examples that follow, [-o -] will be omitted. nmsgtool uses the nmsgtool.chalias configuration file to determine the proper network interface to collect data from. This configuration file contains the SIE channel number, IP address, and UDP port mappings. When a channel number is specified on the command line, nmsgtool looks it up in the nmsgtool.chalias file and listens on the specified network interface. The header from an NMSG datagram is the first line displayed for an NMSG message. Breaking this down, the header fields for SIE Channel 212 (Newly Observed Domains) follow: * \[72\]: Message size in bytes * \[2015-02-03 13:35:29.678474903\]: UTC timestamp with nanosecond resolution * \[2:5 SIE newdomain\]: Vendor and message ID, vendor and message type * \[a1ba02cf\]: Optional source identifier * \[\]: Optional operator code * \[\]: Optional group code The message payload contains key-value pairs that conform to a schema specified by the vendor for the message type. In the preceding example, the vendor is SIE and the message type is newdomain. nmsgtool includes dynamically loadable modules that enable it to display the data as you see in the preceding example and also enable NMSG-based programs or scripts to load the key-value pairs into structures. These concepts will be explained in more detail in future NMSG articles. ### Acquire Data from SIE and Write to Binary NMSG Files Another use-case for SIE customers is to acquire real-time data from an SIE channel and save it to the local filesystem in a binary NMSG file for analysis at a later time/date. The following invocation of nmsgtool acquires 100,000 NMSG payloads [-c 1000000] from SIE Channel 208, DNSDB Verified Data (deduplicated, verified, and prior to filtering), [-C ch208] and writes them to a binary NMSG file [-w ch208.nmsg]. The result is a file approximately 15.25 megabytes in size. ``` $ nmsgtool -C ch208 -c 100000 -w ch208.nmsg $ stat -c "%n %s" ch208.nmsg ch208.nmsg 15246581 ``` The binary file can then be read by nmsgtool using [-r ch208.nmsg] and it will display one (1) NMSG payload [-c 1] containing a dnsdedupe message. The message is emitted to stdout in ASCII presentation data. ``` $ nmsgtool -r ch208.nmsg -c 1 [72] [2015-02-01 00:07:53.596907788] [2:1 SIE dnsdedupe] [a1ba02cf] [] [] type: EXPIRATION count: 2 time_first: 2015-01-31 07:29:37 time_last: 2015-01-31 07:29:37 bailiwick: rrname: rrclass: IN (1) rrtype: A (1) rrttl: 43200 rdata: ``` ### NMSG Payload Compression nmsgtool can optionally compress the payload of each NMSG container using zlib compression (the same algorithm used by gzip), either to a file or for NMSG data transmitted across the network. Note: Compression is performed on each payload in an NMSG container, not across the entire file. To demonstrate the on-disk storage benefit that compression provides, we can compress the data acquired in the previous example. The following invocation reads the binary file [-r ch208.nmsg] from the previous example, compresses each payload using [-z], and then writes the output to a new file [-w ch208z.nmsg]. The result is a file approximately 6.43 megabytes in size, which is a 58% decrease in file size. ``` $ stat -c "%n %s" ch208.nmsg ch208.nmsg 15246581 $ nmsgtool -r ch208.nmsg -w ch208z.nmsg -z $ stat -c "%n %s" ch208z.nmsg ch208z.nmsg 6428829 ``` ### Kicker Scripts and Rotating Output Files Another useful capability that nmsgtool provides is the ability to perform automatic file rotation (rolling) based on a duration of time or payload count. Additionally, the user can specify a kicker script or command to run on output files. An example shell script follows. The script, count.sh, counts the number of dnsdedupe payloads from a binary NMSG file using grep -c. ``` #!/bin/sh echo "$1: " `nmsgtool -r "$1" | grep -c "\[2:1 SIE dnsdedupe\]"` ``` The following invocation of nmsgtool acquires data from SIE Channel 208, DNSDB Verified Data (deduplicated, verified, and prior to filtering), [-C ch208] and writes compressed payloads [-z] to a binary NMSG file [-w ch208] that is prefixed with ch208. Every two (2) seconds [-t 2] the file is closed, rotated, and the kicker script [-k count.sh] is run on the output file. The output from each count.sh invocation is the filename followed by the number of NMSG payloads that each file contains. The count.sh script is invoked in the following example: ``` $ nmsgtool -C ch208 -w ch208 -t 2 -z -k count.sh ./ch208.20150202.0110.1422839406.364843292.nmsg: 49404 ./ch208.20150202.0110.1422839408.013136741.nmsg: 80446 ./ch208.20150202.0110.1422839410.024261700.nmsg: 91067 ./ch208.20150202.0110.1422839412.024284315.nmsg: 86070 ./ch208.20150202.0110.1422839414.033887391.nmsg: 85490 ./ch208.20150202.0110.1422839416.014162500.nmsg: 90793 ``` Note: The -k cmd or --kicker cmd arguments make [-t] and [-c] continuous. In this mode, output file names are suffixed with a timestamp and nmsgtool runs continuously, rotating output files as payload counts are reached or a specified duration of time expires. ### Transport NMSG Data Across the Network nmsgtool can transport NMSG payloads across an IP network to either a unicast or broadcast address IPv4 address or a unicast IPv6 address. For this example, two (2) nmsgtool sessions on run on two (2) separate systems. On the receiving system, we run nmsgtool as follows: System Receiving NMSG Payloads The following invocation of nmsgtool listens on a network socket for NMSG payloads sent to IPv4 address 10.0.1.52 on UDP port 9430 [-l 10.0.1.52/9430]. When NMSG payloads are observed, they will be displayed as ASCII presentation data to stdout. ``` $ nmsgtool -l 10.0.1.52/9430 ``` System Sending NMSG Payloads The following invocation of nmsgtool reads two (2) NMSG payloads [-c 2] from the binary NMSG file created in the previous example [-r ch208...]. The NMSG payloads are then sent to IPv4 unicast address 10.0.1.52 on UDP port 9430 [-s 10.0.1.52/9430]. On the sending system, we run nmsgtool as follows: ``` $ nmsgtool -r ch208.20150202.0110.1422839406.364843292.nmsg -c 2 -s 10.0.1.52/9430 ``` System Receiving and Displaying NMSG Payloads On the receiving system, the output from one (1) of the two (2) dnsqr NMSG payloads is displayed to stdout in ASCII presentation format. The other is redacted and cropped for publication. ``` [293] [2015-02-02 01:08:21.902736000] [1:9 base dnsqr] [e9b019b8] [] [] type: UDP_QUERY_RESPONSE query_ip: response_ip: proto: UDP (17) query_port: 31211 response_port: 53 id: 7644 qname: qclass: IN (1) qtype: AAAA (28) rcode: NOERROR (0) delay: 0.182413 udp_checksum: ABSENT [...] [352] [2015-02-02 01:08:22.095911000] [1:9 base dnsqr] [e9b019b8] [] [] [...] ``` Note: The system sending NMSG payloads has the option to tune network performance, including setting the NMSG container maximum transmission unit size (note this is distinct from IP MTU buffering, and rate limiting). For more information: * https://en.wikipedia.org/wiki/Maximum_transmission_unit\#IP_.28Internet_protocol.29 ### Payload Striping vs Mirroring When multiple outputs are specified, nmsgtool defaults to striping https://en.wikipedia.org/wiki/Data_striping payloads across each output. However, nmsgtool can also be configured to mirror https://en.wikipedia.org/wiki/Disk_mirroring payloads to each output. The following invocation acquires 100 NMSG payloads [-c 100] from SIE Channel 211, Newly Active Domains, and mirrors [--mirror] the data to two (2) outputs: * \[-o \-\]: ASCII presentation format is sent to stdout and displayed on the screen * \[-s 10.0.1.52/9430\]: Binary NMSG payloads sent to a network socket, IPv4 destination address 10.0.1.52 on UDP port 9430 ``` $ nmsgtool -C ch211 -c 100 -o - -s 10.0.1.52/9430 --mirror [94] [2015-02-03 08:50:19.277158975] [2:5 SIE newdomain] [a1ba02cf] [] [] [...] ``` ### Acquire Data from a Network Interface or Read a PCAP File with BPF filtering Perhaps you would like to acquire and create your own stream of NMSG payloads sourced from live network traffic. But more specifically, you only want to observe DNS traffic. When running nmsgtool, you can tell it to acquire IP datagrams directly from a network interface or it can read a PCAP file. Additionally, an optional user-defined Berkeley_Packet_Filter (BPF) can be specified to filter packets. When acquiring data from a network interface or reading a PCAP file, nmsgtool requires the user to specify the vendor and message type so it knows how to properly encode each NMSG payload. The following invocation of nmsgtool acquires data from a network interface [-i eth1] and filters packets for UDP port 53 [-b "udp 53"]. The NMSG payloads are encoded as base/dnsqr [-V base] and [-T dnsqr] and emitted in ASCII presentation format to stdout. ``` $ nmsgtool -i eth1 -V base -T dnsqr -b "udp 53" [220] [2010-05-09 05:08:54.951124000] [1:9 base dnsqr] [00000000] [] [] [...] ``` Reading from a PCAP file is syntactically similar, just substitute [-i eth1] with [-p example.pcap]. ## Python module: pynmsg Farsight is the maintainer of a Python module named pynmsg, a Python 2.7 extension module implemented in Cython for the nmsg C library. See Farsight's Network Message, Volume 5: The Python Programming API for more information. ### Introduction ### Installation * To build from source, install Python 2.7 and required dependencies. ``` $ sudo apt-get install python2.7 python-pip cython ``` Install pynmsg from source code. ``` $ wget "https://github.com/farsightsec/pynmsg/archive/tags/v0.4.0.tar.gz" -O "pynmsg-tags-v0.4.0.tar.gz" $ tar xzvf "pynmsg-tags-v0.4.0.tar.gz" $ cd pynmsg-tags-v0.4.0/ $ python setup.py build $ sudo python setup.py install $ cd .. ``` Install pywdns from source code. ``` $ wget "https://github.com/farsightsec/pywdns/archive/tags/v0.10.0.tar.gz" -O "pywdns-tags-v0.10.0.tar.gz" $ tar xzvf "pywdns-tags-v0.10.0.tar.gz" $ cd pywdns-tags-v0.10.0/ $ python setup.py build $ sudo python setup.py install $ cd .. ``` ## Perl module: Net::Nmsg ### Perl ### Introduction Net::Nmsg is a perl binding to libnmsg, the reference implementation of the NMSG binary structured message interchange format. See https://metacpan.org/release/Net-Nmsg for additional information. ### Installation For Debian/Ubuntu, Farsight maintains a package called libnet-nmsg-perl. ``` $ apt-get install libnet-nmsg-perl ``` For FreeBSD, Net::Nmsg is available as an official binary package. ``` $ pkg install p5-Net-Nmsg ``` For other operating systems it is possible to install Net:Nmsg using CPAN. Prior to installing, it requires libpcap development header files or libpcap to be installed from source. ``` $ perl -MCPAN -e shell cpan> install Bundle::CPAN cpan> install Net::Nmsg ``` Some software package dependencies may ask installation questions, like hitting [enter] when asked for a mathematic expression, or entering some minimal system information when configuring IO::Socket. ## Contacting Support To request a demonstration of DNSDB or to inquire about a trial API key please contact the DomainTools sales team. ## Appendix A - Installing NMSG Software ### Compile and install from source code Source code tarballs for the software packages below are also available from . ##### Example installation instructions from source on Ubuntu 16.04 #### Install wdns, nmsg, and sie-nmsg * Install dependencies from Ubuntu repositories ``` $ sudo apt-get install build-essential pkg-config libpcap0.8-dev libprotobuf-c-dev protobuf-c-compiler libxs-dev libyajl-dev zlib1g-dev ``` * Install wdns from source code. ``` $ wget "https://github.com/farsightsec/wdns/archive/tags/v0.10.0.tar.gz" -O "wdns-tags-v0.10.0.tar.gz" $ tar xzvf "wdns-tags-v0.10.0.tar.gz" $ cd wdns-tags-v0.10.0/ $ ./configure $ make $ sudo make install $ cd .. ``` * Install nmsg from source code. ``` $ wget "https://github.com/farsightsec/nmsg/archive/tags/v0.15.1.tar.gz" -O "nmsg-tags-v0.15.1.tar.gz" $ tar xzvf "nmsg-tags-v0.15.1.tar.gz" $ cd nmsg-tags-v0.15.1/ $ ./configure $ make $ sudo make install $ cd .. ``` * Install sie-nmsg from source code. ``` $ wget "https://github.com/farsightsec/sie-nmsg/archive/tags/v1.2.1.tar.gz" -O "sie-nmsg-tags-v1.2.1.tar.gz" $ tar xzvf "sie-nmsg-tags-v1.2.1.tar.gz" $ cd sie-nmsg-tags-v1.2.1/ $ ./configure $ make $ sudo make install $ cd .. ``` * Run ldconfig to update the shared library cache. ``` $ sudo ldconfig ```