Security Information Exchange (SIE): User Guide¶

Introduction¶

SIE gives you real-time access to data from our global sensor network. That data includes over half a million passive DNS (pDNS) observations per second, as well as other key security data points. DomainTools processes this data into usable formats, stream it over real-time channels, and provides you with tools for your use case.

Access SIE Channels through the following mechanisms:

SIE Batch is a web interface and REST API with access to the last 12-18 hours of data from your subscribed feeds.
SIE Remote Access creates a tunnel from SIE to the analyst’s system, and supports a REST API (see AXAMD, below).
SIE Direct Connect is a leased blade server with pre-installed SIE tools that provides direct access to the SIE network.

Data formats are detailed in the File and Data Formats section. Feeds are typically available in NMSG and JSON formats, depending on the access method (some JSON availability is through conversion with nmsgtool).

NMSG is a streaming binary format, based on Google Protocol Buffers, that handles high volume, real-time traffic. SIE delivers most channels with NMSG. The included nmsgtool (detailed below) can convert NMSG to JSON.
JSON, typically in JSONL/NDJSON format, is provided directly by certain feeds.
Packet Capture (libpcap) is used for channels that preserve packet structure.

Access¶

Access to SIE is provisioned by DomainTools Enterprise Support at enterprisesupport@domaintools.com. Batch and SIE Remote Access is provisioned with an API key. SIE Direct Connect requires the customer's public key and originating IP address(es) for SSH access.

System Requirements¶

A server configured by DomainTools for SIE Direct Connect has the following operating system (OS) and hardware specifications.

Component	Specification
Compute	One (1) x Intel Quad Core Xeon E3 3.40Ghz
Memory	16GB RAM
Storage	Two (2) x 2TB 7200RPM drives (configured for RAID 1, 2TB available for customer use)
Network (Internet connection)	100Mbps
Network (SIE connection)	10gigabit

These hardware specifications are adequate for acquiring, processing, compressing, and buffering data from high throughput SIE channels. Intensive processing and analysis on data from SIE channels with the highest data volumes may require additional memory (RAM).

Channels Overview¶

The following table provides the channel, name, description, bitrate, payload rate, and notes on each channel. All channels are available in NMSG and JSON formats unless otherwise noted.

Number	Name	Description	Bitrate	Payloads	Notes
24	Spam-Full	Spam-Full shares full copies of spam emails sent to email spamtrap systems. The honeypots have been configured to collect and store all email messages for analysis and they use email addresses that have never been used to receive email, or are no longer in use and should not be receiving email.	2.4kbps (104kbps)	2/sec (14/sec)
25	Spam-Select	Spam-Select shares key fields from spam emails sent to email honeypot/spamtrap systems. The spamtraps have been configured to collect and store all email messages for analysis and they use email addresses that have never been used to receive email, or are no longer in use and should not be receiving email.	2kbps (105kbps)	1/sec (3/sec)
27	Phishing URLs	Includes URLs, the brand target, and other information related to phishing campaigns.	\<1kbps (2kbps)	\<1/sec (2/sec)
42	IDS and Firewall Log Data	Information about network traffic that is blocked by Intrusion Detection Systems (IDS) and firewall devices. The data is anonymized and batched every five minutes.	7mbps (35mbps)	390/sec (2kbps)	Not available over AXAMD.
80	Conficker Sinkhole	HTTP connection information from Conficker-infected clients to ‘sinkholed’ command and control servers.	385kbps (520kbps)	210/sec (280/sec)	Not available over AXAMD.
115	DDos Events	Evidence of DDoS and DRDoS (Distributed Reflection Denial of Service) attacks based on analysis of data from captured network packets destined from unused network space.	\<1kbps (\<1kbps)	\<1/sec (1.5/sec)
207	DNSDB De-Duplicated Data	Passive DNS observations after the deduplication processing phase and immediately prior to the verification phase.	144mbps (170mbps)		Not available over Remote Access or AXAMD; Batch files available as NMSG.
208	DNSDB Verified Data	Deduplicated passive DNS data with low value entries removed, and verified for balliwick-appropriate data (misleading resource record information is removed).	88mbps (117mbps)		Not available over Remote Access or AXAMD; Batch files available as NMSG.
211	Newly Active Domains (NAD)	Previously observed domains (via Channel 204) becoming active after 10 or more days of inactivity.	62kbps (900kbps)	53/sec (760/sec)
212	Newly Observed Domains (NOD)	Domains not previously observed by SIE. It can also be accessed by: NOD RPZ, NOD rbldnsd; See also: NOD FAQ	3kbps (18kbps)	2/sec (13/sec)
213	Newly Observed Hostnames (NOH)	Fully Qualified Domain Names (FQDNs) not previously seen when compared to the DNSDB historical database.	Under review	TBA
214	DNS Changes	Domains, hostnames, or record data that is unknown to DNSDB, either because the data is for a new domain or hostname or because the record data for a domain or hostname has changed. These changes may include new RR types, new or changed IP addresses, or a change in the authoritative name servers for a domain. More information: The DNS Changes Channel	4.9mbps (8mbps)	TBA	Not available over AXAMD.
220	DNS Errors	Non-deduplicated DNS responses that returned a non-zero Response Code (RCODE). This includes NXDomain, ServFail, Refused, FormErr, NotImp, and other RCODEs.	570mbps (860mbps)	TBA	Only available over Direct Connect`.`
221	NX Domains	Derived from Channel 220 DNS Errors, but only includes NXDomain RCODEs. More information: Introducing NXD	52mbps(73mbps)	TBA	Not available through AXAMD.
255	Heartbeat (Testing)	For monitoring purposes.	1kbps (1kbps)	TBA

Installation¶

Debian GNU/Linux 11 Bullseye¶

Debian GNU/Linux is supported through an apt package repository. Ubuntu and other Debian-based systems may be able to use this repository as well.

Enable the bullseye-farsightsec package repository by copying the GPG key in place and adding a sources.list entry:

$ wget -O debian-farsightsec.gpg https://dl.farsightsecurity.com/debian/debian-farsightsec.gpg
$ echo deb [arch=amd64] http://dl.farsightsecurity.com/debian bullseye-farsightsec main | sudo tee -a /etc/apt/sources.list.d/debian-farsightsec.list
$ sudo cp debian-farsightsec.gpg /etc/apt/trusted.gpg.d/
$ sudo apt-get update

Package Signing Key¶

Our open source Debian packages are hosted at https://dl.farsightsecurity.com/debian/. The GPG keyid is 9B4F9753 (https://dl.farsightsecurity.com/debian/debian-farsightsec.gpg). Its GPG key details are:

pub rsa4096 2025-05-05 [SC] [expires: 2030-05-04]
    6BE3C9395BF00DFF2B1A7A45AEEA86469B4F9753
uid DomainTools, LLC (Public Package Signing Key)

The previous package signing key was A511AE06.

For "apt-get update" use, this key file can be copied to the /etc/apt/sources.list.d/ directory or imported using apt-key add.

Package Installation¶

Install the core nmsg packages and SIE plugins:

$ sudo apt-get install nmsgtool nmsg-msg-module-sie

Install packages for developing C application:

$ sudo apt-get install libnmsg-dev nmsg-msg-module-sie-dev libwdns-dev

Install packages for developing Python applications:

$ sudo apt-get install python-nmsg python-wdns

Install packages for SIE remote access (sratool, sratunnel):

$ sudo apt-get install axa-tools

From Source on Rocky 9 (RHEL-compatible)¶

Rocky Linux is compatible with Red Hat Enterprise Linux (RHEL).

This installation was tested on Rocky Linux 9.2 with default install options for the Server with GUI. These instructions should work for all RHEL-compatible distributions which have access to the EPEL and CRB repositories or their equivalents. It results in these changes:

A local user was created with sudo rights
VM name was set to rocky9.local
A password was set for root to enable su

All updates and patches were applied:

$ sudo dnf update

Setup, Dependencies, and Configuration¶

The following dependencies were installed. Note that both the “Extra Packages for Enterprise Linux: (a.k.a. EPEL) and "Code Ready Builder" (a.k.a crb) repositories containing extra libraries and developer tools are required to compile required packages.

$ sudo dnf install epel-release
$ sudo /usr/bin/crb enable   # OR sudo dnf config-manager --set-enabled crb
$ sudo dnf group install "Development Tools"
$ sudo dnf install protobuf-c-devel libedit-devel yajl-devel json-c-devel
$ sudo dnf install libpcap-devel zeromq-devel libbsd-devel lmdb-devel

Note: you should not need the following, as the group install should cover it:

$ sudo dnf install autoconf automake libtool pkg-config git zlib

Ensure paths are set correctly:

$ su
# echo "/usr/local/lib" > /etc/ld.so.conf.d/local.conf
# exit
$ export PKG_CONFIG_PATH=/usr/local/lib/pkgconfig

Create a space to work in:

$ mkdir ~/fsi

You are now ready to install axa and its dependencies.

Install `wdns`¶

Clone the repository and install:

$ cd ~/fsi/
$ git clone https://github.com/farsightsec/wdns.git
$ cd wdns
$ ./autogen.sh
$ ./configure
$ make
$ sudo make install

Validate the install was successful. You should now see at least the following files. Note that other files could also be in this directory.

$ ls /usr/local/lib
libwdns.a  libwdns.la  libwdns.so  libwdns.so.1  libwdns.so.1.3.1  pkgconfig

Ensure the libraries are available from /usr/local/lib:

$ su
# ldconfig -v | grep libwdns
libwdns.so.1 -> libwdns.so.1.3.1
# exit

Install `nmsg`¶

Clone the repository and install:

$ cd ~/fsi/
$ git clone https://github.com/farsightsec/nmsg.git
$ cd nmsg
$ ./autogen.sh
$ ./configure
$ make
$ sudo make install

Validate the install was successful. You should now see at least the following files. Note that other files could also be in these directories:

$ ls /usr/local/lib
libnmsg.a   libnmsg.so   libnmsg.so.8.1.1   libwdns.la   libwdns.so.1   nmsg`   `libnmsg.la   libnmsg.so.8  libwdns.a   libwdns.so   libwdns.so.1.3.1   pkgconfig
$ ls /usr/local/lib/nmsg
nmsg_flt1_sample.la  nmsg_flt1_sample.so  nmsg_msg9_base.la  nmsg_msg9_base.so

Ensure the libraries are available from /usr/local/lib:

$ su
# ldconfig -v | grep libnmsg
libnmsg.so.8 -> libnmsg.so.8.1.1
# exit

Install `sie-nmsg`¶

Clone the repository and install:

$ cd ~/fsi/
$ git clone https://github.com/farsightsec/sie-nmsg.git
$ cd sie-nmsg
$ ./autogen.sh
$ ./configure
$ make
$ sudo make install

Validate the install was successful. You should now see at least the following files. Note that other files could also be in these dirs.

$ ls /usr/local/lib
libnmsg.a   libnmsg.so   libnmsg.so.8.1.1  libwdns.la  libwdns.so.1      nmsg   libnmsg.la   libnmsg.so.8  libwdns.a   libwdns.so   libwdns.so.1.3.1   pkgconfig
$ ls /usr/local/lib/nmsg
nmsg_flt1_sample.la   nmsg_msg9_base.la  nmsg_msg9_sie.a   nmsg_msg9_sie.so   nmsg_flt1_sample.so  nmsg_msg9_base.so  nmsg_msg9_sie.la

Install `axa` client¶

Clone the repository and install:

$ cd ~/fsi/
$ git clone https://github.com/farsightsec/axa.git
$ cd axa
$ ./autogen.sh
$ ./configure
$ make
$ sudo make install

Validate the install was successful. You should now see at least the following files. Note that other files could also be in this directory.

$ ls /usr/local/bin
axa_link_certs  axa_server_cert  nmsgtool  sratunnel`  `axa_make_cert   sratool

Ensure the libraries are available from /usr/local/lib:

$ su
# ldconfig -v | grep libaxa
    libaxa.so.3 -> libaxa.so.3.0.0
# exit

Validate the tool can successfully execute. Note that running sratool like this only proves all libraries and files are installed correctly; it is still necessary to configure access to stream data successfully.

$ sratool -V
sratool built using AXA library 3.0.1, supporting AXA protocols v1 to v2;
currently using v2
client HELLO: {"hostname":"rocky9.local","uname_sysname":"Linux",
"uname_release":"5.14.0-284.30.1.el9_2.x86_64",
"uname_version":"#1 SMP PREEMPT_DYNAMIC Sat Sep 16 09:55:41 UTC 2023",
"uname_machine":"x86_64","origin":"sratool","libaxa":"3.0.1","libnmsg":"1.1.2",
"libwdns":"0.12.0","libyajl":20100,"OpenSSL":"OpenSSL 3.0.7 1 Nov 2022", "AXA protocol":2}

FreeBSD¶

Binary Package Installation¶

Install the core nmsg packages and SIE plugins:

pkg install nmsg sie-nmsg

Optionally: Install packages for developing Python applications:

pkg install py-pynmsg py-pywdns

Optionally: Install packages for SIE remote access (sratool, sratunnel):

pkg install axa

Installation from Source using FreeBSD Ports¶

If you do not require the Doxygen generated API documentation, you may wish to disable the DOXYGEN option when building these ports to avoid building Doxygen and its many dependencies.

(cd /usr/ports/net/nmsg && make install clean)
(cd /usr/ports/net/sie-nmsg && make install clean)
(cd /usr/ports/net/py-pynmsg && make install clean)
(cd /usr/ports/dns/py-pywdns && make install clean)
(cd /usr/ports/net/axa && make install clean)

Uninstallation¶

pkg delete wdns nmsg sie-nmsg py-pynmsg py-pywdns axa

MacOS 14 Sonoma¶

These instructions are for installing AXA Tools (to enable SIE Remote Access) on Mac computers running MacOS 14.x Sonoma. See additional notes at the end for older Intel-based Macs.

Setup, Dependencies, and Configuration¶

The following tools and dependencies were installed. This configuration depends on the third-party brew package manager to install all required dependent libraries. Install Xcode command line tools. This shell command opens up a new UX window to confirm the install of the XCode tools. If you have the full version of XCode installed, you can skip this step.

$ xcode-select --install

Install Homebrew from brew.sh:

$ /bin/bash -c "$(curl -fsSL` `https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"

Homebrew may require you to make modifications to your shell’s profile. Make these changes before proceeding to ensure brew is set up and configured correctly.

Use Homebrew to install all required tools and libraries for AXA and its dependencies:

$ brew install wget openssl@1.1
$ brew install autoconf automake libtool pkgconfig
$ brew install protobuf protobuf-c libpcap yajl zmq lmdb json-c

Create a space to work in:

$ mkdir ~/fsi

OpenSSL Configuration¶

OpenSSL requires some extra configuration to ensure we use the brew installed version instead of the version pre-installed by macOS. Alter your ~/.profile or ~/.zprofile file with the following or set these exports directly in the terminal:

export LDFLAGS="-L/opt/homebrew/opt/openssl@1.1/lib"
export CPPFLAGS="-I/opt/homebrew/opt/openssl@1.1/include"
export PKG_CONFIG_PATH="/opt/homebrew/opt/openssl@1.1/lib/pkgconfig"

After setting these OpenSSL config changes, you may need to source your profile changes or restart your shell.

You are now ready to install axa and its dependencies.

Install `wdns`¶

Clone the repository and run the install:

$ cd ~/fsi
$ git clone https://github.com/farsightsec/wdns.git
$ cd wdns
$ ./autogen.sh
$ ./configure
$ make
$ sudo make install

Validate the install was successful. You should now see at least the following files. Note that other files could also be in this dir.

$ ls /usr/local/lib
libwdns.1.dylib  libwdns.a  libwdns.dylib  libwdns.la  pkgconfig

Install `nmsg`¶

Clone the repository and run the install. Note the change to configure:

$ cd ~/fsi
$ git clone https://github.com/farsightsec/nmsg.git
$ cd nmsg
$ ./autogen.sh
$ ./configure --with-libpcap=/opt/homebrew/opt/libpcap
$ make
$ sudo make install

Validate the install was successful. You should now see at least the following files. Note that other files could also be in these directories:

$ ls /usr/local/lib
libnmsg.8.dylib  libnmsg.dylib    libwdns.1.dylib  libwdns.dylib    nmsg   libnmsg.a   libnmsg.la   libwdns.a   libwdns.la   pkgconfig
$ ls /usr/local/lib/nmsg
nmsg_flt1_sample.la  nmsg_flt1_sample.so  nmsg_msg9_base.la  nmsg_msg9_base.so

Install `sie-nmsg`¶

Clone the repository and run the install.

$ cd ~/fsi
$ git clone https://github.com/farsightsec/sie-nmsg.git
$ cd sie-nmsg
$ ./autogen.sh
$ ./configure
$ make
$ sudo make install

Validate the install was successful. You should now see at least the following files. Note that other files could also be in these dirs.

$ ls /usr/local/lib
libnmsg.8.dylib  libnmsg.dylib    libwdns.1.dylib  libwdns.dylib    nmsg   libnmsg.a   libnmsg.la   libwdns.a   libwdns.la   pkgconfig
$ ls /usr/local/lib/nmsg
nmsg_flt1_sample.la   nmsg_msg9_base.la   nmsg_msg9_sie.a   nmsg_msg9_sie.so   nmsg_flt1_sample.so   nmsg_msg9_base.so   nmsg_msg9_sie.la

Install `axa` client¶

Clone the repository and run the install. Note the change to configure:

$ cd ~/fsi
$ git clone https://github.com/farsightsec/axa.git
$ cd axa
$ ./autogen.sh
$ ./configure --with-libpcap=/opt/homebrew/opt/libpcap
$ make
$ sudo make install

Validate the install was successful. You should now see at least the following files. Note that other files could also be in this directory.

$ ls /usr/local/bin
axa_link_certs  axa_server_cert  nmsgtool  sratunnel`   `axa_make_cert   sratool

Validate the tool can successfully execute. Note that running sratool like this only proves all libraries & files are installed correctly, you will still need to set up access to stream data successfully.

$ sratool -V
sratool built using AXA library 3.0.2, supporting AXA protocols v1 to v2; currently using v2
client HELLO:` `{"hostname":"My-M1-Mac.local","uname_sysname":"Darwin","uname_release":"23.2.0",
"uname_version":"Darwin Kernel Version 23.2.0: Wed Nov 15 21:53:34 PST 2023;
root:xnu-10002.61.3~2/RELEASE_ARM64_T8103","uname_machine":"arm64","origin":"sratool",
"libaxa":"3.0.2","libnmsg":"1.1.2","libwdns":"0.12.0","libyajl":20100,
"OpenSSL":"OpenSSL 3.1.4 24 Oct 2023","AXA protocol":2}

Intel-based Mac Differences¶

Apple Silicon (M1) Macs and older Intel-based Macs have some differences to be aware of since brew operates differently on the two platforms.

brew installs all installed packages to /opt/homebrew on Apple Silicon Macs, it installs into /usr/local for Intel-based Macs. All references to /opt/homebrew above will need to be changed.

Sudo is not required for make install on Intel Macs as brew changes /usr/local permissions. Sudo is required for Apple Silicon Macs.

Since brew operates inside /usr/local on Intel-based Macs and is more permissive with directory permissions, other users of the Mac may have access to these tools by default

Intel-based Mac sratool output for V3.x:

$ sratool -V  # on an Intel Mac
sratool built using AXA library 3.0.1, supporting AXA protocols v1 to v2; currently using v2
client HELLO: {"hostname":"My-Mac.local","uname_sysname":"Darwin","uname_release":"23.1.0",
"uname_version":"Darwin Kernel Version 23.1.0: Mon Oct  9 21:27:27 PDT 2023;
root:xnu-10002.41.9~6/RELEASE_X86_64","uname_machine":"x86_64","origin":"sratool",
"libaxa":"3.0.1","libnmsg":"1.1.2","libwdns":"0.12.0","libyajl":20100,
"OpenSSL":"OpenSSL 1.1.1w  11 Sep 2023","AXA protocol":2}

Access Methods¶

This section outlines SIE access methods and links to detailed documentation for each. Information on the file and data formats is available in the Data Formats section.

SIE Batch Web Interface¶

Connect: https://batch.sie-remote.net/
More information:
- SIE Batch User Guide
- What’s SIE Batch

The Batch web interface is a frontend for the REST API that provides full access to the last 12-18 hours of batched SIE data. This batch method is designed for periodic updates that can (with short delay periods) approach real-time. For real-time feeds, consult Remote Access and Direct Connect options, below.

Log in with your API key, which you obtain from enterprisesupport@domaintools.com. The interface will present each subscribed channel, the channel’s data format, and default download windows.

To use the web interface, select the channel and date range. The SIE web interface will confirm the channel, display its average traffic, and set the default date range.

To download current data, set the end date-time to your current time, minus \~10 seconds. SIE Batch does not de-duplicate overlapping downloads.

SIE Batch REST API¶

Connect: https://batch.sie-remote.net/siebatchd/v1/
More information:

The Batch REST API is the backend to the Batch Web Interface. Like the web interface, it provides full access to the last 12-18 hours of batched SIE data. This batch method is designed for periodic downloads, rather than as a source of real-time data. For real-time feeds, see Remote Access and Direct Connect options, below.

Validating an API Key¶

Obtain your API key from enterprisesupport@domaintools.com. Use the key to issue a POST command. For example, with curl:

APIKEY=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
curl -d '{"apikey":"'$APIKEY'"}' https://batch.sie-remote.net/siebatchd/v1/validate

In response, SIE will issue a response similar to:

HTTP/1.1 200 OK
{
  "profile": {
    "username": "siebatch-customer",
    "siebatch": {
      "ch212": {
        "description": "Newly Observed Domains"
      },
      "ch213": {
        "description": "Newly Observed Fully Qualified Domain Names"
      },
    }
  },
  "_status": "OK",
  "_message": ""
}

SIE Remote Access (AXA Toolkit)¶

Connect: Depending on the channel format, you may need to perform further processing after implementing Remote Access. See the Data Formats section below.
More information:

SIE Remote Access (SRA) provides real-time streams of all but the highest bandwidth SIE channels. It uses a suite of tools referred to collectively as the AXA (Advanced Exchange Access) toolkit, which implement the Remote Access transport layer (also known as the AXA Transport Layer). In order to reduce bandwidth, SRA provides subscribers with the ability to invoke a server-side filtering capability across a set of channels, selecting only that subset of records that match specific domain name / IP address search criteria.

Use Remote Access (AXA) tools sratool and sratunnel to select and define search or filtering criteria for your SIE channels, control rate limiting, and receive accounting messages.

SIE Remote Access Tool (`sratool`)¶

sratool is used to test, debug, inspect, or stream SIE Remote Access connections. In its most common invocation, it connects to a Remote Access server, issues AXA protocol messages, and displays the responses. sratool can be automated, but is typically used for interactive user-supplied commands.

More information:

SIE Remote Access Tunnel (`sratunnel`)¶

sratunnel is a production command-line tool that streams SIE data to the local network. It automates Remote Access server connections and updates.

More information:

Core Features¶

Watches¶

There are four kinds of watches you can set with Remote Access:

IP Watches: used to express interest in SIE messages containing a specified IP address or CIDR block. AXA understands both IPv4 and IPv6 address types.
DNS Watches: used to express interest in SIE messages containing a specified hostname, domain, or wildcard.
Channel Watches: used to express interest in SIE messages from an entire channel. This is useful to enable “the firehose” for a given channel and ask SRA to send everything from the specified channel rather than matching IP address information or DNS names.
Error watches: used to express interest in SIE messages that cannot be decoded by the server.

Rate Limiting¶

Rate limiting is used to limit the rate of incoming AXA messages as emitted from the server to the client.

Accounting¶

More information:
- Understanding Accounting

Accounting tracks traffic totals. Server-side, SIE maintains a series of per-client packet counters, and emits periodic accounting messages. The command is available from sratool and sratunnel via the -A command line option.

SIE Remote Access REST API (AXAMD)¶

More Information:
- axamd Client GitHub repository
- Using AXAMD with Newly Observed Domains

The AXA Middleware Daemon (AXAMD) provides a REST API interface for the Remote Access service. A Python module, axamd_client, is also available in the axamd GitHub repository.

SIE Direct Connect¶

More information:
- Introduction to nmsgtool

Direct Connect provides the fastest connection to the SIE network, with a dedicated, co-located blade server. DomainTools provides you with a pre-configured server (with root access). Contact enterprisesupport@domaintools.com to discuss your options.

With Direct Connect, you use nmsgtool to access and process SIE data on your blade server before transferring it to your network.

Data Formats¶

SIE data can be available in NMSG, JSON (and NDJSON), and packet capture (pcaplib) formats.

The NMSG format, introduced next, is unique to DomainTools and includes libraries and reference implementations for processing SIE data.

NMSG (Network Message)¶

More information:

NMSG is a file and transmission format that encapsulates typed, structured data into payloads which are packed into containers. The NMSG format relies on Google Protocol Buffers to encode the payload header. Each payload is associated with a specific message schema. nmsg can be extended at runtime with new message types, and can be converted to JSON with the included nmsgtool.

NMSG Libraries and Tools¶

libnmsg is the reference implementation of this format and provides an extensible interface for creating and parsing messages in NMSG format. Individual NMSG payloads are distinguished by assigned vendor ID and message type values and libnmsg provides a modular interface for registering handlers for specific message types. libnmsg makes it easy to build new message types using a Protocol Buffers compiler.

C Library: libnmsg¶

NMSG is delivered to the application programmer as a C library called libnmsg. The library presents a rich API for the programmer to build NMSG-capable applications and configure, tune, and/or tweak its many options and features.

The reference implementation of libnmsg is the included nmsgtool, a thin wrapper around libnmsg that provides powerful NMSG functionality at the Unix command-line.

Python Library: pynmsg¶

Also available is a Python extension module, pynmsg, that enables NMSG development using the Python programming language. See the pynmsg distribution and the pynmsg Python extension source.

JSON and NDJSON/JSONL¶

To see which channels and methods are available in JSON or Newline Delimited JSON (also known as JSON Lines), review the Channels and Formats table (above).

nmsgtool can natively convert from NSMG to JSON; consult the NMSG section above.

Packet Capture Format (PCAP)¶

A small number of channels arrive in packet capture format.

Detailed Channel Information¶

Channels 204, 208, 209: Value-Added Passive DNS¶

These channels consist of:

Channel 207: DNSDB Deduplicated Data
Channel 208, DNSDB Verified Data
Channel 204, Processed DNS Data (used by DNSDB)

Using Passive DNS Channel Data¶

Data acquired from Channel 204, 207, or 208 is returned in NMSG format for all access methods. NMSG is an adaptable container format that allows for consistent or variable message types.

The nmsgtool program is a tool for acquiring a variety of different inputs, like data streams from the network, capturing data from network interfaces, reading data from files, or even standard input and making NMSG payloads available to one or more outputs. The nmsgtool program can acquire data from SIE Channel 220 and convert it to a ND-JSON (newline-delimited JSON) text format for display or additional processing and analysis. nmsgtool is a program written by Farsight and released as open source.

After data for Channel 220 has been acquired, written, and saved to a file, you need to decode it to ND-JSON using nmsgtool. The [-r pdns-data.nmsg] option tells nmsgtool to read binary NMSG data from a file, [-c 1] limits the output to single NMSG payload, and [-J -] displays the record in ND-JSON format to stdout, which is typically the screen.

$ nmsgtool -r pdns-data.nmsg -c 1 -J -
(returned ND-JSON record)

Once the data has been formatted to ND-JSON, a record from the DNS Changes channel will look similar to the following. The following output can be sent to another tool for additional processing.

{"time":"2020-04-06 21:48:59.039279480","vname":"SIE","mname":"dnsdedupe",
"message":{"type":"EXPIRATION", "count":2,"time_first":"2020-04-06 18:47:22",
"time_last":"2020-04-06 18:47:22","bailiwick":"example.com.",
"rrname":"www.example.com.",
"rrclass":"IN","rrtype":"CNAME","rrttl":3600,"rdata":["dns.example.com."]}}

The open source software package is available on Debian and can be installed using $ sudo apt-get install jq. The output from nmsgtool in JSON format [-J -] can be piped to jq using the following:

$ nmsgtool -r pdns-data.nmsg -c 1 -J - | jq -r '.'
{
  "time": "2020-04-06 21:48:59.039279480",
  "vname": "SIE",
  "mname": "dnsdedupe",
  "message": {
    "type": "EXPIRATION",
    "count": 2,
    "time_first": "2020-04-06 18:47:22",
    "time_last": "2020-04-06 18:47:22",
    "bailiwick": "example.com.",
    "rrname": "www.example.com.",
    "rrclass": "IN",
    "rrtype": "CNAME",
    "rrttl": 3600,
    "rdata": [
      "dns.example.com."
    ]
  }
}

Data Format for SIE Passive DNS Channels 204, 207, 208¶

The SIE NMSG dnsdedupe schema is a DNS Query and Response resource record (RR) schema that observes and collects data returned from a query.

The data available from these channels contain NMSG SIE:dnsdedup type messages that include the following fields:

KEY	VALUE
time	Time when hostname was first observed in Channel 204.
vname	Vendor Name, SIE.
mname	Message type, dnsdedupe.
group	Reason DNS message was rejected.
message	Embedded JSON record describing the observed DNS Query and Response RR.

The embedded NMSG message payload is JSON formatted and includes the following fields:

KEY	VALUE
type	Types are INSERTION or EXPIRATION.
count	Number of times an RRset was observed since the last message was sent to the channel.
time_first	Indicates first time the RRset was observed by pDNS. Unix epoch timestamps with second granularity in UTC. Field is not present if the RRset was only observed from a zone file import.
time_last	Indicates last time the RRset was observed by pDNS. Unix epoch timestamps with second granularity in UTC. Field is not present if the RRset was only observed from a zone file import.
response_ip	IP address of the name server replying to the query. Field always exists in Channel 207 and optional in Channels 204 and 208.
rrname	Domain name or hostname of the query observed by pDNS or extracted from a zone file import.
rrclass	RR CLASS is always "Internet (IN)", which is decimal value "1".
rrtype	RR TYPE describes the type of RR, e.g., A(1), NS(2), CNAME(5).
rrttl	Time to live (TTL) of the RR.
rdata	Data that describes the RR type (may repeat).
bailiwick	The domain under which the RRset answer was given. Field always exists in channels 204 and 208. It is not returned in Channel 207. For example, an authoritative generic TLD (gTLD) name server for "com." may respond with different answers for the same query than the authoritative name servers for "farsightsecurity.com." would respond with.

Understanding Passive DNS INSERTION & EXPIRATION Mesages¶

DNS data sent to channel DNSDB Deduplicated Data (207), DNSDB Verified Data (208), or Processed DNS Data (204) will be either INSERTION or EXPIRATION type messages.

To understand what INSERTION and EXPIRATION mean, we need to discuss how deduplication is implemented in SIE. During processing, the waterfall model maintains a cache table of observed RRsets as a large ring buffer in memory.

When DNS data is received by SIE, the cache table is checked to see if the RRset already exists. If the RRset exists in the cache table, the cache entry's count is incremented and the time_last field is updated, and the record discarded as a duplicate. If the RRset does not exist in the cache table, the record is inserted into the cache, and an "INSERTION" record is sent from the deduplicator to the next phase of processing. This causes the oldest record in the ring buffer to be expired from the cache table, and an "EXPIRATION" record is sent from the deduplicator to document the removal. These records are broadcast to the DNSDB Deduplicated Data (207) channel and for the verification phase of the waterfall mode.

If you are primarily interested when an RRset is first observed, you can focus on "INSERTION" records and if you are interested in how often an RRset is observed, you should monitor "EXPIRATION" records.

Example Message -- INSERTION Record¶

The time_first and time_last fields for INSERTION records are always the same and the count is always 1. In the following example, the query was received from IP address 10.10.10.10 (which is acting as the authoritative name server for com.) and the message indicates an SOA record was observed in the response. rrttl displays Time to Live (TTL) value for the record that would be used when caching the data, and rdata is the data returned for the query.

{
  "time": "2020-04-06 22:39:55.429865036",
  "vname": "SIE",
  "mname": "dnsdedupe",
  "message": {
    "type": "INSERTION",
    "count": 1,
    "time_first": "2020-04-06 22:38:48",
    "time_last": "2020-04-06 22:38:48",
    "response_ip": "10.10.10.10",
    "bailiwick": "com.",
    "rrname": "com.",
    "rrclass": "IN",
    "rrtype": "SOA",
    "rrttl": 86400,
    "rdata": [
      "dns.example.com. dns2.example.com. 1586212699 1800 900 604800 86400"
    ]
  }
}

Example Message -- EXPIRATION Record¶

The following example message is for a AAAA resource record. The data returned in the rdata field are the IPv6 addresses for the domain in the rrname field, which is www.example.com.. With the site acting as its own authoritative name server in the bailiwick. Since count is 1, time_first will match time_last, indicating only one query was seen before record expired from the hash table. If value of count was more than 1, time_last may or may not be the same as time_first.

{
  "time": "2020-04-06 22:39:57.420893762",
  "vname": "SIE",
  "mname": "dnsdedupe",
  "message": {
    "type": "EXPIRATION",
    "count": 1,
    "time_first": "2020-04-06 19:32:42",
    "time_last": "2020-04-06 19:32:42",
    "bailiwick": "www.example.com.",
    "rrname": "www.example.com.",
    "rrclass": "IN",
    "rrtype": "AAAA",
    "rrttl": 120,
    "rdata": [
      "2001:db8::1"
      "2001:db8::2"
      "2001:db8::3"
      "2001:db8::a"
      "2001:db8::b"
      "2001:db8::c"
    ]
  }
}

Security Information Exchange (SIE): User Guide¶

Introduction¶

Access¶

System Requirements¶

Channels Overview¶

Installation¶

Debian GNU/Linux 11 Bullseye¶

Package Signing Key¶

Package Installation¶

From Source on Rocky 9 (RHEL-compatible)¶

Setup, Dependencies, and Configuration¶

Install wdns¶

Install nmsg¶

Install sie-nmsg¶

Install axa client¶

FreeBSD¶

Binary Package Installation¶

Installation from Source using FreeBSD Ports¶

Uninstallation¶

MacOS 14 Sonoma¶

Setup, Dependencies, and Configuration¶

OpenSSL Configuration¶

Install wdns¶

Install nmsg¶

Install sie-nmsg¶

Install axa client¶

Intel-based Mac Differences¶

Access Methods¶

SIE Batch Web Interface¶

SIE Batch REST API¶

Validating an API Key¶

SIE Remote Access (AXA Toolkit)¶

SIE Remote Access Tool (sratool)¶

SIE Remote Access Tunnel (sratunnel)¶

Core Features¶

Watches¶

Rate Limiting¶

Accounting¶

SIE Remote Access REST API (AXAMD)¶

SIE Direct Connect¶

Data Formats¶

NMSG (Network Message)¶

NMSG Libraries and Tools¶

C Library: libnmsg¶

Python Library: pynmsg¶

JSON and NDJSON/JSONL¶

Packet Capture Format (PCAP)¶

Detailed Channel Information¶

Channels 204, 208, 209: Value-Added Passive DNS¶

Using Passive DNS Channel Data¶

Data Format for SIE Passive DNS Channels 204, 207, 208¶

Understanding Passive DNS INSERTION & EXPIRATION Mesages¶

Example Message -- INSERTION Record¶

Example Message -- EXPIRATION Record¶

Install `wdns`¶

Install `nmsg`¶

Install `sie-nmsg`¶

Install `axa` client¶

Install `wdns`¶

Install `nmsg`¶

Install `sie-nmsg`¶

Install `axa` client¶

SIE Remote Access Tool (`sratool`)¶

SIE Remote Access Tunnel (`sratunnel`)¶