Skip to content

SIE AXA User Guide

Introduction

Security Information Exchange (SIE) is the world's largest real-time threat intelligence platform — it aggregates, filters and broadcasts diverse Internet-security related information so security professionals can more accurately and quickly identify, map, and protect from cybercriminal activity.

There are multiple delivery mechanisms to consume data on SIE--consult the SIE User Guide for more information.

This user guide will discuss and illustrate the tools found in the Farsight Advanced Exchange Access Toolkit to connect and consume data from SIE.

Advanced Exchange Access Toolkit

The Advanced Exchange Access (AXA) toolkit contains tools and a C library to bring Farsight's real-time data and services directly from the Farsight Security Information Exchange (SIE) to the subscriber's network edge.

AXA enables subscribers to connect to Farsight's subscription-based SRA (SIE Remote Access) and RAD (Real-time Anomaly Detector) servers. These servers provide access to data and services built from Farsight's SIE.

SRA streams real-time SIE data while RAD streams real-time anomaly detection data (from services such as Brand Sentry and Domain Sentry).

Requirements

Operating System

Linux, FreeBSD or other POSIX compliant operating systems.

Hardware

The minimum hardware requirements to get started with tools in the Advanced Exchange Access Toolkit are listed below. Depending on the amount of data being processed, the resources may need to be increased accordingly.

  • 1 CPU
  • 1 GB Memory
  • 1 GB Disk

Network

Tools in the Advanced Exchange Access Toolkit require permitted outbound to sra.sie-remote.net and rad.sie-remote.net over TCP using port 22.

Service Entitlement

Subscribers must have purchased a SIE service entitlement from Farsight Security and have been provisioned access using a SSH key.

Contents of the Advanced Exchange Access Toolkit

The Advanced Exchange Access Toolkit distribution contains the following:

  • sratool: A test/debug/instructional command-line tool used to connect to an SRA server, set watches, enable SIE channels, and stream data.
  • radtool: A test/debug/instructional command-line tool used to connect to a RAD server, set watches, enable anomaly detection modules, and stream data.
  • sratunnel: A production command-line tool that streams SIE data to the local network.
  • radtunnel: A production command-line tool that streams anomaly data to the local network.
  • libaxa: A C library providing an API for the AXA protocol including:
    • connection instantiation/teardown,
    • message encapsulation/decapsulation,
    • watch parsing/loading,
    • trie storage and lookup,
    • control packet rate limits, sampling rates, window sizes, and many other AXA-specific functions.

For usage details on sratool, radtool, sratunnel, and radtunnel, please see their respective man pages (included in the distribution).

Installing Advanced Exchange Access Toolkit (axa-tools)

Debian 8 and Ubuntu 14.04/16.04

These instructions use Debian packages created, maintained and hosted by DomainTools.

  1. Download the Farsight Apt signing key.
$ sudo wget -O /etc/apt/trusted.gpg.d/debian-farsightsec.gpg \
    https://dl.farsightsecurity.com/debian/archive.pubkey
  1. Add the Farsight Debian repository.
$ echo "deb http://dl.farsightsecurity.com/debian wheezy-farsightsec main" \
    | sudo tee -a /etc/apt/sources.list.d/debian-farsightsec.list
  1. Resynchronize the package index files.
$ sudo apt update
  1. Install the Advanced Exchange Access Toolkit (axa-tools).
$ sudo apt install axa-tools

Build from Source

See the section titled Building manually https://github.com/farsightsec/axa/blob/master/README.md#building-manually in the README file found on the GitHub repository for the Farsight Advanced Exchange Access Toolkit https://github.com/farsightsec/axa

Configuring Advanced Exchange Access with SSH

  1. At the time of provisioning you would have been asked to generate a SSH key pair used for authentication. The following steps will reference this key, make sure you reference the correct directory path when configuring the key.
$ ssh-keygen -t rsa -b 4096 -C farsight_security -f ~/.ssh/farsight_security
  1. Create or edit the SSH config file with the following:
$ vim ~/.ssh/config

Add the following:

Host sra.sie-remote.net rad.sie-remote.net
  IdentityFile ~/.ssh/farsight_security

Usage Examples

Prerequisites

  • A host with Linux, FreeBSD or other POSIX compliant operating system installed
  • A SIE Remote Access entitlement from Farsight Security
  • Having already exchanged a SSH keypair with Farsight Security
  • Having installed and configured the Advanced Exchange Access Toolkit

sratool

sratool is a test/debug/instructional command-line tool used to connect to an SRA server, set watches, enable SIE channels, and stream data.

Stream SIE traffic with sratool

An example using sratool to emit five messages seen on SIE Channel 255 (SIE Heartbeat Channel):

  1. $ sratool
  2. sra> connect ssh:sra-service@sra.sie-remote.net: connect to an SRA server using the SSH transport. SSH used its keyring to prove the user's identity, so there was no 'password:' prompt. The HELLO response from the remote end displays its version number and the protocol level.
  3. sra> count 5: instruct the sratool client to stop after five messages are output.
  4. sra> channel 255 on: instruct the remote end to listen to SIE channel 255 which was OK'd by the server indicating that it is provisioned for this channel according to the authentication and authorization level.
  5. sra> 1 watch ch=255: watch all content on channel 255 (with no rate limiting or filtering).
$ sratool
sra> connect ssh:sra-service@sra.sie-remote.net
HELLO srad version 1.2.1 sra AXA protocol 1
sra> count 5
sra> channel 255 on
OK CHANNEL ON/OFF channel ch255 on
sra> 1 watch ch=255
1 OK WATCH started
1 ch255  base encode   TEXT
1 ch255  base encode   TEXT
1 ch255  base encode   TEXT
1 ch255  base encode   TEXT
1 ch255  base encode   TEXT
packet count limit exceeded
sra> exit

sratunnel

sratunnel is a production command-line tool that streams SIE data to the local network.

Create a persistent connection to SIE

An example using sratunnel as a background process to stream nmsg messages from SIE Channel 255 (SIE Heartbeat Channel) to the loopback interface on port 8000.

  1. Invoke sratunnel with the following arguments.
$ sratunnel -s 'ssh:sra-service@sra.sie-remote.net' -c 255 \
    -w ch=255 -o nmsg:udp:127.0.0.1,8000 &
  1. Use tcpdump to confirm messages are being streamed.
$ sudo tcpdump -i lo -c 5 -nn port 8000
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on lo, link-type EN10MB (Ethernet), capture size 262144 bytes
11:18:41.204425 IP 127.0.0.1.36707 > 127.0.0.1.8000: UDP, length 941
11:18:58.672776 IP 127.0.0.1.36707 > 127.0.0.1.8000: UDP, length 941
11:19:16.312962 IP 127.0.0.1.36707 > 127.0.0.1.8000: UDP, length 941
11:19:33.833821 IP 127.0.0.1.36707 > 127.0.0.1.8000: UDP, length 941
11:19:51.277784 IP 127.0.0.1.36707 > 127.0.0.1.8000: UDP, length 941
5 packets captured
10 packets received by filter
0 packets dropped by kernel
  1. Bring the background process to the foreground.
$ fg
  1. Kill the sratunnel process by pressing Control-C.

Process messages with nsmgtool

The nmsgtool program is a single tool for taking inputs from a variety of different inputs like data streams from the network, capturing data from network interfaces, reading data from files or even standard input and making NMSG payloads available to one or more outputs.

Installing nmsgtool

  1. Install https://www.farsightsecurity.com/technical/SIE-user-guide/sie-debian/ nmsgtool:
$ sudo apt install nmsgtool nmsg-msg-module-sie

Viewing a stream of nmsg's from sratunnel with nmsgtool

An example using sratunnel as a background process to stream nmsg messages from SIE Channel 255 (SIE Heartbeat Channel) to the loopback interface on port 8000; using nmsgtool to connect to the loopback interface and print the nmsg to the terminal in presentation format.

  1. Invoke sratunnel with the following arguments:
$ sratunnel -s 'ssh:sra-service@sra.sie-remote.net' -c 255 -w ch=255 -o nmsg:udp:127.0.0.1,8000 &
  1. Invoke nmsgtool to connect to the loopback interface on port 8000, process three payloads and print the output to the terminal using the presentation format.
$ nmsgtool -l 127.0.0.1/8000 -c 3 -o -
[23] [2017-06-28 19:53:51.844574928] [1:11 base encode] [1ba02cfd] [] []
type: TEXT
payload: <BYTE ARRAY LEN=19>

[23] [2017-06-28 19:53:52.345241069] [1:11 base encode] [1ba02cfd] [] []
type: TEXT
payload: <BYTE ARRAY LEN=19>

[23] [2017-06-28 19:53:52.845875978] [1:11 base encode] [1ba02cfd] [] []
type: TEXT
payload: <BYTE ARRAY LEN=19>
  1. Bring the background process to the foreground.
$ fg
  1. Kill the sratunnel process by pressing Control-C.

Saving a stream of nmsg's from sratunnel with nmsgtool

An example using sratunnel as a background process to stream nmsg messages from SIE Channel 255 (SIE Heartbeat Channel) to the loopback interface on port 8000; using nmsgtool to connect to the loopback interface and saving the output to a rotating set of files using the nmsgtool kicker function.

  1. Invoke sratunnel with the following arguments:
$ sratunnel -s 'ssh:sra-service@sra.sie-remote.net' -c 255 \
    -w ch=255 -o nmsg:udp:127.0.0.1,8000 &
  1. Invoke nmsgtool to connect to the loopback interface on port 8000, save nmsg files to disk every sixty seconds as a background process.
$ nmsgtool -l 127.0.0.1/8000 -t 60 -k '/bin/true' -w ch255 &
  1. List the saved files using ls.
$ ls -l
total 16
-rw-r--r-- 1 demo demo 5518 Jun 28 16:03
    ch255.20170628.2002.1498698127.548592412.nmsg
-rw-r--r-- 1 demo demo 6436 Jun 28 16:04
    ch255.20170628.2003.1498698180.574404303.nmsg
  1. Read one of the files using nmsgtool and outputting the results to the terminal in JSON:
$ nmsgtool -r ch255.20170628.2003.1498698180.574404303.nmsg -J -
{"time":"2017-06-28 20:03:02.061745882","vname":"base",
    "mname":"encode","source":"1ba02cfd",
    "message":{"type":"TEXT","payload":"IkZTSSBTSUUgaGVhcnRiZWF0Ig=="}}
{"time":"2017-06-28 20:03:02.562045097","vname":"base",
"mname":"encode","source":"1ba02cfd",
"message":{"type":"TEXT","payload":"IkZTSSBTSUUgaGVhcnRiZWF0Ig=="}}
{"time":"2017-06-28 20:03:03.062705039","vname":"base",
"mname":"encode","source":"1ba02cfd",
"message":{"type":"TEXT","payload":"IkZTSSBTSUUgaGVhcnRiZWF0Ig=="}}
  1. Bring the nmsgtool background process to the foreground.
$ fg

  1. Kill the nmsgtool process by pressing Control-C.

  2. Bring the sratunnel background process to the foreground.

$ fg
  1. Kill the sratunnel process by pressing Control-C.

AXA Protocol

The AXA protocol is documented in the section titled AXA Protocol https://github.com/farsightsec/axa/blob/master/README.md#axa-protocol in the README file found on the GitHub repository for the Farsight Advanced Exchange Access Toolkit https://github.com/farsightsec/axa.

Limits

Some of the channels offered by the SIE network burst to an extremely high bitrate (some over 500Mbps). AXA has two ways to deal with such network-hungry situations: optional filtering and loss-tolerance built into the protocol.

Filtering can take one of the following forms:

  • Via the rate limit option to reduce the flow of ingress data to a certain number of packets per second.
  • Via one or more IP-based or DNS-based "watches" to limit the flow of data to specific assets the subscriber wishes to observe.

Finally, AXA is a deliberately lossy protocol. If a subscriber requests more data than the network can carry, data overruns will occur. When this happens, loss markers are transmitted reliably within the AXA stream to inform the subscriber via the AXA accounting subsystem https://www.farsightsecurity.com/2015/09/24/mschiffm-axa-accounting/. At this point, the subscriber's possible mitigation strategies include:

  • ask for less data via rate limiting
  • increase network capacity and/or other host resources
  • treat the SRA stream as a chunky and non-representative sample of the total SIE data
  • pursue Direct Access to SIE