SIE Batch endpoints and examples

Conventions

Request Format

All API requests must:

Use HTTP POST method
Include Content-Type: application/json header
Send parameters as JSON in the request body

API Authentication

All endpoints require authentication via the apikey parameter:

Include apikey in the JSON request body (not in headers or query string)
API keys are in UUID format: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
Obtain your API key from DomainTools Enterprise Support
Use the validate endpoint (below) to verify your API key and check provisioned channels

Rate Limiting

For information about rate limits and quotas, contact enterprisesupport@domaintools.com.

Response Format

Every response from the SIE Batch API contains two status indicating keys:

_status: A string containing either OK or NOK indicating whether the API call was successful
_message: A string containing a description of what went wrong (empty string if _status is OK)

If _status is NOK, additional fields giving details of the exception are present, conforming to RFC 7807.

Exception response examples are provided for each API endpoint.

Authentication

Validate - Validate an API Key

Validates a given API Key, returns profile information associated with it.

POST

https://batch.sie-remote.net/siebatchd/v1/validate

Example cURL

APIKEY=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
curl -d '{"apikey":"'$APIKEY'"}' \
  -H "Content-Type: application/json" \
  https://batch.sie-remote.net/siebatchd/v1/validate

Example Request

{
  "apikey": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
}

Request Parameters

Field	Type	Description
apikey	String	Your Farsight SIE Batch API Key

Response Fields

Field	Type	Description
profile	Dictionary	A dictionary containing user profile information (provisioning). The following keys are only some of the possible keys available.
username	String	The username associated with your API Key.
siebatch	Dictionary	The SIE batch channels you have access to.
_status	String	OK
_message	String	An account status message ("" [empty] if no message).

Success-Response

HTTP/1.1 200 OK

{
  "profile": {
    "username": "siebatch-customer",
    "siebatch": {
      "ch212": {
        "description": "Newly Observed Domains"
      },
      "ch213": {
        "description": "Newly Observed Fully Qualified Domain Names"
      },
    }
  },
  "_status": "OK",
  "_message": ""
}

Error 4xx 5xx

Field	Type	Description
apikey	String	Your Farsight SIE Batch API Key
channels	List	List (array) of one or more channel numbers (integers). Valid channels are 1-255.

MissingAPIKey

HTTP/1.1 403
{
    "_message": "API key not present",
    "_status": "NOK",
    "logid": "30fd25e0-0130-4ab1-a375-fabf1d57430e",
    "status": 403,
    "type": "missing-api-key"
}

Batched Channel Access

siebatch chdetails - Get Channel Details

Request the details of a channel such as possible datetime ranges, download filetype, and the size of data available.

These details will update as data is stored and pruned on the server every minute; channel data is always in motion so the details represent a moving window of data that is available for download. This means that the size and earliest/latest times will change frequently. Because channels are continuously recording new data it’s not possible to always be up to date if you were to repeatedly request the latest “earliest” and “latest” details for a channel; it’s an archival reference only.

POST

https://batch.sie-remote.net/siebatchd/v1/siebatch/chdetails

Example cURL

APIKEY=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx

curl -d '{"apikey":"'$APIKEY'","channels":[212, 213]}' \
  -H "Content-Type: application/json" \
  -X POST https://batch.sie-remote.net/siebatchd/v1/siebatch/chdetails

Example Request

{
   "apikey": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
   "channels": [212, 213]
}

Request Parameters

Field	Type	Description
apikey	String	Your Farsight SIE Batch API Key
channels	List	List (array) of one or more channel numbers (integers). Valid channels are 1-255.

Success 200

Field	Type	Description
channels	Dictionary	Dictionary of channel objects keyed by channel name (e.g., “ch212”)
earliest	String	Earliest available data timestamp in “YYYY-MM-DD HH:MM:SS” format
latest	String	Latest available data timestamp in “YYYY-MM-DD HH:MM:SS” format
size	Integer	Size of available data in bytes
mimetype	String	MIME type of the data (e.g., “application/x-ndjson”)
_status	String	OK
_message	String	"" (empty if no message)

Success Response Example

HTTP/1.1 200 OK

{
  "channels": {
    "ch212": {
      "earliest":"2020-01-28 00:40:00",
      "latest":"2020-01-28 20:06:00",
      "size":83837922,
      "mimetype": "application/x-ndjson"
    },
    "ch213": {
      "earliest":"2020-01-28 00:40:00",
      "latest": "2020-01-28 20:06:01",
      "size":10397567992,
      "mimetype": "application/x-ndjson"
    }
  },
  "_status": "OK",
  "_message":""
}

Error 4xx 5xx

Name	Type	Description
_status	String	NOK
_message	String	A short description of what went wrong.
status	Integer	HTTP status code
type	String	A hyphenated mnemonic for the error. Use this to key on a local i18n translation.
detail	String	Optional String that contains more details about the specific error.

SessionFailure

HTTP/1.1 403
{
    "_message": "<Message will contain details>",
    "_status": "NOK",
    "logid": "30fd25e0-0130-4ab1-a375-fabf1d57430e",
    "status": 403,
    "type": "session-failure"
}

InvalidParameters

HTTP/1.1 403
{
    "_message": "<message will be specific wrt to which parameter(s) is missing>",
    "_status": "NOK",
    "logid": "30fd25e0-0130-4ab1-a375-fabf1d57430e",
    "status": 403,
    "type": "invalid-parameters",
    "invalid_parameters": {
       "name": "foobar",
       "reason": "should be an integer but received a string"
    }
}

MissingAPIKey

HTTP/1.1 403
{
    "_message": "API key not present",
    "_status": "NOK",
    "logid": "30fd25e0-0130-4ab1-a375-fabf1d57430e",
    "status": 403,
    "type": "missing-api-key"
}

siebatch chfetch - Fetch Channel Data

Download data associated with a given channel.

As mentioned in chdetails, the earliest data available is pruned frequently, so you are not guaranteed to retrieve that data if you request it.

The seconds portion of the start/end times are ignored, but for API consistency, we require seconds to be given. This means that if you ask for “12:10:30” through “12:45:30”, you get “12:10:00” through “12:45:00”.

If no data is available then an HTTP 404 is returned - otherwise it returns 200, sets the appropriate HTTP headers, and the download begins.

POST

https://batch.sie-remote.net/siebatchd/v1/siebatch/chfetch

Example cURL

APIKEY=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx

curl -d '{"apikey":"'$APIKEY'","channel":213,"start_time":"2020-01-28 12:00:00","end_time":"2020-01-28 12:01:00"}' \
  -H "Content-Type: application/json" \
  -X POST https://batch.sie-remote.net/siebatchd/v1/siebatch/chfetch

Example Request

{
   "apikey": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
   "channel": 213,
   "start_time": "2020-01-28 12:00:00",
   "end_time": "2020-01-28 12:01:00"
}

Request Parameters

Field	Type	Description
apikey	String	Your Farsight SIE Batch API Key
channel	Integer	Channel Number
start_time	String	A Start Date, “YYYY-mm-dd HH:MM:SS” in UTC
end_time	String	An End Date, “YYYY-mm-dd HH:MM:SS” in UTC

Error 4xx 5xx

Name	Type	Description
_status	String	NOK
_message	String	A short description of what went wrong.
status	Integer	HTTP status code
type	String	A hyphenated mnemonic for the error. Use this to key on a local i18n translation.
detail	String	Optional String that contains more details about the specific error.

SessionFailure

HTTP/1.1 403
{
    "_message": "<Message will contain details>",
    "_status": "NOK",
    "logid": "30fd25e0-0130-4ab1-a375-fabf1d57430e",
    "status": 403,
    "type": "session-failure"
}

InvalidParameters

HTTP/1.1 403
{
    "_message": "<message will be specific wrt to which parameter(s) is missing>",
    "_status": "NOK",
    "logid": "30fd25e0-0130-4ab1-a375-fabf1d57430e",
    "status": 403,
    "type": "invalid-parameters",
    "invalid_parameters": {
       "name": "foobar",
       "reason": "should be an integer but received a string"
    }
}

MissingAPIKey

HTTP/1.1 403
{
    "_message": "API key not present",
    "_status": "NOK",
    "logid": "30fd25e0-0130-4ab1-a375-fabf1d57430e",
    "status": 403,
    "type": "missing-api-key"
}

siebatch makeurl - Make URL

Generate a URL to download data associated with a channel later or independently of the SIE Batch webserver.

As mentioned in chdetails, the earliest data available is pruned frequently, so you are not guaranteed to retrieve it if it is about to be pruned.

Using makeurl as opposed to chfetch is advantageous because generated URLs are more easily shared and accessed later on different computers and by fellow users, independently of the user that originally requested them and without revealing credentials. Generated URLs are the preferred method for sharing snippets of channel data.

Data Availability

The server retains a rolling window of data per channel, typically 12-18 hours depending on the channel’s data rate. Use chdetails to check a channel’s earliest and latest timestamps before generating a URL.

The makeurl call itself succeeds even if the requested time range extends beyond the retention window. The error occurs when you download from the returned URL:

HTTP Status	Error	Cause
404	NoSuchKey	Requested data is outside the channel’s retention window.
403	Unable to authenticate	The generated URL has expired. Generate a new one.

POST

https://batch.sie-remote.net/siebatchd/v1/siebatch/makeurl

Example cURL

APIKEY=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx

curl -d '{"apikey":"'$APIKEY'","channel":213,"start_time":"2020-01-28 12:00:00","end_time":"2020-01-28 12:01:00"}' \
  -H "Content-Type: application/json" \
  -X POST https://batch.sie-remote.net/siebatchd/v1/siebatch/makeurl

Example Request

{
   "apikey": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
   "channel": 213,
   "start_time": "2020-01-28 12:00:00",
   "end_time": "2020-01-28 12:01:00"
}

Request Parameters

Field	Type	Description
apikey	String	Your Farsight SIE Batch API Key
channel	Integer	Channel Number
start_time	String	A Start Date, “YYYY-mm-dd HH:MM:SS” in UTC
end_time	String	An End Date, “YYYY-mm-dd HH:MM:SS” in UTC

Success 200

Field	Type	Description
_status	String	OK
_message	String	"" (empty if no message)
url	String	The presigned shareable URL to the specified SIE data

Success-Response

HTTP/1.1 200 OK

{"_status": "OK",
 "_message": "",
 "url": "https://batch-dl.sie-remote.net/range/ch213/20200128.1200.20200128.1201?
 X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=..."
}

Error 4xx 5xx

Name	Type	Description
_status	String	NOK
_message	String	A short description of what went wrong.
status	Integer	HTTP status code
type	String	A hyphenated mnemonic for the error. Use this to key on a local i18n translation.
detail	String	Optional String that contains more details about the specific error.

SessionFailure

HTTP/1.1 403
{
    "_message": "<Message will contain details>",
    "_status": "NOK",
    "logid": "30fd25e0-0130-4ab1-a375-fabf1d57430e",
    "status": 403,
    "type": "session-failure"
}

InvalidParameters

HTTP/1.1 403
{
    "_message": "<message will be specific wrt to which parameter(s) is missing>",
    "_status": "NOK",
    "logid": "30fd25e0-0130-4ab1-a375-fabf1d57430e",
    "status": 403,
    "type": "invalid-parameters",
    "invalid_parameters": {
       "name": "foobar",
       "reason": "should be an integer but received a string"
    }
}

MissingAPIKey

HTTP/1.1 403
{
    "_message": "API key not present",
    "_status": "NOK",
    "logid": "30fd25e0-0130-4ab1-a375-fabf1d57430e",
    "status": 403,
    "type": "missing-api-key"
}