SIE Batch API Reference¶

Conventions¶

Request Format¶

All API requests must:

Use HTTP POST method
Include Content-Type: application/json header
Send parameters as JSON in the request body

API Authentication¶

All endpoints require authentication via the apikey parameter:

Include apikey in the JSON request body (not in headers or query string)
API keys are in UUID format: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
Obtain your API key from DomainTools Enterprise Support
Use the validate endpoint (below) to verify your API key and check provisioned channels

Rate Limiting¶

For information about rate limits and quotas, contact enterprisesupport@domaintools.com.

Response Format¶

Every response from the SIE Batch API contains two status indicating keys:

_status: A string containing either OK or NOK indicating whether the API call was successful
_message: A string containing a description of what went wrong (empty string if _status is OK)

If _status is NOK, additional fields giving details of the exception are present, conforming to RFC 7807.

Exception response examples are provided for each API endpoint.

Authentication¶

Validate - Validate an API Key¶

Validates a given API Key, returns profile information associated with it.

POST¶

https://batch.sie-remote.net/siebatchd/v1/validate

Example cURL¶

APIKEY=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
curl -d '{"apikey":"'$APIKEY'"}' \
  -H "Content-Type: application/json" \
  https://batch.sie-remote.net/siebatchd/v1/validate

Example Request¶

{
  "apikey": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
}

Request Parameters¶

Field	Type	Description
apikey	String	Your Farsight SIE Batch API Key

Response Fields¶

Field	Type	Description
profile	Dictionary	A dictionary containing user profile information (provisioning). The following keys are only some of the possible keys available.
username	String	The username associated with your API Key.
siebatch	Dictionary	The SIE batch channels you have access to.
_status	String	OK
_message	String	An account status message ("" [empty] if no message).

Success-Response¶

HTTP/1.1 200 OK

{
  "profile": {
    "username": "siebatch-customer",
    "siebatch": {
      "ch212": {
        "description": "Newly Observed Domains"
      },
      "ch213": {
        "description": "Newly Observed Fully Qualified Domain Names"
      },
    }
  },
  "_status": "OK",
  "_message": ""
}

Error 4xx 5xx¶

Field	Type	Description
apikey	String	Your Farsight SIE Batch API Key
channels	List	List (array) of one or more channel numbers (integers). Valid channels are 1-255.

MissingAPIKey¶

HTTP/1.1 403
{
    "_message": "API key not present",
    "_status": "NOK",
    "logid": "30fd25e0-0130-4ab1-a375-fabf1d57430e",
    "status": 403,
    "type": "missing-api-key"
}

Batched Channel Access¶

siebatch chdetails - Get Channel Details¶

Request the details of a channel such as possible datetime ranges, download filetype, and the size of data available.

These details will update as data is stored and pruned on the server every minute; channel data is always in motion so the details represent a moving window of data that is available for download. This means that the size and earliest/latest times will change frequently. Because channels are continuously recording new data it's not possible to always be up to date if you were to repeatedly request the latest "earliest" and "latest" details for a channel; it's an archival reference only.

POST¶

https://batch.sie-remote.net/siebatchd/v1/siebatch/chdetails

Example cURL¶

APIKEY=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx

curl -d '{"apikey":"'$APIKEY'","channels":[212, 213]}' \
  -H "Content-Type: application/json" \
  -X POST https://batch.sie-remote.net/siebatchd/v1/siebatch/chdetails

Example Request¶

{
   "apikey": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
   "channels": [212, 213]
}

Request Parameters¶

Field	Type	Description
apikey	String	Your Farsight SIE Batch API Key
channels	List	List (array) of one or more channel numbers (integers). Valid channels are 1-255.

Success 200¶

Field	Type	Description
channels	Dictionary	Dictionary of channel objects keyed by channel name (e.g., "ch212")
earliest	String	Earliest available data timestamp in "YYYY-MM-DD HH:MM:SS" format
latest	String	Latest available data timestamp in "YYYY-MM-DD HH:MM:SS" format
size	Integer	Size of available data in bytes
mimetype	String	MIME type of the data (e.g., "application/x-ndjson")
_status	String	OK
_message	String	"" (empty if no message)

Success Response Example¶

HTTP/1.1 200 OK

{
  "channels": {
    "ch212": {
      "earliest":"2020-01-28 00:40:00",
      "latest":"2020-01-28 20:06:00",
      "size":83837922,
      "mimetype": "application/x-ndjson"
    },
    "ch213": {
      "earliest":"2020-01-28 00:40:00",
      "latest": "2020-01-28 20:06:01",
      "size":10397567992,
      "mimetype": "application/x-ndjson"
    }
  },
  "_status": "OK",
  "_message":""
}

Error 4xx 5xx¶

Name	Type	Description
_status	String	NOK
_message	String	A short description of what went wrong.
status	Integer	HTTP status code
type	String	A hyphenated mnemonic for the error. Use this to key on a local i18n translation.
detail	String	Optional String that contains more details about the specific error.

SessionFailure¶

HTTP/1.1 403
{
    "_message": "<Message will contain details>",
    "_status": "NOK",
    "logid": "30fd25e0-0130-4ab1-a375-fabf1d57430e",
    "status": 403,
    "type": "session-failure"
}

InvalidParameters¶

HTTP/1.1 403
{
    "_message": "<message will be specific wrt to which parameter(s) is missing>",
    "_status": "NOK",
    "logid": "30fd25e0-0130-4ab1-a375-fabf1d57430e",
    "status": 403,
    "type": "invalid-parameters",
    "invalid_parameters": {
       "name": "foobar",
       "reason": "should be an integer but received a string"
    }
}

MissingAPIKey¶

HTTP/1.1 403
{
    "_message": "API key not present",
    "_status": "NOK",
    "logid": "30fd25e0-0130-4ab1-a375-fabf1d57430e",
    "status": 403,
    "type": "missing-api-key"
}

siebatch chfetch - Fetch Channel Data¶

Download data associated with a given channel.

As mentioned in chdetails, the earliest data available is pruned frequently, so you are not guaranteed to retrieve that data if you request it.

The seconds portion of the start/end times are ignored, but for API consistency, we require seconds to be given. This means that if you ask for "12:10:30" through "12:45:30", you get "12:10:00" through "12:45:00".

If no data is available then an HTTP 404 is returned - otherwise it returns 200, sets the appropriate HTTP headers, and the download begins.

POST¶

https://batch.sie-remote.net/siebatchd/v1/siebatch/chfetch

Example cURL¶

APIKEY=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx

curl -d '{"apikey":"'$APIKEY'","channel":213,"start_time":"2020-01-28 12:00:00","end_time":"2020-01-28 12:01:00"}' \
  -H "Content-Type: application/json" \
  -X POST https://batch.sie-remote.net/siebatchd/v1/siebatch/chfetch

Example Request¶

{
   "apikey": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
   "channel": 213,
   "start_time": "2020-01-28 12:00:00",
   "end_time": "2020-01-28 12:01:00"
}

Request Parameters¶

Field	Type	Description
apikey	String	Your Farsight SIE Batch API Key
channel	Integer	Channel Number
start_time	String	A Start Date, "YYYY-mm-dd HH:MM:SS" in UTC
end_time	String	An End Date, "YYYY-mm-dd HH:MM:SS" in UTC

Error 4xx 5xx¶

Name	Type	Description
_status	String	NOK
_message	String	A short description of what went wrong.
status	Integer	HTTP status code
type	String	A hyphenated mnemonic for the error. Use this to key on a local i18n translation.
detail	String	Optional String that contains more details about the specific error.

SessionFailure¶

HTTP/1.1 403
{
    "_message": "<Message will contain details>",
    "_status": "NOK",
    "logid": "30fd25e0-0130-4ab1-a375-fabf1d57430e",
    "status": 403,
    "type": "session-failure"
}

InvalidParameters¶

HTTP/1.1 403
{
    "_message": "<message will be specific wrt to which parameter(s) is missing>",
    "_status": "NOK",
    "logid": "30fd25e0-0130-4ab1-a375-fabf1d57430e",
    "status": 403,
    "type": "invalid-parameters",
    "invalid_parameters": {
       "name": "foobar",
       "reason": "should be an integer but received a string"
    }
}

MissingAPIKey¶

HTTP/1.1 403
{
    "_message": "API key not present",
    "_status": "NOK",
    "logid": "30fd25e0-0130-4ab1-a375-fabf1d57430e",
    "status": 403,
    "type": "missing-api-key"
}

siebatch makeurl - Make URL¶

Generate a URL to download data associated with a channel later or independently of the SIE Batch webserver.

As mentioned in chdetails, the earliest data available is pruned frequently, so you are not guaranteed to retrieve it if it is about to be pruned.

The seconds portion of the start/end times are ignored, but for API consistency, we require seconds to be given. This means that if you ask for "12:10:30" through "12:45:30", you get "12:10:00" through "12:45:00".

Using makeurl as opposed to chfetch is advantageous because generated URLs are more easily shared and accessed later on different computers and by fellow users, independently of the user that originally requested them and without revealing credentials. Generated URLs are the preferred method for sharing snippets of channel data.

Data Availability¶

The server retains a rolling window of data per channel, typically 12-18 hours depending on the channel's data rate. Use chdetails to check a channel's earliest and latest timestamps before generating a URL.

The makeurl call itself succeeds even if the requested time range extends beyond the retention window. The error occurs when you download from the returned URL:

HTTP Status	Error	Cause
404	NoSuchKey	Requested data is outside the channel's retention window.
403	Unable to authenticate	The generated URL has expired. Generate a new one.

POST¶

https://batch.sie-remote.net/siebatchd/v1/siebatch/makeurl

Example cURL¶

APIKEY=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx

curl -d '{"apikey":"'$APIKEY'","channel":213,"start_time":"2020-01-28 12:00:00","end_time":"2020-01-28 12:01:00"}' \
  -H "Content-Type: application/json" \
  -X POST https://batch.sie-remote.net/siebatchd/v1/siebatch/makeurl

Example Request¶

{
   "apikey": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
   "channel": 213,
   "start_time": "2020-01-28 12:00:00",
   "end_time": "2020-01-28 12:01:00"
}

Request Parameters¶

Field	Type	Description
apikey	String	Your Farsight SIE Batch API Key
channel	Integer	Channel Number
start_time	String	A Start Date, "YYYY-mm-dd HH:MM:SS" in UTC
end_time	String	An End Date, "YYYY-mm-dd HH:MM:SS" in UTC

Success 200¶

Field	Type	Description
_status	String	OK
_message	String	"" (empty if no message)
url	String	The presigned shareable URL to the specified SIE data

Success-Response¶

HTTP/1.1 200 OK

{"_status": "OK",
 "_message": "",
 "url": "https://batch-dl.sie-remote.net/range/ch213/20200128.1200.20200128.1201?
 X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=..."
}

Error 4xx 5xx¶

Name	Type	Description
_status	String	NOK
_message	String	A short description of what went wrong.
status	Integer	HTTP status code
type	String	A hyphenated mnemonic for the error. Use this to key on a local i18n translation.
detail	String	Optional String that contains more details about the specific error.

SessionFailure¶

HTTP/1.1 403
{
    "_message": "<Message will contain details>",
    "_status": "NOK",
    "logid": "30fd25e0-0130-4ab1-a375-fabf1d57430e",
    "status": 403,
    "type": "session-failure"
}

InvalidParameters¶

HTTP/1.1 403
{
    "_message": "<message will be specific wrt to which parameter(s) is missing>",
    "_status": "NOK",
    "logid": "30fd25e0-0130-4ab1-a375-fabf1d57430e",
    "status": 403,
    "type": "invalid-parameters",
    "invalid_parameters": {
       "name": "foobar",
       "reason": "should be an integer but received a string"
    }
}

MissingAPIKey¶

HTTP/1.1 403
{
    "_message": "API key not present",
    "_status": "NOK",
    "logid": "30fd25e0-0130-4ab1-a375-fabf1d57430e",
    "status": 403,
    "type": "missing-api-key"
}