SIE Batch API Reference

Conventions¶

Every response from the SIE Batch API will contain two status indicating keys.

The first is _status which is a string that will contain either OK or NOK indicating whether the API was successful or not. The second field is _message which will contain a short description of what went wrong. If the _status is OK, then _message will be present but empty.

If _status is NOK then additional fields giving details of the exception will be present, conforming to RFC 7807.

Exception response examples are provided for each API below where appropriate.

Authentication¶

Validate - Validate an API Key¶

Validates a given API Key, returns profile information associated with it.

POST¶

https://batch.sie-remote.net/siebatchd/v1/validate

Permission: public

Example cURL¶

APIKEY=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
curl -d '{"apikey":"'$APIKEY'"}' https://batch.sie-remote.net/siebatchd/v1/validate

Example JS XMLHttpRequest¶

const apikey = "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx";

let xhr = new XMLHttpRequest();
xhr.open("POST", "https://batch.sie-remote.net/siebatchd/v1/validate");
xhr.send(JSON.stringify({apikey: apikey}));

xhr.onload = function() {
  do_something();
};

xhr.onprogress = function(event) {
  do_something_else();
};

xhr.onerror = function() {
  do_something_error();
};

Example JS React, Axios¶

const apikey = "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx";

axios({
  url: "https://batch.sie-remote.net/siebatchd/v1/validate",
  method: "POST",
  data: {
    apikey: apikey,
  },
}).then(
  response => {
    if(response.status === 200) {
      do_something();
    } else {
      do_something_else();
    }
  },
  error => {
    do_something_error();
  }
);

Example Object¶

{
  "apikey": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
}

Parameter¶

Field	Type	Description
profile	Dictionary	A dictionary containing user profile information (provisioning). The following keys are only some of the possible keys available.
username	String	The username associated with your API Key.
siebatch	Dictionary	The SIE batch channels you have access to.
_status	String	OK
_message	String	An account status message ("" [empty] if no message).

Success-Response¶

HTTP/1.1 200 OK

{
  "profile": {
    "username": "siebatch-customer",
    "siebatch": {
      "ch212": {
        "description": "Newly Observed Domains"
      },
      "ch213": {
        "description": "Newly Observed Fully Qualified Domain Names"
      },
    }
  },
  "_status": "OK",
  "_message": ""
}

Error 4xx 5xx¶

Field	Type	Description
apikey	String	Your Farsight SIE Batch API Key
channels	List	List (array) of one or more channel numbers (integers). Valid channels are 1-255.

MissingAPIKey¶

HTTP/1.1 403
{
    "_message": "API key not present",
    "_status": "NOK",
    "logid": "30fd25e0-0130-4ab1-a375-fabf1d57430e",
    "status": 403,
    "type": "missing-api-key"
}

Batched Channel Access¶

siebatch chdetails - Get Channel Details¶

Request the details of a channel such as possible datetime ranges, download filetype, and the size of data available.

These details will update as data is stored and pruned on the server every minute; channel data is always in motion so the details represent a moving window of data that is available for download. This means that the size and earliest/latest times will change frequently. Because channels are continuously recording new data it's not possible to always be up to date if you were to repeatedly request the latest "earliest" and "latest" details for a channel; it's an archival reference only.

POST¶

https://batch.sie-remote.net/siebatchd/v1/siebatch/chdetails

Permission: user

Example cURL¶

APIKEY=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx

curl -d '{"apikey":"'$APIKEY'","channels":[212, 213]}' \

-X POST https://batch.sie-remote.net/siebatchd/v1/siebatch/chdetails

Example JS XMLHttpRequest¶

const apikey = "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx";

let xhr = new XMLHttpRequest();
xhr.open("POST", "https://batch.sie-remote.net/siebatchd/v1/siebatch/chdetails");
xhr.send(JSON.stringify({apikey: apikey, channels: [212, 213]}));

xhr.onload = function() {
  do_something();
};

xhr.onprogress = function(event) {
  do_something_else();
};

xhr.onerror = function() {
  do_something_error();
};

Example React, Axios¶

const apikey = "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx";

axios({
  url: "https://batch.sie-remote.net/siebatchd/v1/siebatch/chdetails",
  method: "POST",
  data: {
    apikey: apikey,
    channels: [212, 213]
  },
}).then(
  response => {
    if(response.status === 200) {
      do_something();
    } else {
      do_something_else();
    }
  },
  error => {
    do_something_error();
  }
);

Example Object¶

{
   "apikey": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
   "channels": \[212, 213\]
}

Parameter¶

Field	Type	Description
apikey	String	Your Farsight SIE Batch API Key
channels	List	List (array) of one or more channel numbers (integers). Valid channels are 1-255.

Success 200¶

Field	Type	Description
_status	String	NOK
_message	String	A short description of what went wrong.
status	Integer	HTTP status code
type	String	A hyphenated mnemonic for the error. This can be used to key on a local i18n translation.
detail	String	Optional String that contains more details about the specific error.

Success Response Example¶

HTTP/1.1 200 OK

{
  "channels": {
    "ch212": {
      "earliest":"2020-01-28 00:40:00",
      "latest":"2020-01-28 20:06:00",
      "size":83837922,
      "mimetype": "application/x-ndjson"
    },
    "ch213": {
      "earliest":"2020-01-28 00:40:00",
      "latest": "2020-01-28 20:06:01",
      "size":10397567992,
      "mimetype": "application/x-ndjson"
    }
  },
  "\_status": "OK",
  "\_message":""
}

Error 4xx 5xx¶

Name	Type	Description
_status	String	NOK
_message	String	A short description of what went wrong.
status	Integer	HTTP status code
type	String	A hyphenated mnemonic for the error. This can be used to key on a local i18n translation.
detail	String	Optional String that contains more details about the specific error.

SessionFailure¶

HTTP/1.1 403
{
    "_message": "<Message will contain details>",
    "_status": "NOK",
    "logid": "30fd25e0-0130-4ab1-a375-fabf1d57430e",
    "status": 403,
    "type": "session-failure"
}

InvalidParameters¶

HTTP/1.1 403
{
    "_message": "<message will be specific wrt to which parameter(s) is missing>",
    "_status": "NOK",
    "logid": "30fd25e0-0130-4ab1-a375-fabf1d57430e",
    "status": 403,
    "type": "invalid-parameters",
    "invalid_parameters": {
       "name": "foobar",
       "reason": "should be an integer but received a string"
    }
}

MissingAPIKey¶

HTTP/1.1 403
{
    "_message": "API key not present",
    "_status": "NOK",
    "logid": "30fd25e0-0130-4ab1-a375-fabf1d57430e",
    "status": 403,
    "type": "missing-api-key"
}

siebatch chfetch - Fetch Channel Data¶

Download data associated with a given channel.

As mentioned in chdetails, the earliest data available is pruned frequently, so you are not guaranteed to retrieve that data if you request it.

The seconds portion of the start/end times are ignored, but for API consistency, we require seconds to be given. This means that if you ask for "12:10:30" thru "12:45:30" you will get "12:10:00" thru "12:45:00".

If no data is available then an HTTP 404 is returned - otherwise it returns 200, sets the appropriate HTTP headers, and the download begins.

POST¶

https://batch.sie-remote.net/siebatchd/v1/siebatch/chfetch

Permission: User

Example cURL¶

APIKEY=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx

curl \-d '{"apikey":"'$APIKEY'","channel":213,"start\_time":"2020-01-28 12:00:00","end\_time":"2020-01-28 12:01:00"}' \\
\-X POST \[https://batch.sie-remote.net/siebatchd/v1/siebatch/chfetch\](https://batch.sie-remote.net/siebatchd/v1/siebatch/chfetch)

Example JS XMLHttpRequest¶

const apikey = "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx";

let xhr = new XMLHttpRequest();
xhr.open("POST", "https://batch.sie-remote.net/siebatchd/v1/siebatch/chfetch");
xhr.send(JSON.stringify({apikey: apikey, channel: 213, start_time: "2020-01-28 12:00:00",end_time:"2020-01-28 12:01:00"}));

xhr.onload = function() {
  do_something();
};

xhr.onprogress = function(event) {
  do_something_else();
};

xhr.onerror = function() {
  do_something_error();
};

Example JS React, Axios¶

const apikey = "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx";

axios({
  url: "https://batch.sie-remote.net/siebatchd/v1/siebatch/chfetch",
  method: "POST",
  data: {
    apikey: apikey,
    channel: 213,
    start_time: "2020-01-28 12:00:00",
    end_time: "2020-01-28 12:01:00",
  },
}).then(
  response => {
    if(response.status === 200) {
      do_something();
    } else {
      do_something_else();
    }
  },
  error => {
    do_something_error();
  }
);

Example Object¶

{
   "apikey": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
   "channel": 213,
   "start_time": "2020-01-28 12:00:00",
   "end_time": "2020-01-28 12:01:00"
}

Parameter¶

Field	Type	Description
_status	String	NOK
_message	String	A short description of what went wrong.
status	Integer	HTTP status code
type	String	A hyphenated mnemonic for the error. This can be used to key on a local i18n translation.
detail	String	Optional String that contains more details about the specific error.

Error 4xx 5xx¶

Name	Type	Description
_status	String	NOK
_message	String	A short description of what went wrong.
status	Integer	HTTP status code
type	String	A hyphenated mnemonic for the error. This can be used to key on a local i18n translation.
detail	String	Optional String that contains more details about the specific error.

SessionFailure¶

HTTP/1.1 403
{
    "_message": "<Message will contain details>",
    "_status": "NOK",
    "logid": "30fd25e0-0130-4ab1-a375-fabf1d57430e",
    "status": 403,
    "type": "session-failure"
}

InvalidParameters¶

HTTP/1.1 403
{
    "_message": "<message will be specific wrt to which parameter(s) is missing>",
    "_status": "NOK",
    "logid": "30fd25e0-0130-4ab1-a375-fabf1d57430e",
    "status": 403,
    "type": "invalid-parameters",
    "invalid_parameters": {
       "name": "foobar",
       "reason": "should be an integer but received a string"
    }
}

MissingAPIKey¶

HTTP/1.1 403
{
    "_message": "API key not present",
    "_status": "NOK",
    "logid": "30fd25e0-0130-4ab1-a375-fabf1d57430e",
    "status": 403,
    "type": "missing-api-key"
}

siebatch makeurl - Make URL¶

Generate a URL that can be used to download data associated with a channel later or independently of the SIE Batch webserver.

As mentioned in chdetails, the earliest data available is pruned frequently, so you are not guaranteed to retrieve it if it's about to be pruned.

The seconds portion of the start/end times are ignored, but for API consistency, we require seconds to be given. This means that if you ask for "12:10:30" thru "12:45:30" you will get "12:10:00" thru "12:45:00".

Using makeurl as opposed to chfetch is advantageous as generated URLs can be more easily shared and accessed later on different computers and by fellow users, independently of the user that originally requested it to be generated and without credentials being revealed. Generated URLs are the preferred method for sharing snippets of channel data.

The expire keyword allows you to specify how long the URL is valid for. The maximum is 168 hours (1 week), the minimum is 1 hour. If you specify an expiration that does not make sense then a 404 will be returned. For example, if a channel collects the last 12 hours of data and you request a URL with an expiration of 13 hours then no data would be available for the final hour that the URL is valid. As a rule of thumb, the maximum requested expiration time should be no more than chXXX["latest"] - chXXX["earliest"] where chXXX is the channel you're generating URLs for.

POST¶

https://batch.sie-remote.net/siebatchd/v1/siebatch/makeurl

Permission: user

Example cURL¶

APIKEY=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx

curl -d '{"apikey":"'$APIKEY'","channel":213,"start_time":"2020-01-28 12:00:00","end_time":"2020-01-28 12:01:00"}' \
-X POST https://batch.sie-remote.net/siebatchd/v1/siebatch/makeurl

Example JS XMLHttpRequest¶

const apikey = "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx";

let xhr = new XMLHttpRequest();
xhr.open("POST", "https://batch.sie-remote.net/siebatchd/v1/siebatch/makeurl");
xhr.send(JSON.stringify({apikey: apikey, channel: 213, start_time: "2020-01-28 12:00:00",end_time:"2020-01-28 12:01:00"}));

xhr.onload = function() {
  do_something();
};

xhr.onprogress = function(event) {
  do_something_else();
};

xhr.onerror = function() {
  do_something_error();
};

Example JS React, Axios¶

const apikey = "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx";

axios({
  url: "https://batch.sie-remote.net/siebatchd/v1/siebatch/makeurl",
  method: "POST",
  data: {
    apikey: apikey,
    channel: 213,
    start_time: "2020-01-28 12:00:00",
    end_time: "2020-01-28 12:01:00"
  },
}).then(
  response => {
    if(response.status === 200) {
      do_something();
    } else {
      do_something_else();
    }
  },
  error => {
    do_something_error();
  }
);

Example Object¶

{
   "apikey": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
   "channel": 213,
   "start_time": "2020-01-28 12:00:00",
   "end_time": "2020-01-28 12:01:00"
}

Parameter¶

Field	Type	Description
_status	String	OK
_message	String	"" (empty if no message)
url	String	The presigned shareable URL to the specified SIE data

Success 200

Field	Type	Description
_status	String	OK
_message	String	"" (empty if no message)
url	String	The presigned shareable URL to the specified SIE data

Success-Response¶

HTTP/1.1 200 OK

{"_status": "OK",
 "_message": "",
 "url": "https://batch-dl.sie-remote.net/range/ch213/20200128.1200.20200128.1201?
 X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=..."
}

Error 4xx 5xx¶

Name	Type	Description
_status	String	NOK
_message	String	A short description of what went wrong.
status	Integer	HTTP status code
type	String	A hyphenated mnemonic for the error. This can be used to key on a local i18n translation.
detail	String	Optional String that contains more details about the specific error.

SessionFailure¶

HTTP/1.1 403
{
    "_message": "<Message will contain details>",
    "_status": "NOK",
    "logid": "30fd25e0-0130-4ab1-a375-fabf1d57430e",
    "status": 403,
    "type": "session-failure"
}

InvalidParameters¶

HTTP/1.1 403
{
    "_message": "<message will be specific wrt to which parameter(s) is missing>",
    "_status": "NOK",
    "logid": "30fd25e0-0130-4ab1-a375-fabf1d57430e",
    "status": 403,
    "type": "invalid-parameters",
    "invalid_parameters": {
       "name": "foobar",
       "reason": "should be an integer but received a string"
    }
}

MissingAPIKey¶

HTTP/1.1 403
{
    "_message": "API key not present",
    "_status": "NOK",
    "logid": "30fd25e0-0130-4ab1-a375-fabf1d57430e",
    "status": 403,
    "type": "missing-api-key"
}