Skip to content

Base configuration

Configure the essential settings to start enriching your Splunk data with threat intelligence.

Configuration

The following sections provide details to help configure the application to provide the most value in your environment.

The base search is a Splunk Search Processing Language (SPL) query that defines which log sources the DomainTools App monitors. SPL is Splunk's query language for searching and analyzing data. The base search outputs the required fields the DomainTools App uses to populate dashboards and enrich events.

The app includes a pre-configured, performance-optimized base query. This query works well in environments where data sources follow the Common Information Model (CIM). CIM is Splunk's standard for normalizing data from different sources into a consistent format.

To configure the base search:

  1. Visit DT Settings → Configure Log Source.
  2. Configure the required fields: url, src, dest, log_source, domain, and _time.
Expand for detailed instructions on expanding the base search

First, identify relevant CIM-compliant data sources for ingestion. For example, you might use data sources already configured to capture web proxy events. If your data sources aren't CIM-compliant, go to configuring the base search with custom SPL.

To use the pre-configured query:

  1. Visit DT Settings → Configure Log Source page.

  2. Confirm the details of the pre-configured Splunk search query.

    The app natively supports logs with multivalue URLs contained in a single event entry. This is commonly seen in Proofpoint logs. Users who previously used mvexpand url to work around this issue can remove that command. This reports the URLs more accurately as a single event.

    A regex-based dtdomainextract2 macro is available for high-throughput environments. It can significantly increase performance of the URL-to-domain conversion with a slight trade-off in accuracy. Some wildcard and exception top-level domains (TLDs) like *.np or \!city.kawasaki.jp can be misidentified as a domain. Note that mvexpand url would still be needed with dtdomainextract2 in environments processing multivalue URLs. The updated base search would look like this:

    tstats summariesonly=true count FROM datamodel=Web BY Web.url Web.src Web.dest source _time | rename Web.url AS url | rename Web.src AS src | rename Web.dest AS dest | rename source AS log_source | mvexpand url | `dtdomainextract2` | eval domain=lower(domain) | fields url src dest log_source domain _time

  3. If needed, customize the pre-configured base search to ensure the required fields are available.

  4. Select Save.

  5. Select the Timeframe for the Base Search.

    Base search requirements and recommendations:

    • You need domain and _time as output fields to operate the app.
    • We recommend adding the optional fields Source, Destination, and Log Source in your base search. They provide additional contextual information on the events. The app won't error out if these fields are missing.

    Performance considerations:

    • Turn on acceleration for the Web data model. Acceleration pre-computes search results to speed up queries.
    • If acceleration isn't turned on but you mapped data to the CIM, you can modify the base search to use summariesonly=false. This may reduce performance.

    If you already extracted domains in the CIM, you can remove the dtdomainextract function from your base search. This will improve performance.

Expand for detailed instructions on configuring the base search with custom SPL

You may need to configure Base Search using your own custom Search Processing Language query (SPL) if data isn't yet CIM compliant, or input data sources are from ingested data from several different sources.

First, identify relevant data sources for ingestion. It may contain URLs or hostnames, in addition to domain names. This app doesn't support IP addresses for processing.

The DomainTools solution provides data on domain names, not IPs, subdomains, or full URLs. IPs sent to the Iris APIs won't return useful data and consume rate-limited resources. Querying a URL will result in inconsistent results. The default base search converts URLs to apex domains for the enrichment.

To configure using custom SPL:

  1. Craft the Splunk search query that efficiently finds events from your preferred data source(s).
  2. You don't need to write regular expressions or other parsing rules to extract domains in logs filled with hostnames or URLs or de-duplicate logs. DomainTools queue builder search jobs handle this task.
  3. We recommend using tstats. We also use it in the pre-configured base search for optimization.
    1. If you don't use the default tstats base search, there's a known issue in clustered SH environments for both Splunk 7.x and Splunk 8.x; consult the workaround and known issue section and Splunk 8.1.0 known issues documentation.
  4. Add the query as the base search via DT Settings → Configure Log Source page.
  5. Add the query in the Base Search input field and select Save.

Your custom SPL must meet these criteria:

  • If your custom search doesn't begin with tstats you must add search to ensure proper functionality once it's merged into the DomainTools scheduled searches.
  • Don't start with a pipe | character.
  • Ensure the result contains a field named domain or use rename function.
  • The search must efficiently return results from the last 10 minutes of events. Ideally in a few seconds, but no longer than two or three minutes.

Example:

search index=mycustomindex | rename url AS domain

The query should be performant in your environment. At a minimum, the query should return ten minutes of events in no more than two minutes of search execution time, with minimal impact on your search head or search head cluster.

Tradeoffs: Pre-Configured and Custom SPL for Base Search

Ensure to select the correct base search method for your environment before proceeding with the installation.

Pros:

  • No post-install customization or configuration required, other than the API username and key initial app setup
  • Finds domain names in every CIM-compliant data source from the Web data model
  • Fast for most environments

Cons:

  • Requires CIM compliant data sources that use the Web data model with hostnames in the Web.url field
  • DomainTools Threat Hunting Dashboard will be empty if the base search can't find domains
  • You must turn on acceleration for the Web data model

Pros:

  • Doesn't require your data source to be CIM compliant
  • You can optimize it to your environment and data sources

Cons:

  • May cause performance problems if you don't manually optimize the search
  • May require additional tuning after installation, making this option unsuitable for rigorous change management cycles
  • Must return the hostname or domain name in a field explicitly named domain

Configure and add saved searches

Saved Searches (Table: Saved Searches) automate some operational tasks within Splunk.

Manage saved searches in DT Settings --> Configure Saved Searches.

Upon clicking Test Connection after entering your API key, if the "Queue Builder for Iris Enrich KV Store" saved search is turned off, you're prompted to turn on the default set of saved searches. Selecting enable turns on the set of seven minimum required saved searches for the Core App functionality noted in the saved searches table.

To turn on Iris Investigate and Detect capabilities or alerting in Splunk Enterprise Security, turn on the saved searches outlined in the table.

To enable a Saved Search, select Edit, and select Edit Schedule.

Enable mass enrichment

Leave the current settings as default. Visit DT Settings → Configure Enrichment and Alerting to change these settings.

The Queue Wait Time is how often the app enriches domain information, and defaults to 5 minutes. Decrease the frequency to reduce API usage or if the enrichment takes longer than 5 minutes to run on a higher volume Splunk cluster.

The Cache Settings control the cache that DomainTools maintains to reduce API query usage. Turn off or reduce the cache retention times (for example, when monitoring volatile domains) in Cache Settings:

  • DomainTools maintains a cache to reduce API query usage. Turn off or reduce the cache retention period when monitoring volatile domains.
  • Enable Cache: Enabled by default to optimize API consumption. Turn off the cache to monitor for changes less than 1 day old. (CAUTION: this can result in high API consumption.)
  • Add the Cache Retention Period: Sets how long domain enrichment should live in the cache before being re-queried. 30 days is the default.

Enable enrichment history

The DomainTools App supports enrichment history with the custom dt_enrich_history index. This index stores all Enrich values for all domains. When a domain is re-enriched, this index records the new values. The index can be used to compare changes in values across WHOIS info, IPs, SSL fields, risk scores, and other indicators. Indexes including dt_enrich_history can be created and configured in Configure DT Indexes.

Next steps

After you configure the base settings, proceed to:

  • Advanced Features - Enable threat feeds, Iris Detect, alerts, and more
  • Dashboards - Explore the threat intelligence and monitoring dashboards