Configuration tables and troubleshooting

This reference provides detailed configuration tables, troubleshooting guidance, and historical release notes for the DomainTools App for Splunk.

Configuration tables

Main configuration files

These configuration files are relevant for using the app and DomainTools datasets.

Note: The configuration files are relevant for this version only. The configuration files, stanzas, and fields are different in other versions.

Conf File	Stanza Tag	Fields	Description
app.conf	package	id	Add details for the Splunk App.
	install	is_configured
	ui	is_visible, label
	launcher	author, description, version
commands.conf	dtaccountinfo	chunked, filename	These are helper commands for the app. The most commonly used commands outside the app are described in greater detail in the Commands Reference and in the in-app documentation. The chunked parameter indicates the search command supports Splunk's "chunked" custom protocol, used by all of these stanzas. The filename parameter indicates the location of the Python .py filenames for these commands.
	dtimportirisdetectmonitors	chunked, filename
	dtimportirisdetectresults	chunked, filename
	dtirisdetectescalate	chunked, filename
	dtirisdetectchangestate	chunked, filename
	dtsyncirisdetectwatchlist	chunked, filename
	dtirisinvestigate	chunked, filename
	dtirisenrich	chunked, filename
	dtformatenrich	chunked, filename
	dtformatinvestigate	chunked, filename
	dtexpirecache	chunked, filename
	dtdomainextract	type, filename, streaming, local, passauth, chunked	The template to modify the DomainTools domainextract function to use Splunk SDK SCP1 if the latest SDK faces throughput issues.
	dtdnsdb	filename, retainevents, supports_multivalues, streaming, overrides_timeorder, passauth	Queries DNSDB for Passive DNS information for a given IP, Domain, Hostname, or Subnet.
	dtdnsdbflex	filename, retainevents, supports_multivalues, streaming, overrides_timeorder, passauth	Performs a DNSDB Passive DNS Flexible Search.
	dtdnsdbenrich	chunked, filename	Enriches Splunk events returned by a given SPL_QUERY with Passive DNS information reported by Farsight DNSDB, part of DomainTools.
	validateip	filename, retainevents, supports_multivalues, streaming, overrides_timeorder
	dtdnsdblimit	filename, retainevents, supports_multivalues, streaming, overrides_timeorder, passauth	Returns the DNSDB API query limit, number of queries remaining, and the time when the remaining queries will reset.
	flushcache	filename, retainevents, supports_multivalues, streaming, overrides_timeorder, passauth
searchbnf.conf	dtaccountinfo-command	syntax, shortdesc, usage, comment1, example1	The syntax (shorter name), description, and whether the usage is public.
	dtirisinvestigate-command	syntax, shortdesc, usage, comment1, example1, comment2, example2, related
	dtirisdetectmonitors-command	syntax, shortdesc, usage, comment1, example1
	dtsyncirisdetectwatchlist-command	syntax, shortdesc, usage, comment1, example1
	dtirisenrich-command	syntax, shortdesc, usage, comment1, example1, comment2, example2, related
	dtformatinvestigate-command	syntax, shortdesc, usage, comment1, example1, related
	dtformatenrich-command	syntax, shortdesc, usage
	dtdomainextract-command	syntax, shortdesc, usage, comment1, example1, comment2, example2
	dtexpirecache-command	syntax, shortdesc, usage, comment1, example1
	dtdnsdb-command	syntax, description, shortdesc, example1, example2, example3, usage
	dtdnsdbflex-command	syntax, shortdesc, example1, example2, example3, usage
	dtdnsdbenrich-command	syntax, description, shortdesc, example1, example2, example3, usage
	dtdnsdblimit-command	syntax, description, shortdesc, example1, usage
server.conf	shclustering	conf_replication_include.domaintools	Default value is set to true.
transforms.conf	dt_iris_enrich_queue	external_type, collection, fields_list, case_sensitive_match	These are KV store fields. Please see the KV Store table for the array of fields_list for each stanza.
	dt_iris_enrich_data	external_type, collection, fields_list, case_sensitive_match
	dt_stats	external_type, collection, fields_list, case_sensitive_match
	dt_allowlist	external_type, collection, fields_list
	dt_monitoring_list	external_type, collection, fields_list
	dt_tags_list	external_type, collection, fields_list
	dt_iris_detect_monitors	external_type, collection, fields_list
	dt_iris_detect_results	external_type, collection, fields_list
	dt_iris_investigate	external_type, collection, fields_list
	dt_public_suffix_list	filename, match_type, max_matches	Needed for macro dtdomainextract2
domaintools.conf	domaintools	proxy_enabled	Use a proxy when connecting to the DomainTools API. To enable, set to 1.
		proxy_server	The proxy server address to use.
		proxy_port	The proxy server port to use.
		ssl_enabled	Use SSL when connecting to the DomainTools API. To enable, set to 1.
		custom_certificate_enabled	Use a custom SSL certificate for the SSL connection. To enable set to 1.
		custom_certificate_path	The path to the custom SSL certificate.
		guided_pivot_threshold	The Guided Pivot Threshold on the Domain Profile page. Set a lower value to narrow investigations. 500 is the default and recommended value.
		bulk_enrichment_batch_size	Number of domains batched in an API call. Set the value from 1 to 100.
		optimize_enrichment_searches	This setting enables quicker correlation of cached data of known domains from the Enrichment table. Requires additional disk space. Disabling will reduce disk space consumption but will slow down searches. Set 1 to enable.
		populate_scores	Checks for whether or not to use the Risk Score over lower tiered scores. Turned off (set to 0) by default.
		logging_on	Toggles whether or not to write logs to file.
macros.conf	See Key Macros table
savedsearches.conf	See Saved Searches table
collections.conf	See KV Store table
distsearch.conf	replicationWhitelist	domainextract	Path to domainextract custom search command to copy to indexers
		lib	Path to python libs to copy to indexers
workflow_actions.conf		dt_iris_lookup	Lookup domain using Iris Investigate
		dt_domain_profile	Lookup domain using Domain Profile
		dt_dnsdb	Lookup passive dns using Farsight pDNS Standard Search

KV store names and fields

KV Store/ Collection Name	Fields
dt_iris_enrich_queue	_key, domain, queued, observed
dt_iris_enrich_data	_key,_raw, dt_queued, dt_retrieved, dt_observed, en_domain_name, en_is_active, en_adsense_code, en_google_analytics_code, en_alexa_ranking, en_domain_create_date, en_domain_updated_timestamp, en_domain_expiration_date, en_tld, en_website_response_code, en_redirect_url, en_registrant_name, en_registrant_org, en_registrar, en_spf_info, en_additional_whois_email, en_additional_soa_email, en_additional_ssl_raw, en_ssl_info_1_hash, en_ssl_info_1_organization, en_ssl_email, en_ssl_info_1_subject, en_risk_score, en_proximity_score, en_threat_profile_type, en_threat_profile_malware, en_threat_profile_phishing, en_threat_profile_spam, en_threat_profile_evidence, en_additional_name_servers_raw, en_name_server_1_domain, en_name_server_1_host, en_name_server_1_ip, en_name_server_2_domain, en_name_server_2_host, en_name_server_2_ip, en_additional_mx_raw, en_mx_1_domain, en_mx_1_host, en_mx_1_priority, en_mx_1_ip, en_additional_ips_raw, en_ip_1_address, en_ip_1_country_code, en_ip_1_isp, en_ip_1_asn, en_ip_2_address, en_ip_2_country_code, en_ip_2_isp, en_ip_2_asn, en_admin_contact_city, en_admin_contact_country, en_admin_contact_fax, en_admin_contact_name, en_admin_contact_org, en_admin_contact_phone, en_admin_contact_postal, en_admin_contact_state, en_admin_contact_street, en_admin_contact_email, en_billing_contact_city, en_billing_contact_country, en_billing_contact_fax, en_billing_contact_name, en_billing_contact_org, en_billing_contact_phone, en_billing_contact_postal, en_billing_contact_state, en_billing_contact_street, en_billing_contact_email, en_technical_contact_city, en_technical_contact_country, en_technical_contact_fax, en_technical_contact_name, en_technical_contact_org, en_technical_contact_phone, en_technical_contact_postal, en_technical_contact_state, en_technical_contact_street, en_technical_contact_email, en_registrant_contact_city, en_registrant_contact_country, en_registrant_contact_fax, en_registrant_contact_name, en_registrant_contact_org, en_registrant_contact_phone, en_registrant_contact_postal, en_registrant_contact_state, en_registrant_contact_street, en_registrant_contact_email, en_tag, en_tag_raw
dt_stats	_key, dt_last_enriched_datetime, dt_num_of_times_enriched, dt_num_of_AdhocLookups, dt_fooyn_timestamp, dt_looyn_timestamp, en_attribute_name, en_attribute_type, en_risk_score
dt_allowlist	_key, en_attribute_type,_dt_updated,_dt_updated_by, _dt_created, _dt_created_by
dt_monitoring_list	_key, en_attribute_type,_dt_updated,_dt_updated_by, _dt_created, _dt_created_by,_dt_source
dt_tags_list	_key, en_attribute_type,_dt_updated,_dt_updated_by, _dt_created, _dt_created_by
dt_iris_detect_monitors	_key, monitor_id, term, state, match_substring_variations, nameserver_exclusions, text_exclusions, created_date, updated_date, status, created_by, discover_new_domains, dt_updated
dt_iris_detect_results	_key, dt_domain, dt_state, dt_status, dt_discovered_date, dt_escalations, dt_risk_score, dt_risk_status, dt_mx_exists, dt_tld, dt_domains_id, dt_monitor_ids, dt_create_date, dt_ip_address_1, dt_ip_address_2, dt_ip_raw, dt_nameServer_1, dt_nameServer_2, dt_nameServer_raw, dt_mailServer_1, dt_mailServer_2, dt_mailServer_raw, dt_registrar, dt_registrant_contact_email, dt_proximity_score, dt_threat_profile_malware, dt_threat_profile_phishing, dt_threat_profile_spam, dt_threat_profile_evidence, dt_monitor_flag, dt_imported
dt_iris_investigate	_key, dt_pivot_type, dt_pivot_value, dt_investigate_raw,_dt_created
dt_rrset_kvstore
dt_rdata_kvstore

Key macros for enrichment

Macro Field Name	Default Value	Description
dt_basesearch		The defined value is the base search. The system pulls data directly from the datamodel. We use this search to search for and queue up domains for the app and certain features such as the dashboards.
enable_cache	1 (enabled)	Enrichment setting to determine caching of enriched data. DomainTools will always enrich every domain in the queue. When turned off (set to 0), the system makes an API call for every domain.
dt_cache_retention_period	30 (in days)	Enrichment setting. Set the value to how many days back before removing older data from the enrichment kvstore. There is also a saved search that will remove records that are over 30 days old.
dt_proximity_score_threshold	65	Enrichment setting. Set the threshold throughout the app when filtering based on the Proximity score.
dt_threat_profile_score_threshold	85	Enrichment setting. Set the threshold throughout the app when filtering based on the Threat Profile score.
dt_high_risk_threshold	90	Enrichment setting. Set the threshold throughout the app.
dt_medium_risk_threshold	70	Enrichment setting. Set the threshold throughout the app.
dt_refresh_interval	15 (in minutes)	The refresh interval.
dtdomainextract2	See note below table	Alternative to dtdomainextract that does regular expression-based matching for TLDs. It is higher performance for high-throughput environments, with a small accuracy trade-off. Notably, some multi-level tlds (for example, edu.np) can be mis-identified as a domain.
dt_risk_score_threshold	75	Enrichment setting. Set the threshold throughout the app when filtering based on the Risk Score.
dt_young_domain_age	7 (in days)	Enrichment setting. The number of days the app considers a domain to be young.
dt_include_allowlisted_domains	0 (false)	Allowlist setting. Set to 1 (enabled) to exclude showing domains in the allowlist in our dashboards.
dt_include_monitoring_list_domains	0 (false)	Setting to include monitoring list domains.
dt_enrich_to_stats_lookup		A partial search that the saved searches use to update the enriched data KV Store.
dt_include_allowlisted_domains_in_notable_events	0 (false)	Enrichment alert setting for notable events.
dt_only_monitored_domains_in_notable_events	1 (enabled)	Enrichment alert setting for notable events.
dt_use_risk_threshold_in_notable_events	0 (false)	Enrichment alert setting for notable events.
dt_use_threatprofile_threshold_in_notable_events	0 (false)	Enrichment alert setting for notable events.
dt_ignore_iris_detect_in_notable_events	0 (false)	Enrichment alert setting for notable events.
dt_monitor_tags_in_notable_events	0 (false)	Enrichment alert setting for notable events.
dt_notable_events		Search for notable events provided by the DomainTools App for Splunk ES.
dt_rename_base_fields		Renames the base search fields. For example, rename src to Source, dest as Destination, log_source as Log Source and domain as Domain Name.
dt_rename_iris_fields
unknown_domain_retry	1 (enabled)	Retry enrichment of domains that are unknown to DomainTools.
unknown_domain_retry_time	60 (in minutes)	Number of minutes to wait before trying to re-enrich a domain.
toEpoch(1)	if(isnull(round(relative_time(time(), "$reltime$"))), "$reltime$", round(relative_time(time(), "$reltime$")))	Changes timestamp to epoch.

dtdomainextract2 default value:

rex field=url "(.*:\/\/)?(?P<temp_domain>[^:#\/?]+)" \
| lookup dt_public_suffix_list wildcard_tld AS temp_domain OUTPUT tld AS tld \
| where match(temp_domain, "(.*[.|@])?([\p{L}\w-]+[.]".tld."$)") \
| eval domain = replace(temp_domain, "(.*[.|@])?([\p{L}\w-]+[.]".tld."$)", "\2")

Saved searches

Access saved searches in DT Settings → Configure Saved Searches.

Name	Type	Description	Required	App Functions
DomainTools - DNSDB Cache Cleanup	Reports	Clears all cached responses that are over a day old	Required for DNSDB	DNSDB
DomainTools - Expire Old Iris Detect Data	Reports	Removes domains imported more than 14 days ago. Default cron_schedule = 0 0 * * *	Required for Iris Detect	Iris Detect
DomainTools - Expire Old Iris Enrich Data	Reports	Removes Iris Enrich enrichment data from the dt_iris_enrich_data collection based on the cache retention settings configured in the app. Default cron_schedule = 0 0 * * *	Required for Iris enrichment	Core App
DomainTools - Expire Old Pivot Data	Reports	Removes Iris Investigate pivot results older than 24 hours.	Required to use the pivot feature through the Domain Profile dashboard	Core App
DomainTools - Expire Old Queue Data	Reports	Removes Iris Enrich domains from the dt_iris_enrich_queue collection that are over a day old. Default cron_schedule = 0 0 * * *	Required for Iris enrichment	Core App
DomainTools - Feed - Domain Discovery	Reports	Retrieves results from the real-time Domain Discovery feed	Optional: Required for the Domain Discovery feed	Domain Discovery feed
DomainTools - Feed - NAD	Reports	Retrieves results from the real-time Newly Active Domains feed	Optional: Required for the Newly Active Domains feed	Newly Active Domains feed
DomainTools - Feed - NOD	Reports	Retrieves results from the real-time Newly Observed Domains feed	Optional: Required for the Newly Observed Domains feed	Newly Observed Domains feed
DomainTools - Feed - RDAP	Reports	Retrieves results from the real-time Parsed Domain RDAP feed	Optional: Required for the Parsed Domain RDAP feed	Parsed Domain RDAP feed
DomainTools - Import Iris Detect Monitors	Reports	Imports newly discovered and watched domains from Iris Detect monitors in the app. Default cron_schedule = 0 0 * * *	Optional: Required for Iris Detect	Iris Detect
DomainTools - Import Iris Detect Results	Reports	Imports newly discovered domains from Iris Detect for monitors enabled in the app. Default cron_schedule = 45 */2 * * *	Optional: Required for Iris Detect	Iris Detect
DomainTools - Iris Enrich History	Report	Saves historical Iris Enrich results	Optional: Required for Iris Detect	Iris Detect
DomainTools - Iris Enrich Monitored Domains	Reports	Refreshes enrichment data for monitored domains based on the frequency configured in the app. Default cron_schedule = 0 0 * * *	Required to enrich monitored domains on a cron schedule	Core App
DomainTools - Iris Enrich Monitored Domains Live	Reports	Refreshes enrichment data for Iris Enrich monitored domains whenever your network sees them. Default cron_schedule = 5 * * * *	Reqired to enrich monitored domains as the Splunk environment (base search) sees them	Core App
DomainTools - Iris Enrichment	Reports	Enriches domains found in dt_iris_enrich_queue and stores results in the dt_iris_enrich_data collection. By default, the search runs every 5 minutes and pulls data over the past 30 minutes. Customize this frequency in the app. Default cron_schedule = */5 * * * *	Required for Iris enrichment*	Core App
DomainTools - Queue Builder for Iris Enrich KV Store	Reports	Extracts domains from raw events based on your configured base search and stores them in the dt_iris_enrich_queue KV store for enrichment. Default cron_schedule = */2 * * * *	Required for Iris enrichment*	Core App
DomainTools - Summary - Timechart count by domain with latest time	Reports	Summarizes events from the base search when the selected time window is greater than 2 hours in any DomainTools dashboard. Default cron_schedule = */5 * * * *	Required for dashboard views of 4h or more	Core App
DomainTools - Sync Iris Detect Watchlist	Reports	Syncs the Iris Detect Watchlist with the DomainTools Monitoring List inside of Splunk. Default cron_schedule = 0 0 * * *	Required for Iris Detect	Iris Detect

* The app will function with "DomainTools - Queue Builder for Iris Enrich KV Store" and "DomainTools - Iris Enrichment" disabled, but won't automatically enrich events. Some customers choose to disable these when building their own enrichment pipelines, using the DomainTools app for ad hoc search or monitoring only.

Alerts

Enable these alerts to create Notable Events from the criteria specified in DT Settings → Configure Enrichment & Alerting.

Name	Type	Description	Required
DomainTools - DomainTools Domain Monitoring - Rule	Alert	Creates events based on enrichment and alerting configuration. Customers wanting to create Notable Events within Enterprise Security must either enable this saved search or enable the correlation search inside Splunk ES. Default cron_schedule = */30 * * * *	Required for alerting at configured thresholds without the Splunk ES correlation search.
DomainTools - DomainTools Iris Detect - Rule	Alert	Creates Iris Detect alerts	Required for alerting at configured thresholds without the Splunk ES correlation search.
DomainTools - DomainTools Risk Score Increase - Rule	Alert	Creates alerts for Risk Score increases	Required for alerting at configured thresholds without the Splunk ES correlation search.
DomainTools - DomainTools Young Domains - Rule	Alert	Creates events based on enrichment and alerting configuration. Default cron_schedule = */30 * * * *	Required for alerting at configured thresholds without the Splunk ES correlation search.

Troubleshooting

Enable logging

Logging is turned off by default. To enable logging to help with diagnostics, go to DT Settings → Diagnostic Panel and select Enable Diagnostic Panel. Allow a few minutes for logs to populate, then refresh the page.

Splunk Cloud configuration during install

The self-service app installation might run into issues installing components required on indexers. The installation proceeds normally, but attempts to run dtdomainextract return an error. This may be due to Splunk self-service only installing apps on search heads.

Check the status of saved searches

Having one or more required saved searches turned off is a common customer issue that could manifest itself as incomplete app functionality.

To check on the status of saved searches:

1. Select the DT Settings menu within the app.
2. Select Configure Saved Searches to load the list of saved searches used by the DomainTools app

Compare the scheduled time on the resulting set of reports against the table of Saved Search Names and Descriptions to ensure the required core app saved searches, as well as the additional ones if required for Iris Investigate or Alerting in Splunk Enterprise Security are enabled.

Non-production environments

Validating the app in non-production environments: if you use a staging environment or development environment to test new Splunk apps, ensure the same data sources you plan to use in production are also available to the Splunk search heads in the test environment.

System architecture

Architecture overview

High-level topology of both Splunk and DomainTools resources

The Saved Searches configuration file (savedsearches.conf) defines the processes for enrichment and the Queue Builder for the Iris Enrich KV store. In the Queue Builder process, the system queries raw logs in the Splunk Indexes from the Web data model as the DomainTools base search configuration (dt_basesearch) defines.

This process includes checking to see if the domain already exists when comparing to existing Iris Enrich data, as that would indicate if DomainTools already enriched the domain. If not, the system queues the new domain for enrichment. The KV store stores each domain with the enriched data.

Domain Enrichment Process between DomainTools and Splunk Indexes

Prior 5.x release notes

Release notes for version 5.3

New in version 5.3

• Support for Newly Active Domains (NAD) and Newly Observed Domains (NOD) Threat Feeds

Fixed in version 5.3

• Public suffix list in Enrichment Summary
• Guided pivots failing for email addresses
• Newly observed domains race condition

Release notes for version 5.2

New in version 5.2

• Enrichment Explorer visualization and filters
• Enrichment dashboard panels: Top ASN; Top Registrars; Top SSL Expired Certificates; Top Nameservers; Top ISPs; Top IP

Updated in version 5.2

• Additional fields in Enrichment Explorer summary view
• Error handling for Iris Investigate
• Allow multi-value inputs to dtirisenrich command

Fixed in version 5.2

• dtwhoishistory documentation
• DNSDB error message specificity

Release notes for version 5.1

Updated in version 5.1

• Improved API usage reporting
• Improved information and responses around required API products

New in version 5.1

• Support for new Iris fields

Release notes for version 5.0

New in version 5.0

• Send alerts sent to any SOAR platform with the new dt_alerts index
• New configurable Risk Score Increase alert from Iris Detect results.
• Log all domain enrichment values, and compare enrich values over time, using the new dt_enrich_history index.
• Track changes to WHOIS data with the dtwhoishistory custom search script
• Iris Investigate and Enrich API responses now include website_title, first_seen and server_type, as well as the SSL fields ssl.alt_names, ssl.duration, ssl.common_name, ssl.issuer_common_name, ssl.not_after, and ssl.not_before. These are available in the domain profile, enrichment explorer, and in Enrich and Investigate custom search commands.