SPLK-2002 Splunk Enterprise Certified Architect Question and Answers

According to the Indexer Clustering Administration Guide, configuration bundles pushed from the Cluster Manager (Master Node) overwrite the contents of the $SPLUNK_HOME/etc/slave-apps/ directory on each search peer (indexer). However, Splunk provides a special persistent location — the _cluster app’s local directory — for files that must survive bundle redeployments.

Specifically, any configuration files placed in:

$SPLUNK_HOME/etc/slave-apps/_cluster/local/

will persist after future bundle pushes because this directory is excluded from the automatic overwrite process.

This is particularly useful for maintaining local overrides or custom configurations that should not be replaced by the Cluster Manager, such as environment-specific inputs, temporary test settings, or monitoring configurations unique to that peer.

Other directories under slave-apps are overwritten each time a configuration bundle is pushed, ensuring consistency across the cluster. Likewise, master-apps exists only on the Cluster Manager and is used for deployment, not persistence.

Thus, the _cluster/local folder is the only safe, Splunk-documented location for configurations that need to survive bundle redeployment.

References (Splunk Enterprise Documentation):

• Indexer Clustering: How Configuration Bundles Work

• Maintaining Local Configurations on Clustered Indexers

• slave-apps and _cluster App Structure and Behavior

• Splunk Enterprise Admin Manual – Cluster Configuration Management Best Practices

According to Splunk’s Search Scheduler and Job Management documentation, the scheduler.log file, located within the _internal index, records the execution of scheduled and saved searches. This log provides a detailed record of when each search is triggered, how long it runs, and its success or failure status.

Each time a scheduled search runs (for example, alerts, reports, or summary index searches), an entry is written to scheduler.log with fields such as:

sid (search job ID)

app (application context)

savedsearch_name (name of the saved search)

user (owner)

status (success, skipped, or failed)

run_time and result_count

By searching the _internal index for sourcetype=scheduler (or directly viewing scheduler.log), administrators can confirm whether a specific saved search executed as expected and diagnose skipped or delayed runs due to resource contention or concurrency limits.

Other internal logs serve different purposes:

metrics.log records performance metrics.

kvstore.log tracks KV Store operations.

btool.log does not exist — btool outputs configuration data to the console, not a log file.

Hence, scheduler.log is the definitive and Splunk-documented source for validating scheduled search activity.

References (Splunk Enterprise Documentation):

• Saved Searches and Alerts – Scheduler Operation Details

• scheduler.log Reference – Monitoring Scheduled Search Execution

• Monitoring Console: Search Scheduler Health Dashboard

• Troubleshooting Skipped or Delayed Scheduled Searches

Data retention is optional in the data source assessment because it is not directly related to the ingestion process. Data retention is determined by the index configuration and the storage capacity of the Splunk platform. Data format, data location, and data volume are all essential information for planning how to collect, parse, and index the data source.

In the context of splunkd.log events written to the _internal index, the field that identifies the specific log channel is the "channel" field. This information is confirmed by the Splunk Common Information Model (CIM) documentation, where "channel" is listed as a field name associated with Splunk Audit Logs​​.

 A Technical Add-On (TA) is a Splunk app that contains configurations for data collection, parsing, and enrichment. It can also enable event data for a data model, which is useful for creating dashboards and reports. Therefore, before installing a TA, it is important to identify the number of scheduled or real-time searches that will use the data model, and to validate if the TA enables event data for a data model. The number of forwarders that the TA can support is not relevant, as the TA is installed on the indexer or search head, not on the forwarder. The installation location of the TA depends on the type of data and the use case, so it is not a fixed requirement

The splunk clean kvstore command is used to clear the KV store. This command will delete all the collections and documents in the KV store and reset it to an empty state. This command can be useful for troubleshooting KV store issues or resetting the KV store data. The splunk clear kvstore, splunk delete kvstore, and splunk reinitialize kvstore commands are not valid Splunk commands. For more information, see Use the CLI to manage the KV store in the Splunk documentation.

The Splunk server name of the member can typically be determined by the serverName attribute in the server.conf file, which is not explicitly shown in the provided snippet. However, based on the provided configuration snippet, we can infer that this search head cluster member is configured to communicate with a cluster master (master_uri) located at node1 and a management node (mgmt_uri) located at node3. The serverName is not the same as the master_uri or mgmt_uri; these URIs indicate the location of the master and management nodes that this member interacts with.

Since the serverName is not provided in the snippet, one would typically look for a setting under the [general] stanza in server.conf. However, given the options and the common naming conventions in a Splunk environment, node3 would be a reasonable guess for the server name of this member, since it is indicated as the management URI within the [shclustering] stanza, which suggests it might be the name or address of the server in question.

For accurate identification, you would need to access the full server.conf file or the Splunk Web on the search head cluster member and look under Settings > Server settings > General settings to find the actual serverName. Reference for these details would be found in the Splunk documentation regarding the configuration files, particularly server.conf.

According to the Splunk Enterprise Capacity Planning and Hardware Sizing Guidelines, a distributed Splunk environment’s minimum search head specification must ensure that the system can efficiently manage search parsing, ad-hoc query execution, and knowledge object replication. Splunk officially recommends using a 64-bit x86 architecture system with a minimum of two physical CPU cores (or four vCPUs) running at 2 GHz or higher per core for acceptable performance.

Search heads are CPU-intensive components, primarily constrained by processor speed and the number of concurrent searches they must handle. Memory and disk space should scale with user concurrency and search load, but CPU capability remains the baseline requirement. While 128 GB RAM (Option C) is suitable for high-demand or Enterprise Security (ES) deployments, it exceeds the minimum hardware specification for general distributed search environments.

Splunk no longer supports 32-bit architectures (Option B). While a 1Gb Ethernet NIC (Option A) is common, it is not part of the minimum computational specification required by Splunk for search heads. The critical specification is processor capability — two physical cores or equivalent.

References (Splunk Enterprise Documentation):

• Splunk Enterprise Capacity Planning Manual – Hardware and Performance Guidelines

• Search Head Sizing and System Requirements

• Distributed Deployment Manual – Recommended System Specifications

• Splunk Hardware and Performance Tuning Guide

Two indexers clustered is the recommended deployment for a customer who plans to ingest 600 GB of data per day into Splunk, has six concurrent users, and wants high data availability and high search performance. This deployment will provide enough indexing capacity and search concurrency for the customer’s needs, while also ensuring data replication and searchability across the cluster. The customer can also save on the hardware cost by using only two indexers. Two indexers not in a cluster will not provide high data availability, as there is no data replication or failover. Three indexers not in a cluster will provide more indexing capacity and search concurrency, but also more hardware cost and no data availability. The customer’s data retention period, number of long searches, or volume of saved/scheduled searches are not relevant for determining the number of indexers. For more information, see [Reference hardware] and [About indexer clusters and index replication] in the Splunk documentation.

The Splunk Enterprise Security and Encryption documentation specifies that the primary mechanism for securing data in motion within a Splunk environment is to enable TLS/SSL encryption between forwarders and indexers. This ensures that log data transmitted from Universal Forwarders or Heavy Forwarders to Indexers is fully encrypted and protected from interception or tampering.

The correct configuration involves setting up signed SSL certificates on both forwarders and indexers:

On the forwarder, TLS settings are defined in outputs.conf, specifying parameters like sslCertPath, sslPassword, and sslRootCAPath.

On the indexer, TLS is enabled in inputs.conf and server.conf using the same shared CA for validation.

Splunk’s documentation explicitly states that this configuration protects data-in-transit between the collection (forwarder) and indexing (storage) tiers — which is the critical link where sensitive log data is most vulnerable.

Other communication channels (e.g., deployment server to clients or browser to Splunk Web) can also use encryption but do not secure the ingestion pipeline that handles the indexed data stream. Therefore, TLS should be implemented between Splunk forwarders and indexers.

References (Splunk Enterprise Documentation):

• Securing Data in Transit with SSL/TLS

• Configure Forwarder-to-Indexer Encryption Using SSL Certificates

• Server and Forwarder Authentication Setup Guide

• Splunk Enterprise Admin Manual – Security and Encryption Best Practices

Decreasing the data model acceleration range will reduce the disk size requirements for a cluster of indexers running Splunk Enterprise Security. Data model acceleration creates tsidx files that consume disk space on the indexers. Reducing the acceleration range will limit the amount of data that is accelerated and thus save disk space. Setting the cluster search factor or replication factor to N-1 will not reduce the disk size requirements, but rather increase the risk of data loss. Increasing the number of buckets per index will also increase the disk size requirements, as each bucket has a minimum size. For more information, see Data model acceleration and Bucket size in the Splunk documentation.

According to Splunk Enterprise Search Performance and Deployment Optimization guidelines, the phone home interval (configured for deployment clients communicating with a Deployment Server) has no impact on search performance.

The phone home mechanism controls how often deployment clients check in with the Deployment Server for configuration updates or new app bundles. This process occurs independently of the search subsystem and does not consume indexer or search head resources that affect query speed, indexing throughput, or search concurrency.

In contrast:

Increasing the number of indexers (Option B) improves search performance by distributing indexing and search workloads across more nodes.

Workload Management (Option C) allows admins to prioritize compute and memory resources for critical searches, optimizing performance under load.

Increasing search heads (Option D) can enhance concurrency and user responsiveness by distributing search scheduling and ad-hoc query workloads.

Therefore, adjusting the phone home interval is strictly an administrative operation and has no measurable effect on Splunk search or indexing performance.

References (Splunk Enterprise Documentation):

• Deployment Server: Managing Phone Home Intervals

• Search Performance Optimization and Resource Management

• Distributed Search Architecture and Scaling Best Practices

• Workload Management Overview – Resource Allocation in Search Operations

The correct topology to ensure a scalable and performant deployment for the customer’s use case is two search head clusters, one for ITSI and one for ES. This configuration provides high availability, load balancing, and isolation for each Splunk app. According to the Splunk documentation1, ITSI and ES should not be installed on the same search head or search head cluster, as they have different requirements and may interfere with each other. Having two separate search head clusters allows each app to have its own dedicated resources and configuration, and avoids potential conflicts and performance issues1. The other options are not recommended, as they either have only one search head or search head cluster, which reduces the availability and scalability of the deployment, or they have both ITSI and ES installed on the same search head or search head cluster, which violates the best practices and may cause problems. Therefore, option B is the correct answer, and options A, C, and D are incorrect.

1: Splunk IT Service Intelligence and Splunk Enterprise Security compatibility

The following clarification steps should be taken if apps are not appearing on a deployment client:

Check serverclass.conf of the deployment server. This file defines the server classes and the apps and configurations that they should receive from the deployment server. Make sure that the deployment client belongs to the correct server class and that the server class has the desired apps and configurations.

Check deploymentclient.conf of the deployment client. This file specifies the deployment server that the deployment client contacts and the client name that it uses. Make sure that the deployment client is pointing to the correct deployment server and that the client name matches the server class criteria.

Search for relevant events in splunkd.log of the deployment server. This file contains information about the deployment server activities, such as sending apps and configurations to the deployment clients, detecting client check-ins, and logging any errors or warnings. Look for any events that indicate a problem with the deployment server or the deployment client.

Checking the content of SPLUNK_HOME/etc/apps of the deployment server is not a necessary clarification step, as this directory does not contain the apps and configurations that are distributed to the deployment clients. The apps and configurations for the deployment server are stored in SPLUNK_HOME/etc/deployment-apps. For more information, see Configure deployment server and clients in the Splunk documentation.

The correct answer is C. Knowledge Object replication. This is a valid use case that a search head cluster addresses, as it ensures that all the search heads in the cluster have the same set of knowledge objects, such as saved searches, dashboards, reports, and alerts1. The search head cluster replicates the knowledge objects across the cluster members, and synchronizes any changes or updates1. This provides a consistent user experience and avoids data inconsistency or duplication1. The other options are not valid use cases that a search head cluster addresses. Option A, providing redundancy in the event a search peer fails, is not a use case for a search head cluster, but for an indexer cluster, which maintains multiple copies of the indexed data and can recover from indexer failures2. Option B, search affinity, is not a use case for a search head cluster, but for a multisite indexer cluster, which allows the search heads to preferentially search the data on the local site, rather than on a remote site3. Option D, increased Search Factor (SF), is not a use case for a search head cluster, but for an indexer cluster, which determines how many searchable copies of each bucket are maintained across the indexers4. Therefore, option C is the correct answer, and options A, B, and D are incorrect.

1: About search head clusters 2: About indexer clusters and index replication 3: Configure search affinity 4: Configure the search factor

 The logical first step when starting a deployment plan is to collect the initial requirements for the deployment from all stakeholders. This includes identifying the business objectives, the data sources, the use cases, the security and compliance needs, the scalability and availability expectations, and the budget and timeline constraints. Collecting the initial requirements helps to define the scope and the goals of the deployment, and to align the expectations of all the parties involved.

Inventorying the currently deployed logging infrastructure, determining what apps and use cases will be implemented, and gathering statistics on the expected adoption of Splunk for sizing are all important steps in the deployment planning process, but they are not the logical first step. These steps can be done after collecting the initial requirements, as they depend on the information gathered from the stakeholders.

According to the Splunk documentation1, the “Unknown sid” error message means that the search job associated with the link has expired or been deleted. The sid (search ID) is a unique identifier for each search job, and it is used to retrieve the results of the search. If the sid is not found, the search cannot be displayed. The other options are false because:

The users having insufficient permissions would result in a different error message, such as “You do not have permission to view this page” or "You do not have permission to run this search"1.

An add-on needing to be updated would not affect the validity of the sid, unless the add-on changes the search syntax or the data source in a way that makes the search invalid or inaccessible1.

One or more indexers being down would not cause the “Unknown sid” error, as the sid is stored on the search head, not the indexers. However, it could cause other errors, such as “Unable to distribute to peer” or "Search peer has the following message: not enough disk space"1.

In Splunk Enterprise, manually editing the serverclass.conf file on a Deployment Server can lead to the loss of UI edit functionality for server classes in Splunk Web.

The Deployment Server manages app distribution to Universal Forwarders and other deployment clients through server classes, which are defined in serverclass.conf. This file maps deployment clients to specific app configurations and defines filtering rules, restart behaviors, and inclusion/exclusion criteria.

When this configuration file is modified manually (outside of Splunk Web), the syntax, formatting, or logical relationships between entries may not match what Splunk Web expects. As a result, Splunk Web may no longer be able to parse or display those server classes correctly. Once this happens, administrators cannot modify deployment settings through the GUI until the configuration file is corrected or reverted to a valid state.

Other files such as deploymentclient.conf, inputs.conf, and deploymentserver.conf control client settings, data inputs, and core server parameters but do not affect the UI-driven deployment management functionality.

Therefore, Splunk explicitly warns administrators in its Deployment Server documentation to use Splunk Web or the CLI when modifying serverclass.conf, and to avoid manual editing unless fully confident in its syntax.

References (Splunk Enterprise Documentation):

• Deployment Server Overview – Managing Server Classes and App Deployment

• serverclass.conf Reference and Configuration Best Practices

• Splunk Enterprise Admin Manual – GUI Limitations After Manual Edits

• Troubleshooting Deployment Server and Serverclass Configuration Issues

Within the [clustering] stanza of the server.conf file, the mode attribute defines the functional role of a Splunk instance within an indexer cluster. Splunk documentation identifies three valid modes:

mode = manager

Defines the node as the Cluster Manager (formerly called the Master Node).

Responsible for coordinating peer replication, managing configurations, and ensuring data integrity across indexers.

mode = peer

Defines the node as an Indexer (Peer Node) within the cluster.

Handles data ingestion, replication, and search operations under the control of the manager node.

mode = searchhead

Defines a Search Head that connects to the cluster for distributed searching and data retrieval.

The value “deployer” (Option C) is not valid within the [clustering] stanza; it applies to Search Head Clustering (SHC) configurations, where it is defined separately in server.conf under [shclustering].

Each mode must be accompanied by other critical attributes such as manager_uri, replication_port, and pass4SymmKey to enable proper communication and security between cluster members.

References (Splunk Enterprise Documentation):

• Indexer Clustering: Configure Manager, Peer, and Search Head Modes

• server.conf Reference – [clustering] Stanza Attributes

• Distributed Search and Cluster Node Role Configuration

• Splunk Enterprise Admin Manual – Cluster Deployment Architecture

When using the props.conf LINE_BREAKER attribute to delimit multi-line events, the SHOULD_LINEMERGE attribute should be set to false. This tells Splunk not to merge events that have been broken by the LINE_BREAKER. Setting the SHOULD_LINEMERGE attribute to true, auto, or none will cause Splunk to ignore the LINE_BREAKER and merge events based on other criteria. For more information, see Configure event line breaking in the Splunk documentation.

According to the Splunk documentation1, multi-site clustering is an indexer cluster that spans multiple physical sites, such as data centers. Each site has its own set of peer nodes and search heads. Each site also obeys site-specific replication and search factor rules. The use cases that are made possible by multi-site clustering are:

Greatly reduce WAN traffic by preferentially searching assigned site (search affinity). This means that if you configure each site so that it has both a search head and a full set of searchable data, the search head on each site will limit its searches to local peer nodes. This eliminates any need, under normal conditions, for search heads to access data on other sites, greatly reducing network traffic between sites2.

Seamlessly route searches to a redundant site in case of a site failure. This means that by storing copies of your data at multiple locations, you maintain access to the data if a disaster strikes at one location. Multisite clusters provide site failover capability. If a site goes down, indexing and searching can continue on the remaining sites, without interruption or loss of data2.

The other options are false because:

Use blockchain technology to audit search activity from geographically dispersed data centers. This is not a use case of multi-site clustering, as Splunk does not use blockchain technology to audit search activity. Splunk uses its own internal logs and metrics to monitor and audit search activity3.

Enable a forwarder to send data to multiple indexers. This is not a use case of multi-site clustering, as forwarders can send data to multiple indexers regardless of whether they are in a single-site or multi-site cluster. This is a basic feature of forwarders that allows load balancing and high availability of data ingestion4.

Indexed extraction configurations are processed in the indexing phase of the Splunk Enterprise data pipeline. The data pipeline is the process that Splunk uses to ingest, parse, index, and search data. Indexed extraction configurations are settings that determine how Splunk extracts fields from data at index time, rather than at search time. Indexed extraction can improve search performance, but it also increases the size of the index. Indexed extraction configurations are applied in the indexing phase, which is the phase where Splunk writes the data and the .tsidx files to the index. The input phase is the phase where Splunk receives data from various sources and formats. The parsing phase is the phase where Splunk breaks the data into events, timestamps, and hosts. The search phase is the phase where Splunk executes search commands and returns results.

The master node distributes configuration bundles to peer nodes in the slave-apps directory under $SPLUNK_HOME/etc. The configuration bundle method is the only supported method for managing common configurations and app deployment across the set of peers. It ensures that all peers use the same versions of these files1. Bundles typically contain a subset of files (configuration files and assets) from $SPLUNK_HOME/etc/system, $SPLUNK_HOME/etc/apps, and $SPLUNK_HOME/etc/users2. The process of distributing knowledge bundles means that peers by default receive nearly the entire contents of the search head’s apps3.

 The parallelIngestionPipelines setting can be adjusted on the indexers and forwarders to improve Splunk performance. The parallelIngestionPipelines setting determines how many concurrent data pipelines are used to process the incoming data. Increasing the parallelIngestionPipelines setting can improve the data ingestion and indexing throughput, especially for high-volume data sources. The parallelIngestionPipelines setting can be adjusted on the indexers and forwarders by editing the limits.conf file. The parallelIngestionPipelines setting cannot be adjusted on the search head or the cluster master, because they are not involved in the data ingestion and indexing process.

The site_mappings attribute should be added to the master node’s server.conf file when decommissioning a site in an indexer cluster. The site_mappings attribute is used to specify how the master node should reassign the buckets from the decommissioned site to the remaining sites. The site_mappings attribute is a comma-separated list of site pairs, where the first site is the decommissioned site and the second site is the destination site. For example, site_mappings = site1:site2,site3:site4 means that the buckets from site1 will be moved to site2, and the buckets from site3 will be moved to site4. The available_sites attribute is used to specify which sites are currently available in the cluster, and it is automatically updated by the master node. The site_search_factor and site_replication_factor attributes are used to specify the number of searchable and replicated copies of each bucket for each site, and they are not affected by the decommissioning process

When Splunk indexes data in a non-clustered environment, it creates index and .tsidx files by default. The index files contain the raw data that Splunk has ingested, compressed and encrypted. The .tsidx files contain the time-series index that maps the timestamps and event IDs of the raw data. The rawdata and index files are not the correct terms for the files that Splunk creates. The compressed and .tsidx files are partially correct, but compressed is not the proper name for the index files. The compressed and meta data files are also partially correct, but meta data is not the proper name for the .tsidx files.

 The following statements are true regarding Splunk Enterprise performance:

Adding search peers increases the search throughput as search load increases. This is because adding more search peers distributes the search workload across more indexers, which reduces the load on each indexer and improves the search speed and concurrency.

Adding search heads provides additional CPU cores to run more concurrent searches. This is because adding more search heads increases the number of search processes that can run in parallel, which improves the search performance and scalability. The following statements are false regarding Splunk Enterprise performance:

Adding search peers does not increase the maximum size of search results. The maximum size of search results is determined by the maxresultrows setting in the limits.conf file, which is independent of the number of search peers.

Adding RAM to an existing search head does not provide additional search capacity. The search capacity of a search head is determined by the number of CPU cores, not the amount of RAM. Adding RAM to a search head may improve the search performance, but not the search capacity. For more information, see Splunk Enterprise performance in the Splunk documentation.

The walklex command in Splunk is a specialized administrative search command used to translate and display LISPY (Splunk’s internal representation of search terms). LISPY is the logical search syntax Splunk uses to parse and execute search queries, and examining it helps administrators and developers debug search optimization, field extraction behavior, and index-time search efficiency.

When you run the command | walklex search="your_search_string", Splunk outputs how it tokenizes and interprets that query internally. This is particularly useful for understanding how Splunk’s search language maps to index-time fields and for diagnosing performance issues caused by inefficient search term parsing.

For example:

| walklex search="error OR failure host=server01"

Displays the corresponding LISPY translation used by Splunk’s search subsystem.

Other options are unrelated:

dbinspect provides index bucket metadata.

Monitoring Console shows performance metrics and health status.

Search Job Inspector analyzes search execution phases but doesn’t expose LISPY.

Thus, the correct and Splunk-documented tool for LISPY inspection is the walklex command.

References (Splunk Enterprise Documentation):

• walklex Command Reference – LISPY and Search Debugging

• Understanding Search Language Parsing in Splunk

• Search Internals: How Splunk Interprets Queries

• Splunk Search Performance Troubleshooting Tools

inputs.conf is a configuration file that contains settings for various types of data inputs, such as files, directories, network ports, scripts, and so on1.

initCrcLength is a setting that specifies the number of characters that the input uses to calculate the CRC (cyclic redundancy check) of a file1. The CRC is a value that uniquely identifies a file based on its content2.

crcSalt is another setting that adds a string to the CRC calculation to force the input to consume files that have matching CRCs1. This can be useful when files have identical headers or when files are renamed or rolled over2.

When troubleshooting a situation where some files within a directory are not being indexed, the ignored files are discovered to have long headers, the first thing that should be added to inputs.conf is to increase the value of initCrcLength. This is because by default, the input only performs CRC checks against the first 256 bytes of a file, which means that files with long headers may have matching CRCs and be skipped by the input2. By increasing the value of initCrcLength, the input can use more characters from the file to calculate the CRC, which can reduce the chances of CRC collisions and ensure that different files are indexed3.

Option C is the correct answer because it reflects the best practice for troubleshooting this situation. Option A is incorrect because decreasing the value of initCrcLength would make the CRC calculation less reliable and more prone to collisions. Option B is incorrect because adding a crcSalt with a static string would not help differentiate files with long headers, as they would still have matching CRCs. Option D is incorrect because adding a crcSalt with the attribute would add the full directory path to the CRC calculation, which would not help if the files are in the same directory2.

When adding or decommissioning a member from a Search Head Cluster (SHC), the proper order of operations is:

Delete Splunk Enterprise, if it exists.

Install and initialize the instance.

Join the SHC.

This order of operations ensures that the member has a clean and consistent Splunk installation before joining the SHC. Deleting Splunk Enterprise removes any existing configurations and data from the instance. Installing and initializing the instance sets up the Splunk software and the required roles and settings for the SHC. Joining the SHC adds the instance to the cluster and synchronizes the configurations and apps with the other members. The other order of operations are not correct, because they either skip a step or perform the steps in the wrong order.

The correct answer is C. Increase the parallellngestionPipelines attribute. This is an option in limits.conf that may provide performance benefits at the forwarding tier, as it allows the forwarder to process multiple data inputs in parallel1. The parallellngestionPipelines attribute specifies the number of pipelines that the forwarder can use to ingest data from different sources1. By increasing this value, the forwarder can improve its throughput and reduce the latency of data delivery1. The other options are not effective options to provide performance benefits at the forwarding tier. Option A, enabling the indexed_realtime_use_by_default attribute, is not recommended, as it enables the forwarder to send data to the indexer as soon as it is received, which may increase the network and CPU load and degrade the performance2. Option B, increasing the maxKBps attribute, is not a good option, as it increases the maximum bandwidth, in kilobytes per second, that the forwarder can use to send data to the indexer3. This may improve the data transfer speed, but it may also saturate the network and cause congestion and packet loss3. Option D, increasing the max_searches_per_cpu attribute, is not relevant, as it only affects the search performance on the indexer or search head, not the forwarding performance on the forwarder4. Therefore, option C is the correct answer, and options A, B, and D are incorrect.

1: Configure parallel ingestion pipelines 2: Configure real-time forwarding 3: Configure forwarder output 4: Configure search performance

A single-site indexer cluster is a group of Splunk Enterprise instances that index and replicate data across the cluster1. A bucket is a directory that contains indexed data, along with metadata and other information2. A replication factor is the number of copies of each bucket that the cluster maintains1. A search factor is the number of searchable copies of each bucket that the cluster maintains1. A searchable copy is a copy that contains both the raw data and the index files3. A search head is a Splunk Enterprise instance that coordinates the search activities across the peer nodes1.

Option D is the correct answer because it reflects the definitions of replication factor and search factor. The cluster will ensure that there are at least three copies of each bucket, one on each peer node, to satisfy the replication factor of 3. The cluster will also ensure that there are at least two searchable copies of each bucket, one primary and one searchable, to satisfy the search factor of 2. The primary copy is the one that the search head uses to run searches, and the searchable copy is the one that can be promoted to primary if the original primary copy becomes unavailable3.

Option A is incorrect because it confuses the replication factor and the search factor. The cluster will ensure there are at least three copies of each bucket, not two, to meet the replication factor of 3. The cluster will ensure there are at least two copies of searchable metadata, not three, to meet the search factor of 2.

Option B is incorrect because it uses the wrong terms. The cluster will ensure there are at least, not at most, three copies of each bucket, to meet the replication factor of 3. The cluster will ensure there are at least, not at most, two copies of searchable metadata, to meet the search factor of 2.

Option C is incorrect because it has nothing to do with the replication factor or the search factor. The cluster does not limit the number of search heads that can access the bucket at the same time. The search head can search across multiple clusters, and the cluster can serve multiple search heads1.

1: The basics of indexer cluster architecture - Splunk Documentation 2: About buckets - Splunk Documentation 3: Search factor - Splunk Documentation

To activate replication for an index in an indexer cluster, the repFactor attribute must be configured in indexes.conf on all peer nodes. This attribute specifies the replication factor for the index, which determines how many copies of raw data are maintained by the cluster. Setting the repFactor attribute to auto will enable replication for the index. The replicate attribute in indexes.conf is not a valid Splunk attribute. The repFactor attribute in outputs.conf and the replicate attribute in deploymentclient.conf are not related to replication for an index in an indexer cluster. For more information, see Configure indexes for indexer clusters in the Splunk documentation.

The minimum reference server specification for a Splunk indexer is 12 CPU cores, 12GB RAM, and 800 IOPS. This specification is based on the assumption that the indexer will handle an average indexing volume of 100GB per day, with a peak of 300GB per day, and a typical search load of 1 concurrent search per 1GB of indexing volume. The other specifications are either higher or lower than the minimum requirement. For more information, see [Reference hardware] in the Splunk documentation.

All of the forwarders would still be phoning home to the old deployment server, because the forwarders are configured to use the old deployment server in $SPLUNK_HOME/etc/system/local. This is the local configuration directory that contains the settings that override the default settings in $SPLUNK_HOME/etc/system/default. The deploymentclient.conf file in the local directory specifies the targetUri of the deployment server that the forwarder contacts for configuration updates and apps. If the forwarders have the old deployment server’s targetUri in the local directory, they will ignore the updated deploymentclient.conf file that was deployed by the old deployment server, because the local settings have higher precedence than the deployed settings. To fix this issue, the forwarders should either remove the deploymentclient.conf file from the local directory, or update it with the new deployment server’s targetUri. Option C is the correct answer. Option A is incorrect because a version mismatch between the forwarders and the new deployment server would not prevent the forwarders from phoning home to the new deployment server, as long as they are compatible versions. Option B is incorrect because the new deployment server is configured and running, and there is no indication that it is not accepting connections from the forwarders. Option D is incorrect because the pass4SymmKey is the shared secret key that the deployment server and the forwarders use to authenticate each other. It does not affect the forwarders’ ability to phone home to the new deployment server, as long as it is the same on both sides12

1: https://docs.splunk.com/Documentation/Splunk/9.1.2/Updating/Configuredeploymentclients 2: https://docs.splunk.com/Documentation/Splunk/9.1.2/Admin/Wheretofindtheconfigurationfiles

The following logs are included in the _introspection index, which contains data that the Splunk Enterprise deployment logs for platform instrumentation:

disk_objects.log. This log contains information about the disk objects that Splunk creates and manages, such as buckets, indexes, and files. This log can help monitor the disk space usage and the bucket lifecycle.

resource_usage.log. This log contains information about the resource usage of Splunk processes, such as CPU, memory, disk, and network. This log can help monitor the Splunk performance and identify any resource bottlenecks. The following logs are not included in the _introspection index, but rather in the _internal index, which contains data that Splunk generates for internal logging:

audit.log. This log contains information about the audit events that Splunk records, such as user actions, configuration changes, and search activity. This log can help audit the Splunk operations and security.

metrics.log. This log contains information about the performance metrics that Splunk collects, such as data throughput, data latency, search concurrency, and search duration. This log can help measure the Splunk performance and efficiency. For more information, see About Splunk Enterprise logging and [About the _introspection index] in the Splunk documentation.

All of the options are possible causes of a crash in Splunk. According to the Splunk documentation1, incorrect ulimit settings can lead to file descriptor exhaustion, which can cause Splunk to crash or hang. Insufficient disk IOPS can also cause Splunk to crash or become unresponsive, as Splunk relies heavily on disk performance2. Insufficient memory can cause Splunk to run out of memory and crash, especially when running complex searches or handling large volumes of data3. Running out of disk space can cause Splunk to stop indexing data and crash, as Splunk needs enough disk space to store its data and logs4.

1: Configure ulimit settings for Splunk Enterprise 2: Troubleshoot Splunk performance issues 3: Troubleshoot memory usage 4: Troubleshoot disk space issues

According to Splunk’s Enterprise Security (ES) Installation and Sizing Guide, the majority of processing and computational load generated by the Splunk App for Enterprise Security is concentrated on the Search Head(s).

This is because Splunk ES is built around a search-driven correlation model — it continuously runs scheduled correlation searches, data model accelerations, and notables generation jobs. These operations rely on the search head tier’s CPU, memory, and I/O resources rather than on indexers. ES also performs extensive data model summarization, CIM normalization, and real-time alerting, all of which are search-intensive operations.

While indexers handle data ingestion and indexing, they are not heavily affected by ES beyond normal search request processing. The Cluster Manager only coordinates replication and plays no role in search execution, and Heavy Forwarders serve as data collection or parsing points with minimal analytical load.

Splunk officially recommends deploying ES on a dedicated Search Head Cluster (SHC) to isolate its high CPU and memory demands from other workloads. For large-scale environments, horizontal scaling via SHC ensures consistent performance and stability.

References (Splunk Enterprise Documentation):

• Splunk Enterprise Security Installation and Configuration Guide

• Search Head Sizing for Splunk Enterprise Security

• Enterprise Security Overview – Workload Distribution and Performance Impact

• Splunk Architecture and Capacity Planning for ES Deployments

A lookup table is a file that contains a list of values that can be used to enrich or modify the data during search time1. Lookup tables can be stored in CSV files or in the KV Store1. Troubleshooting lookup tables involves identifying and resolving issues that prevent the lookup tables from being accessed, updated, or applied correctly by the Splunk searches. Some of the tools and methods that can help with troubleshooting lookup tables are:

web_access.log: This is a file that contains information about the HTTP requests and responses that occur between the Splunk web server and the clients2. This file can help troubleshoot issues related to lookup table permissions, availability, and errors, such as 404 Not Found, 403 Forbidden, or 500 Internal Server Error34.

btool output: This is a command-line tool that displays the effective configuration settings for a given Splunk component, such as inputs, outputs, indexes, props, and so on5. This tool can help troubleshoot issues related to lookup table definitions, locations, and precedence, as well as identify the source of a configuration setting6.

search.log: This is a file that contains detailed information about the execution of a search, such as the search pipeline, the search commands, the search results, the search errors, and the search performance. This file can help troubleshoot issues related to lookup table commands, arguments, fields, and outputs, such as lookup, inputlookup, outputlookup, lookup_editor, and so on .

Option B is the correct answer because web_access.log is the best place to start troubleshooting lookup table issues, as it can provide the most relevant and immediate information about the lookup table access and status. Option A is incorrect because btool output is not a log file, but a command-line tool. Option C is incorrect because health.log is a file that contains information about the health of the Splunk components, such as the indexer cluster, the search head cluster, the license master, and the deployment server. This file can help troubleshoot issues related to Splunk deployment health, but not necessarily related to lookup tables. Option D is incorrect because configuration_change.log is a file that contains information about the changes made to the Splunk configuration files, such as the user, the time, the file, and the action. This file can help troubleshoot issues related to Splunk configuration changes, but not necessarily related to lookup tables.

 To reduce the captain’s work load in a search head cluster, the setting that will prevent scheduled searches from running on the captain is captain_is_adhoc_searchhead = true (on the current captain). This setting will designate the current captain as an ad hoc search head, which means that it will not run any scheduled searches, but only ad hoc searches initiated by users. This will reduce the captain’s work load and improve the search head cluster performance. The adhoc_searchhead = true (on all members) setting will designate all search head cluster members as ad hoc search heads, which means that none of them will run any scheduled searches, which is not desirable. The adhoc_searchhead = true (on the current captain) setting will have no effect, as this setting is ignored by the captain. The captain_is_adhoc_searchhead = true (on all members) setting will have no effect, as this setting is only applied to the current captain. For more information, see Configure the captain as an ad hoc search head in the Splunk documentation.

According to Splunk Enterprise documentation for the Monitoring Console (MC), the data displayed in its dashboards is sourced primarily from two internal mechanisms — REST API calls and metrics.log.

The Monitoring Console (formerly known as the Distributed Management Console, or DMC) uses REST API endpoints to collect system-level information from all connected instances, such as indexer clustering status, license usage, and search head performance. These REST calls pull real-time configuration and performance data from Splunk’s internal management layer (/services/server/status, /services/licenser, /services/cluster/peers, etc.).

Additionally, the metrics.log file is one of the main data sources used by the Monitoring Console. This log records Splunk’s internal performance metrics, including pipeline latency, queue sizes, indexing throughput, CPU usage, and memory statistics. Dashboards like “Indexer Performance,” “Search Performance,” and “Resource Usage” are powered by searches over the _internal index that reference this log.

Other tools listed — such as btool (configuration troubleshooting utility) and diag (diagnostic archive generator) — are not used as runtime data sources for Monitoring Console dashboards. They assist in troubleshooting but are not actively queried by the MC.

References (Splunk Enterprise Documentation):

• Monitoring Console Overview – Data Sources and Architecture

• metrics.log Reference – Internal Performance Data Collection

• REST API Usage in Monitoring Console

• Distributed Management Console Configuration Guide

 The tailing_processor.log file would be the best place to search if you suspect there is a problem interpreting a regular expression in a monitor stanza. This log file contains information about how Splunk monitors files and directories, including any errors or warnings related to parsing the monitor stanza. The splunkd.log file contains general information about the Splunk daemon, but it may not have the specific details about the monitor stanza. The btool.log file contains information about the configuration files, but it does not log the runtime behavior of the monitor stanza. The metrics.log file contains information about the performance metrics of Splunk, but it does not log the event breaking issues. For more information, see About Splunk Enterprise logging in the Splunk documentation.

According to Splunk Enterprise Distributed Clustering documentation, when you add a search head to an indexer cluster, you must configure it to communicate with the Cluster Manager (previously known as Master Node). The proper way to initialize this connection is by editing the cluster configuration using the splunk edit cluster-config command.

The correct syntax for a search head is:

splunk edit cluster-config -mode searchhead -manager_uri https:// :8089 -secret

Here:

-mode searchhead specifies that this node will function as a search head that participates in distributed search across the indexer cluster.

-manager_uri provides the management URI of the cluster manager.

-secret defines the shared secret key used for secure communication between the manager and cluster members.

Once this configuration is applied, the search head must be restarted for the changes to take effect.

Using -mode peer (Option B) is for indexers joining the cluster, not search heads. The add cluster-manager command (Options C and D) is not a valid Splunk command.

References (Splunk Enterprise Documentation):

• Configure the Search Head for an Indexer Cluster

• Indexer Clustering: Configure the Cluster Manager, Peer, and Search Head Nodes

• Splunk Enterprise Admin Manual: splunk edit cluster-config Command Reference

Migration from single-site to multisite index replication only affects new data, not existing data. Multisite policies apply to new data only, meaning that data that is ingested after the migration will follow the multisite replication and search factors. Existing data, or data that was ingested before the migration, will retain the single-site policies, unless they are manually converted to multisite buckets. Single-site buckets do not instantly receive the multisite policies, nor do they automatically convert to multisite buckets. Multisite total values can exceed any single-site factors, as long as they do not exceed the number of peer nodes in the cluster. A master node is not required at each site, only one master node is needed for the entire cluster

Multiple search pipelines should be enabled only if CPU and memory resources are significantly under-utilized. Search pipelines are the processes that execute search commands and return results. Multiple search pipelines can improve the search performance by running concurrent searches in parallel. However, multiple search pipelines also consume more CPU and memory resources, which can affect the overall system performance. Therefore, multiple search pipelines should be enabled only if there are enough CPU and memory resources available, and if the system is not bottlenecked by disk I/O or network bandwidth. The number of concurrent users, the disk IOPS, and the Splunk Enterprise version are not relevant factors for enabling multiple search pipelines

According to the Splunk Search Job Inspector documentation, the search.log file for a completed search job can be accessed through Splunk Web by navigating to the job’s detailed inspection view.

To access it:

Open the completed search in Splunk Web.

Click the Job menu (top right of the search interface).

Select Inspect Job.

In the Job Inspector window, click the search.log link.

This log provides in-depth diagnostic details about how the search was parsed, distributed, and executed across the search head and indexers. It contains valuable performance metrics, command execution order, event sampling information, and any error or warning messages encountered during search processing.

The search.log file is generated for every search job—scheduled, ad-hoc, or background—and is stored in the job’s dispatch directory ($SPLUNK_HOME/var/run/splunk/dispatch//).

Other listed options are incorrect:

Option A queries the _internal index, which does not store per-search logs.

Option B is used to view search configurations, not logs.

Option C is not a valid Splunk Web navigation option.

Thus, the only correct and Splunk-documented method is via Job → Inspect Job → search.log.

References (Splunk Enterprise Documentation):

• Search Job Inspector Overview and Usage

• Analyzing Search Performance Using search.log

• Search Job Management and Dispatch Directory Structure

• Splunk Enterprise Admin Manual – Troubleshooting Searches

According to the indexes.conf reference in Splunk Enterprise, the parameter maxDataSize controls the maximum size (in GB or MB) of a single hot bucket before Splunk rolls it to a warm bucket. When the value is set to auto_high_volume on a 64-bit system, Splunk automatically sets the maximum hot bucket size to 10 GB.

The “auto” settings allow Splunk to choose optimized values based on the system architecture:

auto: Default hot bucket size of 750 MB (32-bit) or 10 GB (64-bit).

auto_high_volume: Specifically tuned for high-ingest indexes; on 64-bit systems, this equals 10 GB per hot bucket.

auto_low_volume: Uses smaller bucket sizes for lightweight indexes.

The purpose of larger hot bucket sizes on 64-bit systems is to improve indexing performance and reduce the overhead of frequent bucket rolling during heavy data ingestion. The documentation explicitly warns that these sizes differ on 32-bit systems due to memory addressing limitations.

Thus, for high-throughput environments running 64-bit operating systems, auto_high_volume = 10 GB is the correct and Splunk-documented configuration.

References (Splunk Enterprise Documentation):

• indexes.conf – maxDataSize Attribute Reference

• Managing Index Buckets and Data Retention

• Splunk Enterprise Admin Manual – Indexer Storage Configuration

• Splunk Performance Tuning: Bucket Management and Hot/Warm Transitions

The following security option must be explicitly configured, as it is not enabled by default:

Certificate authentication between forwarders and indexers. This option allows the forwarders and indexers to verify each other’s identity using SSL certificates, which prevents unauthorized data transmission or spoofing attacks. This option is not enabled by default, as it requires the administrator to generate and distribute the certificates for the forwarders and indexers. For more information, see [Secure the communication between forwarders and indexers] in the Splunk documentation. The following security options are enabled by default:

Data encryption between Splunk Web and splunkd. This option encrypts the communication between the Splunk Web interface and the splunkd daemon using SSL, which prevents data interception or tampering. This option is enabled by default, as Splunk provides a self-signed certificate for this purpose. For more information, see [About securing Splunk Enterprise with SSL] in the Splunk documentation.

Certificate authentication between Splunk Web and search head. This option allows the Splunk Web interface and the search head to verify each other’s identity using SSL certificates, which prevents unauthorized access or spoofing attacks. This option is enabled by default, as Splunk provides a self-signed certificate for this purpose. For more information, see [About securing Splunk Enterprise with SSL] in the Splunk documentation.

Data encryption for distributed search between search heads and indexers. This option encrypts the communication between the search heads and the indexers using SSL, which prevents data interception or tampering. This option is enabled by default, as Splunk provides a self-signed certificate for this purpose. For more information, see [Secure your distributed search environment] in the Splunk documentation.

The __introspection log file contains data about the impact of the Splunk software on the host system, such as CPU, memory, disk, and network usage, as well as KV store performance1. This log file is monitored by default and the contents are sent to the _introspection index1. The other options are not related to the __introspection log file. File monitor input configurations are stored in inputs.conf2. File monitor checkpoint offset is stored in fishbucket3. User activities and knowledge objects are stored in the _audit and _internal indexes respectively4.

According to the Splunk blog1, the Universal Forwarder is ideal for collecting data from high-velocity data sources, such as a syslog server, due to its smaller footprint and faster performance. The Universal Forwarder performs minimal processing and sends raw or unparsed data to the indexers, reducing the network traffic and the load on the forwarders. The other options are false because:

When most of the data requires masking, a Heavy Forwarder is needed, as it can perform advanced filtering and data transformation before forwarding the data2.

When data comes directly from a database server, a Heavy Forwarder is needed, as it can run modular inputs such as DB Connect to collect data from various databases2.

When a modular input is needed, a Heavy Forwarder is needed, as the Universal Forwarder does not include a bundled version of Python, which is required for most modular inputs2.

 The splunk diag --exclude command is a way to exclude search artifacts when creating a diag. A diag is a diagnostic snapshot of a Splunk instance that contains various logs, configurations, and other information. Search artifacts are temporary files that are generated by search jobs and stored in the dispatch directory. Search artifacts can be excluded from the diag by using the --exclude option and specifying the dispatch directory. The splunk diag --debug --refresh command is a way to create a diag with debug logging enabled and refresh the diag if it already exists. The splunk diag --disable=dispatch command is not a valid command, because the --disable option does not exist. The splunk diag --filter-searchstrings command is a way to filter out sensitive information from the search strings in the diag

According to the Splunk Enterprise Search Head Clustering (SHC) Deployer documentation, the “local_only” push mode is the correct option when deploying built-in apps. This mode ensures that the deployer only pushes configurations from the local directory of built-in Splunk apps (such as search, learned, or launcher) without overwriting or merging their default app configurations.

In an SHC environment, the deployer is responsible for distributing configuration bundles to all search head members. Each push can be executed in different modes depending on how the admin wants to handle the app directories:

full: Overwrites both default and local folders of all apps in the bundle.

merge_to_default: Merges configurations into the default folder (used primarily for custom apps).

local_only: Pushes only local configurations, preserving default settings of built-in apps (the safest method for core Splunk apps).

default only: Pushes only default folder configurations (rarely used and not ideal for built-in app updates).

Using the “local_only” mode ensures that default Splunk system apps are not modified, preventing corruption or overwriting of base configurations that are critical for Splunk operation. It is explicitly recommended for pushing Splunk-provided (built-in) apps like search, launcher, and user-prefs from the deployer to all SHC members.

References (Splunk Enterprise Documentation):

• Managing Configuration Bundles with the Deployer (Search Head Clustering)

• Deployer Push Modes and Their Use Cases

• Splunk Enterprise Admin Manual – SHC Deployment Management

• Best Practices for Maintaining Built-in Splunk Apps in SHC Environments

A best practice to maximize indexing performance is to minimize configuration generality. Configuration generality refers to the use of generic or default settings for data inputs, such as source type, host, index, and timestamp. Minimizing configuration generality means using specific and accurate settings for each data input, which can reduce the processing overhead and improve the indexing throughput. Using automatic source typing, using the Splunk default settings, and not using pre-trained source types are examples of configuration generality, which can negatively affect the indexing performance

When converting from a single-site to a multi-site cluster, existing single-site clustered buckets will maintain replication as required according to the single-site policies, but never age out. Single-site clustered buckets are buckets that were created before the conversion to a multi-site cluster. These buckets will continue to follow the single-site replication and search factors, meaning that they will have the same number of copies and searchable copies across the cluster, regardless of the site. These buckets will never age out, meaning that they will never be frozen or deleted, unless they are manually converted to multi-site buckets. Single-site clustered buckets will not continue to replicate within the origin site, because they will be distributed across the cluster according to the single-site policies. Single-site clustered buckets will not be replicated across all peers in the multi-site cluster, because they will follow the single-site replication factor, which may be lower than the multi-site total replication factor. Single-site clustered buckets will not stop replicating within the single-site and remain on the indexer they reside on, because they will still be subject to the replication and availability rules of the cluster

Increasing the search factor in the cluster will best address the requirement of high availability for searchable data. The search factor determines how many copies of searchable data are maintained by the cluster. A higher search factor means that more indexers can serve the data in case of a failure or a maintenance event. Increasing the replication factor will improve the availability of raw data, but not searchable data. Increasing the number of search heads or CPUs on the indexers will improve the search performance, but not the availability of searchable data. For more information, see Replication factor and search factor in the Splunk documentation.

When using ingest-based licensing, there are no Splunk roles that require the license manager to scale, because the license manager does not need to handle any additional load or complexity. Ingest-based licensing is a new licensing model that allows customers to pay for the data they ingest into Splunk, regardless of the data source, volume, or use case. Ingest-based licensing simplifies the licensing process and eliminates the need for license pools, license stacks, license slaves, and license warnings. The license manager is still responsible for enforcing the license quota and generating license usage reports, but it does not need to communicate with any other Splunk instances or monitor their license usage. Therefore, option C is the correct answer. Option A is incorrect because search peers are indexers that participate in a distributed search. They do not affect the license manager’s scalability, because they do not report their license usage to the license manager. Option B is incorrect because search heads are Splunk instances that coordinate searches across multiple indexers. They do not affect the license manager’s scalability, because they do not report their license usage to the license manager. Option D is incorrect because deployment clients are Splunk instances that receive configuration updates and apps from a deployment server. They do not affect the license manager’s scalability, because they do not report their license usage to the license manager12

1: https://docs.splunk.com/Documentation/Splunk/9.1.2/Admin/AboutSplunklicensing 2: https://docs.splunk.com/Documentation/Splunk/9.1.2/Admin/HowSplunklicensingworks

The client filters available in serverclass.conf are DNS name, IP address, and platform (machine type). These filters allow the administrator to specify which forwarders belong to a server class and receive the apps and configurations from the deployment server. The Splunk server role is not a valid client filter in serverclass.conf, as it is not a property of the forwarder. For more information, see [Use forwarder management filters] in the Splunk documentation.