CTA

Splunk is among the best load management and analysis solutions in the IT industry. It is one of the topmost analytics and Big Data tools and has an extremely high demand in the corporate world, so is the case with Splunk professionals. If you wish to become a successful Big Data professional, having expert knowledge and skills in Splunk is important. This is why the competition in the market is challenging for Splunk jobs. Here, we have created a list of Splunk interview questions with the help of industry experts so that you can be prepared for your interview. Take a look at the following Splunk interview questions and get ready to crack your interview:

Q1. Compare Splunk with Spark.

Q2. What is Splunk?

Q3. What are the common port numbers used by Splunk?

Q4. What are the components of Splunk? Explain Splunk architecture.

Q5. Which is the latest Splunk version in use?

Q6. What is Splunk Indexer? What are the stages of Splunk Indexing?

Q7. What is a Splunk Forwarder? What are the types of Splunk Forwarders?

Q8. Can you name a few most important configuration files in Splunk?

Q9. What are the types of Splunk Licenses?

Q10. What is Splunk App?

The Splunk Interview Questions blog is largely divided into the following parts:

1. Basic

2. Intermediate

3. Advanced

Check out this video on Splunk Tutorial:

Top Splunk Interview Questions and Answers Top Splunk Interview Questions and Answers

Learn for free ! Subscribe to our youtube Channel.

Basic Interview Questions

1. Compare Splunk with Spark.

Criteria Splunk Spark Deployment area Collecting large amounts of machine-generated data Iterative applications and in-memory processing Nature of tool Proprietary Open-source Working mode Streaming mode Both streaming and batch modes

2. What is Splunk?

Splunk is ‘Google’ for our machine-generated data. It’s a software/engine that can be used for searching, visualizing, monitoring, reporting, etc. of our enterprise data. Splunk takes valuable machine data and turns it into powerful operational intelligence by providing real-time insights into our data through charts, alerts, reports, etc.

3. What are the common port numbers used by Splunk?

Below are the common port numbers used by Splunk. However, we can change them if required.

Service Port Number Used Splunk Web port 8000 Splunk Management port 8089 Splunk Indexing port 9997 Splunk Index Replication port 8080 Splunk Network port 514 (Used to get data from the Network port, i.e., UDP data) KV Store 8191

4. What are the components of Splunk? Explain Splunk architecture.

Below are the components of Splunk:

Search Head : Provides the GUI for searching

: Provides the GUI for searching Indexer : Indexes the machine data

: Indexes the machine data Forwarder : Forwards logs to the Indexer

: Forwards logs to the Indexer Deployment Server: Manges Splunk components in a distributed environment

5. Which is the latest Splunk version in use?

Splunk 6.3

6. What is Splunk Indexer? What are the stages of Splunk Indexing?

Splunk Indexer is the Splunk Enterprise component that creates and manages indexes. The primary functions of an indexer are:

Indexing incoming data

Searching the indexed data

Picture

7. What is a Splunk Forwarder? What are the types of Splunk Forwarders?

There are two types of Splunk Forwarders as below:

Universal Forwarder (UF) : The Splunk agent installed on a non-Splunk system to gather data locally; it can’t parse or index data.

: The Splunk agent installed on a non-Splunk system to gather data locally; it can’t parse or index data. Heavyweight Forwarder (HWF): A full instance of Splunk with advanced functionalities.

It generally works as a remote collector, intermediate forwarder, and possible data filter, and since it parses data, it is not recommended for production systems.

Enroll in our Splunk Course in London to get a clear understanding of Splunk!

8. Can you name a few most important configuration files in Splunk?

props.conf

indexes.conf

inputs.conf

transforms.conf

server.conf

9. What are the types of Splunk Licenses?

Enterprise license

Free license

Forwarder license

Beta license

Licenses for search heads (for distributed search)

Licenses for cluster members (for index replication)

CTA

10. What is Splunk App?

Splunk app is a container/directory of configurations, searches, dashboards, etc. in Splunk.

11. Where is Splunk Default Configuration stored?

$splunkhome/etc/system/default

12. What are the features not available in Splunk Free?

Splunk Free does not include below features:

Authentication and scheduled searches/alerting

Distributed search

Forwarding in TCP/HTTP (to non-Splunk)

Deployment management

13. What happens if the License Master is unreachable?

If the license master is not available, the license slave will start a 24-hour timer, after which the search will be blocked on the license slave (though indexing continues). However, users will not be able to search for data in that slave until it can reach the license master again.

14. What is Summary Index in Splunk?

A summary index is the default Splunk index (the index that Splunk Enterprise uses if we do not indicate another one).

If we plan to run a variety of summary index reports, we may need to create additional summary indexes.

Learn more about Splunk from this Splunk Training in New York to get ahead in your career!

15. What is Splunk DB Connect?

Splunk DB Connect is a generic SQL database plugin for Splunk that allows us to easily integrate database information with Splunk queries and reports.

Intermediate Interview Questions

16. Can you write down a general regular expression for extracting the IP address from logs?

There are multiple ways in which we can extract the IP address from logs. Below are a few examples:

By using a regular expression:

rex field=_raw "(?<ip_address>\d+\.\d+\.\d+\.\d+)"

OR

rex field=_raw "(?<ip_address>([0-9]{1,3}[\.]){3}[0-9]{1,3})"

17. Explain Stats vs Transaction commands.

The transaction command is the most useful in two specific cases:

When the unique ID (from one or more fields) alone is not sufficient to discriminate between two transactions. This is the case when the identifier is reused, for example, web sessions identified by a cookie/client IP. In this case, the time span or pauses are also used to segment the data into transactions.

When an identifier is reused, say in DHCP logs, a particular message identifies the beginning or end of a transaction.

When it is desirable to see the raw text of events combined rather than an analysis of the constituent fields of the events.

In other cases, it’s usually better to use stats.

As the performance of the stats command is higher, it can be used especially in a distributed search environment

If there is a unique ID, the stats command can be used

18. How to troubleshoot Splunk performance issues?

The answer to this question would be very wide, but mostly an interviewer would be looking for the following keywords:

Check splunkd.log for errors

Check server performance issues, i.e., CPU, memory usage, disk I/O, etc.

Install the SOS (Splunk on Splunk) app and check for warnings and errors in its dashboard

Check the number of saved searches currently running and their consumption of system resources

Install and enable Firebug, a Firefox extension. Log into Splunk (using Firefox) and open Firebug’s panels. Then, switch to the ‘Net’ panel (we will have to enable it). The Net panel will show us the HTTP requests and responses, along with the time spent in each. This will give us a lot of information quickly such as which requests are hanging Splunk, which requests are blameless, etc.

19. What are Buckets? Explain Splunk Bucket Lifecycle.

Splunk places indexed data in directories, called ‘buckets.’ It is physically a directory containing events of a certain period.

A bucket moves through several stages as it ages. Below are the various stages it goes through:

Hot : A hot bucket contains newly indexed data. It is open for writing. There can be one or more hot buckets for each index.

: A hot bucket contains newly indexed data. It is open for writing. There can be one or more hot buckets for each index. Warm : A warm bucket consists of data rolled out from a hot bucket. There are many warm buckets.

: A warm bucket consists of data rolled out from a hot bucket. There are many warm buckets. Cold : A cold bucket has data that is rolled out from a warm bucket. There are many cold buckets.

: A cold bucket has data that is rolled out from a warm bucket. There are many cold buckets. Frozen: A frozen bucket is comprised of data rolled out from a cold bucket. The indexer deletes frozen data by default, but we can archive it. Archived data can later be thawed (data in a frozen bucket is not searchable).

By default, the buckets are located in:

$SPLUNK_HOME/var/lib/splunk/defaultdb/db

We should see the hot-db there, and any warm buckets we have. By default, Splunk sets the bucket size to 10 GB for 64-bit systems and 750 MB on 32-bit systems.

Interested in learning Splunk? Enroll in Intellipaat’s Splunk Training today!

20. What is the difference between stats and eventstats commands?

The stats command generates summary statistics of all the existing fields in the search results and saves them as values in new fields.

command generates summary statistics of all the existing fields in the search results and saves them as values in new fields. Eventstats is similar to the stats command, except that the aggregation results are added inline to each event and only if the aggregation is pertinent to that event. The eventstats command computes requested statistics, like stats does, but aggregates them to the original raw data.

21. Who are the top direct competitors to Splunk?

Logstash, Loggly, LogLogic, Sumo Logic, etc. are some of the top direct competitors to Splunk.

22. What do Splunk Licenses specify?

Splunk licenses specify how much data we can index per calendar day.

23. How does Splunk determine 1 day, from a licensing perspective?

In terms of licensing, for Splunk, 1 day is from midnight to midnight on the clock of the license master.

24. How are Forwarder Licenses purchased?

They are included with Splunk. Therefore, no need to purchase separately.

Interested in learning Splunk? Go for the online instructor-led Splunk Training in Toronto!

25. What is the command for restarting Splunk web server?

We can restart Splunk web server by using the following command:

splunk start splunkweb

26. What is the command for restarting Splunk Daemon?

Splunk Deamon can be restarted with the below command:

splunk start splunkd

27. What is the command used to check the running Splunk processes on Unix/Linux?

If we want to check the running Splunk Enterprise processes on Unix/Linux, we can make use of the following command:

ps aux | grep splunk

28. What is the command used for enabling Splunk to boot start?

To boot start Splunk, we have to use the following command:

$SPLUNK_HOME/bin/splunk enable boot-start

29. How to disable Splunk boot-start?

In order to disable Splunk boot-start, we can use the following:

$SPLUNK_HOME/bin/splunk disable boot-start

Learn the complete concepts of Splunk from Intellipaat’s Splunk Training at Hyderabad in just 26 hours!

30. What is Source Type in Splunk?

Source type is Splunk way of identifying data.

Advanced Interview Questions

31. How to reset Splunk Admin password?

Resetting Splunk Admin password depends on the version of Splunk. If we are using Splunk 7.1 and above, then we have to follow the below steps:

First, we have to stop our Splunk Enterprise

Now, we need to find the ‘passwd’ file and rename it to ‘passwd.bk’

Then, we have to create a file named ‘user-seed.conf’ in the below directory:

$SPLUNK_HOME/etc/system/local/

In the file, we will have to use the following command (here, in the place of ‘NEW_PASSWORD’, we will add our own new password):

[user_info] PASSWORD = NEW_PASSWORD

After that, we can just restart the Splunk Enterprise and use the new password to log in

Now, if we are using the versions prior to 7.1, we will follow the below steps:

First, stop the Splunk Enterprise

Find the passwd file and rename it to ‘passw.bk’

Start Splunk Enterprise and log in using the default credentials of admin/changeme

Here, when asked to enter a new password for our admin account, we will follow the instructions

Note: In case we have created other users earlier and know their login details, copy and paste their credentials from the passwd.bk file into the passwd file and restart Splunk.

32. How to disable Splunk Launch Message?

Set value OFFENSIVE=Less in splunk_launch.conf

Learn more from Intellipaat’s insightful Splunk Tutorial!

33. How to clear Splunk Search History?

We can clear Splunk search history by deleting the following file from Splunk server:

$splunk_home/var/log/splunk/searches.log

34. What is Btool?/How will you troubleshoot Splunk configuration files?

Splunk Btool is a command-line tool that helps us troubleshoot configuration file issues or just see what values are being used by our Splunk Enterprise installation in the existing environment.

35. What is the difference between Splunk App and Splunk Add-on?

In fact, both contain preconfigured configuration, reports, etc., but Splunk add-on do not have a visual app. On the other hand, a Splunk app has a preconfigured visual app.

36. What is .conf files precedence in Splunk?

File precedence is as follows:

System local directory — highest priority

App local directories

App default directories

System default directory — lowest priority

37. What is Fishbucket? What is Fishbucket Index?

Fishbucket is a directory or index at the default location:

/opt/splunk/var/lib/splunk

It contains seek pointers and CRCs for the files we are indexing, so ‘splunkd’ can tell us if it has read them already. We can access it through the GUI by searching for:

index=_thefishbucket

Are you interested in learning Splunk from experts? Intellipaat’s Splunk Course in Bangalore is the right choice!

38. How do I exclude some events from being indexed by Splunk?

This can be done by defining a regex to match the necessary event(s) and send everything else to NullQueue. Here is a basic example that will drop everything except events that contain the string login:

In props.conf:

<code>[source::/var/log/foo] # Transforms must be applied in this order # to make sure events are dropped on the # floor prior to making their way to the # index processor TRANSFORMS-set= setnull,setparsing </code>

In transforms.conf:

[setnull] REGEX = . DEST_KEY = queue FORMAT = nullQueue [setparsing] REGEX = login DEST_KEY = queue FORMAT = indexQueue

39. How can I understand when Splunk has finished indexing a log file?

We can figure this out:

By watching data from Splunk’s metrics log in real time:

index="_internal" source="*metrics.log" group="per_sourcetype_thruput" series="<your_sourcetype_here>" | eval MB=kb/1024 | chart sum(MB)

By watching everything split by source type:

index="_internal" source="*metrics.log" group="per_sourcetype_thruput" | eval MB=kb/1024 | chart sum(MB) avg(eps) over series

If we are having trouble with a data input and we want a way to troubleshoot it, particularly if our whitelist/blacklist rules are not working the way we expected, we will go to the following URL:

https://yoursplunkhost:8089/services/admin/inputstatus

For more on these, visit our Splunk Community!

40. How to set the default search time in Splunk 6?

To do this in Splunk Enterprise 6.0, we have to use ‘ui-prefs.conf’. If we set the value in the following, all our users would see it as the default setting:

$SPLUNK_HOME/etc/system/local

For example, if our

$SPLUNK_HOME/etc/system/local/ui-prefs.conf file

includes:

[search] dispatch.earliest_time = @d dispatch.latest_time = now

The default time range that all users will see in the search app will be today.

The configuration file reference for ui-prefs.conf is here:

http://docs.splunk.com/Documentation/Splunk/latest/Admin/Ui-prefsconf

41. What is Dispatch Directory?

$SPLUNK_HOME/var/run/splunk/dispatch

contains a directory for each search that is running or has completed. For example, a directory named 1434308943.358 will contain a CSV file of its search results, a search.log with details about the search execution, and other stuff. Using the defaults (which we can override in limits.conf), these directories will be deleted 10 minutes after the search completes—unless the user saves the search results, in which case the results will be deleted after 7 days.

42. What is the difference between Search Head Pooling and Search Head Clustering?

Both are features provided by Splunk for the high availability of Splunk search head in case any search head goes down. However, the search head cluster is newly introduced and search head pooling will be removed in the next upcoming versions.

The search head cluster is managed by a captain, and the captain controls its slaves. The search head cluster is more reliable and efficient than the search head pooling.

43. If I want to add folder access logs from a windows machine to Splunk, how do I do it?

Below are the steps to add folder access logs to Splunk:

Enable Object Access Audit through group policy on the Windows machine on which the folder is located Enable auditing on a specific folder for which we want to monitor logs Install Splunk universal forwarder on the Windows machine Configure universal forwarder to send security logs to Splunk indexer

44. How would you handle/troubleshoot Splunk License Violation Warning?

A license violation warning means that Splunk has indexed more data than our purchased license quota. We have to identify which index/source type has received more data recently than the usual daily data volume. We can check the Splunk license master pool-wise available quota and identify the pool for which the violation has occurred. Once we know the pool for which we are receiving more data, then we have to identify the top source type for which we are receiving more data than the usual data. Once the source type is identified, then we have to find out the source machine which is sending the huge number of logs and the root cause for the same and troubleshoot it, accordingly.

45. What is MapReduce algorithm?

MapReduce algorithm is the secret behind Splunk’s faster data searching. It’s an algorithm typically used for batch-based large-scale parallelization. It’s inspired by functional programming’s map() and reduce() functions.

46. How does Splunk avoid the duplicate indexing of logs?

At the indexer, Splunk keeps track of the indexed events in a directory called fishbucket with the default location:

/opt/splunk/var/lib/splunk

It contains seek pointers and CRCs for the files we are indexing, so splunkd can tell us if it has read them already.

See more at:

http://www.learnsplunk.com/splunk-indexer-configuration.html#sthash.t1ixi19P.dpuf.

CTA

47. What is the difference between Splunk SDK and Splunk Framework?

Splunk SDKs are designed to allow us to develop applications from scratch and they do not require Splunk Web or any components from the Splunk App Framework. These are separately licensed from Splunk and do not alter the Splunk Software.

Splunk App Framework resides within the Splunk web server and permits us to customize the Splunk Web UI that comes with the product and develop Splunk apps using the Splunk web server. It is an important part of the features and functionalities of Splunk, which does not license users to modify anything in Splunk.

48. For what purpose inputlookup and outputlookup are used in Splunk Search?

The inputlookup command is used to search the contents of a lookup table. The lookup table can be a CSV lookup or a KV store lookup. The inputlookup command is considered to be an event-generating command. An event-generating command generates events or reports from one or more indexes without transforming them. There are many commands that come under the event-generating commands such as metadata, loadjob, inputcsv, etc. The inputlookup command is one of them.

Syntax:

inputlookup [append=<bool>] [start=<int>] [max=<int>] [<filename> | <tablename>] [WHERE <search-query>]

Now coming to the outputlookup command, it writes the search results to a static lookup table, or KV store collection, that we specify. The outputlookup command is not being used with external lookups.

Syntax:

outputlookup [append=<bool>] [create_empty=<bool>] [max=<int>] [key_field=<field_name>] [createinapp=<bool>] [override_if_empty=<bool>] (<filename> | <tablename>)

CTA