Mastering Splunk: A Beginner's Guide to Big Data Analysis

Welcome to SDN TechForum.

So let's continue from our
earlier video, ASA Essential.

And in ASA Essential, we
did essential configuration,

like SSH, SNMP, NetFlow, SPAN,
syslog, and packet tracer.

So now, what happened, this
ASA is sending all the logs

to a syslog server.

And in this video, we are
going to start a new topic.

And that topic is
Fun with Splunk.

So you may be thinking like,
am I watching old video?

No, we'll continue
from there and continue

building up what we did so far.

OK.

Fun with Splunk, as you can see.

What we are going to
do, as a side note,

I'm going to show
you how to configure

a syslog-ng server on Ubuntu,
with a Raspberry Pi on laptop.

And then we will
start with Splunk.

So what I'm going
to do, I'm going

to install the Splunk server.

I'm going to use a
free trial version.

I'll install it on
a Windows machine.

All right.

Then we will have a Splunk
universal forwarder,

which is a small CPU.

I mean, it's not very
resource-intensive.

It's a lightweight agent
setting on the places

where you have the
data, the data which

you want to input
to the Splunk server

so that you can index it,
crunch it, and visualize it.

So this universal
forwarders, they

will be sitting on the sources
where we have our data storage,

and which is generally a
Linux machine or Windows.

Then I'll show you how to input
data from forwarder-- so what

rules you need to create.

And then, finally, we'll come
back to the Splunk Dashboard

again and see if we can
manage the forwarder status.

And also, we can crunch
that data, index that data,

search that data,
all those things.

So this is not going to be
very Splunk-intensive video,

but a lightweight good
for you to get started.

And at the same
time, finally, I'll

give you some forwarder
troubleshooting tips

because, many times,
forwarders, once you set up

the channel, after
some time, you

may see that forwarder
is not sending data.

So how to troubleshoot
that, I'll show you that.

OK.

So quickly, let's first
review the syslog-ng server

configuration requirement.

So this is for a
Linux Ubuntu machine.

So what you have to
do, you have to get

apt-get install syslog-ng.

And then, basically, that will
install the syslog-ng server

and then validate if it is
listening to port number 514.

You can also watch
validate the status

by using sudo services
status syslog-ng.

So syslog-ng is started.

It's listening on
port number 514.

Now what we have to do?

We already did that.

Actually, ASA is sending
syslogs to port number 514

or to this server, all right?

So I'll show you that.

And then we will talk about how
to do in universal forwarder

config.

But let's first validate
syslog-ng server.

So this is our ASA.

Mind it, this is going to be
a little demo-intensive video.

So please try to
follow along with me.

So as you can see, ASA
is sending this logs

to 192.168.1.22.

And that is our Ubuntu server.

All right, I'm going to
show you that IP address.

IP address is 192.1 in this.

And let's do netstat grep 514.

So you can see, this is already
listening on port number 514

for TCP/UDP and receiving
all the syslog details, OK?

Now, it is our turn to
install Splunk forwarder.

Before we do Splunk forwarding,
let's go to the Splunk website,

all right?

So here I am on Splunk website,
and I want free Splunk.

So I created an account here and
downloaded the Splunk Enterprise

software, OK?

Not the cloud one,
the Splunk 8.5,

which is the current software.

You can say Free Splunk,
and you can download.

I already downloaded it, so I'm
not going to download it again.

As you can see, this is a
60-day free trial for Splunk

Enterprise.

This is what I downloaded and
installed on a Windows machine,

OK?

So here is your main
Splunk dashboard.

What we are going to do, we are
going to do a couple of things.

First is we are going to
make this server listen

for data stream, right?

Multiple ways, you can add data.

Like here, if you
click on Add Data,

there are multiple options.

I'm going to skip the tour.

You can do networking,
you can do OS, and upload.

You can actually
upload the data.

So if you have a
compressed file,

CSV file, you can
actually upload it.

But that's not
very scalable way.

We want our data to be
continuously sent as a stream,

and then Splunk to do
all those indexing so

that we can run our searches.

So for that, what
you have to do,

you have to prepare your Splunk
to listen on certain ports.

And that is called
receiving here, OK?

Forwarding and receiving--
configure receiving, OK?

We don't want to
configure forwarding here

because we will be
using forwarding agents.

Only thing is, I want
this Splunk server

to listen on certain port.

And that is port
number triple 97, 9997.

And that's the default
port for Splunk.

So I kept it default. OK,
that's all you want to do here.

Now, since this server is
listening on the designated

port, so now it is
our turn to configure

the universal forwarder.

And for that, what
do you have to do?

You have to download
the forwarder, OK?

And I'll show you from where
to download the forwarder.

So you can do sudo wget and
the wget IP on this part

so that it will get
downloaded to your machine.

And then what you can do is
you can copy that forwarder,

what you downloaded, to
a third-party directory

or a third-party
software directory,

which is /opt cd plus cp.

And then whatever you
downloaded, copy it to /opt.

Then go to that
directory, sudo/opt,

and do a sudo D package,
which is like a package

manager for Ubuntu.

And this is what you have to do.

You may need ins curl
because it is running

some background curl checks.

So make sure you have the
curl utility installed.

And if not, then you have to
do dpkg-reconfigure again, OK?

Once you do that, it will
install the software.

Now, finally, what
you can do, when

you do a list of
directories, you

can see there is a directory
created called splunkforwarder.

Go to splunkforwarder there.

Under that, go to bin directory.

And that is the directory where
you can start, stop, or restart

your Splunk instance,
universal forwarder instance.

So we are going to go to cd
bin and say sudo splunk start.

And make sure you accept the
license from command line,

like this.

Otherwise, you will have to read
the entire license by pressing

Page Up and Page Down.

And, finally, you can validate
the Splunk status check.

All right.

So I'm going to show you all
this on the forwarder itself.

So let's go.

As you can see, I downloaded
this forwarder here

and then parked it to opt.

And here you can see
Splunk forwarder is there.

Under Splunk forwarder, we
have lot of directories, right?

All the local
configuration-related things

are stored in etc, just like
any Ubuntu Linux system.

But this is only for
Splunk-related files, all right?

But right now, we are interested
in checking the status.

So what you can do, you can
just go to bin directory

and then do a sudo.

So Splunk status.

When you install, it will
ask you to create a username

and password.

And that's the-- but this is
the sudo username, password.

OK, Splunk command not found.

OK, pwd/bin.

I am not in the
correct file, OK?

That is the reason.

So let me start over.

I'm going to say
cd/opt/splunkforwarder bin.

That's it.

And then Splunk status.

That's it, Splunk is running.

So my universal forwarder
is properly installed.

As I was telling you that
all the configuration files

are stored in etc.

So let's quickly revisit
the st. Go to system.

Or maybe just list everything
here and see all the--

OK.

All Splunk configuration
related files are here.

And you can read instance
config, licenses.

And even you can go
to system, 3D system.

And you can look at
the local, cd local.

And here is your output config.

Where this universal forwarder
will send the config?

What is the server
config look like?

All those information are here.

But again, as I
mentioned, I'm not

going to go deep into this, OK?

So this is up to you.

Now, what we are
going to do, we are

going to configure
the forwarding rules.

So again, we are going to go to
splunk dot forwarder/bin here.

And let's go back and look
at the configuration again.

So this is for installation.

Now, the rule setting, right?

So what you are going to say?

You are going to say
sudo/splunk add forward-server.

And this is the Splunk
Enterprise IP address, slash

or colon add port.

If you remember, we created
a receiving port 9997.

So put your Splunk Enterprise
IP address colon port number.

Make sure you have
the networking

or reachability between
forwarder and Enterprise server.

And there is no firewall
blocking and other things.

So this is how you will point
your universal forwarder

to the Splunk Enterprise server.

Next, what you want to do?

You want to monitor
the data, right?

The data thing, what you
want to send to the server.

And for that, we have to do
splunk add monitor and then

the file and location.

So here, what I'm doing, I'm
sending my ASA logs, which

is coming to the syslog server.

At this folder, I'm going to
send this to Splunk Enterprise.

And when you
configure these rules,

you may have to
restart the Splunk.

And to do that, you can
just say splunk restart.

That's it.

You can come back always and
check if your forwarder is

active or no.

And if something is wrong,
by using this command, OK?

So now let's go back
and check our forwarder.

Splunk.

Let's look at the
command list forwarder.

You can always do help.

So we are going to say list--

too bad it doesn't do
a tab complete, but--

your session is
invalid, so you have

to log in to your
universal forwarder, OK?

So this is the log-in.

Your username and
password, you will

create, while installing
the universal forwarder, not

your Enterprise Splunk
username and password.

But, for me, both are same.

I re-use the username
and password.

And here you can see, after
putting the credentials,

I can see this is my
active forwarder, what

I configured using port number.

And is there any
inactive forwarder?

No.

So we are good.

So this is how you are going
to create the forwarder.

And now let's validate if
this data is showing up

or if this forwarder is showing
up in Splunk Enterprise or not.

And for that, what you can
do, you can go to Dashboard.

Your dashboard may be empty, OK?

So what you can do, you
can create a dashboard.

OK.

Let's go back to Search first.

And here you can come
and say Data Summary.

A quick way to test
your data inputs

are by setting--
click on Data Summary.

Once you click on
Data Summary, it

is going to look
how many hosts--

that means forwarder-- is
talking to this Enterprise

server.

And if you click on
that, I have two.

One, which is sending the 121,
which is sending [INAUDIBLE].

And that is defined by
this naming convention.

And then another is Ubuntu Pi.

So these two data are being
sent to Enterprise server.

Sources, what source
we are monitoring?

All those things
are listed here.

And source type,
it automatically

tries to classify by reading
the files by some existing rules

and say these are
the source type.

There are various pre-built
source type, like ASS.

Not all those pre-built
source type is there.

You can also build a
custom-built source type.

So let's look at the host
and try to load this.

So here, you can see
all my var/log/firewall,

the place which we are
monitoring on syslog-ng server.

All these logs
started showing here.

And based on these logs here, it
has created some selected field.

You can select those field
and create a new search query.

Right now, it is just
searching on the host name.

And you can see all those events
nicely getting populated here.

You can go back in timeline--

24 hour, 30 minute, five minute.

Everything you can see.

You can create your
own search pattern,

and you can also do
some visualization.

And at the same time, you
can create a table view.

So different ways of
visualization, table format,

bar chart format,
and all those things.

But the nice, cool
thing about Splunk,

which need a little
bit of education

about Splunk Processing
Language, SPL,

so that you can
actually use these

logs to create your search
query or create a pattern,

so that you can present these
logs in a meaningful way.

And that's the end goal, right?

Right now, in today's
video, I'm just

going to making you familiar
with Splunk distributed model.

What is universal forwarder?

What is the Enterprise?

And how you can
bring your logs here.

But you can do much more
by learning a few tricks

in SPL language.

OK.

What else I want to show you?

I want to show you--

if you go to the home
page, Splunk, here I

created the forwarder instance.

So it's a snapshot.

When I come to the
home page, it quickly

gives me a snapshot
of my forwarders,

which are the
forwarders available,

and how their data
pattern looks like.

So, as I mentioned,
I have two of them.

And I can load them here.

I can watch their data patterns.

And I can also click
on any of these

and see who is my receiver.

So this Windows machine
itself is a receiver.

So this is a cool thing
to monitor your forwarder,

if they are sending your
data in the real-time or not.

Finally, if you want to know
something about the Splunk

utilization or the Enterprise
utilization itself,

so what you can do, you can
always go to Monitoring Console

and see here how your
Enterprise server is

doing resource-wise right now.

So basically, these are the
license usage, disk usage,

CPU usage, and all those
things for Enterprise server.

That means how the server
instance installation

is doing health-wise.

Is there any memory pressure?

Is there any CPU pressure?

Are we hitting any license or
disk or throughput indexing rate

threshold?

All those things, you
can manage from here.

All right.

But mostly, why you come
here, if you are not a Splunk

administrator,
you will come here

to parse log for
your application.

And for that, mostly, you
want to create some search

and reporting, create
some cool search indexes,

so that you can find a
needle in a haystack.

So with that, I'm going
to stop this video.

And I'll continue
learning Splunk.

And I hope you will find
it interesting also.

So let's continue this journey.

Thank you.