Splunk Enterprise Security Free Training | Correlation Searches

[Music]

Alright, welcome to my enterprise security

video playlist. This time we're going to

be covering correlation searches. This is

a fancy word for a safe search that

creates an alert. That's really what it

comes down to. They call them notables—

there's a lot of terminology involved—

but the ultimate concept is a

correlation search is a search that

fires off at predefined periods of time,

maybe every five minutes, every hour,

searches back across your logs for

certain behaviors, and if it sees it, it

creates a...it creates an alert. You can

make it create a notable. Technically, it

doesn't have to create a notable, and

I'll explain how that works, but it's

really just a safe search. So let's go

break right into enterprise security, and

let's talk about that.

So I come into enterprise security. We're

going to show what is already outcomes

out of the box. So if
I go 'configure', I'm

in my Enterprise security
and I come into...

'content', and I go to
'content management',

these are all the knowledge objects that

come with enterprise security, and I'm

going to flip this to a
correlation search.

I click that...

we can see that it's going to come back

with lots and lots of results, 58 pages

plus of them and multiple to a page. You

can read this, so I'm just going to go

into the very first one. And this is

'abnormally high number of endpoint

changes by a user'. If I go and open this

up a little bit...

'detects an abnormally high number of

endpoint changes by user account as it

relate to restarts, audits, file system,

user, registry, notifications".

If I go into this...

I'm actually going to be able to see

the query. I'm not going to go explain it

because I can already tell you, it's

probably going to be written with lots

of data models and macros, but out of the

box, you can see: here's the query. And

it's basically...it's going to look at

your data model. You'll hear me talk

about data models. I've discussed data

model, but this is going to be the

endpoint data model, and it's going to be

looking at file systems for changes by the

user, it's going to do a bunch of other

things that ultimately it's going to

come back and say...if you meet a certain

criteria, and you can see that it's

actually using the machine learning

toolkit, so down here it's actually

building a threshold saying, what is the

normal amount of use of changes, and is

it jumping out of that at normal level.

It's really cool, put some really cool

analytics out there for you. You can just

use what they've got. What I love is I

don't want to...I hear, oh

well aren't correlation searches

attached to now frameworks? Well, you can

see the very first ones.
Sometimes they

are. But here, these are frameworks. I've

heard this in my own work,
oh, well they're

all mapped to the miter. Well,

are they? I'll just grab the very first

one, and...there's no miter technique

mapped. What should it be? Well, there's a

lot of things that could cause a miter

technique to...uh...if there's endpoint

changes, it could be many different types

of tact. Then I'll have it mapped. You

could come in here and you could map it,

we'll discuss that later, but point is, we

come down here...

make that go away, that's all...

we can see that it's looking back 1,450

minutes, and the latest time is zero. This

runs at five after the hour, that's how I

read that, five after the hour.

It's...if the results are greater than

zero, it groups by user and change type,

and we see that it creates...it does not

create a notable, it actually just

provides a risk analysis. And we'll

discuss risk analysis when we talk about

RBA. But the point is, you can make it do

a bunch of adaptive responses.

My job here is not to help you

understand every correlation search that

comes out of the box, I'm here to discuss

the part that most people don't know how

to do: create your own. So I've shown you

that you can go look through...there's

the documentation on Splunk, says 1400

plus, I don't know how they define what a

correlation search is. I'm going to tell

you that it's a lot. There's a

lot of them. And by default,

enterprise security is smart. They do

not come enabled. If I look at the

enabled correlation searches,

this is mine that I was using as I

started to help understand

enterprise security,

and these two were turned on

and this is for risk-based approach.

Other than that, there are no correlation

searches that come out of the box. Why?

Well, one, they don't want to turn

something on that doesn't fit your data

set; two, often you have
to tweak them, the

correlation search is great, but it's not

always going to be perfect for your

environment, and so as a general rule,

they're there as a guidance. Use them

when they make sense,
turn one on, test it,

see how it works.
If it doesn't, modify it,

and typically you'll just clone the

correlation search and build your own.

Anyway, enough talking about that, let's

talk about actually building my own

correlation search. So I'm in 'configure

content' and I went to
'content management'.

If I do 'create new content', that's how

I'm going to build one. And so we're

going to create a new content,

we're going to make a correlation search.

This is the way that I
do correlation searches.

That doesn't mean it's the way

that it has to be done,
but it's the way it works for me.

I'm going to call this, I

would hopefully have a much better name

for this, but I'm going to do 'YouTube

Correlation Search'.

Horrible name, because someone who comes

across this will have no idea what it's

for, but for me, when I need to purge

stuff from my system, it's really easy

and it stands out. So I'm going to put it

that way. Then here in my description, I'm

going to go...

'Grab one event from network logs'.

I'm not actually going to build

something that I'm looking for.

That's not the point of this video.

I'm just showing how
to build one, and I want

them to always fire, so I'm going to

fudge the numbers so that I always

get what I want. And so the first thing I

do is I don't try to build a search

through here. You can use a guided.

Guided's cool, it'll allow you it'll pick

data models, you can pick fields from it,

so if I enable the guided mode, you'll

see the data, it'll say alright, what

data model do you want to look at?

I might come down to 'network traffic'...

and what data set do I want to use...

'all traffic'. Do I want
to use 'summaries only'?

I'll discuss summaries only later

this is not the place for it. Time range.

And there is your basic query. I can run

the search and see how it looks.

Then I'm going to hit

'filter', and filter would be like

All.Traffic...

AllTraffic.destIP...

oh.

it's a boolean. Where...

and I actually don't know how to make

this work. All_Traffic...

I'd have to go look this up. Well that's

not very good...helpful
there. The point is,

I'm not actually going through the

guided search tour. I'm going to stay

right here with a manual query where I

can write it. It does have guided, again,

you got to understand exactly what

you're pulling. Guided is nice if you

know, follow the docs. I'm not here for

following the docs, I'm here to take a

query. This is my home network. I'm going

to look at the correlate logs. I'm going

to look at my correlate conn logs. I'm

going to say...where source IP is

192.1680.*. That is only so I make

sure that I'm looking at a specific

subnet section of my network. This is

primarily my network designed for doing

Splunk videos, and so this isn't my...

this is part of my home network, but it's

a subnet on my network that I use for

testing, pen testing, setup of systems

that I tear up and pick up and tear down,

and so I just want to know what they're

doing. And so I wanted the source IP

Maybe you don't want the source IP.

All I really cared about though, is I just

wanted this, because ultimately, later

down, I'm going to do inventory, and I'm

going to have a very simple inventory of

that subnet, and so I only want IPs that

at least one piece of the data

ties to my inventory. And so, as you can

see, this here has nothing to do with my

network, but this one does. And I'm going

to do a head 1, because I don't

want lots and lots of results.

Basically, I want a query

and I'm always going to return one

result...and that's what I built.

This isn't bad. This isn't actually a

known bad, I just wanted data to come

back, so then I can put other stuff on it.

I'm doing this as a demo for you guys to

understand how

to build a query. You would want to build

a query that actually is looking for

something malicious. Right now, I just

want a query to return a result, so that

I can...when I do my next video about

triage and the triage system, there are

actually tickets coming in. If I write a

query that's looking for bad, well, that

bad better be occurring on my network or

it's not going to fire. And so it's a lot

harder to troubleshoot if the thing is

working if you're building queries right,

If you build something that isn't...

you hope to not actually
see on your network.

So I actually hope to see
correlate conn logs.

I sure hope so. That means my

network has traffic. Anyway, and I'm just

going to put the head 1, because I only

want it to create one alert. If I let it

come back, it's every event that comes

back in here would be a notable alert.

I don't want my triage
system getting inundated.

So I'm just going to do this head 1.

Now I'm going to map it. I'm going to go

to miter, and I'm going to

put in some

tickets. So I'm going to go 'T1143'. I

actually can't remember what all these

mean off the top of my head. You can go

look them up. I'm going to say this, and

this has note, no bases whatsoever, but

again, these videos are

going to build on themselves. And so I'm

building these miter attacks so when I

go to the RBA section of this video

playlist, you'll see how it maps all the

different techniques together. And so I'm

going to put this down here,

and actually, because
I want this to work on

my system, I'm going to actually do...

I want it always to be 0.128,

that way I'm only going to get alerts

that are relating to this system.

That means my risk-based
approach will cross

the threshold. That actually makes a lot

more sense for me. I'll explain that when

we actually get to RBA, but basically, I'm

going to give me...
give me an alert every time

0.128 is the source of network traffic.

And that should fire off
quite frequently.

Ignore the picture up in the top.

We're just going to move on.
Head 1.

My videos are done rendering.
Anyway, so I'm going

to map it to these TTPs. Again, this is

all for demo purposes, so I just pick

some TTPs, and I can come down here and

I can put a confidence score,
an impact score,

contacts, analytics, we're just gonna

leave that alone for now.

I can create my own framework

And now here it's going to say

how far back do I want to look? Do I

want to look back 24 hours?

I could, but I know how often

my logs are firing. I'm going

to look back one hour. Doesn't really

matter, because I'm just grabbing head 1.

And...I have...I probably get

hundreds of events every...probably

thousands of events every hour

on this particular subnet. And so it's

not going to be a problem getting data.

I'm going to go look back one hour to

now. And how often do I want it to run?

You know what? I'm going to let it run

every five minutes. And that's going to

be important so that I actually have

events. And that'll work.

I'm going to come down here,
and I'm going to say do I

want it to run as real time or

continuous. We'll just
leave it at its default.

What's my scheduling window? Again,

these are...I'm not going over these, this

is just basically how you want to run

your times. I'm going to run this

every five minutes. Schedule priorities

in case there's conflicts. Hopefully with

your enterprise security, you actually do

not overload your system so these become

a big deal.

Trigger conditions, number of results

greater than zero, that's always going to

be the case because I'm getting back one.

But if I was doing this, if I want to do

thresholds I could make it...the thing has

to occur at least 10 times, or 15 times,

or whatever. Then windows durations

filled to group by...that's it. That's all

I want to deal with. Really, the only

places I put around with this is I wrote

a query in the most basic format to get

your correlation searches going. Pick a

search. I would tie it to an annotation,

but you don't have to, not required.

You come down, here pick your time window,

these three boxes, how far back do you

want to look, latest time, earliest time,

and your cron schedule, and then you

really don't have to touch anything else,

except this 'add adaptive response'. I'm

going to come and modify this in a

minute. There is, when we talk about RBA,

I'm going to put a risk analysis. For the

sake of keeping this simple, I am only

going to do

notables for now. So I'm going to come in

here and I'm going to click a notable.

A notable is an alert that goes to

your triage system.

Gonna go...'YouTube

notable'. Give it a description.

I can actually use...

variable substitution, so I'm going to do

'Alert for $src_Ip'.

I need to make sure that field comes

back, and this does have a source IP, so I

can use it, and you just call it like you

do in with the dollar sign on both sides

of a variable, and that'll be dynamic. And

so my description will come back with

this. And just because I

want to, what if I...yeah, we'll just

leave it at that.

YouTube notable security domain. There

are a bunch of domains. This is dealing

with access areas, that would be

authentication, endpoint, a lot of your

host logs, network logs, threat, identity,

and audit. And so those are the six areas

Splunk has as security domains. We'll

just leave it as a...
we'll put as a network.

In the network domain, I'm going to put

the severity

as low.

And default owner, I can put in these,

I can leave it unassigned.

I'm going to put it as
unassigned to start with.

Again, you don't have to.

Default status, I'm going to
put it as unassigned.

And I could put a drill down search in

there, and let's do that.

We're going to take this very same query.

Just to keep things really simple, one of

the very first drill downs
I want to put in there

is the actual query

that created this log.

But in this case, I'm not going to put

head 1, I'm going to put...I'm going to

take the head out.

Oh, it looks like I've lost the 128 on

there. 128.

Make sure 128 is up here.

Yeah, it is. Okay, and I can choose...

the drill down search will be

'See...

what caused alert'.

There are other ways of doing this I'll

show, but I'm just going to

create a few add drill down searches.

And here, we're going to just do

'why does

this

drilldown exist'.

I just want to show
I can go search anything.

'Index equals internal'.

Why would you be looking at your

internal logs? It doesn't really matter.

Well, actually, let's just do this.

I'm going to put in '$src_ip$'.

So I'm basically looking in my internal

logs, and I'm going to see if I find that

IP address popping up. It's just kind

of an interesting way you can add

additional searches to your information.

So I'm going to be searching my internal

logs for the source IP.

And I hope you saw this earliest offset,

latest offset. You can change this, or you

can you can let it just go by its

default. Or you can say, for here I'm

going to go

plus, this is a earliest,
for example, one hour

and I'm going to leave
the other one as zero.

Does that make sense? So I hope

this helps. I can change my time.

It's basically going to look in this

window one hour back, based off of

the time this event occurred.

So this might actually look a little bit

in the future, this is
gonna look a little bit

in the future.

It's going to use time in the back.

So let's go...

we're going to go one hour....

this is going to go one hour in the

future and one hour in the past.

Sounds good. I'm going to leave my

investigation profile alone. And these

are...extractions, and what it's

going to do is it's going to

identify identities, these are users

and stuff like that on your network.

Assets would be like IPs, and machines,

and files, and URLs that it might have

found. I'm going to...we got assets here.

Source dest.

Does my log, do my logs contain

source and dest?

Well, let's go look. Had one, do I actually

have a source and a dest here?

I have a source IP, but no source.

So I don't have the
field it's looking for to

be able to identify it. So what I need to

do is I need to come in here,
and I'm going to go

'$src_ip$',

except it's on identity.

The identity...it's an asset so I'm going,

to come in here and I'm going to go

'source IP'.

And just because we might

want to identify the other

machine in question.

We're going to put dest
IP in there as well.

So I'm going to have my source IP

and my destination IP.

They're going to be assets that are

extracted. And that's all I'm going to do.

I just want to make sure that

anything that might be identifiable in

these queries...not these queries,

the query up here. Let's call them out.

And I hope all this
will make more sense as

you actually see the stuff come back.

There's just a lot of capabilities here.

I can write steps if I want to, I can set

things up to, for example, send an

email, stream capture if you have

Splunk Stream, nbstat and it's...

You can make your system do a lot of

things. Like, I could have Splunk go ping

an IP address. You know what?

In a little bit, I'll actually show me

doing that. I can have it do a risk

analysis, run a scripts,
send a UBA, send a

Splunk mobile. Splunk mobile is really

cool. Now it's being sent to my phone. Add

thread intelligence from it, web hooks,

whatever. You have a lots of capabilities,

don't need to do it. The minimum you

need for a notable:

title, description,

you don't even need these drilldowns,

you can let this be set as default,

probably should pick a security domain,

and literally, that's it.
Make sure...it's a

lot more helpful if you can identify

your stuff coming back as identities and

sources. And I'm going to show you that

in the next video with workbenches and

stuff like that, but for the sake of this,

don't worry about it.

Just know that it's it's good if you

can call it out, but if you don't,

it's not like the query will break.

I'm going to hit save,

and I should have a
correlation search done.

Now I'm going to have to wait.

I probably just missed my window. It's

supposed to be kicking off
five minutes after the hour,

so I can almost guarantee that if I come

to incident review,
I will not find an alert

called 'YouTube notable'.

I'm gonna have to wait 'til five more

minutes go by, but let's go ahead and

check that. So I can come down, I can

refresh the page here, or I can refresh

the page here. But either way, that is not

the purpose of this video is to look at

the incidents coming in. Mine was to talk

about correlation searches and how to

make my own. I have set up a correlation

search, and so I've accomplished my task.

I'm gonna come see it here

with a configure content.

Configure content, content management.

My new correlation search is in here.

We can see that when I go
'all correlation search'...

And when you create them, by default,

they are enabled.

So if I come in here and I enable,

I can see 'YouTube correlation search'
for Lame Creations.

If I want to make any changes to it,

I just hit search. Now, that's interesting

that it doesn't say that
it's actually scheduled.

Alright, well, probably because it

hasn't run the very first time. Once it

runs, I should see

here the next schedule time.

But it's really easy,

just keep it under the enabled

and correlation searches.

So...yep, there it is.

Now I've got a time for
the next scheduled time.

stored in the Enterprise Security app.

What have we covered?

We've talked about correlation searches,

what they are, they're safe

searches that can be used to create

notables. Notables fill out tickets that

will go into a ticket triaging

system, which we will cover in the next

video in this playlist. Please look at

the link below, notice that this is a

playlist. Go ahead and join the playlist

and watch the videos. This is meant to be

a comprehensive training to help you

understand enterprise security.

Click that link. We have now....I've

shown you how to see the correlation

search that come out of the box, and I've

shown you how to create your own from

scratch. I hope this has been helpful, I

hope this helps you move from being a

lame analyst to a Splunk Ninja, that

you'll keep following, particularly this

playlist, watch the videos in it, and that

they're helpful.

Anyway, hope to see you around.