CompTIA Linux+: Logical Volumes & Filesystem Hierarchy #5

You're watching ITPRO.TV.

Hello, and thank you for watching ITPRO.TV,

helping you level up with IT learning

everywhere you go. I'm your host, Zach

Memos, for this episode of CompTIA Linux Plus.

Logical Volume Manager is the name of

this episode, but you knew that, didn't

you? And the person we know and we love

and we trust, Don Pezet, is here to help us out.

Don, good to see you. Hey. Glad to

be back. And, you know, tackling a pretty

cool technology this episode, we're going

to be looking at the LVM, or the Logical

Volume Manager. It's a really neat

utility that allows us to have a lot of

flexibility with our file systems.

In previous episodes, I talked about how

we don't want to change file systems

after we create them. LVM actually makes

that a lot easier. So in this episode,

we're going to see,

you know, what the LVM is, obviously, but

how to install it, how to get it

configured, how to arrange our disks

using the LVM, and then provide us that

flexibility down the road. So that's all

coming up right here in this episode.

So what do we need to know to get our

system ready for the LVM? Alright. So

the odds are that your distro supports

the LVM. It might not be installed, but

you should have support for it. Most of

them do have it installed by default.

Unless you do, like, a minimal install, it might not.

We can check that really easily, so that's

going to be kind of step one. The other

thing is we're going to need some hard

drives. Right? The LVM is a Logical

Volume Manager. It's requiring, to

create logical volumes or kind of

virtual disks, and those have to lay on

top of actual physical disks.

So we're going to need some in order to do this.

Now, you can use LVM with your, you know,

with your root directory and your home

and so on. So you can have a lot of

things mounted that way. In fact, when you

install Linux, you probably had the

choice to pick to use the LVM. If you

watched our installation episodes, I didn't choose that.

So my disks are currently set up without using the LVM.

If I want to create it from scratch, I'll

just have to know a handful of

commands. So let me run through a couple

of the basics here. The thing, I'm

going to make sure that LVM is installed.

There's a few ways to do it, but the

easiest way is to use your package

manager. So I'm going to do a

yum list lvm*. The main package

you're looking for is called lvm2

because it's version 2 of the Logical

Volume Manager. And so when I look at

mine, I see I've got installed packages:

lvm2 and lvm2-libs. The libraries--

those are already installed, so I'm good

to go. If they weren't installed, I would

just do a sudo yum install lvm2.

If you're on an Ubuntu machine, I believe

the package there is actually just

called lvm. I don't think it has the

number, but it is version 2 that it

installs by default. So I've got it. It's right there.

And so now I know that I've got the

tools. The next part are the disks.

I'm going to need some disks, and I'm going

to be attaching them to the LVM system.

So when I do that, it actually expects

you to already have a partition on the

disk. You don't have to format it, but it at

least needs to have a partition. And,

normally, we fret about the size of the

partition and the disk type and all that

stuff. LVM really doesn't care about that.

It just needs to have a partition in

place. So if you've already got one,

you're in business. I'm going to do an lsblk,

and I've got three disks: sdb, sdc, and sdd. And each of them--

they're one-terabyte disks, and

they've all got a single partition on

them that is one terabyte in size. That's

perfect for what I need here in this lab.

If your disks don't have a partition, you

just need to create one, and we saw that

in the last episode--or actually two

episodes ago. You can use the fdisk utility.

You know, sometimes we worry about using

GPT versus MBR. Here, none of that's

really going to matter because the

logical volumes are what we're going to

manipulate. So pick whatever partition type you like,

throw it on there. You don't need to

format it. We don't need to drop a file

system on it. Just have the partition

present, and then you'll be ready to

launch it with LVM and start to get your

volumes set up. So now that we're

prepared, how do we actually create

logical volumes, Don? Alright. So now we

can go in and we start to create this, but

it is actually a little bit--

I'm going to call it complex. It's not

that complex, though, but there's actually

three steps we have to get through.

Let me bring up a diagram here to show you

so we can kind of have a better understanding of this.

Inside of the LVM, there's three

different things you work with: there's

physical volumes, there's volume groups,

and there's logical volumes. Alright?

We have to create all three.

We're going to start with the physical volumes.

Now, notice it says physical volumes, not

physical disks. Right? I have

three physical disks, and they've got a

partition on them, but they're not

considered volumes yet. We're going to

bring those physical volumes--those disks--

into LVM. So we've got to create physical

volumes, and these will match up with our

physical disks. I've got three disks,

so I'm going to create three physical volumes. Right?

Then we create a volume group.

A volume group is where we gather the physical disks

together to create a combined pool of storage.

Alright? In the diagram here,

there's three two-terabyte drives, so

they get gathered together into six terabytes of storage.

Once that's done, then we can start

slicing and pulling from that storage

however we want. I can say, alright,

I need a three-terabyte disk, and so we can

slice that out of the volume group by

creating a logical volume. Now,

three terabytes is bigger than any one of

these physical disks. LVM will manage

spreading that data across the disks to

be able to allow for that space. And

what's really neat is down the road, if I

decide, man, I'm running out of space,

I need to add another terabyte to that

disk--we can go in and expand that logical volume.

We can add another terabyte onto the side of it--just slice

it right out of the volume group. Or what

they've done here in the example is

sliced out a second three-terabyte disk

and pulled it right out. So LVM lets us

do that, but it requires these three components. We're going to have to create each one.

Fortunately, we've got tools that do all

that. It's all when you install that lvm2

package--it installs a whole collection of tools.

To help you with this,

if you memorize these three names, which

you definitely want to do for the exam,

the tools almost always start with

letters that match up with what you're

doing. So the physical volume tools

almost all start with pv, and then the

logical volumes start with lv, and the volume

groups start with vg. So when you take a

look at your system--let me just jump over here.

If I take a look inside of /usr/sbin,

and I look at--I want to work with

physical volumes, so I'll just look for pv*.

Alright? I can see right here, I've

got pvcreate, pvchange, pvremove, pvresize.

These are all my physical volume

utilities that let me mess around with

them. And then if I want to look at

volume groups, I can look for vg*, and

I'll find all of them--almost the same as

what we saw for physical volumes, except

we have some new options in here for,

like, vgimport, vgreduce for shrinking and growing--

you know, all those different ones.

And then lastly, we had logical volumes--that was lv*,

and I'll see each of them tucked away

inside of there, like the

lvdisplay and so on. Some of them are

aliases or shortcuts. A lot of them are

interchangeable. Like, lvdisplay will

show me all of my logical volumes.

lvs actually stands for logical volume show.

It will show me my logical volume.

So, some of these kinds of overlap with each

other, but it’s pretty easy to spot the

utilities, and we just need to go through

creating them. You create from the

ground up. We’re going to create our

physical volumes, then we’re going to

create our logical volume--sorry--

we create our physical volumes, then

we create our volume groups, then we create

our logical volumes on top.

So that’s the basic process that we’re going to go

through. Let me just run through

that. I’m going to start a little

more simple than in this diagram:

I’m just going to do two disks. I’ve got two

one-terabyte disks. I’m going to combine them

into a two-terabyte volume group that

I can then slice out into a logical volume. Alright?

So what I can do is I’m going to start with pvcreate, alright?

And actually, let me sudo this so that

I’m an administrator.

So I'm going to sudo the pv create command, and I need to

tell it the disks that I'm bringing in.

Now remember I said we need a partition

on each one. We saw I already have some

partitions. So I'm going to bring in /dev/sdb1 and /dev/sdc1. Alright.

Those are the two disks. I have a 1/sdd1.

I'm going to hold on to it because I

want to show you guys adding more

storage after the fact. So I'm just going

to grab these two. So when I do that, it

sees the disk. And remember how I said

you don't have to format it? Here's why.

It sees I've already got a format on

there, and it's going to wipe it.

It's going to erase whatever's on there.

It needs the partition. It doesn't need the

file system. So I'll say yes to let it

wipe it. It's going to do the same thing

on the other disk. There we go. So they're

wiped, and now they've been brought into

the LVM. And I should be able to see that by doing pv display,

which is my physical volume display. Oops.

Non-root user. So let me just rerun that. Whoops.

And there we go. And so now it's going to

show me those two physical volumes, sdb1

and sdc1. They're one terabyte. They each

have a UUID that's been assigned, and

I can see all of that right there. So

they've been brought in the way that I want them.

Now I can create a volume group to

combine these two together, and I'm just going to use,

you guessed it, vg create. Now this

command is a little bit different.

I'm going to create a volume group, and that

volume group needs to have a logical

name that I can map to. When I have a

hard drive, the hard drive is referenced

as /dev/something, right, a device ID.

Well, our volume groups need to have device IDs also, and there's a

service that's running in the background

called the Logical Volume Manager

Mapper, the mapper service. And the mapper

service is what's going to create /dev

entries for our logical volume, so

it's really important. So when I start to

create a volume group, I have to give it

a name for what I want the volume group

to be, and that's going to become part of

the device name for my logical volumes.

They'll be /dev/mapper/ and then the

volume group name followed by / and the

logical volume name. So it gets kind of

long. So when I create this one, I need to give it a name.

I would normally name this based off of

what I intended the pool storage to

be used for. But here on the show, just to

keep things a little bit easy, I'm going to

call it vg1. This is volume group one.

And that way, you guys will easily be able to

tell, you know, that I'm working with a

volume group. So I'll stick with vg1, and

then I need to attach the physical

volumes to it. My physical volumes were

/dev/sdb1 and /dev/sdc1.

Alright. So that's going to create the

volume group, and it's going to attach

those two physical volumes to it. If I do a vg display...oh,

let me sudo that. There we go. I can see

that it's here. I can see that it has somewhere in here,

my size here, my volume group size.

It's only showing 15 GB right now, so I need

to give that a moment for it to kind of

sync up. But if I scroll back up here...oh,

sorry. I'm looking at the wrong volume

group. I already have a volume group for

something else. If I looked at the VG

name, see how that one's called CentOS? Mhmm.

That's one of my, you know, one of the

ones the operating system created, not

mine. So I need to scroll up. Here's vg1,

and it's got the right storage. Thank

goodness. So you can see right here, I have

1.95 terabytes of storage available

inside of this volume group,

ready to be sliced out into logical

volumes. And to create those logical

volumes, we're going to use lv create,

right, which I'm going to sudo. One of

these days, I'll remember to do that right off the bat.

With a logical volume, we're slicing

out storage and turning that into a disk.

So I'm going to do lv create, and I need to tell it the size

that I want this to be. Okay? I want mine

to be, let's say I'll just do 200 GB

or, you know, I could do the full 2

terabytes if I wanted or 1.5 terabytes.

You know, maybe I'll do 1,500

gigabytes. So I'll put in 1500g,

and then I need to tell it the volume

group that I'm getting the storage from.

I might have more than one volume group.

I just have one, so I'll type vg1.

Then I need to give it a name. And, again,

normally, I would name this, like, web

storage or file storage or, you know, something

that told me what it was for. But in this

case, to keep things easy, I'm going to

call it lv1. This is logical volume number one. Alright.

So when I go to create that, it builds it,

and we can use lv display to be able to verify that. Alright?

So when I take a look here again, I'm going to

ignore the ones the operating system

created and find mine. Here's lv1.

That's a part of volume group one, and it is 1.46

or basically one and a half terabytes in

size. Bigger than one of my disks. Right?

And so I sliced it out. I still have more

space available, but that one's grabbed

right there. And if you look at the

device name that it assigned,

we've got /dev/vg1/lv1. Right? So it's not so hard to remember.

It's not /dev/sda or /dev/sdb because

there's no physical device. These are

virtual mappings, /dev/vg1/lv1.

Now in Red Hat and CentOS, they do a really

good job of mapping it that way, so it's

easy to remember. You just do the volume

group followed by the logical volume.

In other distros, they'll nest it under a

mapper folder, which I likely have.

If I go into /dev and take a look,

yeah. Here's that mapper folder that I

was talking about, and I don't know if

it'll let me go into there. Oh, it did.

So if I go into the mapper folder, see

there's a folder in there called vg1/lv1.

So that's another name for that device.

I could do /dev/mapper/vg1-lv1 like that.

That would also be a

device name that points to the same

place, but it's kind of nice having

/dev/vg1/lv1 like that.

That's easier to remember. It makes a little

more sense for us humans. Just know that

not every distro does that, so look for

that mapper folder if you need to.

But at this point, I've created the physical

volume. I've created the logical volumes,

you know, the volume group in the

middle. We've created all that. It's

running. I used the set of commands that

are the most common for me.

There are more commands, though, that are available.

So, for example, I did pv display, vg

display, and so on. I could have also done pvs

and, oops, it is

not liking the fact that I left a bunch

of other stuff on that command. So let me

redo that. Here we go. pvs, which gives

us kind of an abbreviated view of the

volumes. When I did pv display, we got a

lot more information. pvs, it's kind of

summarized. And so I can see the sdb1,

sdc1 that are attached. I also have

vgs for volume group, and so we can see that, again, kind of summarized.

And lastly was lvs to see the

logical volumes. So if you just want a

quick view, those tools are also really

handy to be able to look at.

And then there's, like, pvscan, which will

scan your system in trying to find

physical volumes to bring in. So there are

a number of other utilities too, but the basic creation

is done with those handful of commands

right there. Are we able to format

and mount the logical volumes like

regular disks now? Oh, yeah. Yeah. So after this,

it acts just like a normal disk. It's got a

slightly weird name, but otherwise, it's a

normal disk. So, for example, if I want to

format this one, let's go ext4.

Right? So I would do sudo mkfs.ext4.

So I'm going to format the disk, and I'll point to /dev/vg1/lv1.

And it's going to find

that disk, and the mkfs command will

think it's a physical disk, and it'll go

in and format it. And now I've got a one

and a half terabyte disk that's ready.

I can then mount that. So maybe I'll,

let me create a folder here. I'm going to

mount that in /mnt/storage.

And so I will mount /dev/sg1, not sg1, lv1

and I'm going to mount that into /mnt/storage like that.

So now it's mounted, and I can go and start putting

files in there and working with it. If I

do df -h, I can see /mnt/storage right there. One and a half

terabytes of space available. It's ready

for me to start dumping things in there,

and I can work with it like normal. Do notice on

the left side, it's not showing /dev/vg1/lv1.

It's showing that mapper folder

that I mentioned, /dev/mapper/vg1-lv1.

So you may see that notation

kind of switch as you work with it.

But they both function the same way. I mounted it, it's online.

I can forget about the LVM at this point

and just go on with my life and never have to think twice about it.

It functions just like any other disk.

Well, are they persistent, or do we need to

add them to the file system table?

Oh, just like a normal disk, we would need to

add this to the file system table or /etc/fstab.

It's not really any different. Although, I will tell you,

you need to be a little bit careful with

the file system table because

a lot of disks get mounted right away

when the system boots, and sometimes the

LVM service hasn't started yet. In which case,

the mapper might not have had time to do

its job. So, when you're mounting inside

of the file system table, it's usually

better to use the /dev/mapper name

as opposed to using /dev/vg1/whatever because that name might

not exist yet at boot time. You can try

it out on yours to see what works, but

basically, you would just add that to

/etc/fstab just like any other disk except for the device name.

Well, here, you can see they've got one already mounted: /var/log/audit is /dev/mapper/whatever.

So, as long as you use the mapper name, you'll

be in good shape, and you can mount it,

and it'll be persistent. So, Don, how would

we go about adding more storage?

Alright. So, I mentioned at the beginning of the

episode that I had that disk that I was

holding. I got another terabyte. So, let's

just say, over time, I fill up this one and a half terabyte

storage drive that I just made, and

I want to add more storage. Well, I actually

I've got 500 GB more storage in this

volume group already that I could slice

out and add and expand to two terabytes.

But maybe I know that I'm going to add

another terabyte of data. So I need to

add more than that 500 GB that's available.

I can go and add another

physical disk to my server, and I can

incorporate that into the LVM.

I can bring it in as another physical volume

and expand out my volume group. Once I've

done that, I can expand out my logical

volume. And then lastly, I can expand out

my file system. And before you know it,

I'll have more storage. So, let's see what that looks like.

Basically, I'm going to go to my command

prompt. And off, I would need to

shut my system down, add another hard

drive, right? And then we would need to

create a partition on it. I mentioned we

always need to have a partition there.

If I do lsblk, I can see that SDD disk right there.

It's already got a partition on it.

And, actually, I didn't bring this up earlier,

but I can see the volume group and

logical volume names have been attached

inside of lsblk, so I can see those guys

are tied up into logical volumes.

Alright. So, I want to incorporate SDD1

into my physical volumes. So I'm going to

bring that in just like we did at the

beginning of the show. I'll do a sudo pvcreate.

I'm creating a physical volume. Yes. And that'll be /dev/sdd1. Alright? So that's going to bring it in, alright.

Now, that just makes it a physical volume. If it had a partition on it,

it would have formatted it and gotten it ready, but it didn't actually

connect it to the volume group. So now I

need to add it to the volume group. And for that,

it's actually a little bit different.

When I did it earlier, I was creating the

volume group, and you can add physical

volumes at the time. Well, now I've

already got a volume group in place, so I can't kind of use the same command from before.

I've already created it. So instead, I need to use a different command,

which is the vgextend command.

Alright? So vgextend says take the existing volume group and

add more to it. I do need to sudo this, which I've literally forgotten every single time.

Yep. [Laughs].

And so, I'm going to be extending vg1, and

then I need to tell it the disk that I'm

adding to it. I'm adding /dev/sdd1. And so when I run that, I'll see

"Volume group 1 successfully extended."

So what does that mean? Well, if we run PVS,

we'll see my physical volumes. Right?

And I can see sdd1 right here has been added.

Alright? Under free space, I see it's

completely free, versus the other ones that have had space used up.

If I do a VGS, then I'll see my volume groups.

And notice how volume group one

shows that it's actually 2.93 terabytes.

It's three terabytes, and that it's got one

and a half terabytes free. There's the

500 GB from the previous volumes that

I didn't use, and now there's a whole

extra terabyte that's been stuck onto it.

So it's expanded. Quite a bit.

But now that I've got that space, the volume

group has been updated. I need to upgrade

or update the logical volume. And for

that, I'm going to be extending out the

logical volume. Now, before I do that, let me just run LVS.

And so, looking at it here, I can see lv1

right now is one and a half terabytes.

So I'm going to run sudo

lvresize, and I'm going to resize that

partition. I want to grow it by one

terabyte. I'm going to add some more space to it.

So I'll come in and just say -l and then +1G.

So I'm adding one terabyte to it.

And then, I don't want to do one giga, I want to do one terabyte.

There we go. Something actually

is meaningful. Ah-ha.

I'm going to add a terabyte to it,

+1T, and then I'll specify this is /dev/vg1/lv1.

That's the logical volume that I want to expand, or

I could use the mapper, you know, whichever one I wanted.

Alright. And when I run that,

it's going to go and extend it.

And it tells me that it extended it from

1.46 terabytes to 2.46 terabytes.

It added that extra space to expand that out.

You might think you're done.

And, technically, as far as the LVM is

concerned, you are done. Right? If I do a

LVS and look at my logical volumes, it's

now two and a half terabytes. So great. We're done.

Except the file system won't recognize

that yet. If you do a df -h

and take a look at it, it still shows as

being a one and a half terabyte partition. And that's because I formatted

it with ext4. Right? And the ext4

partition that's on there is still one

and a half terabytes. It's got an extra

terabyte of free space on it, but it doesn't know that.

The LVM has expanded everything that it

can expand. Now it's up to your file

system. So if you're running XFS or

ext4 or whatever, you'll need to use that

file system's tooling to be able to

expand it. So this is where things

start to differentiate. I'm running ext4

on this one, so I can choose its tools.

With it, you have a utility called resize2fs.

With XFS, you have to use the xfs_growfs

command. Or, no. Actually, it's xfs_resize, I think, is the one for that.

Be careful with it. If you're running Red Hat,

Red Hat will tell you they do not

officially support resizing XFS partitions.

So, you can do it, but they don't support it because they tell you

you just need a backup, format, restore.

So that's kind of how they are.

But, with EXT, we can just use the resize2fs command, and I'll point that at /dev/vg1/lv1.

And if you don't provide any other options, this utility will just expand it

to fill all the available space. So when I run that, it sees there's more space.

It goes and acquires that. And if I take a look at my storage now,

that disk shows as being two and a half terabytes in size. And I did it

while it was online. I never unmounted it.

Right? It stayed mounted the whole time.

Users could have been in the system

writing and so on, and it just tacks it

on, adds that space, and we just grew it.

And that's really the big advantage of

the LVM: that we can grow our partitions

without actually having to kick our

users out, without having to reboot a

server. We can do it all dynamically

while people are in the system working--

you know, happy, shiny people holding

hands and all that stuff--and it's done.

If I do an lvs or lvdisplay, I can see

I've got that two and a half terabytes

is available there, and the file system sees it as well.

So we are now expanded and done. So after all that, I'm going to

ask: is it difficult to remove the LVM if

we no longer need it? You know, it's

not difficult to remove it.

If you're done with it--well, if you want

to move it to another computer. Right?

Maybe I've got this on one server. I want

to move it somewhere else. You can just unmount all this stuff

and then take it over to another

computer and reincorporate it. The pvscan utility is designed to find your physical volumes.

When it finds them all, it'll bring it

into the LVM, and it'll see the volume

group and the logical volumes are all

there. So it's kind of easy to

reincorporate somewhere else. But if I'm

actually done with it, I want to turn

these back into regular disks. Right?

Because right now, it's a bit of a mess.

If I do lsblk, you'll see all these different

kind of notes in here about what it created with the volume groups and the logical volumes and so on.

So if I want to just reset these back to becoming regular

disks, we can move through a couple of

steps. I need to unmount the folder

that it's tied to, which is /mnt/storage.

So I'm going to unmount it

using the umount command. So umount /mnt/storage.

And then we basically just go through and destroy everything

backwards. So we start at the top of the

diagram with the logical volumes.

We destroy that, then we destroy the volume

group, then we destroy the physical

volumes. And when you're done, now you've

got regular old disks available again.

So what that looks like is: we do sudo lvremove,

and I'll remove /dev/vg1/lv1.

So I'm going to remove that logical

volume, and I would remove each one.

If I had more than one--I just have the one--

that was lvremove. Then we do

vgremove to remove the volume group, and that's /dev/vg1 for me.

So we'll remove it. Mhmm. It's gone. If it has no logical

volumes, then you don't get a prompt to

confirm because there's no data at that

point, so it just removes. And then lastly,

we remove the physical volumes, which is

a little bit harder because we have to

type each one. So I'll do sudo pvremove,

and then I'm going to remove /dev/sdb1, /dev/sdc1, and /dev/sdd1. Alright?

Each one, removing them. And see how it

says "successfully wiped"?

It actually kills off that file system. And now if I do an lsblk,

I'll see where those guys are just three

regular old disks with regular old

partitions sitting on them, waiting for

me to format them and put them back in use as non-LVM disks. So, pretty easy to remove all this stuff.

Logical Volume Manager. Another wonderful

episode Don. Thank you very much.

And before we move on, what else would you

like to say? You know, most of the

distros that are out there turn LVM

on by default if you're doing a desktop install.

So you likely already have it.

On servers, it's usually not on by

default. They're just standard

assignments. And, honestly, if you're not

going to be expanding your disk storage,

then the LVM doesn't really bring you

any benefits. So, there's some

snapshotting features that are kind of

nice that we didn't really get into,

but you can actually do that stuff

without the LVM too. So when I do server

deployments, it is rare that I use the

LVM because there's just no real benefit

to it. So keep that in mind. Just because

the feature's there doesn't mean you

have to use it. On desktops, though, it is

kind of nice. If you've got a storage

disk and you're running out of space,

then you can add more. If you've got a

laptop, though, it's pretty hard to add extra

storage to a laptop. You're usually

replacing a disk, and the LVM doesn't

help with that. So do understand that it's not

always the right solution. Sometimes it's

perfectly fine to just go with regular

disks and, you know, create your partitions

on there. It can be a little easier to

manage sometimes too. Great advice. Thanks,

Don. And here's some more great advice: watch

every episode of CompTIA Linux+.

You're going to be very glad you did.

It's going to help you now and into the future.

And don't forget about the supplementary

information in our course library that

will do only one thing: help you be more

successful. So check that out too.

And tell everybody you know about ITPRO.TV.

ITPRO.TV is binge-worthy. Thanks for

watching. I'm Zach Memos. And I'm Don

Pezet. We will see you again soon.

[Music].

Thank you for watching ITPRO.TV.

Welcome to ITPRO.TV. I'm your host, Don Pezet. [Crosstalk].

Live from San Francisco’s home country.

You're watching ITPRO.TV.

Hello, hello, and thank you for watching ITPRO.TV,

helping you level up with IT learning

everywhere you go. I'm your host, Zach

Memos, for this episode of CompTIA Linux+. File System Hierarchy

Standard is the name of this episode.

And once again, Don Pezet is here to show us

the way. Don, good to see you. Nice to work

with you again. Yep. Glad to be here.

And, you know, we're going to tackle a pretty

neat topic here, which is that File

System Hierarchy Standard. Many of you,

as you learn Linux, have probably

experienced where you browse to the hard

drive and you see all these crazy

folders--folders that oftentimes just

have the three-letter names that don't make

a whole lot of sense--and

that's a pretty big obstacle to

getting in and learning a new operating

system. So what we're going to do right

here is we're going to take a look at

the File System Hierarchy Standard and

see what exactly are those folders, what

do they do, what is their purpose, how are

they arranged, how it varies from distro

to distro, and get a little bit more

familiarity with that file system.

If you're coming from something like Windows,

well, you just have a couple of folders

that it uses. Inside of Linux, though,

there's actually a lot of different

folders, and we need to be familiar with them.

It'll help you better use the system, but

it will also help you with

troubleshooting and locating

configuration. So definitely a good skill

to have, and that's what we're going to

be learning about right here in this

episode. So, Don, who created the FHS?

Alright. So the FHS has actually

been around a long time, right? And that's

easier to say than "File System Hierarchy Standard," I guess--

which, oddly enough, is "file system." Say that:

file system is actually two words. Right?

File and system. Yeah. But in the standard,

they decided to combine them, which is

not dictionary correct, but they did.

So it is the FHS. And, basically, what

happened is--remember that a lot of the

Linux stuff that we work with came from

the UNIX world. So, back in the 1970s at AT&T Bell Labs, they created the UNIX

operating system, and they had to use a

file system to identify practically

everything. Even hardware was represented

as files. Well, it was just a couple of

guys, and they were making this stuff up

as they went. Fast forward to today, where you've got

Linux and FreeBSD and OpenBSD and all these different

platforms that are all reliant on that

same file system structure. How are you

supposed to create applications that are

compatible from one distro to another if

you don't even know where the files are supposed to go?

So they teamed up at one point--this was

decades ago now--and came up with a

standard that said, "Here are the folders

that are the locations that we expect

things to be in, in order for your system

to work." And by making consistency, that

helps with compatibility. You know, with

Linux, anybody can go out there. So you,

right now, if you want, could go and

take the Linux kernel source code and

grab some GNU utilities or any other

number of utilities that are out there

and package them together and create

your own distro. And you can map the file

system however you want. You can create

files with whatever name you want,

folders with whatever name you want, put

configurations wherever you want.

You have that flexibility. It's free, right?

But if you're expecting party applications to run,

they're expecting to see certain folders.

And that's where we end up where we

are today--where we have a standard.

And the standard was put together--I kind of

got off tangent--so going back to your

question, who created this thing?

It actually came from the Linux Foundation.

And we've talked briefly about the Linux

Foundation. I think I've got their web page pulled up here.

Whoops. The Linux Foundation is the

organization that is--or was--headed up by

Linus Torvalds himself. So, you know,

obviously, the creator of the Linux

kernel is going to be involved here.

But it's actually got members from a number

of organizations. Companies like Red Hat

and Canonical are a part of it.

The individuals that create projects

like Slackware and other things--they're

all a part of it. And so it was a group

effort amongst all the major distros at

the time to come up with this standard.

And it's not like they were really

inventing the standard--most of it just

came from UNIX--and everybody just agreed,

"Hey, from now on, we're going to follow

this standard." So when I log into a Red

Hat Enterprise Linux box, I can expect

/etc to be there, and I can expect

configuration files to be in that folder.

And then when I go to an Ubuntu box,

I can expect /etc to be there and

configuration files to be there. And even

if I jump outside of Linux and go to a

BSD system, I can still expect /etc

to be there and that it's going to have

configuration files--because that's what

was agreed upon. So, the Linux Foundation--

they're the ones who put this all together.

Almost everybody adheres to it, so it

works out pretty well. But it does

really kind of come from Linus Torvalds

himself as part of his leadership in

the Linux Foundation. Well, Don, where can

we view the official standard? So they

do publish the standard. It's online in

what is a spectacularly old web page.

This is it right here.

So--it does look a bit old. If you want to

look at it, I'll put the link in the show

notes, but it's pathname.com.

Pathname.com. pathname.com/fhs

That'll take you right here to this

amazing web page, which is pretty neat.

What's really neat about it is how

static it's been. It hasn't changed much.

I can see right here, the current version is

2.3, which was announced January 29, 2004--

which means, as of the filming of

this episode, that was 15 years ago.

There has not been a change to the FHS in 15

years. That's a long time. And right on

here, you can actually come through and

view the standard. They have it in

PDF, raw text, and also HTML, which is

probably the easiest. And if you browse

into that, you'll see the actual standard

laid out right here. And they're

basically defining all the different

folders, defining which ones are required,

which ones are optional, and what role

that folder is supposed to serve. What is

it supposed to do? And you can kind of

see that here, like /dev is supposed

to contain device files, or /etc is

supposed to have host-specific system

configuration. And if you drill into it a little more deeply,

we can come in and find specific options

about it--variations, types of files that

might be expected to be found in there.

In earlier episodes, we've seen files

like the passwd file. So right here, it's

defined. The FHS doesn't just define

folders. It also defines individual files

that are inside of it. And so if you've

ever found yourself, like, scratching your

head: why is the passwd file--why is it the same everywhere?

It's because it's right here. It's part

of the standard. Everybody adheres to

that, and you can kind of browse

through and see. It's all freely posted

right there online. Do all Linux

distributions adhere to the FHS? You know,

I'm going to say that right eventually.

The FHS. Thank you. They should,

but--as is the case with most

technologies--they don't all do it.

Most vendors are actually really good

about it. So I mentioned Ubuntu, or

Canonical, and Red Hat. Those are kind of

the two biggest commercial players in

the Linux world. They adhere to it

pretty well, but they both deviate a

little. In fact, I'm not sure I could

name a Linux distro that completely

conforms to the standard because there's

some weird stuff in the

standard that maybe was relevant back in 2004

and today is not so relevant--

especially when it comes to security,

there are some things that have changed,

and then the way that we need to

lay things out. So, when I talk about variations,

let me give you an example. I don't know

if I can find this real quick in the

standard, but there's a folder /sbin, and here, they define it

right here--for system binaries. Okay?

So the /sbin folder, it's got--well--

system binaries. Great. Mhmm. Well, we also have a

/usr/sbin, right, which is under /usr. So here we've got the

/usr hierarchy under section 4,

and you can find /usr/sbin in

the list there. Nonessential

standard system libraries. So both

folders have system libraries, but one

folder is for essential stuff, and one is nonessential.

Well, why do we need to separate those out?

Now, in the old days, there was a case where

hard drive space was at a premium.

Maybe you're allocating out partition spaces

to say, "I've got this much space for

/sbin and this much more space for /usr/sbin," or whatever. There was a use

case for that. But today, storage is cheap.

You know, we don't necessarily need separate

partitions. And so, a lot of distros have

started combining those. And, you know, I'm

on a CentOS system. Let me show you here.

This would be the same--I believe

Ubuntu is doing the same today--so it

should be the same on either one. But if

I navigate in my CentOS box, I can go to cd /sbin, right?

And I can take a look inside of

there, and I see a bunch of

executables. Right? So here are these

system binaries and libraries. So there it is.

And then I can go into /usr/sbin,

and I can take a look, and I see a bunch of files and binaries.

If you're a sharp observer, you might

notice that they're the same. Right?

I'm seeing the same files in both folders, and

that's because /usr/sbin is a real folder.

If I pull up the full listing here, see

how I've got sbin, and it's flagged as a

directory right there? It's a real folder.

But if I take a look at /sbin, it's a link. It's a symlink.

So what happened is the CentOS team--really Red

Hat and the RHEL team--they made the

decision that there really wasn't a need

to separate these out. It just creates

confusion. So they combined everything

into /usr/sbin, and then they just symlinked /sbin.

And that's important to note. They symlinked it. They said it is

still important to have /sbin

because many applications expect that,

but we're not going to maintain separate

files anymore. We're going to go ahead

and combine it. And you'll see that.

Like, up here, I've got /bin--same kind of thing--

/lib, /lib64, where if a vendor decides to deviate

from the standard, they normally put symlinks in place so that they still

technically conform to the standard even

though the files are in a different

location. So if you learn the standard,

you'll be pretty safe. You know, if I'm

navigating my file system and I go to /sbin, not /usr/sbin, thanks to that

symlink, I still find everything that I

want. I can still use the system the way

that I expect. So it's pretty rare that

somebody will deviate from the standard

and not put symlinks in place to

preserve that compatibility. It's pretty

rare for somebody to step in and say, "I

don't care about compatibility." And that

usually only happens on really,

really specialized distributions--things

like pfSense. Right? That’s--well, take

BSD, not Linux. But anyhow, it's

specialized for firewalls, and they say,

"We don't need compatibility. You're not

loading other applications." It's just

going to run what we've put on there. And so,

those are scenarios where they might

deviate, but most people do conform to

the standard. Well, Don, where would we

normally find applications?

Alright. So our applications that we

run, right? I just actually showed two

examples right there: /sbin or sorry /usr/sbin.

But applications can actually come from a

number of places. There's really, I'd say, five locations by default, but

you can put an application wherever you

want. And when you install an application,

it can go anywhere. It can end up all

over the place. That makes it hard to

find. And that's why, in an earlier episode, we

learned about some commands to be able

to locate applications. But, for example,

if I go into my root folder here,

there are a couple of folders that are

really designed to be able to store

those applications. And--let

me zoom in a little bit here--

alright, one is /bin, right?

/bin--bin is short for

binaries. Right? And applications are

typically binary files that we flag as

executable, and they can run. So /bin

seems like a really great place for

those to be. Yeah. And then there's the user variant of it, which is

/usr/bin. Same thing for /sbin and /usr/sbin. So those are a couple places.

But these folders--these aren't where, like, if I install OpenOffice, OpenOffice

isn't going into /sbin, or at least not normally.

So applications that I install--

those are considered optional

applications, and they will normally go

into /opt—the OPT right there.

So that's kind of that place: /bin, /sbin, /usr/bin, /usr/sbin, and /opt. Now, when

you install an application, there are

actually a number of vendors that will

maybe not put it in /opt, let you

pick somewhere--maybe it runs from your

home folder. And they may actually create

links to their application inside of /usr/sbin or somewhere, and that

creates that uncertainty of like,

"Where the heck is this app? How am I

going to find it?" But let me run over

the official standard and what these

folders are for. Right? The one is /bin. This one is what's

considered essential command-line

utilities that are available to all

users. Alright? So this is stuff that

is really considered essential for the

system to work. And if you were to browse

around on a system where these were

separated and poke around in there, it's

going to be things like, system CTL, we'll actually that'd be an

administrative commanded that'd be an sbin, so there's a little variation.

But commands that are critical

just to the regular operation--things like

mount, right? In order to mount a file

system, I need the mount command, so it

would need to be in /bin. /sbin, on the other hand, that

stands for system binaries, and system

binaries are required for the system to

boot up. If it's missing a particular

binary, it can't even boot anymore.

When the Linux kernel runs, it then needs to

fire up the operating system. It needs to perform operations, and

it needs a certain set of commands.

Those are going to be found inside of /sbin.

Then you've got the user one, /usr/bin.

That one is going to have the--the--the rest

of our user commands. User commands that

aren't considered essential--we don't have to

have them, but they're nice to

have. So they end up in /usr/bin. And then /usr/sbin, those are

nonessential system binaries. So they're

ones for the system to use--for the kernel to use, really--

but they're not essential. The system

would boot up if those were all missing.

Now, those four right there really all

pertain to system commands, not

applications. And that's where we have

/opt. That's where we're going to find things like

OpenOffice or whatever if we installed it.

In fact, here--let’s take a look at mine.

So on my system, I know it's symlinked.

So if I go to /bin, I'm actually seeing

/usr/bin. They're really the combination of the two.

And if we look around inside of there,

we're going to find things like the zip

utility. If I want to create an archive,

right? Or I'll probably find gzip in here as well. There's yum to be able to do

package updates and operating system updates.

I don't know--there's a ton of junk in

here. The cat utility and things.

And these are all--these are all really

useful utilities, ones we use all the

time. Right? But they're not essential. I--I

don't--I don't have to have them in order

for the system to boot. But I might have

to have them in order to perform some

kind of operation. Right? But it's not

like the system depends on them to boot.

If I look inside of /sbin, on the

other hand, I'm going to see a bunch of

things in here that deal with, like,

interacting with the file system. Right?

Things like fdisk, and so on. We'll see

other utilities in here, like for

interacting with the CPU, for the printing

subsystem, and so on. This is stuff that

users aren't normally going to run.

Sometimes we might, though. If you watched

our storage episodes, we used all these

logical volume commands when we were

working with the Logical Volume Manager, or

LVM. Well, if you look--see how they're all

glowing blue? Mhmm. They're not actually here

in this folder. They're linked from

somewhere else. If I do a -l on that, I can see that--well, actually, most of

these are just aliases, aren't they?

So they're all just linked to the lvm

command, which actually is right here in

this folder--so not--not somewhere else.

So some of these aren't even really

commands. They're just aliases or

shortcuts. The mkfs commands are like

that. If we pull up all of those--

well, they actually do show up as

binaries: mkfs.ext2, and so on. But some of

them, like these guys right here, are just

aliases off of a base command. So you'll

see variations there. If I look in /opt--actually, I don't think I've

installed anything on this system.

Looks like it has some kind of Red Hat

binaries in here. Whoops.

And, actually, I don't. Yeah. That's an

empty--empty little world right here.

I haven't installed any applications on

this system because we, you know, just

installed it for the show. But if I start

installing things like OpenOffice and

Firefox and so on--if they come from

system packages--then they may actually

end up in the main binary folders. But if

I'm installing them from a third-party

site, then they'll typically end up right

here inside of /opt. That's that

optional software. And I've noticed over

the years that Linux distributions are the

worst about using this folder.

BSD systems are usually the best about

always using it, pushing all sorts of things

in there. So you'll--you'll see variations there.

But I guess I've just done a really long

answer to your short question: "Zach,

where are apps?" Those are kind of the

five main places where you're going to

find the application files. "What are some

of the other key folders we need to be

familiar with, Don?" Alright. There are a

lot of other folders in here, and kind

of looking at--actually, you know what? Let me--

I'm going to use the GUI. I'm going to cheat.

I don't normally use the GUI, for whatever reason, but let me just

browse into the file system here. Here we go.

Alright. So, taking a look at these

folders--when you browse in your file

system, there's a lot of crazy folders in

here, some that are super mega insane important. Right?

And then there's some that you don't really

care about. Right? And if you don't know

them--if you're not familiar with them--

then they don't make a whole lot of

sense. So let's just kind of run through

some of the critical ones.

If you read the FHS standard, there's actually something like 50 folders that are defined.

You don't need to know them all

for the exam. You don't even need to know

them all for real life. There's really

just a handful that we deal with on a

regular basis. Probably the most

important one is this guy right here:

/boot. Mhmm. Right? The /boot folder--it's almost

always a separate partition, and so this

is more of a mount point than a folder.

And that's because it contains the Linux kernel

and other files that are basically fired

up when your system boots up.

In the very beginning, this folder gets

called. If you're doing disk encryption,

you can't encrypt the /boot folder

because otherwise, you wouldn't be able

to read your boot files. So this will be

a separate partition that's unencrypted.

It could be combined on a single

partition, but it, you know, causes

problems with things like LUKS

disk encryption. So that's the /boot

folder. And if you ever browse into it,

you'll see things in there--basic

configuration files. You'll see some stuff about vmlinuz. That's the actual kernel file right there.

That's the thing that gets run. And I've got GRUB--that's my

bootloader that helps it find the

partitions of the operating system.

EFI, that's my--my--my UEFI BIOS so that

it's able to locate and boot the

operating system. So all--all really critical

stuff, and that's found in /boot.

Now, you rarely mess with what's in /boot. In fact, on a lot of distros,

this will be mounted read-only because you

don't want somebody messing with the

kernel on the disk, and so it's kind of

flagged that way. It depends--yours may

or may not be flagged read-only. And you can

always mount it read-write, and that's

what happens when you do an update or

you need to change your bootloader,

it'll have to get mounted that way.

There's /dev, which is short for

"device," and this is where all of our device definitions go.

In Linux, every piece of hardware is

represented as a file. And if you watched

our storage episodes, we saw that with

hard drives. All of our hard drives were

/dev/sda1 or /dev/--you know--mapper/, or /dev/whatever.

It doesn't just stop with hard drives.

You'll see that with your console, your

displays, your video card, printers--all

sorts of things will show up in here as

a device file. So that's kind of what I'm

looking at here on this screen: that big list of files

for all these different devices. And if

it's not in here, then that means your

system doesn't know about that piece of

hardware. You're probably missing a

driver or something else that's critical

to make that work. So kind of an

important folder when it comes to

troubleshooting hardware.

Also an important folder if you're

trying to mount a partition--

you'll find the name right inside of here.

Alright. Let me go back up.

Let’s see. That was /dev/etc.

That was super important, right? This is

where our system’s configuration files

will go, and we’ve gone into this folder

in a number of episodes already, and

we’ll continue to go into it in some

others ones. But basically,

when you fire up a service or an

application, it needs to know a

configuration, and that configuration is

expected to be stored here. It’s mostly

just packed full of text files, so it’s

not a very big folder, but those text

files are really important because it's

telling your system how to configure itself.

/home—that’s where your user home

folders go, so all your personal files.

This will oftentimes be a separate

partition as well, so that if we fill it

up, it doesn’t mess with the regular OS.

It still has room, but it doesn’t have

to be. In fact, I think on my system it is

actually part of my main partition.

But anyhow, that’s where your user stuff will go. /lib--those

are for libraries, or linkable libraries.

When applications run, they rely on

libraries to provide basic functions, and

they’re usually shared between

applications, so you’ll find those

there. I’m on a 64-bit system, so I also

have lib64. When Linux was created, it was all

all created around a 32-bit ecosystem,

and so when you see a folder like lib, it’s expecting 32-bit libraries to be in there.

Let’s see. Some other ones: /mnt. /mnt is

optional--you might not have it.

It’s short for “mount.” In the olden days,

when we would mount a hard drive or a CD

drive or whatever, we would mount it to a

folder under /mnt. A lot of distros don’t

do that anymore. The ones that do auto-mount will usually mount under the

run folder. I see I’ve got run also.

run is not in the FHS, not in the

official standard. It’s just something

they made up, but it’s there. And it

just shows you can choose to deviate if

you want to. Let’s see. /proc--

whoops, I’m going to break my system.

/proc is a really neat one, because

this looks like a folder actually isn’t.

Right? This is a virtual file system.

Every program that you run is a process,

and the process is represented as a file

inside of /proc. So it looks like

you’re navigating around it. In fact, if I--let me go back here--if I go into

/proc and pull up a listing, I see

all these numbers. Right? What are those

numbers? Well, if I were to pull up a list

of applications running on my

system, I’d see all of these process-ID

numbers right here. That’s the process ID

of that binary running in RAM. Well, in

the file system, it all gets

represented right here as files, and you

can even browse inside of them.

If I pick one at random, like 10971,

and pull up the listing, I can start to find

out information about that application

that’s running. These are not real files--

this is all virtual.

It's being dynamically generated as I navigate

through, and it’s part of the whole idea

of representing everything as a file.

Not everything in Linux is represented as a

file. There’s an operating system called

Plan 9 where they went hardcore--and

everything, literally everything, is

represented as a file. In Linux,

most everything is represented as a file,

and so that’s kind of what we see in /proc.

Similar to that is /sys. If you go into /sys,

this one doesn’t show information about

processes, but showing me information

about the system: the Linux

kernel and what it’s running. And so,

you’ll see things like device detection.

If you're running a hypervisor, kernel

data. That's all tucked away in here.

And so, when you run something like

you name -a to find out what kernel version

you’re running, that’s actually coming from inside /sys.

It’s looking in there to find it and

represent that, putting that on your screen.

I already talked about /usr a little bit--for /usr/bin and /usr/sbin.

This one is supposed to contain our

really, binaries and any other kind of

data designed to be shared

between users. I don’t mean shared

across the network--I mean shared between

users right here on the system.

If there’s a library file and I have a user

account and Zach has one, there’s no

point in storing two copies of that

library. We can put it in /usr, and now we

both have access to it. That was the original intent of /usr.

On servers, that makes a lot of sense; on

desktops, it doesn’t, so it really

just kind of became the de facto dumping

ground for everything. Everything is

considered shared amongst users

these days, and that’s why you’re seeing

more distros linking to the

/usr folder. But if you look, there’s

more than binaries in there: there are libraries--lib and lib64.

You might have an etc folder for shared configurations, a src folder,

games if you’ve installed any. I didn’t

think I had, but apparently--oh, it’s empty.

Alright.

So... [Laugh]. I don't have anything no there,

but these are things that every user would then have access to inside of /usr.

A neat fact: this is supposed to be mounted read-only

based on the FHS, so users can’t

tamper with binaries other users might be accessing.

Very few distros actually mount this

read-only today, though, so this is

another area where a lot of people

deviate from the standard.

And another important folder for us, let's see,

that was /usr, is /var, that one, which is short for “various” and this is a various folder.

But it's not too much variety. There's

actually a couple of things we expect to

be in there. First off, our log files, log

files for applications, for the

system, for journal D, for all these other

different things. Those log files get

dumped into /var/log, but so does our

print spooler. When we print a job,

it gets written to the hard disk first

before being sent to the printer, and so

that’s inside /var/spool.

If you poke around in there, you’ll find a

lot of things--there’s spool, as I mentioned, and log is right above it.

But you’ll see other things. If you’re

running an email server, your mail queues

will be in there. The point about /var is

that it’s supposed to be considered

temporary. The stuff in there

isn’t supposed to be mission-critical.

If I lose my logs, the system is actually

fine--I’ll be okay. If I lose my principle,

it’s fine--I’ll be okay. If I lose email

in transit, that kind of thing, it’s not really critical. It's just temporary.

And in older days, you’d put

cheaper storage in /var because you

didn’t care about it as much.

Today, storage is so cheap and reliable that

/var is usually pretty stable.

And so, people will put things like security

logs in there that they intend to

keep long-term, even though that’s not

the original intent of the folder. Done.

Do we really need to memorize all these folders--

the ones you just went over?

Yeah. For the exam, yes--you do.

At least the ones I named. Right?

Don’t go memorize the whole

FHS--that’s crazy. The ones I just highlighted are kind of the

critical ones. You want to know those for

the exam. In real life, you can

actually cheat a bit. If you don’t

remember these folders, it’s fine,

because it’s so easy to find files.

In fact, we have another episode

coming up on how to locate stuff in

Linux. But to give you a preview, let me get to a console here. So let’s say I want to find a file.

Yeah. I’m going to use a good example: the SSH command, right? I’m going to SSH into a

server, and I want to make sure that SSH

is a certain version, so I need to

take a look at the SSH file itself and,

you know, test it. Well, where is it stored? I might have to think to myself,

“Is that an essential binary or a nonessential one? Is it in /bin, /sbin, or /usr/bin?”

It could be any number of places. Or is it in /opt? Has it been overwritten?

Where is it? You can use commands like which.

And the which command, if you say which ssh, it’s going to figure out which

actual binary would be run if you type

that command. It tells me this one is /usr/bin/ssh. Now, there could be more than one

copy of SSH, so we can use a different command, whereis ssh. And then I'm going to find if

there's multiple copies, and it'll show me

the multiple copies. And even things like

the help documentation and the

configuration. It tells me right

here the binary is /usr/bin/ssh, the configuration is /etc/ssh,

and the help documentation is /usr/share/man/man1/ssh.1.gz.

Rolls right off the tongue. So it does--it

just rolled right off. We don’t have to

remember it if we can remember the whereis command, right? You can run that into

it. There’s also locate and find. I’ll cover those in another episode,

but they’re useful for finding not just

binaries, but any file. If I'm just

trying to find a file, I forget where I saved something, I can use locate or find; they’ll find it as long as it’s listed in the name,

and then we can get it. If you don’t have it memorized, you’ll

get by just fine in most scenarios, but

it takes time. If you have it memorized,

and, you know, exactly where to go, you

save a lot of time, and you're more

efficient. So, that's the way I

look at it when it comes to memorization.

Awesome, Don--File System Hierarchy

Standard. This is another great

episode. FHS, if you please. And Don,

before we move on, anything else you’d like to say?

You know, this is all fairly static; it

doesn’t change much, and I know this

is our Linux+ show. But if you learn

the File System Hierarchy Standard, when

you jump to macOS, they actually conform to a lot of

this. And if you jump to BSD, a lot of them conform, too. So, you learn

it once, and it applies to more than one

operating system. That’s a pretty cool

thing. Definitely spend some time and

get familiar with it. Great advice--thank

you, Don, and a great episode as usual.

Linux+ is a wonderful series, and it’s designed to help you.

Make sure you watch every single episode--

you’ll be glad you did. And don’t forget

about our course library, where there's

supplementary information is there

to help you go even further.

Check that out as well, and tell

everybody you know about ITPRO.TV. ITPRO.TV is Binsworthy. Thanks for watching.

I’m Zach Memos, and I’m Don Pezet. We’ll see you again soon.

[Music].

Thank you for watching ITPRO.TV.