CompTIA Linux+: Managing Services #11

[Music].

You're watching ITPRO.TV.

Hello. I think you're watching ITPRO.TV,

helping you level up with IT learning

everywhere you go. I'm Zach Memos, your

host for this episode of CompTIA Linux

Plus. And in this episode, we're managing services with Systemd.

And we've asked our expert, Don Pezet,

to tell us more. Don, good to see you.

Thank you for being here. Glad to be here,

Zach. I'm ready to dive into the world of Systemd, which is a really powerful

new technology. I'm going to say "new."

It's actually been around since, I think, 2008,

but really gained prominence in the last

five to six years. It is responsible

for powering the majority of Linux

distros that are out there. So it's really

important that we learn how it

works, what it does, and why--because

everything it's doing is behind the

scenes. So in this episode, what we're

going to do is kind of pull back the

curtain and get a chance to see exactly

what Systemd is like, how it functions, and

how we can interact with it to keep our

system up and running and make sure our

services are operating properly. Well, how

are services controlled under Linux?

So, you know, there are a few different ways

this stuff can be controlled. And we're

going to focus on Systemd, but it's not

the only way. There are many

systems that use the original Unix init

system. So, you know, when Linux boots up,

there's a kernel. Right? And the Linux

kernel gets called. That's really

important. That's the part that Linus

Torvalds wrote. But the kernel itself

doesn't interact with the user much.

It's kind of limited in what it can do.

So it's actually--like, I don't wanna say

it's not impressive--but from a

standpoint of computer technology, it's

very impressive. But from a standpoint of human interaction, it's not very

impressive at all. So what the kernel

does is, as soon as it fires up, the next

thing that it does is it launches what's

called an init process. Right?

And the init process is what actually starts all

the various services and makes your

operating system work. It seems super

impressive to us, but, technologically,

it's really basic. In fact, in the original

versions of Unix, they used what was just

called the SysV init. The SysV--System V--

was, you know, for tracking one of their releases.

And the SysV init--what it would

do is, it just had a folder with a bunch

of scripts in it, and it would run the

scripts one after the other, and that

would start your system up. And then, when

you got to a login prompt, it was because,

at some stage, a script was run that

launched a shell or a console and then

made that available to you.

That's all it was. Right?

Well, that sounds really simple--because it was

really simple--and it had certain

drawbacks, certain problems, certain

things that made it not efficient.

The number one problem

was that SysV init ran everything in

a serial order, one after the other.

So it would run script one, then script two,

then script three. And if script three

took a long time or if it hung, the

system just waited right there. Right?

So it wasn't terribly fast. So that was one

problem that it had. The other problem

was that the scripts were just run when

the system booted up. Right?

And after that, if something changed on the

system, the scripts wouldn't detect it

unless you recalled them. That wasn't a

problem back in the 1970s and in the 1980s. But fast forward to

2019, where we are constantly adding and removing

hardware from our systems. You might have USB thumb drives or webcams or whatever that plug in and go.

Imagine if you plugged in a USB key

and the system didn't even know that

happened. Right? And we had to run these

scripts and tell it to look for that missing hardware. So that is a problem with SysV init.

Systemd was created as the next

generation of that--to try and replace SysV init--and do it in a different

way. So what Systemd does is, it's not

just a series of scripts. Systemd is an

actual program. It's an actual daemon.

It actually stands for "system daemon."

And if you've never heard that term, it's D-A-E-M-O-N--daemon. And in the Unix and

Linux world, any program that runs in the

background as part of the operating

system--that runs unattended from the

user--is called a daemon. It's just, you

know, doing its whole job in the background. It actually comes from a

physics book that described some weird physical motion.

It's a crazy history. Some other book.

Yeah. But, basically, you know, these

daemons are what the system is running in

the background, and it used to be the

init scripts that would call them.

Now it's the Systemd executable. So this

program runs, and it doesn't just run

when the system boots up. It stays

running. So Systemd stays running the

whole time your system is online, and

it's able to do things like--well, one, it

can detect hardware changes because it's always running.

Two, it can do things in parallel. It can say,

"Alright, I'm gonna run these five

routines all at the same time." And if one

hangs, it's no big deal because it's still

doing the other four--and so it's faster.

Right? Systemd is faster, more stable.

There are some people out there that

don't like Systemd. You'll hear some

people complain about it from time to

time because it moves away from

tradition. Right? Traditionally, you could

control your entire Unix-like operating

system by modifying some text files.

With Systemd, it's not like that. It does have

configuration files, but it's actually

doing a lot of stuff in a binary.

And some people--if you're a Unix purist--

that's not something you're happy

about. But I'll tell you, as a Linux

administrator myself, I love it. I think

it's faster. It works really well, and

it's a great way to deal with things. So Systemd

is pretty prominent, and it's typically how we're gonna be

manipulating services. So going back to

Zach's original question--hey. How are these services controlled?

You power on your system. Right? Your BIOS kicks in. It reaches out to the hard drive.

It finds the boot files, loads up

that Linux kernel, then the Linux kernel

launches init. And in this case, it would

be the Systemd that it fires up. And then

Systemd goes through the routine of starting

everything else that you need to

interact with your system and bring it

online if it's supposed to. Right?

If there are things that aren't supposed to

come online, it leaves them off. Systemd

is what's actually controlling that.

So how do we know if we are actually using

Systemd? Alright. So, you know, good

question. I will say that all of the

major Linux distros these days use

Systemd. So if you get Ubuntu, you get Red

Hat Enterprise Linux, CentOS, Fedora, Mint--

it just--I mean, it goes on and on. All of the major

distros use Systemd by default. So the odds are, you are using Systemd.

But it's not 100%. There are some distros

out there that have chosen not to go

with Systemd, that are following the old

model--SysV init. Or there was

actually another init system called

Upstart. Upstart was really just a slight

modification to SysV init. It wasn't a full-on change. Mhmm.

So you may find some systems that do that.

Typically, it's if you're running older versions. Like, if you're running CentOS 7,

you're using Systemd. But if you're

running CentOS 6, you're using Upstart.

And if you're running CentOS 5, you're

running SysV init. Right? So, you know,

it's kind of changed over time.

If you're running Ubuntu, they've been

running Systemd for the last three or

four versions, I think. And then before

that, they were on SysV init.

But there are current distros like

Slackware. Slackware was actually the

very first Linux distro that I ran back

in 1993, 1992, 3, I think 1993. Either

way, it ran SysV init back then, and

it runs SysV init today. They have

not moved over to Systemd because they

follow that model of a pure Unix-type

deployment. There's other ones.

There's variants of Debian. I think it's one

that's called Devlon that actually uses

sysv init instead of systemd, but none of

the major distros. So if you're deploying

something in a business, if you're deploying something where you need

support, all of those companies are using

systemd. But we can verify, right? Let's be

sure, right? So I've got a CentOS box here

in front of me, and I'm running CentOS 7.

And I know that it defaults to systemd.

I shouldn't say default--like, it just runs

systemd. But if we wanna verify that, there's a few different ways.

I mentioned the boot order earlier.

And when I turn the computer on, the kernel

fires up, and the thing it does is

it starts init, right? So, technically, your

init process is the very first process

that runs. It's number one. So if we use a

utility like ps, right--this is the process show command--it shows me all the

processes running in the background. If I do a ps

aux, right, that says show all of the processes with the user that's attached to them,

and x--I can't remember what x does.

It's like services or something. I can't remember

what x is. But if I do ps aux and I run

that, it will show me all the processes

that are running. And if you look at

these process numbers, right--this

second column--I’ve got Process 13,999, Process 14,416. When a process runs,

it gets a number, and when it closes, the number goes away. And then it just kind of

keeps incrementing and growing. So over

14,000 processes have run since I booted

this machine up. Yeah. It's a couple.

Which hasn't been up that long, right?

I've been up for an hour and a half.

And so that's a big number. What if I

scroll up on that list? Let's just scroll,

scroll, scroll. The numbers get smaller

and smaller. These are processes that ran

early on and are still running. And I can

find process number one right there, PID

number one. And if I look to the side

over here, it's /usr/lib/systemd/systemd.

That tells me that I'm running

systemd. I'm on CentOS. Let me jump over

to Ubuntu. So this is an Ubuntu machine,

and I can do the exact same thing here.

If I jump into my terminal,

and I'll just do ps aux, and I'm just

gonna feed it into the head command to

save a little time. And there's my

process ID 1, and I can see--oh, this one's actually a little interesting.

It's telling me /sbin/init.

Alright. Now wait a minute. I said this defaulted to systemd.

That's not systemd. If you see /sbin/init, that's sysv init. Okay?

However, if I relied on this, I might get

tricked. And let me show you what I mean

because I'm, like, 99.9% positive this

should be systemd. I didn't check before

the show, so that's my fault.

But if you pull up a listing--I'm gonna

do an ls -l /sbin--I wanna see

the binaries that are in that file, or in

that folder. Okay? And I go in here and

find init--actually, let me just do /sbin/init. There we go.

They're tricking us, right? It's a symlink.

They have symlinked /sbin/init does exist and it's being pointed to /lib/systemd/systemd.

So there we go. So it looked like in the process

output that we weren't. Now that means that

we can't really trust the process output,

can we? The ps aux--it works if we're

full-on running systemd, but it could

trick us with older stuff. But if we

actually look at /sbin/init, it's in the official standard

that any operating system that's using

the Linux kernel has to have a file in /sbin/init. Even if it's not

what's called, it has to be there. And so

that's a more reliable way to look and

see what we're running. If I jump back over to

my CentOS box and do the same thing

in /sbin, I'm gonna look for--so

I'm running ls -l to get a long output--/sbin/init,

and I can see they've linked it there as

well. The difference is in the process

command--that ps aux. On CentOS, it's

actually showing me the true binary.

On Ubuntu, they're showing me the link

instead of the binary, so they've changed

the way that command works. But I can see

here they're both clearly running sbin, now, I mean systemd.

Now, if I looked in here and it actually said /sbin/init, and it was green and that was

that, then I'm running sysv init or

upstart. Upstart, I mentioned, was like an

upgrade to sysv init--not a true change. So it would be

/sbin/init as well. So I wouldn't be able to tell the difference

between upstart and sysv init, but that

doesn't matter for us. I know we're running systemd.

So, systemd starts right after the

kernel, right? Yeah. Then what happens?

So it's a big deal, right? It's the first

thing. And if systemd crashes, your system crashes, right? It's powering everything. Mhmm.

So once it fires up, then it's gotta get

to work. And it says, "Okay. Now it's time

for me to do my job. I need to start some services.

I need to open up some ports on the system for

network communication. I need to figure out whether we're gonna load a

GUI or go straight to the command line."

It has to do all these different things,

and it does that through the use of unit

files. Alright? Unit files are stored in--

if I change directories here--I'm gonna go into /lib/systemd/system.

Alright? If you go into that folder right there and pull up a

listing, what you're gonna find is a ton of files. And each file will be a

name, and then it'll end in .service, or .timer, .target, .slice. Each one is kind of

something different. Most of them are .service, right?

Those .service files are identifying daemons. Daemons are gonna

run in the background. Those are services

or applications that run and just keep running and do the thing. Alright?

Other ones, though, like .target--that's a user environment, right? So there's a .target file for if I want to have a

GUI, right? It's actually, I think, called "graphical-something-or-another." It's in here.

Remembering the names of these is pretty hard because there's so many.

Yeah--graphical.target, right there. And then there's a command-line one, which is actually called multi-user.target.

There are several others that are like that.

You'll also find .socket, which is identifying a network port that's

going to be open--or possibly a named

pipe or something that'll allow an application to talk to another

application via network service.

Let's see--that was targets, services, and

sockets. Those are the most common.

You also see .mount. So those are where

it'll mount a hard drive partition.

You still have the file system table--the /etc/fstab file--but you don't need

it anymore if you don't want to, because

systemd can use .mount unit files to

be able to specify what to reach out to.

So when systemd starts, it's actually parsing all of the stuff in this

directory and looking at it and saying,

"Alright. What do I need to do?" And if we pick—let me pick something easy--

SSH. Alright. So SSH--that's the Secure Shell daemon, and it's

running by default on most Linux systems

so that we can create a secure remote

shell, connect in, and get a command prompt, right? So it should be starting when the system starts.

If I look inside of that file--so let me just cat sshd.service.

There we go. There's not a whole heck of a lot in here, right? So it starts with this

block quote right here where it says [Unit].

And from there, it goes on to specify

information about that daemon. So I can

see that this is the OpenSSH server daemon,

and it has documentation, right? There's a

man file that's available for it, so it's

got that registered here. And then it's

saying, we wanna start it--but don't

start it right away. You need to make

sure that the network.target environment

has started, right? We have to have networking

working before we can start SSH, and we

need the sshd-keygen service running.

So this is setting up a hierarchy, right--

like an order and dependencies that

systemd is able to use. So it says, okay.

I can't start this. I need to

start network.target, and I need to start sshd-keygen.service.

So once those are started, then I can

come back and start this one.

And remember, it's doing all this in parallel,

so it can reach out and try and do multiple things at once.

And then after that, it says once. Right?

After it says, "I can't start until these are

started," but once means, "Once I start, I

want this other thing to start as well."

Now sshd-keygen.service is listed twice.

And what that's saying is it needs to be started when I start.

But if something goes wrong, I need that

one started again because I depend on it.

Right? That's establishing that dependency too. So we've got those

entries. And then after that, we see the

actual service. Right? And so it's telling

me I can find my configuration in /etc/sysconfig/sshd. That's where the

configuration is gonna be. And then

when we execute, when we start, here's the

command that it runs: /usr/sbin/sshd. That's an executable.

-d tells it to run as a daemon, run in the

background. Otherwise, when that shell

closes, the application would close.

We can see what to do when we close it,

and it's gonna run the kill command,

and it's gonna shut that one down.

You can see how to kill it--you just

use it as a process execute. Sounds really violent, but it's just exiting.

And then restart--if there's a failure, if

it crashes, it's gonna try and restart

it. And if it does a restart, it'll wait

42 seconds. Right? This is all defined

right here. When we install this service,

if I want it running every time, it's

gonna install it and attach it to multi-user.target. So when that starts,

which is our command line, when the

command line starts, it's going to make

sure that SSH starts as well. These are

unit files, and the unit files are just

text. You can actually go in and

edit them if you want. It's not a great

idea to edit these. Most of the time, when

you install it, when you install the SSHD

service, it creates these files for you.

Right? So I didn't create these. They were

made for me. If I want to modify them, I

usually will not modify it here because

if we get an update, right, so SSHD gets

an update and they update this unit file,

it's gonna overwrite my changes.

I'm gonna lose them. So if you're gonna

make changes, don't make them in this

folder. So, let me get back in here.

I actually need to zoom back out. There we go.

So I'm in /lib/systemd/system.

You really should never edit anything in

/lib. It should just be left alone. But if you go into /etc/systemd/system, alright?

And that folder might not exist.

You might have to make it. On CentOS, it's

there by default. So /etc/systemd/system as opposed to /lib/systemd/system.

When you look in here, you'll find service files and target

files and wants and so on. These are where

we're overriding what's done inside of

systemd's folder. So if I want to make a

change, I shouldn't go and change the

files in /lib. I can come right in

here and create a file with the same

name. And when I do that, this file will

override the other one. And when an

update comes along, it'll leave my file

alone. Right? Now, I haven't overridden

anything, and so everything in here was

done by the system. Looks like DBUS is

doing some stuff. We even have sysinit,

which is for backwards compatibility

with SysVinit. So some of these entries

are already in here. Most of these are just links.

In fact, I think all of them are just links.

Well, no. Some of them just have weird

permissions on them. But many of them are

links to other locations, and you'll see

them pointing to /lib or somewhere to override an activity

from somewhere else. So that's how that

kind of functions, and that's really the

heart and soul of systemd. Systemd is a

binary. It's an executable that runs.

In fact, I think I browsed right past it.

If I go into /lib/systemd and we look in here somewhere... right here.

There we go. There's the systemd executable. It's actually not very flashy at all.

It's--yeah. Let me do a human-readable size on this one.

It's a blazing 1.6 megabytes, and yet that 1.6 megabytes is powering

my entire user environment right now.

The kernel is handling my hardware

interaction, and then systemd is making

all of my software work. I mean, this is

super important software, and it's really,

really small. And then it relies on all

those configuration files to do its job.

Well, do we have to create unit files for new services, Don?

You can. So if I'm installing a new

service, you can go in and create and

define them yourself. I've had to do that

on occasion, but normally, the services

will create them automatically. Right?

So let's say--let me give you an example.

I want to be a web server. Right?

So I want to install Apache. Well, if I take a look inside of

my systemd library folder--so I'm back in /lib/systemd/system.

If I look for any file that starts with

httpd, oh, shoot. I actually see it right there: httpd.service. So it looks like

I'm already a web server.

That file is defining the Apache service and getting it set up to do its job.

Right? Well, I didn't create that file, or at

least I don't remember doing it. So when I installed httpd,

it must have created that. Right?

If I were to remove httpd--so if I do a sudo yum remove or we'll just remove httpd.

There we go. So I'm gonna uninstall it.

So I'm uninstalling httpd, and now I'll just take a look again,

and I see there's no unit file defined

for it anymore. So when it uninstalled

httpd, it knew to take away that unit

file. And if I install it back again--

so I'm just gonna come in here and do an install,

and when I do that, it's gonna go

through. It's gonna do the install.

There it goes. And if I take a look, I can see it created that file again.

Right? So most services, if you're installing them from a package file,

whoever the developer is that created

that software will know, "Hey, this is the

unit file I need," and it'll go ahead and

create it for you. So you really don't

have to worry about it. You don't have to

go in and edit these things. It just does it.

Now, I will tell you, sometimes you

install software, and it doesn't create the unit file for you. Right?

And that might be software that's designed to be run on demand. Okay?

You need unit files for software that

runs in the background, that runs

automatically without interaction. I want

this web server running all the time.

It needs a unit file. But if I have something like

OpenOffice, right--OpenOffice, I need

it to run when I'm interacting with it. I don't want it running in the background.

So there's not going to be a unit file

for OpenOffice. Those are just user applications.

Alright? But sometimes when you install a service, it still

doesn't create the unit file because most services don't start automatically.

You've got to configure them. And we do that with systemctl.

systemctl is a utility that is packaged alongside systemd.

It's the system control utility, and it lets us control

systemd and do things that otherwise

might require us to edit some files.

So, for example, if I want httpd to start right now,

I could do sudo systemctl start httpd. Right? And it's gonna read the

unit file. It's actually looking for that unit file, and I should have typed this the whole way and said httpd.service

because that's the unit file name, and

it's gonna look for it, find it, and

start it. Okay. If you leave the extension

off, it looks for services first because I'm telling it to start something.

And that way, it doesn't look for .socket, .target, and those guys. Right?

But if you want, the full name would actually

have .service on the end. So it's gonna

find it. It's gonna start it. Alright?

Now, we can start a service even if it

doesn't have a unit file. If it's got dependencies, things will break.

But even if it didn't have one, you could do that.

But when the system reboots,

that service is not gonna be started

by default unless we tell it. And we

could go in and modify the actual

service file and/or the unit file and actually tell it,

"Yes, the multi-user or graphical target depends on

this file. We need HTTPD to start." Right?

Or you can just do it right here with systemctl. You can say systemctl enable httpd. Right?

And when I run that, look what it did. Created a symlink from /etc/systemd/system/multi-user.target.wants/httpd.service.

So it put a symlink in /etc. Right?

And it's pointing to /usr/lib/systemd/system/httpd.service.

It didn't edit anything inside of /lib or /usr/lib. It edited instead inside of /etc.

And even then, it wasn't really an edit. It was just a link that

says, "Now whenever I start the multi-user target, I will also start httpd."

It did that for me. I didn't have to create unit files. I didn't have to go and modify /etc.

It was just done automatically. And if I say, "You know

what? I want the web server, but I don't want

it to start automatically," I can just come in and say disable.

And when you do that, it removes the link.

It didn't disable Apache. I can still run Apache if I do sudo systemctl start

httpd like that, and then I can do status.

So, sudo systemctl status httpd, and it will

actually show me that it's active and running right now. It's up and operational. Okay?

But looking up here, it says that it's

loaded, but it also tells me that it's

disabled, which means if I reboot, Apache

is not gonna be running anymore. Right?

If I go back and enable it again, let me do that.

And then we pull that status. What I'm

gonna see now is that the service is

enabled. So if I reboot, Apache is going

to come back up, and we'll be in business.

So the important part here is that we do all of this

without ever actually editing the file. systemctl does it all behind the scenes

for us. So this is another area where we kind

of deviate away from the way that Unix

used to be, where we'd edit all these

script files. Now it's all automated

through systemctl. How does systemd keep

track of which services to run?

So we've kind of seen that a little bit

with our dependencies, but it really

boils down to the target files. Right?

The target files are what define what's

gonna get fired up and so on.

So let's take a look at one of those

target files. If I get... what folder am

I in? I'm in the right folder: /lib/systemd/system.

If I pull up a listing of *.target, alright, you'll see a lot of targets.

Right? Targets are like collections of

unit files. It says, "Hey, if I want the

network to function, I've got a network

target, and here's all the stuff I need

for the network to function." And you'll

see other things in here that are like

that. You know, if I want remote encryption to work or reboot to work or

swap files to work, they've all got

different targets, and those targets are

defining all the services needed to make

that work. So if I want to boot to a command line, just to a command line, then

I need what's in multi-user.target. If I take a look at that file--let's use less on multi-user.target,

I can look in there, and it's actually not a very big file at all.

Right? But that's because it's including

some things that it needs. It says, "Alright, multi-user can start whenever it

wants." Right? So I don't see a... well, actually, I do

have a require, don't I? Here's a require that says basic.target.

So before multi-user can start, anything in

basic.target has to start. Then this one

can start. And once it starts, afterwards,

we'll fire up basic.target if it isn't

already, but we've got rescue.service,

rescue.target. We've got a few other

things that can be triggered as well

to, you know, get our system up and

running. Well, let's look at basic.target.

So if I just... I'll cat basic.target, and I

look in here, it's also not very big at all. Right? But it's got that it requires sysinit.target,

and it's calling several other things too. And if I look at

sysinit.target, we're kind of following

this back. Right? Going back, back, back, and we see how it's calling more and more things.

All of this is coming together to build

the environment that is required

for our system to boot. If I want a

graphical user interface, then I would need graphical.target,

and you'll see where it depends on

multi-user. The command line has to work

before I can launch stuff to get the GUI. Right? They all are interdependent.

As a human, it would take me forever to sift through all these

files and build out the tree of dependencies, but systemd does it every

time you boot. It parses through these

files, figures out the dependencies.

It actually doesn't monitor the files on an ongoing basis. If I change a file

right now, it won't recognize it unless I tell it to reload. Right? So you can go in

and... well, you can reboot your system.

It's one way to do it. Or, like with a service,

you use the restart command, and then it'll

reload the unit file for that service.

But, otherwise, it's figuring out all

those dependencies, kind of mapping it

together. It gets pretty complex because of

the sheer amount of files, but at the end

of the day, they are just simple text

files. You can view it all. Well, can we

change between the targets? You can.

Yeah. You know, my system right now, if I were to reboot... Mhmm.

It's set to default to the graphical

user interface. So it's basically set

to boot to graphical.target. Right? But if this is a server, I don't need a GUI.

I might want to have the GUI installed

so when I'm sitting at the server, I can

use it, but the rest of the time, I don't

need a GUI running. So I can actually

switch targets and tell it, "Hey, don't run graphical anymore. Just run

multi-user." And we can do that pretty

easily. So here I am in my command prompt, and I could be in any

prompt. It really wouldn't... or any folder.

It wouldn't matter which one. I'm gonna do a sudo systemctl

isolate. And so I'm telling it I want to target a specific

target. That's a little redundant.

And then I'll just say multi-user.target.

Alright? Multi-user means, yes, it's

command line, but it's not just one

console just for you. It still allows network services and other things to run.

And so when I run that, see how my system freaked out? Yeah. Right?

It recognized immediately. "Okay, we're not

gonna do the GUI target anymore.

We're gonna just do a command line." And so

now I can come in and I can log in,

assuming I can remember my password,

and there we go. I'm not in a GUI.

I'm just in the command line. And so now if I reboot, I'm gonna boot just to

this interface right here. You know, this

is my... well, actually, I need to go and modify my GRUB

loader to tell it exactly what I want.

But at this point, I'm now in just

command line. X Windows and all the GUI stuff is not running at all.

I can switch back with that same command...

well, a little bit different. So if I do sudo systemctl isolate, and then I'll do graphical.target,

and that's gonna switch me back.

And now it's loading X Windows. See how

it's taking a moment? It's not like X Windows

was running in the background. It's got

to relaunch all that stuff and get me in

here and, and get me put back in place.

Now, because this is happening after my

boot operation and all that, you see how it's kind of, like, screwed up my

resolution? Yeah. And all that?

That's one of the trickier parts about moving from one to another.

If something's not defined in the unit

files or some other configuration file,

then stuff gets lost. I... oh, it cleaned

up. Well, sort of. I got large icons now. So, you know, there's a lot of

people like me. If you do this stuff a

lot, you'll end up writing scripts to kind of automate some of the changes.

And so, for example, I have a little script in my home directory.

This will show you how often I do this

to manually set my resolution so that I can just run it real quick,

and then I'm right back in.

Now my icon’s the right size and all.

So, you can switch between targets at the

drop of a hat, but just know that there

are certain penalties to it.

There are certain things that happen on your

system you might not expect.

So, definitely test it out, you know, run

through the routine a few times.

But that’s basically the nuts and bolts of

systemd right there. There’s a lot more to

it, but the basic operation is that there are

a ton of text files. It reads them and

performs as they dictate. Great episode on

managing services with systemd.

Wonderfully done, Don. And before we move

on, what else would you like to say?

You know, if you want to learn more about

systemd, definitely check out the systemd website.

That’ll be like your Google response, I’m sure.

But there’s also a whole suite of other

utilities. I use systemctl.

There’s journald for the special logging system

that systemd uses. There’s hostnamectl for changing your computer’s hostname.

There are a number of other CTL commands

that are all part of what systemd

creates. So, there’s a lot more than what we

went over in this episode. I was just

focusing on really what’s involved in

getting your system up and operational and getting your services running.

Great advice as usual. Here’s some more great

advice: Watch every episode of CompTIA

Linux Plus. You’re going to be glad you

did. It’s going to help you now and in

the future, and keep studying.

Take advantage of the supplementary information in our course library.

It’s there to do one thing: help you be even

more successful, so check that out as

well. And tell everybody you know about

ITPRO.TV. You know, ITPRO.TV is binge-worthy. Thanks for watching.

I’m Zach Memos. And I’m Don Pezet. And we will see you again soon.

[Music].

Thank you for watching ITPRO.TV.

Welcome to ITPRO.TV. I'm your host Don Pezet.

Live from San Francisco.

You're watching ITPRO.TV.

Hello, hello, and thank you for watching ITPRO.TV,

helping you level up with IT learning

everywhere you go. I’m your host, Zach Memos, for this episode of CompTIA Linux Plus.

Managing services with SysV init is

the name of this episode, and Don

Pezet is going to be showing us the way.

Don, good to see you. Thanks for joining

us. Well, glad to be back, Zach. And like I

said, we’re going to be tackling SysV init.

You know, in the last episode, we got a chance to see systemd, which is the most

popular initialization system in Linux

right now. However, if you’re working out

there in the field, there are still many,

many systems deployed that use the older

SysV init. In fact, on servers, it’s

actually pretty common. So, definitely

something we want to learn about, and

that’s what we’re going to do right here

in this episode. We’re going to learn a little bit about how SysV init works.

It’s for you and SysV init. So, it can happen to anybody, to anyone.

We’ll see how it works, we’ll see how to manage it, manipulate it, and just kind

of see how to operate in an environment

that is running SysV init. Fantastic. Don,

how are services controlled under Linux?

Alright, so basically, all of our services inside of Linux depend on an

initialization of some sort, right?

And systemd, we’ve already seen this one way,

right? Your system boots up, and systemd

is, well, kind of follows a boot process,

right? You know, your kernel kicks in, and then it

launches your init. And so, systemd would

launch, and then it would control

services from there. But with SysV init,

it’s a little bit different. Your system

boots up, your kernel fires up, and then

the kernel calls a series of scripts. SysV init is not a binary executable.

Well, I guess it sort of is. It’s a little bit

of a binary. But the majority of what it

does is all being driven by script files.

We saw with systemd, it was driven by

unit files, which aren’t really scripts;

they’re kind of like descriptions, and

then systemd knows what to do with it

and executes. SysV init just runs scripts.

Whatever happens to be in the script, it

runs. And you can put literally anything

in there, like starting a service, passing parameters to a service,

checking to make sure the service is

running, or the permissions are set right.

All of that is different things that

SysV init can do, and it does it in a

significantly different way than systemd does. Now, many people feel that systemd is the

better of the two options for modern

systems. But if you’ve got a server that

millions of users rely upon and it needs

to be rock solid, SysV init’s been around since the late 1970s, early 1980s.

It’s a very stable, tried-and-true system.

And that’s why when you go out and look at servers, you’ll find many still use it.

Let me give you an example:

If you’re deploying Linux servers in Amazon Web Services (AWS), they have what’s called

Amazon Linux. And Amazon Linux is based on CentOS, which is based on Upstart,

which is based on SysV init. So, it’s kind of a long

chain. So, when you go into an Amazon Linux AMI, it’s going to be using

SysV init initialization strings and all of that, so you’ve got to know it.

Amazon Linux 2, which only recently reached production, now uses

systemd, so they’re moving in that direction.

But most people haven’t ported their workload over to it yet, so it is very,

very common to encounter SysV init systems.

So, how do we know if we’re using SysV init?

Yeah. So, you know, if you’ve just been hired on somewhere, you’re taking

over a server, you might not necessarily know whether it’s running systemd or not.

Maybe you’ve got all your distros and version numbers memorized and you just

know offhand. But if not, we can double-check. You know, I mentioned that the

kernel is going to fire off the initialization system, and that might be systemd

or it might be SysV init. Well, that

process of firing off initialization is

the very thing that it does. So, we

can just look at process ID number one,

and that will usually tell us whether we’re running systemd or SysV init.

Right? So, for example, here, let me take

this--I’ve got two boxes here. This one is a CentOS 6 box.

So, it’s running CentOS 6.7, which is currently supported. So, this isn’t like

this is old and crusty. Like, you can run

this in a production environment. And if I come in and do a ps aux, right,

and I’m going to just pipe that into the

head command. So, I want to see the 10 lines of my process list. When I look

at the top processes here, process number

one right here fired off by root, if I

come along to the side here, I can see

that that’s /sbin/init. If it just says that, /sbin/init,

then odds are you’re running SysV init.

Now, it’s not a guarantee, and that’s because

some of the OS distributions, like Ubuntu, will actually show /sbin/init also.

But when you go and look at that binary,

if you do an ls -l /sbin/init, when you look at the binary on an Ubuntu system, on a newer Ubuntu system,

it will actually be a symlink pointing to systemd. Right?

So, be careful with this one. In fact, if I go to, like, my--here’s a CentOS 7 box.

Let me get into a terminal here, and I’ll do the same kind of thing with ps aux,

and I’ll pipe that into head.

And that very process I can see

right here is pointing to /usr/lib/systemd/systemd.

So, this is pretty obvious. Right? Ubuntu, at least in the newest ones in the 18 releases,

they’re a little bit trickier. So always be on the lookout for that. But that’s

probably the easiest way to tell whether

you’re running SysV init or not.

So what you’re looking for is that /sbin/init; that tells you what

you’re running. And once you’ve got that figured out, now you know that you

need to administer this system a

slightly different way than you would a systemd system.

So, SysV init starts right after the

kernel. Right? So, then what happens? Alright. So, SysV init does have a binary,

right? So, it did /sbin/init. Right? So, our BIOS fires up on the

system, and it does all the hardware

detection. Then it reaches out. It finds a hard drive partition.

It finds the /boot partition, finds the kernel, and launches

the kernel. The kernel then turns around

and says, "Okay, I’m a SysV init system," and it launches /sbin/init.

And when it does, it then proceeds to start launching a

series of scripts. That init executable is then just going to run a

series of scripts, and it runs them in

serial, not in parallel. Systemd does stuff in parallel. It’ll do more than one

thing at the same time. But with SysV init, it runs one script at a time.

And when it finishes, then it runs the next

one. When that’s done, then it runs the

next one, and so on. And it all kicks off

with one script. Let me show you here on this system. If you go into /etc,

you’ll find a folder in there called

rc.d, this folder right here. Now this

folder has changed a lot over the years. Originally, there were

different folders. In fact, see all these

light blue folders up here? Yeah. rc, rc0.d, rc.local, rc.assistant.

All these are, like, previous iterations of what SysV init or, like, System V init or System 4 init.

Like, each of them kind of relied on different folders. They’re there for backward

compatibility. But now pretty much everything is tucked away inside rc.d.

And if we go into that folder and look, you’ll see a number of subfolders based on all sorts of

different stuff. And each one of these

kind of influences what SysV init does

when it launches. And the main one that

we want to pay attention to is this guy

right here, rc.sysinit. When SysV init starts, this is the

first thing that it calls. It reaches out

to this script and it executes it.

And then from there, it will proceed to

spawn other things. And there’s a lot of things that influence what it’s going to

spawn, which are all kind of tied to parts of what it does. But this one’s

really the heart and soul of it. So, /sbin/init runs, and it

calls this script to then initialize the

system. Does that one script then call

other scripts? Absolutely. Yep. So, it runs, and as it’s finishing, it’s then launching other things based on a certain criteria. Right? And that criteria is what’s called a run level.
So notice all these other folders over here, rc0.d, rc1.d, rc2.d. Those are tied to run levels. Right? So, in Linux, you have seven run levels, and each one indicates a different status of the machine. So, for example, 0 actually means the system is shut down.

runs and as it's finishing it's then

launching other things based on a

certain criteria right and that criteria

is what's called a run level so notice

all these other folders over here

rc0.d1.d2.d

those are are tied to run levels right

so in in linux you have seven run levels

and each one indicates a different

status of the machine so for example

zero actually means the system is shut

down

now you might wonder to yourself there

like

why would i have a run level for

shutdown well you got to tell the system

to shut down and you might actually have

scripts that are a part of that when

your system is shutting down it needs to

you know basically

perform a few tasks maybe you know stop

rights to the hard disk or so on and

close out what it's doing and those will

be part of the shutdown script you also

have

rc6 here which is reboot and so that's

when your system is rebooting it might

have certain tests they might be the

same they might be totally empty there

might be no tasks tied to reboot and

shutdown but that's what those run

levels are the other run levels

are kind of up to the distro that they

can operate in different modes and

they've they've kind of standardized on

it so most of the distros follow the

same pattern but it's not a guarantee so

always be careful if you're not sure

check your distro's documentation a lot

of them

will document what their run levels are

in their initialization table so there's

a file if i i'm just going to cat slash

etc

init tab that's the initialization table

and when you look in that file most

distros do take the time to document the

run levels and so i can see right here

for centos and this applies to uh centos

and red hat enterprise linux i can see

the seven run levels so see how the zero

is halt halt is when the computer's

turned off uh and then six is reboot one

is single user mode that means that only

one user can be logged into the system

and nobody else can be logged in right

that's actually pretty handy if you're

an administrator sometimes you want to

modify a file but it's always in use

somebody's accessing it or there's a

file share people are accessing it over

the network you need to make sure you

have exclusive access to it so you can

boot in single user mode and you know

you'll have that there's multi-user

without it says without nfs it's

actually multi-user without networking

so it boots up in the full mode but with

no networking services at all and that

way you can ensure nobody over the

network is getting into the system if

you think you've been hacked here's a

great way to boot and shut off your

network and now you can get in and

examine the system and you know you're

not communicating out

you have full multi-user mode which is

run level three and that's where you're

saying look i want to boot up to a

command line interface

but i'm allowing multiple users people

can ssh in or i have more than one

console so a lot of people can be

working in the system that's the run

level that most servers run at

run level four it's actually not used i

i don't think i've seen a distro yet

that makes use of it but it's available

so if you want to create a custom run

level that's usually what run level 4 is

reserved for

run level 5 it's labeled here as x11 but

that means we want to boot a gui we want

to boot a graphical user interface which

is typically the x environment but you

know you might have some alternate user

interface

and that one is multi-user with

networking and a gui so it's kind of

like the

the full shebang right it boots

everything and then you get access to

the system so those are the run levels

and

looking at this slash etc rcd dot d

folders kind of hard to say

all these subfolders rc1.d rc2.d these

contain scripts so when i boot into

multi-user mode command line only

rc dot system it is going to run and

then it's going to look in rc3.d

and it's going to look at what scripts

are in there and look at all these

scripts that i've got i've got where

it's starting the quota daemon and

iptables for the firewall the lvm

monitor for my logical volume manager uh

network manager is starting postfix all

these different things will start when

my system starts these are tied to

various services right postfix uh that's

an email service

ssjshd that's my secure shell daemon

these are all being started that's how

it knows what services to fire up and

it's all controlled by the run level if

i look in uh let's go into rc

5.d

all right so this is if i boot with a

full gui

and i'll see things are a little bit

different there's a lot of overlap right

so i still see sshd right there i still

see post fix right but i'll also see

some things in here that are tied to

x windows let's see i should be seeing

gdm in here somewhere i must be skimming

over it

it's hard when you're under pressure on

the show but somewhere in here is gdm

which starts the graphical device uh

device manager they have to let you get

logged in and all of that so it's

additional things that will get loaded

you may see variations depending on

which run level you boot that's what all

happens right here

then there's one more thing that may not

get loaded but typically doesn't uh back

in rc.d

there's one more file

rc.local okay on really old systems i'm

talking unix all right

it would run rc.systenit

then it would run all the scripts for

the run level and then last it would run

rc.local and that was your way to

override anything else that was run

before you kind of undo or redo stuff

most distros

have the file for backwards

compatibility but don't do anything with

it so it's usually empty or it may not

even be called at all you'd have to

check your run level scripts to see if

they return to it or check rc.cisnet so

like if i take a look at my rc.local

let's see what's in it um yeah it's

pretty much empty in fact it just uh

touches a file to indicate that it's

locked and that's it so

it's not used in this case but it is

there so if you want to make use of it

you can versus if i take a look at

rc.systenit

and take a look at that one there's all

sorts of stuff in here as it goes

through mounting your disks or turning

on se linux right there uh you know

doing all sorts of other stuff this is a

big deal right it's got a lot of work to

do rc.local is empty

do we have to modify the script files

for each uh run level when we install a

service

all right so a good question right so i

showed where sshd was in there right so

my ssh demon is starting automatically

and it's in more than one run level

right so if i install a service like

let's say i want to install a web server

right so i'm going to install apache and

when i install apache i want it to run

on my system boots okay well

if you notice with ssh it was set to

start at run level three and it was set

to start at run level five uh they

skipped run level four because it's not

used and they skipped run level two

because run level two boots with no

networking so why why bother starting a

web server if there's no networking

right

or ssh in that case

i might want a web server to do the same

thing so i want the web server to start

with run level three and run level five

so to do that i mean we could manually

create scripts and put them in there but

what you'll find is that most of this is

handled for us through a handful of

commands all right let me let me just

show you kind of how this works so on my

system i'm going to do a sudo yum

install httpd all right so i'm going to

install apache right

and apparently it's already installed

great

so i've already got it installed now

once it's installed you can go through

and create scripts so if i go back to uh

let's see

rc3.d

and take a look in here i don't recall

seeing http in the list oh it is right

there

k15 httpd so i've already got it

installed and it's already set to start

so i picked a bad example all right

let's do uh maybe we'll do a database

instead is mysql on here let me just

skim the list and make sure i don't make

the same mistake twice all right we'll

do mysql so i'm going to do a sudo yum

install

mysql

and i'll do mysql-server as well and

we'll do a dash-y so it just does it all

right so it's going to install the

database server now when the database

server is installed it's not set to run

automatically or at least it shouldn't

be so i'm going to pull a list back up

here and look

and i do see it actually created a link

right there so it's smart enough to

create the links with what the default

run level should be if we want it on

three and we want it on five it shows up

right there now

what if that's not what we want though i

don't want mysql to run maybe i i only

want it to run when my gui is running

but i don't want it to run when i'm

booting to command line only or you know

something crazy like that right well you

can actually manipulate these pretty

easy without having to write or even

edit the scripts at all right so you

certainly can come in here most of these

are just sim links see how they're light

blue if you uh pull up a long listing

you'll see they're almost all sim linked

to the init.d folder

and in there is where these text files

are kind of being created but we can

manipulate all of this a few different

ways

the easiest way is to use the check

config command chk

co and fig check config what check

config does is it lets you turn services

on or off for different run levels

it defaults to run level three and five

so if i do a check config

and then i say

httpd on

that's going to turn oop i need to sue

do that let me do that command there

there we go that's going to turn on

httpd for run levels 3 and 5. if i boot

at run level 1 or 2 or 4 it's not going

to start but 3 and 5 it will

and that's kind of its default operation

if i say off

then httpd is not going to start in run

level anything right it's going to turn

them all off right but i can go a step

further with this command and i can be

really specific i can say for example

check config httpd

dash dash level

and then specify the levels i want the

default would be three five like that so

run levels three and five that's where

it's going to start maybe i want it to

run well i'll just leave it at 3 and 5.

let's take

mysql

oops

sorry i have to do one more option here

and say on there

so i left off a little on tag so it got

unhappy with me

let's say that i want mysql to run in

run level three and run level five and

run level two right mysql actually

doesn't need the network in a

traditional sense you can use name pipes

to access it so maybe i want that

database up and running even when the

network stack isn't up and going so for

that one i could do a sudo check config

dash dash level 235

mysqld on

and now it's set to go with multiple run

levels so it's kind of a cool one

it's easy to lose track of this stuff

though and we could go and sift through

the scripts and figure out where it's

set to run and where it's set not to so

if i go into uh

slash etc rc.d slash

rc2.d and take a look in there i can

find

wait a minute http is in the list

right it's actually sim linking to all

of these but the script is telling it

only to run at a certain run level and

that run level is left off so if we try

and track it right here through these

sim links

it really doesn't help we don't actually

know if it's going to start or not so a

better way to figure out what's going to

start and what isn't is to use check

config dash dash list

and that's going to list your services

and tell you what run levels they're

going to execute at so if i run that i

get this nice big list right here

and it's got all seven run levels with a

simple on or off next to each one so i

can find http right uh whoops

if i can scroll back up to it i can find

it right here

and the mouse is my biggest challenge

with this stuff

and i can see run levels one

zero one and two it's off run level 3 is

on 4 is off 5 is on 6 is off so just 3

and 5 are on even though i saw an httpd

link in slash etc

rc.d slash rc2.d and if i look at mysql

right here

i can see that it's a little different

in that it's two three and five that are

all turned on okay this is a better way

to see what's being run at each run

level if you just look at the links you

can't tell you'd have to follow the link

and then you have to edit the script or

at least view the script that it links

to and in that script it'll have an if

statement at the beginning that says

like if run level equals and it'll list

the run levels where it's going to start

that's a pain

check config dash list you'll see it all

right there and if you have a huge list

you can always pipe that into the grep

command and once it's in the grip

command uh

you can just say like i'm looking for

mysql and then you just get that one

line that makes life a little bit easier

there so pretty easy to verify uh we

could also just do it manually like

maybe maybe i'm not so concerned about

when the system starts i just need to

use my sql for a little while right now

okay so for that you'll use the service

command and we can just come in and say

sudo service

and then start and whatever the services

we want to start if i want to start

httpd there we go uh oops i got it

backwards uh service

httpd start

there we go

sorry you got to say the service name

first and start second i got it

backwards so sudo service start httpd

and now it starts up and i can do the

same thing with like mysqld which it'll

probably be unhappy that i haven't

secured it yet but we'll do it anyway

we can start that one up yeah it's

warning me that i need to run the

security startup script but uh but there

it goes so now those two services are

running

but

let's say i'm at run level five right

now and those are up and running and i

reboot my system if those aren't set to

start with run level five they won't be

started when i reboot so using the

service command like this is not

permanent it's really just kind of a

temporary thing just to get uh get the

service up and running and so on and you

can do a service

status

for any service

and take a look at like hdbd

and

oops

backwards again darn it service httpd

status there we go

and if we do that when i'll get it right

eventually we can see if the service is

actually actively running right now you

may find where it started with the

script and then something went wrong and

it closed and so even though rc.d is

telling it yeah you need to start this

at boot it may not be booted so this is

a good way to quick check that

and see and i'll do the same thing with

mysqld and i can verify right there

they're both up and running and happy

down how does this v in it know which

run level to boot to

uh you know i glanced over it earlier so

we kind of have some defaults right

where we're saying boot to the graphical

user interface or boot to the command

line uh and your your os your

distribution will typically default to

one or the other that comes from the

slash etc init tab folder that our file

that i opened earlier uh if i cat that

one slash etc

init tab all right so this is the file i

brought up on screen earlier to show the

run levels well there's one tiny little

line at the very end that i didn't

mention this this guy right here that

says id colon

5 colon init default colon all right so

this is telling the system what run

level to boot to by default so mine is

at the default to run level 5 which is a

full graphical user interface if i'm

setting up a server i probably don't

want the gui so i'll tell it to boot to

the command line to boot to multi-user

mode which is run level three so to do

that i would just edit this file and

change that five to a three and then

when you reboot it's gonna boot to the

command line and when you're ready you

can always change it back to a five

again and it'll boot up and and that'll

be kind of the the run level that it

goes to now that's if i want to make a

permanent change

let's say it's just a temporary thing

right maybe i'm having some kind of

crazy issues with x windows and

i just want to

to dump out and get rid of all of x

windows and then come back into x

windows but not

kind of not have to restart all my

services or whatever so we can actually

change the run level whenever we want

while the system is running now it can

have disastrous effects so we need to be

a little bit careful with it you know

all these scripts are going to execute

and do stuff

but if you ever want to change your run

level just temporarily there's a command

you can run which is init

if you run init

followed by a number let me pull out the

man file for that so if you just run

init followed by a number it will

immediately change your system

into whatever that other

init level happens to be now if you're

going to do this be really careful of

other users in the system you can you

can really mess with them especially if

you're about to drop a single user mode

so you might want to use the who command

to see who's logged in and how they're

accessing the system to say all right

here's people i need to contact before i

change this run level but assuming

you're the only one in here let me take

this system and i'm going to go to

multi-user mode right so i'm going to

say sudo init three i want to drop to

run level three and when i run that

my system drops to the itsy-weensy tiny

little console

and so i get to see that and we get out

the magnifying glass yeah yeah we'd have

to use it here uh so we lose all the you

know the cool

um

uh

screen enhancements and so on and now

we're just in straight command line mode

and there we are right so now i'm kind

of operating in this

this different world x windows has been

completely unloaded and it's executed

the scripts found in that rc3.d folder

to move me into this kind of mode and i

can switch back if i want by saying init

well i gotta sue do that sudo in it

and then move back into run level five

and when i run that

there we go i see that it takes me back

to my centos login screen

and i can get logged back in

and now i'm back into the gui so we

quickly and easily change our run level

like that now uh

most of the time we're going to do it

just like i did right there it's going

to happen fast

there may be times where you want to

schedule changing the run level and the

the official recommended way to do that

is use cron you can set up a cron job to

change the run level at a certain time

if you want

that's one technique you can use there

used to be a command called telling

and what telonite would do

is it would let you set a time delay for

when the run level would change and many

systems don't have the command anymore

and let's see if i do but you have to

spell it right if you want to find it i

do have it okay so there it is uh

telling it it lets you change the system

run level so just like the init command

but the difference is it takes some

options the init command really didn't

take any options this one does and one

of the options is dash t which is in

here somewhere well i didn't see it but

it does exist uh where you can come in

you can say

telling it

and then what number you want to go to

so i'll say i'm going to run to jump to

run level three and you can say dash t

followed by a time and you know maybe i

want to wait 30 seconds before i do it

and so i can specify that i would need

to suit it that command

and then i'll do that dash t30 and run

it okay so that's going to start that

timer well notice how mine just totally

ignored the timer that's because the

telenik command it's an old command it's

not supported by most a lot of districts

don't even have it and the ones that do

don't fully support it so it's kind of

the old way so instead if you want to

schedule it you're a little better off

by jumping in and doing uh a cron job to

run in it at a certain time fantastic

information managing services with cis v

in it great episode as usual thank you

don and before we move on i think you

have something else you'd like to say

yeah you know i've mentioned before that

cisvnit is really old and the systemd is

really new and shiny and i personally

like system d

but i think the sysv in it is still

really really important you are going to

find many many servers that are out

there that rely upon it and if you start

jumping into other operating systems

like freebsd and open bsd and those guys

they still use cisv in it too systemd

hasn't made the leap over those sides

yet so learning how this works is a

skill that you'll be able to apply many

different places definitely definitely a

valuable one to have great advice thank

you don here's some more great