0:30
Welcome, friends, to another session of the Kubernetes Fundamental Series
0:40
This is session number 13. And today I'm going to talk about daemon sets and jobs
0:48
Is that daemon sets or daemon sets? So you can pronounce it as both of them, I guess
0:56
So I'm going to say daemon sets or daemon sets, okay? So, let's get started, okay
1:03
Daemon sets are used for applications that you want to run on each of the nodes in the cluster, okay
1:13
Maybe you have a logging application or maybe you have, I don't know
1:20
maybe a security enforcement kind of application that you want to run on each of the nodes that are in your cluster
1:30
So that's why you can use daemon sets. Okay. They ensure that a specific pod runs on all
1:38
or a subset of nodes in a cluster. Okay. That's the gist of daemon sets
1:49
And actually, a lot of applications that you can install nowadays in your Kubernetes clusters
1:54
are running as daemon sets. For instance, log collection is one of the examples that most of the time is using this kind of resources
2:08
So daemon sets are part of the apps forward slash v1 API version kind of resources
2:19
And the kind is daemon set, as you can see here. So it has API version, kind, metadata, and also the spec
2:29
This is very similar to a deployment if you think about it because you have a selector
2:35
which is a match labels kind of selector, and then you have the pod template
2:42
Okay. So inside the template, just specify the pod. In the selector, just specify, of course, the selection of those pods
2:50
that are going to be considered in this daemon set. In the metadata, you can specify the name
2:59
and also labels or annotations or what have you, and API version and kind, okay
3:06
So the difference here is, of course, kind, and also we can use tolerations
3:15
which is a way of telling Kubernetes, is, hey, you know what? I want to run this pod that has this container or this number of containers
3:26
and I want you to bypass or I want you to tolerate some restrictions on the nodes
3:34
For instance, there are some nodes that could have taints, which is the technical name for that
3:41
and things tell the scheduler, hey, you know what? I don't want this and this and this other kind of pods
3:50
or applications or objects in this node. So it's a way to prevent the scheduler
3:57
for scheduling things in those nodes. And tolerations is a way for Kubernetes
4:04
to bypass those restrictions, okay? Actually, I'm not going to talk about tolerations and taints
4:13
That's the topic for another session. But yeah, you can use tolerations
4:20
and most of the time you want to do that in your daemon sets if you want to bypass the taints, okay
4:28
So let's create our first daemon set. So I'm going to open up my terminal
4:37
And let's see what's going on here. So, Cube Cuddle. So, first, let me see if I have Minikube
4:51
Is it running? Minikube status, remember that I'm using, throughout the series, I've been using Minikube
5:00
So, yeah, it looks like it's working. And I have three nodes, okay
5:05
Actually, this Minikube and this laptop, I'm not using my desktop PC
5:12
This laptop I'm using Minikube on top of Docker for Windows. This is another driver that Minikube supports
5:22
I'm not using the Hyper-V version of Minikube. I have three nodes
5:29
Let's verify that this is running, cluster info, so it looks like it's running
5:35
So kubectl, you know what? Let me create an alias, alias k
5:40
Is that PC? Is that P-E-A-S-A-L-I-S? Is that the name or alias
5:49
I can remember the commandlet. P-E-A-S-A-L-I-S. Anyways, I'm going to just write kubectl
5:55
I don't care. So let's create the emmon sets as a folder
6:02
in here, let's just to understand what's going on here. Okay
6:14
I have a deployment, I have a service, I have other things that I've been using
6:19
for some training purposes, so I don't care about those. Let's see if I have a daemon set, get daemon set
6:31
I don't have any kind of daemon set, so that's fine. Let me show you one thing, okay
6:36
API resources. Remember that you can use this command for displaying all the different objects
6:46
that your current Kubernetes version supports. So in this case, I have daemon sets
6:54
It should be right there. It should be right here, okay? jmlsets and the short name is ds so in other words i can say cubicle get ds and of course i don't have
7:11
any kind of jmlsets in this moment so i can create a daemon set using the imperative way by using
7:19
create ds and then the name right my daemon set or something like this right and then outputting this to YAML However let me use Visual Studio Code because I
7:36
guess it's better for these sessions. Instead of me creating those objects by using
7:42
the imperative way and using output the YAML, and then changing things
7:48
let's just use the Visual Studio Code extension. daemonSet.gamo, okay? So daemonSet, do I have a template for daemonSet? That's the question here. DaemonSet, no, I don't. So I'm gonna just copy this, okay
8:16
and I'm going to copy the pod spec as well. Okay. So, of course, the spec, this is the spec for the pod
8:27
So, it should be right here. Let me zoom in. Right. So, the spec is part of the template
8:38
and this is the place where we specify the pod. Let me open up my, what's the name of this
8:45
Epic Pen. Do I have Epic Pen in this laptop? I guess I don't. I guess
8:54
my laptop doesn't have all the tools that I have on my desktop
8:58
PC. If you've been following this series, most of the time I'm in my desktop PC anyway
9:07
So it looks good. This is my daemon set. All regular things, API version, kind, metadata
9:13
Remember that the kind is daemon set. And inside the spec, we have the selector
9:19
Be careful because this label, in this case, app, that's the name of the label
9:26
That's just a random thing that this Visual Studio Code extension created
9:35
It could be any kind of label that you want. And this should match the labels of the pod that you want to run as a daemon set
9:46
Makes sense, right? It's just like a regular deployment. So we have the selector and then the template
9:53
This is a place where we define the pod. And in this case, this is going to use Nginx
10:00
I just want to use the latest version of Nginx, okay? And I want to expose port 80 through TCP
10:11
And toleration, this toleration is telling Kubernetes, specifically the scheduler, hey, you know what
10:19
Let's bypass that restriction, that taint that the master node has, right
10:28
Of course, it depends on your Kubernetes installation. In this case, this toleration obviously is for the master node, and that master node should have obtained in order for this to work
10:47
Anyways, it looks like for this specific demo, the toleration is not needed
10:53
However, I'm going to leave that there, okay? Let me fix this so it looks nicer
11:02
Okay, so looking good. kubectl apply my daemon set. Is that the name
11:10
Yes. Do this, okay? daemon set created. So I can say kubectl get daemon sets
11:20
So as you can see, I have my daemon set running in this moment
11:25
It says, okay, three, my daemon sets are running. Why is that
11:31
Because I have three nodes, right? Get nodes. I showed you that when I executed Minikube status
11:41
So I deployed Minikube in this laptop by using the Docker for Windows driver
11:50
and I created three nodes, okay? So, technically speaking, those nodes are running as Docker containers in this laptop
12:00
This is a great way for us to test Kubernetes and play with this technology
12:08
So, I have three nodes, and the daemon set is running in all of those nodes, okay
12:15
So, pretty easy stuff. So, actually, let me describe my daemonSet. Okay
12:28
So as you can see here, we can... Let me clear the screen and execute this again
12:38
So again, I can see that this desired number of nodes is three and the current number of
12:45
nodes are free okay so i can see that this is running correctly and of course it's running
12:54
nginx it's not doing anything special right now it could be using any kind of application
13:01
or it could be using i don't know the busybox container image for executing any kind of bash
13:07
command in each of those nodes, okay? So, and yeah, the taint, we can see that not the taint
13:19
the toleration, the toleration worked because it's bypassing that taint. You know what? Let me show
13:28
you that cube curl get nodes okay i have the mini cube node which is my control plane node cube curl
13:38
describe node mini cube okay so let me show you that i have do i have taints here i should
13:53
see the taints in this describe taints. Okay, so there are no taints
14:01
Minikube, there are no taints at all. However, for all of those nodes that do have taints
14:09
this is a place where you can list them, okay? This is the place where you're going to see them
14:15
In this case, Minikube didn't add any kind of taint of taint in this code, in this node, okay? Anyways, so the toleration at the end of the
14:26
day, I could remove it and the result will be the same, okay? Okay, so that's daemon
14:36
Let's continue with the presentation. Okay, cool. Jobs
14:51
So jobs they allowed you to run a task or batch process that runs until completion okay And this is useful for I don know maybe something that you need to run
15:10
and you want to be sure that that task completes successfully, okay
15:18
That's the use case for jobs. So they're useful for tasks that need to run only once or a specific number of times, and that they guarantee that the desired number of successful completions will be achieved
15:34
Okay? So jobs are quite useful in that sense. There are some use cases for using those, such as batch processing
15:45
I mean, you can be running jobs all the time for doing some kind of process or task
15:52
Database migrations for running database migration and creation of schemas or schema validations and those kind of things
16:02
Data backups, if you want to regularly say you need to backup a special folder or a special storage that is running in your node, you can be running this job
16:21
And that job could be doing your backup. So machine learning, of course, you could be running machine learning training jobs in Node, and you want to fire up this kind of processes and let them finish and schedule tasks
16:44
I'm going to talk about that when I talk about cron jobs in a few minutes, okay
16:50
Okay. So, one of the, or some key fields for jobs are the ones that I have here in the screen
17:00
Completions, it's the number of times a job spot needs to run successfully to be considered complete
17:06
So, that's just a field that you set. I'm going to show you that in a few minutes
17:12
You can set this field, completions, and the number of completions that you want
17:18
So the job executes that number of times to be considered complete and successful
17:27
Parallelism is the number of pods that can run simultaneously. So this is useful for machine learning
17:35
Again, if you want to execute, I don't know, machine learning models, training
17:43
you could be running those in parallel instead of just waiting for a job to complete
17:50
and then another job to complete. So this is useful for not waiting
17:56
for a previous jobs to end. Back of limit is the number of times
18:04
Kubernetes will retry a job if it fails. So the default, I think it's six
18:13
So six times is going to wait and it's going to retry before Kubernetes says
18:20
you know what, this job is not working. This is not working
18:25
I'm not going to retry anymore. So you can set this back of limit
18:31
field as the number of times that you want. Okay. So yeah, the default is six
18:39
So here's the manifest. Here's the manifest. So as you can see, the API version is batch forward slash v1
18:51
This is a different API version. And this is because jobs were created after a lot of the primitive resources that we've been talking about in this series
19:05
And the kind is job. Okay? Pretty easy stuff. Just remember the API version
19:11
kind of metadata, just as usual, and the spec, again, as usual
19:18
And inside the template, just as you expect, is the place where you define the pod, okay
19:27
So inside the spec of this pod, we have, in this particular example
19:33
it's going to run busybox, and it's going to run this particular command
19:38
which is shell and it's echoing hello world and it's sleeping for 30 seconds pretty easy stuff
19:47
actually this particular example however one of the things that i want to highlight here is that
19:58
restart policy is set to never this is because restart policy the default value is always
20:06
That means that if your pod completes, the pod itself is going to be restarted and you don't want to do that, right
20:18
You want the job, I mean the pod to execute and not restart once it's completed, okay
20:29
So that's why you can use restart policy and use the never value to prevent that
20:39
So in this case, container won't be restarted if it fails or completes
20:44
That's actually very useful for these jobs. Okay. And as I said, backup limit, that's the property that you can use to set the number of retries before marking the job has failed
21:02
So again, the default is six. You can set it as any number you like
21:06
completions is the number of completions that should be complete, right
21:16
In order to this job be considered as successful. Parallelism, that's the one that I told you about
21:24
This is the number of pods that will run at a time
21:30
And TTL, seconds after finished. This is interesting because if you're running jobs, mostly cron jobs, you don't want your cluster to be full of jobs and jobs and jobs and jobs that have completed
21:52
You know what I mean? You don't want to keep those finished jobs all the time, right
22:00
Until the end of time. So maybe you want to delete those
22:06
That's why TTL seconds after finished, it's particularly helpful for deleting the jobs
22:15
after they complete, okay? After, in this case, 10 seconds, for instance
22:21
if the job completes, then after 10 seconds it going to be deleted Let do this Cool Let do this
22:35
My job demo. Let's see if I have a template here in Visual Studio Code
22:48
API version is patch V1. The name is my job. Let me change this to this other place
23:00
So in this case, the Visual Studio Code extension is using Perl and it's running this Perl command
23:11
I don't want to do that. Let's do the bcbox example, bcbox, bcbox, command shell, and that's going to be dash
23:27
C and let's echo LL C sharp corner this
23:40
And let's, I don't know, sleep for 30 seconds or 60 seconds is fine
23:50
So restart policy set as never. That's one of the things that I just mentioned
23:57
and this is the number of seconds that is going to wait for the job to be deleted
24:05
So that's fine. 100 seconds is almost a couple of minutes, and let's do this, okay
24:16
Completions 2. Let's begin by using just one completion. It's fine, okay
24:27
Okay, so, demo sets, actually this is a folder that I'm using today and I created the job
24:38
inside this folder. Anyways, kubectl apply, my job, do I have jobs
24:45
Let me show you the kubectl API resources again. Let me try this again, okay
24:56
it's right here okay so there's no short form for a job because it's it's short right itself
25:08
it's a just a three letter word um anyways kubectl apply f my job so the job is created
25:20
we can query for those jobs job kubectl get job I'm trying to use the k alias so my job
25:35
right at this moment it doesn't have any completions right let me watch this because
25:44
Because the command itself is echoing hello world, hello C-sharp corner, right
25:50
And then it's waiting, is that 60 seconds? Yeah, it's going to wait for 60 seconds
26:00
So in this moment, the job is running, right? It's doing something
26:05
It's processing the task. It's training the machine learning model or what have you, right
26:11
Of course, in this case, it's just echoing that string and it's waiting for 60 seconds
26:18
But this is on purpose, okay? Because I want to show you that this job is going to take one minute, right
26:26
And after a while, after that minute, the job will be completed, okay
26:34
So we can see here in the completions column that the job is completed
26:41
So, let's get up. So we have my job and number of completions
26:51
One of them were expected, right? This is the desired and the current
26:56
So we have current and desired. Right now, those are the same
27:02
And of course, the job is considered complete, right? It took 68 seconds and actually I think that's the information that we have here
27:17
And we have some other things such as what was the container that this job was using
27:25
and the selector itself. So let's query for the job again. Because after 100 seconds, the job will be deleted
27:41
Remember that? That's because of the TTL seconds after finished. This one that is right here. Okay
27:52
So after 100 seconds, this job will be deleted automatically by Kubernetes, which is great
28:01
This is very useful to avoid having hundreds or thousands of jobs in your cluster, okay
28:14
So let's wait for this to be deleted. I guess it was deleted already
28:19
Yeah. So the job, we don't have any other jobs, right? That job was deleted
28:30
Great. All right, so let's try completions. So let's do this shorter, okay
28:41
So completions, let's try Coupe, okay? And let's try this again
28:51
Go back here. TubeCuddle apply. TubeCuddle get jobs. Let's watch this
29:05
So the first completion is running, right? I mean, the first process, right
29:13
The first job, the first task is running. So that's going to take 30 seconds or so
29:22
And then the second completion will run. Okay. But as you can see here in the column, we have the desired number of completions and the current number of completions
29:38
Right at this moment, we have one completion. The job is still running
29:43
It's not complete. Okay. It's not complete because it's waiting for the second completion to be done
29:51
Make sense? Pretty fantastic example and... useful object, right, this job is
30:03
So this is going to take another 30 seconds. And yeah, the job is completed
30:15
So after one minute, that job will be deleted. Okay. So I'm not going to wait for that
30:23
That's going to happen. And of course, backoff limit, This is useful for retrying because what happens if your container fails somehow, right
30:37
Of course, busy bugs, it's not going to fail. I mean, it's just echoing
30:43
But I could create a command for randomly sending errors or something
30:53
And that's where the back of limit field comes in handy. Okay
31:03
And finally, parallelism is the number of jobs that will run at the same time
31:11
So let's explain cron jobs, which is, I think this is an abstraction on top of jobs
31:23
Because those jobs, the job that I created, I run the job manually, right
31:30
I applied the manifest. So I decided manually, I decided to run the job
31:37
That was deliberately executed by me, right? So most of the time, you want those jobs to be executed on a schedule
31:53
Okay. That's where cron jobs are useful. Okay. Those cron jobs are built on top of jobs and are used for running periodic tasks at scheduled intervals
32:07
So in other words, you're going to schedule a particular task to be executed and let Kubernetes do the job and just report if the job was successful or not
32:22
Right. That's a cron job. So the cron job creates jobs. OK, the cron job, you set the schedule. I'm going to show you this
32:35
You create the schedule to define. I want this. This. I don't know
32:39
They want this job to run every one hour or every day at 10 p.m. or those things
32:48
and the cron job itself is going to create jobs, okay? The cron job is going to create job objects, okay
33:00
And of course, job objects themselves, they have pods, okay? So we have the cron job manifest right here
33:14
The most important thing here is the schedule. To be honest, this is the trickiest thing about cron jobs
33:27
The cron expression that is right here, okay, that you use in the schedule field
33:34
This is the most complicated thing about cron jobs. You have to know the Krone expression and you need to read it and understand what's going on there
33:49
Of course, if you understand Krone expressions, you're fine. Otherwise, you can just read the documentation and that's it
33:57
Actually, if you see this, if this is your first time reading cron expressions, you will be like scared, right
34:09
Anyways, this cron expression that is right here, it has minutes, hours, seconds, minutes, hours
34:19
And I think this is the day of the month. And this other one is the day of the week
34:27
or the other way around. We can see the documentation, but actually it's not necessary
34:34
And inside the job template, you specify the job, right? And inside the job
34:43
you specify the pod. That's why we have spec that is inside this other spec, right
34:53
And this spec is inside this other spec. Be careful about that because this is not clean, to be honest
35:03
This is tricky. But anyways this is the way that we define cron jobs So let do this
35:19
My cron job, YAML, cron, do I have a cron job? Oh, no
35:29
I don't, I don't care. I can just copy this and show you this, okay
35:35
So API version is batch forward slash V1, just like a regular job, okay
35:42
The kind is cron job. Let me show you the API resources command
35:48
you've got all API resources. So the short name is CJ, okay? CJ
35:59
And metadata, API version, kind metadata and spec, regular things, right? Inside metadata, you can define the name for this object
36:10
which is the cron job. And then inside the spec, you define the schedule
36:16
That's the most important thing here. So this schedule, it has a cron expression
36:25
Let's see if I can execute. kubectl explain cron job dot spec dot schedule yeah explain cron job spec schedule thank you
36:41
kubernetes documentation it's not telling me anything it's sending me to wikipedia anyways
36:49
um anyways uh that's seconds minutes hours the day of the month and the day of the week
36:57
but you can always go to the Kubernetes official documentation. So if this is true, this will run every one second, okay
37:13
And this will run every 10 seconds. Let's see that. Let's verify if this is going to work
37:25
and then job template, okay? Job template, and you begin specifying the job template
37:35
And here, right here, you can begin specifying the different properties for the job
37:44
not the cron job, okay? The job itself. For instance, TTL after, no, TTL seconds after finished
37:58
Let's do 20. So again, job template, those are the properties for the job
38:09
And then it comes the spec, which is the pod itself. we have the template and inside this spec we can define the pod specification and there's the place
38:25
where we can define the containers and volumes and those kind of things so in this case it's
38:32
running busybox in this other case is gonna say hello world and it's going to say the date
38:41
And the restart policy that I'm using here is on failure. I changed this to be sure that this process will run successfully
38:55
And if that happens, then the job will be considered completed. And then the current job will be considered successful
39:05
Do you see that relationship, that inheritance between pods, jobs, and cron jobs
39:14
So it's kind of tricky to have three specs, one inside the other
39:24
However, if you think about it, it makes sense. I don't like the syntax, but it is what it is right now
39:32
Okay. Okay, cool. So, it makes sense. I guess this is enough for us to test this. So, I have my cron job
39:46
By the way, if I execute get jobs, do you remember that the previous job was waiting for, I guess, one minute or something to be deleted
39:59
We have no jobs at all. So it makes sense. Actually, let's see if something happened
40:07
Let's see if we can see something here. Yeah, created pod, job completed, right
40:16
And executing kubectl get events, to obtain the events from the cluster Anyways you curl apply f my cron job
40:30
so cron job version v1 cannot be handled strictly calling no field spec job template tt
40:36
l seconds after finished so it looks like something's wrong So this should be here, I guess
40:50
TTL seconds after finished. Is that it? TTL seconds after finished. Is that the error
41:05
So it looks like something's wrong. kubectl explain cron job spec and then job template
41:17
Okay, job template has the spec. And this is a job object
41:25
Detail seconds after finished. So it should be inside the spec. And of course, my Jamal syntax is wrong
41:35
That's why, okay? Right? This syntax doesn't make sense. I'm sorry about that
41:43
And actually, the Visual Studio Code was telling me, hey, you know what
41:48
This is not wrong. I mean, this is not right. TTL seconds after finished should be inside the spec
41:57
Thank you, kubectl explain. Let's apply this. And now the cron job was created, get cron jobs
42:08
So this one is supposed to run each 10 seconds. Let's see what's going on here
42:19
is that seconds or maybe 10 the first one is minutes hours days hmm I'm not sure let's see this okay
42:33
so kubernetes.io and cron expressions
42:49
Let's see if I can find the meaning of the cron. Okay, so the first one is minute is not seconds
43:00
Okay, our day of the month, month, day of the week. That's actually the meaning of those
43:07
So I don't want to wait for 10 minutes. Let's delete. It's cron job, delete cron job
43:15
Example cron job. and let's edit this okay. And now let's apply this again
43:30
Google get Chrome jobs. So this is going to run every minute
43:39
So we have to wait for this to run and yeah actually let me show you other thing
43:47
which is the number of jobs. I don't have jobs that are running right now
43:56
So this job, this example, cron job, I don't know if it came from this cron job
44:05
Let's wait for this to run. That cron job is going to create a job
44:14
and the job itself is going to wait for the pods to be completed, okay
44:21
Anyways, let's wait for this, and let's see if I have more information about this
44:28
I don't think so. Let me show you the blue labels that I have here
44:35
So, this is the crown expression. This is the inner job template
44:39
and then the inner pod template, okay? make sense so let's go back here so so yeah i think run you know what let me
45:03
again you call delete cron job my example is that on job
45:14
Then, cubecadl get cron jobs, cubecadl get jobs. So I don't have cron jobs, I don't have jobs
45:26
That's fine. Let's apply this. cubecadl get cron jobs and jobs So this is the job that is going to be executed by the Chrome job
45:49
Yeah, I mean, I have to wait for this minute to pass to see that the job was completed
45:58
I think the previous job that we were seeing there, I'm not sure if I didn't delete the current job correctly
46:17
Anyways, this current job is right there waiting to execute. Let's wait for 20 more seconds
46:24
and then this job will execute. Again, I mean, I think that was executed
46:32
and then it was deleted, right? So let's wait for this. Maybe things are happening faster than my hands
46:44
That's why, right? The current job creates the job, executes the job
46:51
and then waits for 10 seconds to be deleted and then waits for one minute to run again
46:59
Maybe that's what's happening here. So I have the job. It was completed
47:05
So yeah, things are happening very fast. So this job was deleted because of the TTL seconds after finished value
47:16
Maybe if I remove that, Let's try this again
47:51
Crong job. We'll call get crong job. Have the crong job. And of course, you don't have any job right now
48:05
Okay? So let's wait for this to be finished. And then it's going to create the job
48:13
The job will wait for the process. and the job will not be deleted, okay
48:22
So I have to wait for this. So let's just watch the jobs to be created
48:27
Anyways, as you can see here, the cron job is useful for scheduling things
48:33
Of course, this is just doing echo and blah, blah, blah. It's not very useful
48:40
I mean, the command is not doing anything fancy right now. is just creating those jobs
48:47
And I'm just describing the way cron jobs and jobs work. Okay
48:54
So that's great. The job was completed. It's right there. It's not going to be deleted
49:00
So we can get a grasp about how things work. Anyways. very useful stuff for Kubernetes applications to have daemon sets, jobs, and cron jobs
49:20
I think from those three, I think the most useful is the daemon set and, of course, the
49:27
cron job. The job manually firing things, yeah, I mean, it's good to have the option
49:35
However, I think jobs are mostly used by cron jobs. And yeah, take care about the schedule
49:47
As you can see, I was struggling with the syntax of the cron expression because I don't remember the five positions that it has
49:59
And there are some cron expressions that have six positions. I don't know why Kubernetes decided to use the cron job expression, the cron expression with five positions instead of six
50:16
Anyways, be careful about those cron expressions. That's actually the most tricky part of cron jobs
50:25
And with that, that's all I have for today. and thank you for your time
50:36
and see you in the next session where we're going to talk about ingress