words to describe setting up k3s
this is hard
this is so difficult to set up
isn't this overkill
what is the load balancer again
why do i need two load balancers
should i use that cd
so wait i need two load balancers and
keep alive d
what is metal lb again
have you heard of cubevip
um isn't that a single point of failure
i know
i'll automate the whole thing
today we're not only going to set up k3s
with ncd and ha installation with cube
vip and middle of b but we're also going
to automate the whole entire thing so
that we can't really mess this up and so
we're going to fully automate the
installation of k3s so that it's 100
repeatable
and then we're gonna tear it all down as
if it never happened but before we do a
huge thanks to our sponsor microcenter
if you're thinking of building a new pc
you should look no further than
microcenter if you've never been to
microcenter you're missing out on seeing
a huge selection of technology in person
they've got everything for custom pc
builders from ssds and hard drives to
power supplies to memory to air and
water cooling to motherboards to video
cards to processors and more microcenter
is your one-stop shop to totally
customize your next pc build and don't
worry if it's your first time building a
pc they have lots of helpful and
eligible staff that are there to help
you out and will point you in the right
direction so that you don't attempt to
apply thermal paste like this
microcenter has been kind enough to give
all new customers a free ssd and it's
available in store only so see the link
in the description for details so how
did i get here well as you may or may
not know i've been running k3s in my own
environment for quite some time and i
even have a video on setting up k3s with
my sequel now there's nothing wrong with
the k3s version of my sequel it runs
great but at the time the lcd version
wasn't available and the lcd version is
super interesting because it creates a
high availability database on the nodes
instead of hosting it outside of the
cluster and right around that time i saw
jeff galen create a video on ansible and
that sent me down a rabbit hole learning
ansible creating a video on ansible and
automating a lot of tasks well you know
how that goes anyway so i found that
github repo i cloned it and created some
virtual machines and then i tried to
provision a high availability cluster
but there was just one problem the
ansible playbook only supported spinning
up one lcd node and that meant only one
server node which isn't ha i mean it's
configured for ha but i would have to
manually add additional server nodes to
make it ha and that's no fun so
technically it wasn't ha out of the box
so i decided to dig around in the code
and in the branches and i found a fork
where somebody actually fixed that issue
so i could actually create an h a
cluster out of the box with ansible and
i saw they also added support for cube
vip this was awesome because this is
exactly what i was trying to do i love
open source so a huge thank you to user
212 850a
this gave me a nice starting point to
automate the rest again huge thank you
to open source community jeff gearling
and user 212 850a so after poking around
for a little bit i found that most of it
was working but it did need some updates
and some configuration changes to work
with the latest version of qvim along
with some other features i wanted to add
so i decided to roll up my sleeves
and start hacking away at this fork in
my own branch and before making it
public i wanted to accomplish a few
things i wanted to make sure that anyone
using this could start with an unlimited
amount of nodes i wanted to make sure
that qvip was rock solid and then it
would actually create a load balancer
that you could use to make k3s fault
tolerant i also wanted to automate an
external load balancer so that when you
expose the service you get an ip address
for that service from your cluster and
then anyone can use that ip address to
access services within k3s so i had a
few choices for this step and a quick
clarification for these two load
balancers the first load balancer you
typically need in k3s is a load balancer
for your kubernetes api this is the load
balancer for control plane and this
should be fault tolerance so that if you
issue k3s commands you can still get a
response back and the other load
balancer is a service load balancer or
kubernetes for you to expose services on
in most cloud environments they supply a
cloud load balancer for you to expose
services on and this service load
balancer that i'm talking about is for
non-cloud environments in self-hosted
environments and since we don't have a
cloud load balancer to give us ips to
expose our services outside we need to
use something that can emulate a cloud
load balancer that kubernetes can ask
for an ip address from so our services
can be exposed so i had choices to make
for load balancers qvip can actually do
both it can be a service load balancer
or a load balancer for your control
plane for your kubernetes lcd nodes this
sounded like a great solution because
then i didn't have to use metal lb
but i love metal lv but taking on one
less dependency sounded like a good idea
especially when it comes to breaking
changes it's just less to manage so then
of course the other option for exposing
my services was just to use metal albeit
and
honestly after hours and hours of trying
to get qvic service load balancer to be
able to work with my services i decided
to fall back on good old trusty metal lb
and metal lb just works and i could use
my existing configuration for it so it
really wasn't a loss at all so at this
point i had my architecture pretty much
decided qubit for my kubernetes control
plane and metal llb for my service load
balancer and once i solved creating
multiple server nodes configuring cube
vip and configuring middle of b
it was time to do some testing for my
test i created five nodes and these are
standard ubuntu cloud image notes and i
just recently created a video on
provisioning new ubuntu machines using
cloud image and cloud init they're the
perfect ubuntu minimal server for k3s
just really check it out so once i had
these five servers up and running and
made note of their ip addresses
it was time to configure myansible
playbook so here in the groupbars file
is where all of my variables are set for
ansible first you can specify the k3s
version and then you could specify an
ansible user and this is the user that
ansible will run as and another quick
tip if you need to set up ansible i got
a really quick video on the bare minimum
stuff you need to do in order to set up
ansible it's a great primer for this too
next is setting a system directory and
you won't really need to touch this next
is setting a flannel interface of eth0
so flannel's responsible for networking
in k3s and it's pretty dense but if you
want to know more about it you should
totally check out their github repo but
as i understand it it's responsible for
layer 3 communication between nodes in a
cluster and so here i set at 0 because
that's the ethernet interface on these
virtual machines next i'm setting a
server endpoint and this is the ip
address of the vip that will get created
for kubernetes control plane and so this
vip gets created instead of you having
to create external load balancers along
with keepa live d this creates a vip
that is highly available that's exposed
through the kubernetes cluster that we
can communicate with and kubernetes can
too so it's pretty awesome that takes
care of two to three additional virtual
machines that you don't have to maintain
anymore next i set my k3s token and this
should be a secret that you should
obviously keep secret but it's your
password or your token for k3s
and you'll only need this in the
beginning or if you join additional
nodes later i then added some additional
arguments to my server and to my agents
but as far as the server goes i disabled
the service load balancer we'll want to
do that if we're running metal of b or a
service load balancer which we are i'm
telling it not to deploy traffic this is
up to you if you want to deploy traffic
you can just delete that arg but i'm
going to delete it so i can install it
with helm later because i like to
install traffic on my own later with
helm but if you wanted to install it you
could just delete this argument this
next argument is just setting
permissions on coop config and this is
really just for convenience so i don't
have to run sudo when i'm remoted into a
node to run coupe control it's probably
a good idea not to this but i got so
tired of typing in sudo every time i was
testing this the thousand times i spun
this up
that i just changed the permissions of
this file but feel free to remove that
argument if you want and the next string
of arguments are quite a few
but i'll leave these in the
documentation but to summarize
the rest of these args as well as the
agent args you see here is that i found
that i needed most of these args to make
k3s a little more responsive what do i
mean by that one of the defaults for k3s
is that if the node's not ready it won't
schedule additional pods on it until
that node becomes ready but the timeout
is like five minutes long which is a
long time i mean it's not a long time if
you're running multiple replicas of a
pod and you're running pods in aj you
would almost not notice at all
especially in larger installations but
in smaller installations like home labs
i found that five minutes is a really
long time especially if you're running a
replica of one that means your service
is down for at least five minutes so
i've scraped the internet found a lot of
these arguments and i've been using
these in my home production
home lab
for about a year now and they seem to
work pretty well but you might need to
do some tweaking depending on your
services your hardware and what works
best for you and again k3s will work
without any of those arguments i just
mentioned and maybe you should try it
that way first next i set the tag
version for cube vib and this is just
the container image tag the current
version is
v0.4.2 and so that's what i'm specifying
here and i did similar things for metal
lb too so for metal lb there's a speaker
container which the latest version is
0.12.1 and then there's a controller tag
as well which i also set to 0.12.1 now
these should be lockstep in the same
version but i made it configurable in my
template just in case they're not so
that i didn't have to figure that out in
the future and next i chose an ip range
for metal lb so this is the range of ips
that when you expose services they'll be
exposed on and you can communicate with
them i'll show you some examples here in
a little bit but i set a range from 192
168 30.80 all the way up to 90. so i get
10
so i get 11 ips here typically i only
need one or two but i set the range from
80 to 90 just in case after that i
checked my host.ini to make sure i had
all of the ip addresses in here and the
three virtual machines i am going to use
for my masters are 38 39 and 40. these
are also referred to as your server
notes and then my worker nodes or my
agents are going to be 41 and 42. so
this means three servers with kubernetes
control plane and ncd making it highly
available and then two worker nodes to
run my user workloads and if i had more
virtual machines i would just add them
below so with all of this configured i
ran the site playbook and pointed it at
my host.ini but before i did that i
started pinging my vip obviously it's
not there as soon as it comes up it
should respond so i ran the playbook
and it installed and configured k3s on
one of the server nodes
shortly after that the vip started
responding so this means qvip is
installed on that machine and the vip is
up
and then it started joining other
machines to the cluster
and then shortly after that i had a high
availability kubernetes cluster on k3s
and that's a ha cluster with that cd
with a load balancer that's also ha for
my control plane and aha load balancers
for all of my services but we need to
verify hopefully you trust me but let's
also verify so we can ssh into one of
our server nodes once we're there we can
run sudo coop control kit nodes and we
can see we have five nodes and they're
all online you can see i have three
control plane at cd masters
and two workers or agents ready for
workloads
super super awesome
so instead of ssh into this server let's
actually copy our coup config locally so
we can run the rest of the commands so
let's exit out of here you'll want to
make a directory for your coup config
file if you've never done this before or
back up your existing coupe config file
if it's there then we'll just scp or
secure copy that file from one of the
servers back to our local machine after
it transfers we can run a coupe control
get nodes and see the same thing awesome
so now we have cube control running on
this machine next i created a super
simple nginx deployment for kubernetes
this deploys an alpine version of nginx
and sets the replicas to three i did
that by running coop control apply dash
f and then the path to the deployment
manifest and then kubernetes told me
that deployment was created then i
wanted to check to see how this
deployment was doing so i ran coop
control describe deployment nginx and
you can see it is deployed and the
desired state is three and three were
updated three total three available and
zero unavailable so all three of my
nginx pods are up and running but this
doesn't give me access to these pods
outside of kubernetes this is where a
service and a load balancer comes in the
exact reason why we installed metal lb
so then i created a super simple service
file this service file is just a service
pointing to the app of nginx that we
just created that deployment and we tell
this service to expose it on port 80 and
that the target port for that container
is also port 80 and here's where the
magic takes place we tell it that the
type is type load balancer this tells
kubernetes to tell our cloud load
balancer to give us an ip and our cloud
load balancer right now is metal lb
so metal lb should hand us an ip address
that we specified in that range and if
all of that happens we should be able to
get to our service so then i ran coop
control apply dash f and then the path
to the service file kubernetes told me
it created the service for me and then i
wanted to verify that so i ran coop
control describe service nginx and we
could see here that it exposed a load
balancer ingress of one of the ip
addresses that we specified in metal lb
so this means my nginx deployment of
three pods is now exposed on a load
balancer at this ip 192 168 30.80 and if
we go to that ip address we can see the
hello world page from engine x this is
so awesome so this proves all the way
through that middle lb is working but we
never really tested the ha side of cubit
we know that we can issue kubernetes
commands right now with goog control but
we didn't take any of those notes down
so let's do that too so i started
pinging that vip and while doing it i
remote it into my first master node or
the server node that's running the
control plane and it's also one of the
nodes that's running cube vip that's
supplying this vip so i decided to shut
it down and as you can see on the right
i'm still getting responses and you can
see on the left i'm not getting a
response from that machine so this means
we have an h a vip now now i can't shut
down a second node an aja cluster of
only three nodes can only lose one
machine so if i shut down another
machine i won't have access to
kubernetes but i will still have access
to all of my workloads that are running
it's just that i can't change the state
of kubernetes nor access it over coupe
control so
this is so awesome so i started that
other node back up and it's responding
and obviously qbip is still responding
so what does one do after we build the
perfect k3s cluster
we burn it down of course there's also a
playbook to totally reset k3s back to
its initial state so running this
playbook and pointing at the same host
will totally clean it up it'll clean up
all nodes remove all containers and
reset it back to the state it was before
we ran this playbook this was super
handy as i was testing on my changes
must have run this at least a thousand
times and after it's done we're back to
a good state one note you might want to
actually reboot them afterwards i've
noticed that the vip stays up and it
will respond so i have a playbook to
reboot all of these machines and this
playbook will actually wait for them to
respond before it reports a success
just like that and so this is everything
that everyone struggles with when
setting up k3s no more using mysql and
making that ha if you don't want to no
more spinning up additional load
balancers and keep a live d and making
those aj if you don't want to no more
configuring metal lb or installing with
helm if you don't want to just one
simple playbook that spins up all of
that in one shot and then you can burn
it down if you want to too so again a
huge thanks to the k3s community who
made this original playbook along with
jeff gearling thank you so much and also
thank you to github user212 850a
thank you so much i'll have links in the
description to all of the code that i
have in the description below so what do
you think of spinning up a truly ha
version of k3s using ansible is there
anything i should contribute to the
script to make it easier for you let me
know in the comments section below and
remember if you found anything in this
video helpful
don't forget to like and subscribe
thanks for watching fix the lights i if
you weren't here last week small episode
with the lights i couldn't figure out
what was going on with my bottom lights
my bottom lights ended up having a small
issue and it took me a long time to
figure out it ended up being a firewall
rule so if it's not dns it is a firewall
rule all right changing the light as
soon as i mention them
if it's not dns it's a firewall rule now
now you're really testing mines all
right it's gonna happen it's gonna
happen so