128 lines
4.8 KiB
Markdown
128 lines
4.8 KiB
Markdown
# Automated build of HA k3s Cluster with `kube-vip` and MetalLB
|
|
|
|
|
|
|
|
This playbook will build an HA Kubernetes cluster with `k3s`, `kube-vip` and MetalLB via `ansible`.
|
|
|
|
This is based on the work from [this fork](https://github.com/212850a/k3s-ansible) which is based on the work from [k3s-io/k3s-ansible](https://github.com/k3s-io/k3s-ansible). It uses [kube-vip](https://kube-vip.io/) to create a load balancer for control plane, and [metal-lb](https://metallb.universe.tf/installation/) for its service `LoadBalancer`.
|
|
|
|
If you want more context on how this works, see:
|
|
|
|
📄 [Documentation](https://technotim.live/posts/k3s-etcd-ansible/) (including example commands)
|
|
|
|
📺 [Watch the Video](https://www.youtube.com/watch?v=CbkEWcUZ7zM)
|
|
|
|
## 📖 k3s Ansible Playbook
|
|
|
|
Build a Kubernetes cluster using Ansible with k3s. The goal is easily install a HA Kubernetes cluster on machines running:
|
|
|
|
- [x] Debian (tested on version 11)
|
|
- [x] Ubuntu (tested on version 22.04)
|
|
- [x] Rocky (tested on version 9)
|
|
|
|
on processor architecture:
|
|
|
|
- [X] x64
|
|
- [X] arm64
|
|
- [X] armhf
|
|
|
|
## ✅ System requirements
|
|
|
|
- Control Node (the machine you are running `ansible` commands) must have Ansible 2.11+ If you need a quick primer on Ansible [you can check out my docs and setting up Ansible](https://technotim.live/posts/ansible-automation/).
|
|
|
|
- You will also need to install collections that this playbook uses by running `ansible-galaxy collection install -r ./collections/requirements.yml` (important❗)
|
|
|
|
- [`netaddr` package](https://pypi.org/project/netaddr/) must be available to Ansible. If you have installed Ansible via apt, this is already taken care of. If you have installed Ansible via `pip`, make sure to install `netaddr` into the respective virtual environment.
|
|
|
|
- `server` and `agent` nodes should have passwordless SSH access, if not you can supply arguments to provide credentials `--ask-pass --ask-become-pass` to each command.
|
|
|
|
## 🚀 Getting Started
|
|
|
|
### 🍴 Preparation
|
|
|
|
First create a new directory based on the `sample` directory within the `inventory` directory:
|
|
|
|
```bash
|
|
cp -R inventory/sample inventory/my-cluster
|
|
```
|
|
|
|
Second, edit `inventory/my-cluster/hosts.ini` to match the system information gathered above
|
|
|
|
For example:
|
|
|
|
```ini
|
|
[master]
|
|
192.168.30.38
|
|
192.168.30.39
|
|
192.168.30.40
|
|
|
|
[node]
|
|
192.168.30.41
|
|
192.168.30.42
|
|
|
|
[k3s_cluster:children]
|
|
master
|
|
node
|
|
```
|
|
|
|
If multiple hosts are in the master group, the playbook will automatically set up k3s in [HA mode with etcd](https://rancher.com/docs/k3s/latest/en/installation/ha-embedded/).
|
|
|
|
Finally, copy `ansible.example.cfg` to `ansible.cfg` and adapt the inventory path to match the files that you just created.
|
|
|
|
This requires at least k3s version `1.19.1` however the version is configurable by using the `k3s_version` variable.
|
|
|
|
If needed, you can also edit `inventory/my-cluster/group_vars/all.yml` to match your environment.
|
|
|
|
### ☸️ Create Cluster
|
|
|
|
Start provisioning of the cluster using the following command:
|
|
|
|
```bash
|
|
ansible-playbook site.yml -i inventory/my-cluster/hosts.ini
|
|
```
|
|
|
|
After deployment control plane will be accessible via virtual ip-address which is defined in inventory/group_vars/all.yml as `apiserver_endpoint`
|
|
|
|
### 🔥 Remove k3s cluster
|
|
|
|
```bash
|
|
ansible-playbook reset.yml -i inventory/my-cluster/hosts.ini
|
|
```
|
|
|
|
>You should also reboot these nodes due to the VIP not being destroyed
|
|
|
|
## ⚙️ Kube Config
|
|
|
|
To copy your `kube config` locally so that you can access your **Kubernetes** cluster run:
|
|
|
|
```bash
|
|
scp debian@master_ip:~/.kube/config ~/.kube/config
|
|
```
|
|
|
|
### 🔨 Testing your cluster
|
|
|
|
See the commands [here](https://technotim.live/posts/k3s-etcd-ansible/#testing-your-cluster).
|
|
|
|
### Troubleshooting
|
|
|
|
Be sure to see [this post](https://github.com/techno-tim/k3s-ansible/discussions/20) on how to troubleshoot common problems
|
|
|
|
### Testing the playbook using molecule
|
|
|
|
This playbook includes a [molecule](https://molecule.rtfd.io/)-based test setup.
|
|
It is run automatically in CI, but you can also run the tests locally.
|
|
This might be helpful for quick feedback in a few cases.
|
|
You can find more information about it [here](molecule/README.md).
|
|
|
|
### Pre-commit Hooks
|
|
|
|
This repo uses `pre-commit` and `pre-commit-hooks` to lint and fix common style and syntax errors. Be sure to install python packages and then run `pre-commit install`. For more information, see [pre-commit](https://pre-commit.com/)
|
|
|
|
## Thanks 🤝
|
|
|
|
This repo is really standing on the shoulders of giants. Thank you to all those who have contributed and thanks to these repos for code and ideas:
|
|
|
|
- [k3s-io/k3s-ansible](https://github.com/k3s-io/k3s-ansible)
|
|
- [geerlingguy/turing-pi-cluster](https://github.com/geerlingguy/turing-pi-cluster)
|
|
- [212850a/k3s-ansible](https://github.com/212850a/k3s-ansible)
|