Preparing k8s nodes

 I like using RKE to build, destroy, update, upgrade k8s clusters. I have come to understand that there are a lot of small but very impactful things we need to take care of it. If I forget this is what I will come back to check

1. Create a separate user for RKE (let us say the user is rke) and use the ssh keyfile for authentication
2. Install docker using the official docker package from docker.io as per their documentation
   https://docs.docker.com/engine/install/ubuntu/
3. Disable docker apt repository so that when you install updates to the system, the node won't update to the latest docker version and thus breaking your cluster. On ubuntu 18 I first checked the apt repositories list
   cat /etc/apt/sources.list | grep docker
   # deb-src [arch=amd64] https://download.docker.com/linux/ubuntu bionic stablerke@build-hq-45:~$ sudo apt list upradeapt list --upgradable^C

rke@build-hq-45:~$ sudo apt list upradeapt list --upgradable^C

rke@build-hq-45:~$ apt list --upgradable

Listing... Done

docker-ce/bionic 5:20.10.2~3-0~ubuntu-bionic rke64 [upgradable from: 5:19.03.14~3-0~ubuntu-bionic]

docker-ce-cli/bionic 5:20.10.2~3-0~ubuntu-bionic rke64 [upgradable from: 5:19.03.14~3-0~ubuntu-bionic]

linux-generic/bionic-updates,bionic-security 4.15.0.132.119 rke64 [upgradable from: 4.15.0.130.117]

linux-headers-generic/bionic-updates,bionic-security 4.15.0.132.119 rke64 [upgradable from: 4.15.0.130.117]

linux-image-generic/bionic-updates,bionic-security 4.15.0.132.119 rke64 [upgradable from: 4.15.0.130.117]

tar/bionic-updates,bionic-security 1.29b-2ubuntu0.2 rke64 [upgradable from: 1.29b-2ubuntu0.1]

rke@build-hq-45:~$ cat /etc/apt/sources.list | grep dock

# deb-src [arch=rke64] https://download.docker.com/linux/ubuntu bionic stable

rke@build-hq-45:~$ sudo add-apt-repository --remove deb-src [arch=rke64] https://download.docker.com/linux/ubuntu bionic stable

Error: need a single repository as argument

rke@build-hq-45:~$ sudo add-apt-repository --remove 'deb-src [arch=rke64] https://download.docker.com/linux/ubuntu bionic stable'

rke@build-hq-45:~$ cat /etc/apt/sources.list | grep dock

4. sudo chmod 777 /var/run/docker.sock && sudo chown rke /var/run/docker.sock

5. sudo usermod -aG docker rke && sudo usermod -aG rke

Installing docker is also a bit tricky sometimes on ubuntu. Here is what has worked for me. https://docs.docker.com/engine/install/ubuntu/ as per this document, 

1. Uninstall existing docker sudo apt-get remove docker docker-engine docker.io containerd runc

2. sudo apt-get install -y \

    apt-transport-https \

    ca-certificates \

    curl \

    gnupg-agent \

    software-properties-common

3. curl -fsSL https://get.docker.com -o get-docker.sh

4. chmod +x *.sh

5. sudo sh get-docker.sh

6. Uninstall again  sudo apt-get remove docker docker-ce docker-engine docker.io containerd runc

The script has updated the docker apt repository and installed the latest docker version too. We want to retain the repository source but install a specific version of docker.

7. sudo apt-get purge docker-ce docker-ce-cli containerd.io

8. sudo apt-get update

9. apt-cache madison docker-ce (list all versions)

10. DOCKER_VERSION="5:19.03.14~3-0~ubuntu-bionic" [we just want docker version 19 since that is what is compatible with out rke and k8s 1.18…]

11. sudo chmod 777 /var/run/docker.sock && sudo chown rke /var/run/docker.sock

12. sudo usermod -aG docker rke && sudo usermod -aG rke

13. edit /etc/ssh/sshd_config and AllowTcpForwarding yes

 Cleaning up of nodes

When you remove k8s or reinstall, then you might want to do it.

- `docker rm -f $(docker ps -qa)`

- `docker rmi -f $(docker images -q)`

- `docker volume rm $(docker volume ls -q)`

- `for mount in $(mount | grep tmpfs | grep '/var/lib/kubelet' | awk '{ print $3 }') /var/lib/kubelet /var/lib/rancher; do umount $mount; done`

- `sudo rm -rf /etc/ceph \

       /etc/cni \

       /etc/kubernetes \

       /opt/cni \

       /opt/rke \

       /run/secrets/kubernetes.io \

       /run/calico \

       /run/flannel \

       /var/lib/calico \

       /var/lib/etcd \

       /var/lib/cni \

       /var/lib/kubelet \

       /var/lib/rancher/rke/log \

       /var/log/containers \

       /var/log/pods \

       /var/run/calico`