profile for Gajendra D Ambi on Stack Exchange, a network of free, community-driven Q&A sites

Tuesday, January 19, 2021

Preparing k8s nodes

 I like using RKE to build, destroy, update, upgrade k8s clusters. I have come to understand that there are a lot of small but very impactful things we need to take care of it. If I forget this is what I will come back to check

  1. Create a separate user for RKE (let us say the user is rke) and use the ssh keyfile for authentication
  2. Install docker using the official docker package from as per their documentation
  3. Disable docker apt repository so that when you install updates to the system, the node won't update to the latest docker version and thus breaking your cluster. On ubuntu 18 I first checked the apt repositories list
    cat /etc/apt/sources.list | grep docker
    # deb-src [arch=amd64] bionic stable
    rke@build-hq-45:~$ sudo apt list upradeapt list --upgradable^C

    rke@build-hq-45:~$ sudo apt list upradeapt list --upgradable^C
    rke@build-hq-45:~$ apt list --upgradable
    Listing... Done
    docker-ce/bionic 5:20.10.2~3-0~ubuntu-bionic rke64 [upgradable from: 5:19.03.14~3-0~ubuntu-bionic]
    docker-ce-cli/bionic 5:20.10.2~3-0~ubuntu-bionic rke64 [upgradable from: 5:19.03.14~3-0~ubuntu-bionic]
    linux-generic/bionic-updates,bionic-security rke64 [upgradable from:]
    linux-headers-generic/bionic-updates,bionic-security rke64 [upgradable from:]
    linux-image-generic/bionic-updates,bionic-security rke64 [upgradable from:]
    tar/bionic-updates,bionic-security 1.29b-2ubuntu0.2 rke64 [upgradable from: 1.29b-2ubuntu0.1]
    rke@build-hq-45:~$ cat /etc/apt/sources.list | grep dock
    # deb-src [arch=rke64] bionic stable
    rke@build-hq-45:~$ sudo add-apt-repository --remove deb-src [arch=rke64] bionic stable
    Error: need a single repository as argument
    rke@build-hq-45:~$ sudo add-apt-repository --remove 'deb-src [arch=rke64] bionic stable'
    rke@build-hq-45:~$ cat /etc/apt/sources.list | grep dock

    4. sudo chmod 777 /var/run/docker.sock && sudo chown rke /var/run/docker.sock
    5. sudo usermod -aG docker rke && sudo usermod -aG rke

    Installing docker is also a bit tricky sometimes on ubuntu. Here is what has worked for me. as per this document, 

    1. Uninstall existing docker sudo apt-get remove docker docker-engine containerd runc

    2. sudo apt-get install -y \

        apt-transport-https \

        ca-certificates \

        curl \

        gnupg-agent \


    3. curl -fsSL -o

    4. chmod +x *.sh

    5. sudo sh

    6. Uninstall again  sudo apt-get remove docker docker-ce docker-engine containerd runc

    The script has updated the docker apt repository and installed the latest docker version too. We want to retain the repository source but install a specific version of docker.

    7. sudo apt-get purge docker-ce docker-ce-cli

    8. sudo apt-get update

    9. apt-cache madison docker-ce (list all versions)

    10. DOCKER_VERSION="5:19.03.14~3-0~ubuntu-bionic" [we just want docker version 19 since that is what is compatible with out rke and k8s 1.18…]

    11. sudo chmod 777 /var/run/docker.sock && sudo chown rke /var/run/docker.sock

    12. sudo usermod -aG docker rke && sudo usermod -aG rke

    13. edit /etc/ssh/sshd_config and AllowTcpForwarding yes

     Cleaning up of nodes

    When you remove k8s or reinstall, then you might want to do it.
    - `docker rm -f $(docker ps -qa)`
    - `docker rmi -f $(docker images -q)`
    - `docker volume rm $(docker volume ls -q)`
    - `for mount in $(mount | grep tmpfs | grep '/var/lib/kubelet' | awk '{ print $3 }') /var/lib/kubelet /var/lib/rancher; do umount $mount; done`
    - `sudo rm -rf /etc/ceph \
           /etc/cni \
           /etc/kubernetes \
           /opt/cni \
           /opt/rke \
           /run/secrets/ \
           /run/calico \
           /run/flannel \
           /var/lib/calico \
           /var/lib/etcd \
           /var/lib/cni \
           /var/lib/kubelet \
           /var/lib/rancher/rke/log \
           /var/log/containers \
           /var/log/pods \

    1 comment:

    1. Don’t worry about the unemployed tag it's so easy to tear apart. The best way to get a employed tag is by joining Oracle training with placement. How beautiful to hear this tag line training with placement at an affordable price.