profile for Gajendra D Ambi on Stack Exchange, a network of free, community-driven Q&A sites

Tuesday, January 19, 2021

Preparing k8s nodes

 I like using RKE to build, destroy, update, upgrade k8s clusters. I have come to understand that there are a lot of small but very impactful things we need to take care of it. If I forget this is what I will come back to check

  1. Create a separate user for RKE (let us say the user is rke) and use the ssh keyfile for authentication
  2. Install docker using the official docker package from docker.io as per their documentation
    https://docs.docker.com/engine/install/ubuntu/
  3. Disable docker apt repository so that when you install updates to the system, the node won't update to the latest docker version and thus breaking your cluster. On ubuntu 18 I first checked the apt repositories list
    cat /etc/apt/sources.list | grep docker
    # deb-src [arch=amd64] https://download.docker.com/linux/ubuntu bionic stable
    rke@build-hq-45:~$ sudo apt list upradeapt list --upgradable^C

    rke@build-hq-45:~$ sudo apt list upradeapt list --upgradable^C
    rke@build-hq-45:~$ apt list --upgradable
    Listing... Done
    docker-ce/bionic 5:20.10.2~3-0~ubuntu-bionic rke64 [upgradable from: 5:19.03.14~3-0~ubuntu-bionic]
    docker-ce-cli/bionic 5:20.10.2~3-0~ubuntu-bionic rke64 [upgradable from: 5:19.03.14~3-0~ubuntu-bionic]
    linux-generic/bionic-updates,bionic-security 4.15.0.132.119 rke64 [upgradable from: 4.15.0.130.117]
    linux-headers-generic/bionic-updates,bionic-security 4.15.0.132.119 rke64 [upgradable from: 4.15.0.130.117]
    linux-image-generic/bionic-updates,bionic-security 4.15.0.132.119 rke64 [upgradable from: 4.15.0.130.117]
    tar/bionic-updates,bionic-security 1.29b-2ubuntu0.2 rke64 [upgradable from: 1.29b-2ubuntu0.1]
    rke@build-hq-45:~$ cat /etc/apt/sources.list | grep dock
    # deb-src [arch=rke64] https://download.docker.com/linux/ubuntu bionic stable
    rke@build-hq-45:~$ sudo add-apt-repository --remove deb-src [arch=rke64] https://download.docker.com/linux/ubuntu bionic stable
    Error: need a single repository as argument
    rke@build-hq-45:~$ sudo add-apt-repository --remove 'deb-src [arch=rke64] https://download.docker.com/linux/ubuntu bionic stable'
    rke@build-hq-45:~$ cat /etc/apt/sources.list | grep dock

    4. sudo chmod 777 /var/run/docker.sock && sudo chown rke /var/run/docker.sock
    5. sudo usermod -aG docker rke && sudo usermod -aG rke


    Installing docker is also a bit tricky sometimes on ubuntu. Here is what has worked for me. https://docs.docker.com/engine/install/ubuntu/ as per this document, 

    1. Uninstall existing docker sudo apt-get remove docker docker-engine docker.io containerd runc

    2. sudo apt-get install -y \

        apt-transport-https \

        ca-certificates \

        curl \

        gnupg-agent \

        software-properties-common

    3. curl -fsSL https://get.docker.com -o get-docker.sh

    4. chmod +x *.sh

    5. sudo sh get-docker.sh

    6. Uninstall again  sudo apt-get remove docker docker-ce docker-engine docker.io containerd runc

    The script has updated the docker apt repository and installed the latest docker version too. We want to retain the repository source but install a specific version of docker.

    7. sudo apt-get purge docker-ce docker-ce-cli containerd.io

    8. sudo apt-get update

    9. apt-cache madison docker-ce (list all versions)

    10. DOCKER_VERSION="5:19.03.14~3-0~ubuntu-bionic" [we just want docker version 19 since that is what is compatible with out rke and k8s 1.18…]

     Cleaning up of nodes

    When you remove k8s or reinstall, then you might want to do it.
    - `docker rm -f $(docker ps -qa)`
    - `docker rmi -f $(docker images -q)`
    - `docker volume rm $(docker volume ls -q)`
    - `for mount in $(mount | grep tmpfs | grep '/var/lib/kubelet' | awk '{ print $3 }') /var/lib/kubelet /var/lib/rancher; do umount $mount; done`
    - `sudo rm -rf /etc/ceph \
           /etc/cni \
           /etc/kubernetes \
           /opt/cni \
           /opt/rke \
           /run/secrets/kubernetes.io \
           /run/calico \
           /run/flannel \
           /var/lib/calico \
           /var/lib/etcd \
           /var/lib/cni \
           /var/lib/kubelet \
           /var/lib/rancher/rke/log \
           /var/log/containers \
           /var/log/pods \
           /var/run/calico`


    No comments:

    Post a Comment