profile for Gajendra D Ambi on Stack Exchange, a network of free, community-driven Q&A sites

Tuesday, January 19, 2021

Preparing k8s nodes

 I like using RKE to build, destroy, update, upgrade k8s clusters. I have come to understand that there are a lot of small but very impactful things we need to take care of it. If I forget this is what I will come back to check

  1. Create a separate user for RKE (let us say the user is rke) and use the ssh keyfile for authentication
  2. Install docker using the official docker package from as per their documentation
  3. Disable docker apt repository so that when you install updates to the system, the node won't update to the latest docker version and thus breaking your cluster. On ubuntu 18 I first checked the apt repositories list
    cat /etc/apt/sources.list | grep docker
    # deb-src [arch=amd64] bionic stable
    rke@build-hq-45:~$ sudo apt list upradeapt list --upgradable^C

    rke@build-hq-45:~$ sudo apt list upradeapt list --upgradable^C
    rke@build-hq-45:~$ apt list --upgradable
    Listing... Done
    docker-ce/bionic 5:20.10.2~3-0~ubuntu-bionic rke64 [upgradable from: 5:19.03.14~3-0~ubuntu-bionic]
    docker-ce-cli/bionic 5:20.10.2~3-0~ubuntu-bionic rke64 [upgradable from: 5:19.03.14~3-0~ubuntu-bionic]
    linux-generic/bionic-updates,bionic-security rke64 [upgradable from:]
    linux-headers-generic/bionic-updates,bionic-security rke64 [upgradable from:]
    linux-image-generic/bionic-updates,bionic-security rke64 [upgradable from:]
    tar/bionic-updates,bionic-security 1.29b-2ubuntu0.2 rke64 [upgradable from: 1.29b-2ubuntu0.1]
    rke@build-hq-45:~$ cat /etc/apt/sources.list | grep dock
    # deb-src [arch=rke64] bionic stable
    rke@build-hq-45:~$ sudo add-apt-repository --remove deb-src [arch=rke64] bionic stable
    Error: need a single repository as argument
    rke@build-hq-45:~$ sudo add-apt-repository --remove 'deb-src [arch=rke64] bionic stable'
    rke@build-hq-45:~$ cat /etc/apt/sources.list | grep dock

    4. sudo chmod 777 /var/run/docker.sock && sudo chown rke /var/run/docker.sock
    5. sudo usermod -aG docker rke && sudo usermod -aG rke

    Installing docker is also a bit tricky sometimes on ubuntu. Here is what has worked for me. as per this document, 

    1. Uninstall existing docker sudo apt-get remove docker docker-engine containerd runc

    2. sudo apt-get install -y \

        apt-transport-https \

        ca-certificates \

        curl \

        gnupg-agent \


    3. curl -fsSL -o

    4. chmod +x *.sh

    5. sudo sh

    6. Uninstall again  sudo apt-get remove docker docker-ce docker-engine containerd runc

    The script has updated the docker apt repository and installed the latest docker version too. We want to retain the repository source but install a specific version of docker.

    7. sudo apt-get purge docker-ce docker-ce-cli

    8. sudo apt-get update

    9. apt-cache madison docker-ce (list all versions)

    10. DOCKER_VERSION="5:19.03.14~3-0~ubuntu-bionic" [we just want docker version 19 since that is what is compatible with out rke and k8s 1.18…]

    11. sudo chmod 777 /var/run/docker.sock && sudo chown rke /var/run/docker.sock

    12. sudo usermod -aG docker rke && sudo usermod -aG rke

    13. edit /etc/ssh/sshd_config and AllowTcpForwarding yes

     Cleaning up of nodes

    When you remove k8s or reinstall, then you might want to do it.
    - `docker rm -f $(docker ps -qa)`
    - `docker rmi -f $(docker images -q)`
    - `docker volume rm $(docker volume ls -q)`
    - `for mount in $(mount | grep tmpfs | grep '/var/lib/kubelet' | awk '{ print $3 }') /var/lib/kubelet /var/lib/rancher; do umount $mount; done`
    - `sudo rm -rf /etc/ceph \
           /etc/cni \
           /etc/kubernetes \
           /opt/cni \
           /opt/rke \
           /run/secrets/ \
           /run/calico \
           /run/flannel \
           /var/lib/calico \
           /var/lib/etcd \
           /var/lib/cni \
           /var/lib/kubelet \
           /var/lib/rancher/rke/log \
           /var/log/containers \
           /var/log/pods \

    Thursday, January 14, 2021

    Online gaming is rigged against non streamers and non payers

    This was my response to a reddit post by a user complaining about *Crosshairs have disappeared bug???* 

    This happens to only non streamers and non payers to give them a disadvantage so that streamers and payers (spenders) get an advantage to get kills/points against you, so that they keep streaming and spending.

    As someone with more than 1600+ hours on it, I have felt that the AI of the game does the following partiality to cripple non streamer, non payer gamers like me.. These are ephemeral things which are next to impossible to prove and can be claimed as just a bug...
    1. non streamers/non payers ephemerally gets their cross hair disabled
    2. non streamers/non payers are most likely get p2020 or mozambique than others when they land
    3. non streamers/non payers are most likely to get empty guns on landing if they dont get p2020 or mobambique
    4. On an average it takes 1 ring to close to get bare minimum for non streamers/non payers (small backpack, KO shield, helmet, body shield) while streamers and payers get them as soon as they land
    5. Harder to lock to target/head of enemies for non payers and non streamers
    6. non streamers/non payers are more likely to face lag and disconnection than streamers and payers
    The priority of best gaming experience is following
    1. streamers
    2. payers
    3. non streamers
    STREAMERS of course will and have to get the best experience since it is they who make the game famous. If streamers are not getting better kill count and score then they will not get viewership/subscribers and they may start losing subs too and hence it is in streamers best interest to stream only a game where they are good at it, and thus it is in a gaming companies best interest to give streamers the best experience and to their viewers.
    Payers are the 2nd priority since they are the one who keeps the company profitable.
    Non streamers are fed as fodder to streamers and payers.
    In short If you are a streamer then you have at least 10% advantage and if you are non streamer/non payer then you have 10% disadvantage.
    Nobody wants mozambique or p2020 but they are kept for a reason, to make it available for non streamers to cripple their game play. Before you dismiss me as conspiracy theorist, tin foil hat or whatever, just consider this, There are people who take lives for a mere 1000 bucks, commit heinous crime for a million, These games have billions of dollars riding on them. A perfect crime is one where you can never prove it and it is ephemeral where data is next to impossible to gather and prove. What can one do? Go against a billion dollar giant in court? lol. It is like going against a social media giant accusing them of trying to manipulate elections.

    Friday, January 8, 2021

    A production kubernetes cluster


    1. An HA nginx cluster of a minimum of  3 VMs with tls
    2. An HA etcd cluster of a minimum of 3 VMs with tls
    3. Install k8s cluster with at least 3 master nodes using the external etcd cluster 
    4. nginx will be used to load balance amongst all 3 master nodes, All connections to k8s including the api calls will go through nginx 
    5. Use the external ceph cluster for main workloads
    6. Use the local rook-ceph cluster which uses the drives in the physical worker nodes.


    1. Restricted network
    2. Public network


    nodes: 3
    Install a 3 node ceph cluster.


    nodes: 3
    Node 1 will contain the following vms
    1. master1
    2. nginx1
    3. etcd1
    4. worker1
    Node 2 will contain the following vms
    1. master2
    2. nginx2
    3. etcd2
    4. worker2
    Node 3 will contain the following vms
    1. master3
    2. nginx3
    3. etcd3
    4. worker3


    K8s will be installed on 3 VMs + X physical nodes as workers. All these hypervisors, VMs and k8s will be installed in restricted network. Public network ip addresses will be used for metallb loadbalancer and apps on k8s will use these public network ip addresses. The nginx (not the nginx cluster of VMs) ingress controller deployed on the k8s via the helm chart will take care of ingress.

    Tuesday, January 5, 2021

    Docker on windows fails with error 'error during connect: This error may indicate that the docker daemon is not running.: Post http://,***** pen //./pipe/docker_engine: The system cannot find the file specified.

     Error: error during connect: This error may indicate that the docker daemon is not running.: Post http://%2F%2F.%2Fpipe%2Fdocker_engine/v1.24/images/create?fromImage=gajuambi%2Fshorty&tag=latest: open //./pipe/docker_engine: The system cannot find the file specified.

    github issue:

    solution which worked for me:

    1. Launch docker for desktop
    2. settings
    3. check Expose daemon on tcp://localhost:2375 without TLS