Preparing k8s nodes

 I like using RKE to build, destroy, update, upgrade k8s clusters. I have come to understand that there are a lot of small but very impactful things we need to take care of it. If I forget this is what I will come back to check

1. Create a separate user for RKE (let us say the user is rke) and use the ssh keyfile for authentication
2. Install docker using the official docker package from docker.io as per their documentation
   https://docs.docker.com/engine/install/ubuntu/
3. Disable docker apt repository so that when you install updates to the system, the node won't update to the latest docker version and thus breaking your cluster. On ubuntu 18 I first checked the apt repositories list
   cat /etc/apt/sources.list | grep docker
   # deb-src [arch=amd64] https://download.docker.com/linux/ubuntu bionic stablerke@build-hq-45:~$ sudo apt list upradeapt list --upgradable^C

rke@build-hq-45:~$ sudo apt list upradeapt list --upgradable^C

rke@build-hq-45:~$ apt list --upgradable

Listing... Done

docker-ce/bionic 5:20.10.2~3-0~ubuntu-bionic rke64 [upgradable from: 5:19.03.14~3-0~ubuntu-bionic]

docker-ce-cli/bionic 5:20.10.2~3-0~ubuntu-bionic rke64 [upgradable from: 5:19.03.14~3-0~ubuntu-bionic]

linux-generic/bionic-updates,bionic-security 4.15.0.132.119 rke64 [upgradable from: 4.15.0.130.117]

linux-headers-generic/bionic-updates,bionic-security 4.15.0.132.119 rke64 [upgradable from: 4.15.0.130.117]

linux-image-generic/bionic-updates,bionic-security 4.15.0.132.119 rke64 [upgradable from: 4.15.0.130.117]

tar/bionic-updates,bionic-security 1.29b-2ubuntu0.2 rke64 [upgradable from: 1.29b-2ubuntu0.1]

rke@build-hq-45:~$ cat /etc/apt/sources.list | grep dock

# deb-src [arch=rke64] https://download.docker.com/linux/ubuntu bionic stable

rke@build-hq-45:~$ sudo add-apt-repository --remove deb-src [arch=rke64] https://download.docker.com/linux/ubuntu bionic stable

Error: need a single repository as argument

rke@build-hq-45:~$ sudo add-apt-repository --remove 'deb-src [arch=rke64] https://download.docker.com/linux/ubuntu bionic stable'

rke@build-hq-45:~$ cat /etc/apt/sources.list | grep dock

4. sudo chmod 777 /var/run/docker.sock && sudo chown rke /var/run/docker.sock

5. sudo usermod -aG docker rke && sudo usermod -aG rke

Installing docker is also a bit tricky sometimes on ubuntu. Here is what has worked for me. https://docs.docker.com/engine/install/ubuntu/ as per this document, 

1. Uninstall existing docker sudo apt-get remove docker docker-engine docker.io containerd runc

2. sudo apt-get install -y \

    apt-transport-https \

    ca-certificates \

    curl \

    gnupg-agent \

    software-properties-common

3. curl -fsSL https://get.docker.com -o get-docker.sh

4. chmod +x *.sh

5. sudo sh get-docker.sh

6. Uninstall again  sudo apt-get remove docker docker-ce docker-engine docker.io containerd runc

The script has updated the docker apt repository and installed the latest docker version too. We want to retain the repository source but install a specific version of docker.

7. sudo apt-get purge docker-ce docker-ce-cli containerd.io

8. sudo apt-get update

9. apt-cache madison docker-ce (list all versions)

10. DOCKER_VERSION="5:19.03.14~3-0~ubuntu-bionic" [we just want docker version 19 since that is what is compatible with out rke and k8s 1.18…]

11. sudo chmod 777 /var/run/docker.sock && sudo chown rke /var/run/docker.sock

12. sudo usermod -aG docker rke && sudo usermod -aG rke

13. edit /etc/ssh/sshd_config and AllowTcpForwarding yes

 Cleaning up of nodes

When you remove k8s or reinstall, then you might want to do it.

- `docker rm -f $(docker ps -qa)`

- `docker rmi -f $(docker images -q)`

- `docker volume rm $(docker volume ls -q)`

- `for mount in $(mount | grep tmpfs | grep '/var/lib/kubelet' | awk '{ print $3 }') /var/lib/kubelet /var/lib/rancher; do umount $mount; done`

- `sudo rm -rf /etc/ceph \

       /etc/cni \

       /etc/kubernetes \

       /opt/cni \

       /opt/rke \

       /run/secrets/kubernetes.io \

       /run/calico \

       /run/flannel \

       /var/lib/calico \

       /var/lib/etcd \

       /var/lib/cni \

       /var/lib/kubelet \

       /var/lib/rancher/rke/log \

       /var/log/containers \

       /var/log/pods \

       /var/run/calico`

Online gaming is rigged against non streamers and non payers

This was my response to a reddit post by a user complaining about *Crosshairs have disappeared bug???*

https://www.reddit.com/r/apexlegends/comments/ifpbs8/crosshairs_have_disappeared_bug/ 

This happens to only non streamers and non payers to give them a disadvantage so that streamers and payers (spenders) get an advantage to get kills/points against you, so that they keep streaming and spending.

As someone with more than 1600+ hours on it, I have felt that the AI of the game does the following partiality to cripple non streamer, non payer gamers like me.. These are ephemeral things which are next to impossible to prove and can be claimed as just a bug...

1. non streamers/non payers ephemerally gets their cross hair disabled
2. non streamers/non payers are most likely get p2020 or mozambique than others when they land
3. non streamers/non payers are most likely to get empty guns on landing if they dont get p2020 or mobambique
4. On an average it takes 1 ring to close to get bare minimum for non streamers/non payers (small backpack, KO shield, helmet, body shield) while streamers and payers get them as soon as they land
5. Harder to lock to target/head of enemies for non payers and non streamers
6. non streamers/non payers are more likely to face lag and disconnection than streamers and payers

The priority of best gaming experience is following

1. streamers
2. payers
3. non streamers

STREAMERS of course will and have to get the best experience since it is they who make the game famous. If streamers are not getting better kill count and score then they will not get viewership/subscribers and they may start losing subs too and hence it is in streamers best interest to stream only a game where they are good at it, and thus it is in a gaming companies best interest to give streamers the best experience and to their viewers.

Payers are the 2nd priority since they are the one who keeps the company profitable.

Non streamers are fed as fodder to streamers and payers.

In short If you are a streamer then you have at least 10% advantage and if you are non streamer/non payer then you have 10% disadvantage.

Nobody wants mozambique or p2020 but they are kept for a reason, to make it available for non streamers to cripple their game play. Before you dismiss me as conspiracy theorist, tin foil hat or whatever, just consider this, There are people who take lives for a mere 1000 bucks, commit heinous crime for a million, These games have billions of dollars riding on them. A perfect crime is one where you can never prove it and it is ephemeral where data is next to impossible to gather and prove. What can one do? Go against a billion dollar giant in court? lol. It is like going against a social media giant accusing them of trying to manipulate elections.

A production kubernetes cluster

 TLDR

1. An HA nginx cluster of a minimum of  3 VMs with tls
2. An HA etcd cluster of a minimum of 3 VMs with tls
3. Install k8s cluster with at least 3 master nodes using the external etcd cluster 
4. nginx will be used to load balance amongst all 3 master nodes, All connections to k8s including the api calls will go through nginx 
5. Use the external ceph cluster for main workloads
6. Use the local rook-ceph cluster which uses the drives in the physical worker nodes.

Networks

1. Restricted network
2. Public network

Storage

nodes: 3

Install a 3 node ceph cluster.

Hypervisors

nodes: 3

Node 1 will contain the following vms

1. master1
2. nginx1
3. etcd1
4. worker1

Node 2 will contain the following vms

1. master2
2. nginx2
3. etcd2
4. worker2

Node 3 will contain the following vms

1. master3
2. nginx3
3. etcd3
4. worker3

K8s

K8s will be installed on 3 VMs + X physical nodes as workers. All these hypervisors, VMs and k8s will be installed in restricted network. Public network ip addresses will be used for metallb loadbalancer and apps on k8s will use these public network ip addresses. The nginx (not the nginx cluster of VMs) ingress controller deployed on the k8s via the helm chart will take care of ingress.

Docker on windows fails with error 'error during connect: This error may indicate that the docker daemon is not running.: Post http://,***** pen //./pipe/docker_engine: The system cannot find the file specified.

 Error: error during connect: This error may indicate that the docker daemon is not running.: Post http://%2F%2F.%2Fpipe%2Fdocker_engine/v1.24/images/create?fromImage=gajuambi%2Fshorty&tag=latest: open //./pipe/docker_engine: The system cannot find the file specified.

github issue: https://github.com/docker/for-win/issues/1825

solution which worked for me:

1. Launch docker for desktop
2. settings
3. check Expose daemon on tcp://localhost:2375 without TLS