Ceph on kubernetes by rook (rook-ceph) fails with its container doing crashloopbackoff

 So all of a sudden 1/3 ceph nodes became not ready in the k8s and it started evicting the pods including of course the ceph pods which need to access the local storage which lead to the ceph storage malfuntion. Tried all from github issues of rook-ceph helm chart but no go. Finally installed the ceph-toolbox on k8s and ran "ceph status" and that triggered the auto repair tasks on ceph and fixed the ceph pods and thus everything came back online. Strange. I hope rook-ceph also starts offering both filesystems from single deployment on k8s too.

Google is wrong about managing multiple kubernetes configs

So here is the documentation about how you should manager multiple kubernetes clusters from your workstation. I know that now it is mostly CNCF which guidelines the k8s but still google 'baba' is the father of it (or may be mother or both).
https://kubernetes.io/docs/concepts/configuration/organize-cluster-access-kubeconfig/
I was kind of confused when I read this first, not just about the technicality but when they suggest that we should *merge* or have one file for all. In the world of compartmentalization, seggregation of apps or services, microservices, namespaces where application or service distancing is as much necessary as social distancing during the pandemic (read covic19) to minimize the spread, kubernetes options or suggestions to have them all in one file is simply baffling. The dis advantages of official documented methods is

1. It is one file for all, If one file is corrupt then your access to all your clusters is in danger.
2. A misconfigured master kuber config can lead to right operations performed on wrong cluster, leading to disaster.
3. switching between clusters (contexts) is a loooong command
4. Messy
5. Hard to keep a track of contexts once the clusters you manage grows

Solution: 20 year old solution. Yes. Linux/Unix has a built in solution.

In my daily operations I use alias.

Take a look at my  ~/.bashrc file.

alias kl=kubectl
alias skl='sudo kubectl'
alias cici='export KUBECONFIG=/home/batman/.kube/configs/ramci'
alias cist='export KUBECONFIG=/home/batman/.kube/configs/ramstaging'
alias k8ra='export KUBECONFIG=/home/batman/.kube/configs/kube_config_rancher-cluster.yaml'
alias k8cs='export KUBECONFIG=/home/batman/.kube/configs/kube_config_k8s-devel-cluster.yaml'

When I type k8cs in my terminal, I switch to a 'k8s cluster sanjose', When I type k8st 'k8s cluster staging'. you can even label then as
k8a
k8ab
k8ac
k8ad...
k8+2 letter permutation among the 26 alphabets.
With just 3 or 4 keystrokes i can switch between clusters. All kubeconfig files are separate from each other and when a colleague wants access or a kubeconfig file of a particular cluster, I can just pass it on without having to worry about passing on a file which contains access to all clusters. I sometimes also append a command to prints out the cluster-info after it switches to that cluster. 

Add kubernetes cluster to gitlab

1. Create a serviceaccount for namespace kube-system (let us say gitlab is the name of the serviceaccount)
2. kubectl -n kube-system get sa gitlab -o yaml
3. You will find the token from the step 2. Let us say the name of that token is gitlab-token-5g769 kubectl -n kube-system get secret gitlab-token-5g769 -o yaml
4. Decode the base 64 ca.crt value and then convert it to ca.pem, you can use openssl command line or some online converter
5. Decode the token too using base 64
6. Now offer the ca and pem values to gitlab during adding of k8s.

Preparing your kubernetes nodes

I often create, destroy, rebuild k8s nodes and I have noticed a few things which are a must. I personally use RKE to build, rebuild, reconfigure, configure these clusters but irrespective of that the list should apply to you too.
note: replace username ambi with whatever you like. The following list if for ubuntu 18+
1. create a new user called ambi and add him to sudo group.
2. log in as this user, install docker-ce by following the docker's official documentation. Add above user to docker group.
3. Make sure /etc/docker/certs.d/<myprivateregistry.com>/<myprivateregistry.com>.crt is present on all nodes
4. sudo chown <newly created user from step 1> /etc/docker
5. If you have storageclasses then make sure to have a default one.
kubectl patch storageclass <mystorageclassname> -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'

MetalLB load balancer for your Rancher 2 Kubernetes cluster

It is fairly simple to setup a kubernetes baremetal cluster with rancher 2 which comes with all the bells and whistles. I however had some touch time in figuring out why I couldnt get the metallb load balaner working.
First setup your rancher 2 baremetal cluster.
Then go to

After this deploy the metallb on your kubernetes cluster as per
https://metallb.universe.tf/installation/
then deploy a configmap with an ip pool.

kubectl run nginx --image nginx
kubectl expose deployment nginx --type=LoadBalancer --name=nginx-service

Now you will see an exteranl IP from the metallb ip pool. You delete the service and metallb will reclaim it.