awx on kubernetes with operator

 So I was using awx15 on our on prem k8s and I had to reinstall the community helm chart was yanked out. The official awx github page was no good, the awx operator could only install the latest but we needed 15 and it gave 18+. Luckily the older operator 0.6.0 deployed the awx 15 but now without the admin secret to log in. Thanks to this https://github.com/ansible/awx-operator/issues/123#issuecomment-797820294 I could just login, create a new superuser and get on with it. The awx operator has a long way to go.

Battle Royale or Scam Royale? An experience with Apex Legends by EA and Respawn

 Top AAA title to play for free forever, what is the catch? nothing, well at least nothing that you can see or know or prove.

I have been an admirer and a continuous player of apex legends from season 01 and so far I have 1000s of hours on this game with highest level in the game too. I still believe that to my taste it is the best game royale as of today and others should learn from it.

I am a free player (I do not pay or spend for anything in the game), I am not an influencer (not a streaming celebrity where my stream has millions of viewers and they get influenced by me on what they have to play. Here I have taken apex legends of course only as an example and it can be done on any online game.

Perfect Crime

A perfect crime is an act of unethical or immoral or dishonest behavior where proving the crime is almost next to impossible. Some might want to dismiss this as a conspiracy, koo koo, tin foil hat theory, my only answer to that is, in this world there are who commit heinous crimes like murder for few bucks where the chances of getting caught are very high, then what are the chances of some one committing a dishonest, almost impossible to prove crime where tens of crores (or billions) of money is riding and this act even if proven, there is no jail time for it, no lives lost or harmed. The code/game keeps getting updated, new features keep getting added. In this constantly changing state of the game, it is of course next to impossible to prove a crime. A search engine giant constantly updates the search engine and they do not disclose their algorithm, which means if they used their search engine to influence elections, promote their partners over others then there is no way way of proving it since they would have already updated the code/algorithm used by the search engine. These companies are so big that it is unthinkable for any use of them to have resources, energy and time to face such goliaths in court or outside.

So how does the game actually make money then? The only source for them is for player to do micro transactions. They (respawn and EA who are the developers of this game) have claimed that no micro transactions will affect the gameplay or in other words the experience (your spending won't affect your chances of losing or winning) and the game play will be the same for players who spend and players who do not. There are 3 types of players in any free to play video game. 

1. Influencers
2. Spenders
3. free player (or fodder or none of the above)

I feel/think that (of course I do not have any proof of it and it is slowly based on my personal observations and experience) your chances of winning and losing can be increased or decreased or in other words game developers have a system which can give a boost to the players that they want. Currently of course the motivation is money but when money is not the factor it can also be done based on the ideology that they want to promote. conservative, politics, religion etc.,

It is also important that they do this only to extent where the players don't completely quit and never play the game. It should be done only enough where fodder level players still play it. If there are those who play it anyway, then you can always target them and feed them as fodder.

Influencers 

Influencers are the one who bring players to the game, if they are not having a great time playing it, then they lose viewership and influences (streamers) won't stream a game for which they will lose viewers. It is in streamers best interest that they stream games which will get them viewers, subscribers and make them famous. If streamers won't stream a game where they seem to suck at or lose or not doing well then game companies will lose, hence it is in game company's best interest that streamers always have an experience which promotes their growth (in terms of subscribers and viewers). All gamers know what are cheat codes and god modes. It is in game company's best interest that they apply cheat codes to the profiles of the influencer's profiles in their game's database on the server side itself and based on that tag attached to their profile the game server will increase their chances of winning ( I will explain how). It is the influences who bring spenders and free players.

Spenders

Spenders are the ones who actually pay money to buy items like skin and other cosmetics which actually have (or should) no value in the game's outcome (winning or losing). A spender will only spend on a game even if he does not have to, only if he is having a lot of fun. It is in the company's best interest that these spenders are of a high (not higher than influencers of course) priority. An AI based on the whether a person is a spender or a free player, can attach certain tags in the database for the profile of the player and based on that the game server will either degrade or upgrade your experience (think of these as cheat codes, but applied at the server side, none can see them except the developers of that AI). 

Free Player (fodder)

This is me. If Influencers need to get on an average 10+ kills per game and spenders should get 5-10 kills per game then it is the free player and some bots who are being fed as a fodder to the influencers and spenders. 

How?

The following tactics can be used to alter a player's experience and thus the outcome of the player's game (winning or losing or points gained (kill counts, wins, level ups etc.,) and here I have taken the game I play apex legend for reference. The following can/will be altered (increase or decrease) based on what type of player (influencer, spender, free) is playing the game

1. network lag
2. input lag
3. output lag
4. damage done
5. damage taken
6. aim assist
7. availability of the type of gun and ammo on entry
8. availability of the type of ammo during the gameplay
9. reliability of the abilities
10. cool down time
11. heal and recharge items
12. heal and recharge time
13. auto/burst symbols disappear or won't show when you switch modes
14. sound issues
15. can't/won't hear enemies
16. false sound (enemy on the left is heard as enemy on the right or center and vice versa)
17. spawning of items (it is no more random but customized for player types, otherwise one wouldn't be getting the weakest weapons on landing, 7/10 times)

They can always dismiss the above and many more as just a bug.

I will list out some possible ways on how they can cripple gameplayer for type 3 players for different characters

Caustic

1. Gas trap wont deploy
2. Gas trap won't get triggered
3. Gas trap trigger time increases
4. Gas cloud damage done decreases (or increases depending on type of player)
5. Disconnect player and when he reconnects he has been shot by the type 1, 2 players.
6. Make player appear in front of enemy by making him suddenly appear in front of enemy (type 1,2 players)

Watson

1. Her fence wont' deploy

Loba

1. jump drive comes back and won't work when you really need it

The whole thing can be done by AI with no or minimal intervention and a ML model can learn about the player to adopt. Currently AI/ML is used by twitter on whom/what to trend, whom/what to suppress, what/whom to cancel or censor, what/whom to shadow ban etc.., This is no secret. The same is true for these online games. Today the motivation is most possibly money but if you look at the top player of apex legends, if not all but a great majority of them all are western, white and/or Christian and this tends to make one think about other motivations too.

Redis server tuning

 TLDR, Here are some of the performance tuning settings for my giant redis on ubuntu 18.

1.     tcp-keepalive 0 at /etc/redis/redis.conf

2.     sysctl -w net.core.somaxconn=65365 at /etc/rc.local

3.     echo 'vm.overcommit_memory = 1' >> /etc/sysctl.conf

4.     uncomment the following lines at (if you are using redis in clustering mode) /etc/redis/redis.conf
save 300 10
save 300 10
save 60 10000

rdbcompression no
rdbchecksum no

appendonly no

5.     Add the following line echo never > /sys/kernel/mm/transparent_hugepage/enabled  to /etc/rc.local before exit 0

6.     config set  timeout 300

7.     config set loglevel notice

8.     config set maxmemory-policy volatile-lru

9.     config set maxmemory 491000MB 

6-8 are run after you enter redis-cli command line by running redis-cli and to know more about what these are go here.

Preparing k8s nodes

 I like using RKE to build, destroy, update, upgrade k8s clusters. I have come to understand that there are a lot of small but very impactful things we need to take care of it. If I forget this is what I will come back to check

1. Create a separate user for RKE (let us say the user is rke) and use the ssh keyfile for authentication
2. Install docker using the official docker package from docker.io as per their documentation
   https://docs.docker.com/engine/install/ubuntu/
3. Disable docker apt repository so that when you install updates to the system, the node won't update to the latest docker version and thus breaking your cluster. On ubuntu 18 I first checked the apt repositories list
   cat /etc/apt/sources.list | grep docker
   # deb-src [arch=amd64] https://download.docker.com/linux/ubuntu bionic stablerke@build-hq-45:~$ sudo apt list upradeapt list --upgradable^C

rke@build-hq-45:~$ sudo apt list upradeapt list --upgradable^C

rke@build-hq-45:~$ apt list --upgradable

Listing... Done

docker-ce/bionic 5:20.10.2~3-0~ubuntu-bionic rke64 [upgradable from: 5:19.03.14~3-0~ubuntu-bionic]

docker-ce-cli/bionic 5:20.10.2~3-0~ubuntu-bionic rke64 [upgradable from: 5:19.03.14~3-0~ubuntu-bionic]

linux-generic/bionic-updates,bionic-security 4.15.0.132.119 rke64 [upgradable from: 4.15.0.130.117]

linux-headers-generic/bionic-updates,bionic-security 4.15.0.132.119 rke64 [upgradable from: 4.15.0.130.117]

linux-image-generic/bionic-updates,bionic-security 4.15.0.132.119 rke64 [upgradable from: 4.15.0.130.117]

tar/bionic-updates,bionic-security 1.29b-2ubuntu0.2 rke64 [upgradable from: 1.29b-2ubuntu0.1]

rke@build-hq-45:~$ cat /etc/apt/sources.list | grep dock

# deb-src [arch=rke64] https://download.docker.com/linux/ubuntu bionic stable

rke@build-hq-45:~$ sudo add-apt-repository --remove deb-src [arch=rke64] https://download.docker.com/linux/ubuntu bionic stable

Error: need a single repository as argument

rke@build-hq-45:~$ sudo add-apt-repository --remove 'deb-src [arch=rke64] https://download.docker.com/linux/ubuntu bionic stable'

rke@build-hq-45:~$ cat /etc/apt/sources.list | grep dock

4. sudo chmod 777 /var/run/docker.sock && sudo chown rke /var/run/docker.sock

5. sudo usermod -aG docker rke && sudo usermod -aG rke

Installing docker is also a bit tricky sometimes on ubuntu. Here is what has worked for me. https://docs.docker.com/engine/install/ubuntu/ as per this document, 

1. Uninstall existing docker sudo apt-get remove docker docker-engine docker.io containerd runc

2. sudo apt-get install -y \

    apt-transport-https \

    ca-certificates \

    curl \

    gnupg-agent \

    software-properties-common

3. curl -fsSL https://get.docker.com -o get-docker.sh

4. chmod +x *.sh

5. sudo sh get-docker.sh

6. Uninstall again  sudo apt-get remove docker docker-ce docker-engine docker.io containerd runc

The script has updated the docker apt repository and installed the latest docker version too. We want to retain the repository source but install a specific version of docker.

7. sudo apt-get purge docker-ce docker-ce-cli containerd.io

8. sudo apt-get update

9. apt-cache madison docker-ce (list all versions)

10. DOCKER_VERSION="5:19.03.14~3-0~ubuntu-bionic" [we just want docker version 19 since that is what is compatible with out rke and k8s 1.18…]

 Cleaning up of nodes

When you remove k8s or reinstall, then you might want to do it.

- `docker rm -f $(docker ps -qa)`

- `docker rmi -f $(docker images -q)`

- `docker volume rm $(docker volume ls -q)`

- `for mount in $(mount | grep tmpfs | grep '/var/lib/kubelet' | awk '{ print $3 }') /var/lib/kubelet /var/lib/rancher; do umount $mount; done`

- `sudo rm -rf /etc/ceph \

       /etc/cni \

       /etc/kubernetes \

       /opt/cni \

       /opt/rke \

       /run/secrets/kubernetes.io \

       /run/calico \

       /run/flannel \

       /var/lib/calico \

       /var/lib/etcd \

       /var/lib/cni \

       /var/lib/kubelet \

       /var/lib/rancher/rke/log \

       /var/log/containers \

       /var/log/pods \

       /var/run/calico`

Online gaming is rigged against non streamers and non payers

This was my response to a reddit post by a user complaining about *Crosshairs have disappeared bug???*

https://www.reddit.com/r/apexlegends/comments/ifpbs8/crosshairs_have_disappeared_bug/ 

This happens to only non streamers and non payers to give them a disadvantage so that streamers and payers (spenders) get an advantage to get kills/points against you, so that they keep streaming and spending.

As someone with more than 1600+ hours on it, I have felt that the AI of the game does the following partiality to cripple non streamer, non payer gamers like me.. These are ephemeral things which are next to impossible to prove and can be claimed as just a bug...

1. non streamers/non payers ephemerally gets their cross hair disabled
2. non streamers/non payers are most likely get p2020 or mozambique than others when they land
3. non streamers/non payers are most likely to get empty guns on landing if they dont get p2020 or mobambique
4. On an average it takes 1 ring to close to get bare minimum for non streamers/non payers (small backpack, KO shield, helmet, body shield) while streamers and payers get them as soon as they land
5. Harder to lock to target/head of enemies for non payers and non streamers
6. non streamers/non payers are more likely to face lag and disconnection than streamers and payers

The priority of best gaming experience is following

1. streamers
2. payers
3. non streamers

STREAMERS of course will and have to get the best experience since it is they who make the game famous. If streamers are not getting better kill count and score then they will not get viewership/subscribers and they may start losing subs too and hence it is in streamers best interest to stream only a game where they are good at it, and thus it is in a gaming companies best interest to give streamers the best experience and to their viewers.

Payers are the 2nd priority since they are the one who keeps the company profitable.

Non streamers are fed as fodder to streamers and payers.

In short If you are a streamer then you have at least 10% advantage and if you are non streamer/non payer then you have 10% disadvantage.

Nobody wants mozambique or p2020 but they are kept for a reason, to make it available for non streamers to cripple their game play. Before you dismiss me as conspiracy theorist, tin foil hat or whatever, just consider this, There are people who take lives for a mere 1000 bucks, commit heinous crime for a million, These games have billions of dollars riding on them. A perfect crime is one where you can never prove it and it is ephemeral where data is next to impossible to gather and prove. What can one do? Go against a billion dollar giant in court? lol. It is like going against a social media giant accusing them of trying to manipulate elections.