profile for Gajendra D Ambi on Stack Exchange, a network of free, community-driven Q&A sites

Tuesday, August 14, 2018

Deploying a postgresql database instance on aws with terraform

So I needed to get the postgresql instance deployed on aws with terraform. I know it is easier to get this done via aws cli or python boto3 but it is easier with terraform. It is supposed to be non coder friendly. You do however need to search a lot online and official github and documentation.


 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
provider "aws" {
  access_key = "ACCESS_KEY"
  secret_key = "SECRET_KEY"
  region     = "us-east-1"
}

resource "random_string" "password" {
  length = 30
  special = true
  number = true
  lower = true
  upper = true
}

resource "aws_db_instance" "default" {
  allocated_storage    = 10
  storage_type         = "gp2"
  engine               = "postgres"
  engine_version       = "9.5"
  instance_class       = "db.t2.micro"
  name                 = "postgres"
  username             = "postgres"
  password = "${random_string.password.result}"
}

line
1 you are telling hashicorp terraform about where this should be deployed.
2. access key which you can take it from aws console while you were creating the user
3. secret key which you can take it from aws console while you were creating the user
4. if you know aws then you know what it is. You can choose whatever region you want for this testing purpose though.
7-13. This is a terraform's feature to generate a random string and here we have used some parameters from https://www.terraform.io/docs/providers/random/r/string.html which basically creates a random string of 30 characters and it mandates that this string should contain special, numbers, lower case and upper case letters.
15-24. Here you are telling terraform about what resource to deploy. In our case it is "aws_db_instance".
23. Uses the password generated by the code from 7-13 for the psql instance.
Now I have got to change the provider to IBM cloud and use hashicorp's vault for credentials management. That will be another blog when I figure it out.

Thursday, August 9, 2018

Deploying an instance on AWS via terraform

So I am trying to explore terraform and how to deploy instances on cloud using terraform.
first follow this to configure aws cli on your windows
http://www.cloudishes.com/2017/12/amazon-aws-automation.html
Not mandatory but recommended.
 Refer my previous post on installing terraform on windows.
Now

provider "aws" {
  access_key = "<access_key?"
  secret_key = "<secret_key?"
  region = "ap-south-1"
}
resource "aws_instance" "example" {
  ami           = "ami-cce794a3"
  instance_type = "t2.micro"
}

create a aws.tf file and paste this code.
Run

terraform init
terraform apply
from the same directory and it might ask you to type yes for a plan. Do that and you are done. Now you can go see your instance on AWS and it is live.
you will notice that it keeps asking for a build plan every time you do this. Instead you can create a build-plan first and then apply that.

terraform plan -out build-plan
terraform apply build-plan




Wednesday, August 8, 2018

getting started with terraform

So the terraform installation instructions on their site are not straight forward. Here how I did it
Linux (VM) : CentOS 6

Download the binaries
https://www.terraform.io/downloads.html


1
2
3
4
5
yum install -y zip unzip # install zip,unzip if not already present
unzip terraform_*.* # unzip terraform and cd into the zipped directory
echo $PATH # list the runtime path
mv terraform /usr/local/bin/ # move the terraform binary file to the runtime path
terraform -v # check the version of the terraform

windows

  • Download from https://www.terraform.io/downloads.html
  • unzip it.
  • create a terraform directory directly under C:/
  • then move the terraform.exe to the C:/terraform
  • launch our good old command window and run 'set PATH=%PATH%;C:\terraform'
  • Since we want it to be accessible via powershell to, go to environment variables and add the 'C:\terraform' to your PATH.
  • Run terraform -v
good luck.

Friday, July 27, 2018

Kubernetes on windows


  1. I hope you have enough rights on your windows machine to get started.
  2. Turn windows features on or off > disable hyper v;enable containers
  3. Install Virtualbox
  4. Search for ‘docker toolbox for windows’ and install it
  5. The usual method of installing kubernetes cli via powershell won’t work most of the times so install chocolatey

1
Set-ExecutionPolicy Bypass -Scope Process -Force; iex ((New-Object System.Net.WebClient).DownloadString('https://chocolatey.org/install.ps1'))

     6. Install the kubernetes cli (kubectl)

1
choco install kubernetes-cli

Run minikube now on your windows to get started and good luck.

Tuesday, July 24, 2018

Learn with me Python 3.7

Python 3.7 Introduction

So why do you wanna learn python? or why should you.

  1. Because I love it
  2. Beginners language
  3. easy to learn
  4. easy to teach
  5. readable code
  6. do more with less code
  7. faster development
  8. opensource
  9. cost effective
  10. Career or job security
  11. It is the master of many trades and jack of none.....
I can go on and one. It is one of the top tool used in
  1. web development
  2. application development
  3. Artificial Intelligence, Machine Learning, Deep Learning, Data Science
  4. Healthcare
  5. Internet of things [IOT]
  6. robotics
  7. automation (IT or anything else)
  8. mathematics, physics
  9. statistics
  10. finance
  11. weather, earthquake prediction to name a few
I am pretty sure I have missed many aspects of it but you get the point. If you are someone who is planning to have a career and you want something which opens a lot of doors for you, not just one or two then python is the answer since it makes you eligible in a lot of areas. Later you can branch out or up with python in whichever stream you want/like to.



Monday, July 23, 2018

Why containers and why not VMs?! or Vice Versa


I hope the above gives you a visual representation of what the container stuff is all about. Let me also collage some major differences.

Scalability
A VM is limited to the hardware that it is running on. The maximum memory, cpu or network resource a VM can have is limited by the hardware that it is running on. Many a times there is also a limitation of the virtualization layer tech that you might be using. If it is VMware, my favorite one then you have a cap on what is the maximum cpu and memory that you can assign. It is a scale up architecture. It is a like a huge tall building with many floors. Ultimately there is a limitation (call it gravity, sustainibility or whatever) and you can't build a skyscraper tall enough to touch the moon.

Containers are scale out. you can always have a building which can be as wide as the earth allows it to be. Instead of getting big like a VM it multiplies or clones itself as and when needed and the clones disappear automagically when the load is low.
winner:container

Security
Containers are fairly new and thus they aren't as secure as a VM architecture is. It is not the fault of containers but the security itself and how the tools are developed and who is their main target audience. Most of the current IT security tools, software are all designed for a traditional datacenter and not the micro architecture. This can be countered if you develop your applications to be cloud ready or cloud native.
winner:VM

Simplicity
A tradition VM architecture is easier and faster to deploy and get started than a container based implementation unless you are starting off new and you start your development with container based architecture as your platform. The skillgap, industry readiness are also adding to this factor.
winner:VM

Availability
Max availability a VM based architecture provides a node failure. A node1 goes down with VM1 but node2 has a copy of that VM and that takes over without a downtime. What if the 3 nodes go down? or 5 nodes? Containers offer a never go down architecture  since it is a scale out architecture. You can have your copies of the container run on different nodes and you can just mention how many nodes you want to span them across and that is it. An orchestrator engine like kubernetes or docker swarm will take care of the rest.
winner:container

Portability
Let us say you have the need of AI/ML frameworks for your project. You can spin up a kubernetes cluster on AKS (azure kubernetes service) to access azure ML/DL frameworks or do it on google to access their AI/ML/DL frameworks and they can easily span over to your on premise datacenter. You can migrate your workload/container between different cloud providers including your on/off premise datacenters. You start off with azure. Tomorrow you might want to go to google or aws or move back to your on premise hardware. That is all possible with containers and this is a big win.
winner:container

Did I miss anything? Do let me know.

Sunday, July 22, 2018

Django dyanimic url for employees or users

Let us say you are creating a site and you want your users to sign up and you want them to redirected once they log in. Below are some url formats for some well known sites.
linkedin
https://www.linkedin.com/in/<username>/
facebook
https://www.facebook.com/<username>/
How can we do that?
It is extremely inefficient to have an html page created for each user and it is a very heavy load on your site.
How to achieve this?
By default django and allauth redirect the logged in user to domain/profile so let us use that to do some magic. Here is my urls.py


 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
from django.contrib import admin
from django.urls import path, include
from . import views as core_views
from stars import views as star_views

urlpatterns = [
    path('admin/', admin.site.urls),
    path('', core_views.welcome, name="welcome"),
    path('settings/', core_views.settings, name='settings'),

    # allauth
    path('accounts/', include('allauth.urls')),

    # stars
    path('profile/', star_views.profile, name="profile"),
    path('<slug:pid>/', star_views.rprofile, name='rprofile'),
]

Concentrate on the line 15,16. So I am letting the django/allauth redirect the url to domain/profile.html and then use the star_views.profile to redirect the request to star_views.rprofile and that is done like this

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
from django.shortcuts import render
from django.contrib.auth.models import User
from django.http import HttpResponseRedirect, HttpResponse
from django.shortcuts import redirect
##Create your views here.

def profile(request):
    site = 'http://127.0.0.1:8000'
    username = request.user.username
    url = f'{site}/{username}'
    return HttpResponseRedirect(url)

def rprofile(request, pid):
    u = User.objects.get(username=pid)
    if u:
        return render(request, 'profile.html')

Now every user has his own url which will be easy to share.

Django Allauth signal to trigger something

So I am still testing the waters of django and I am still like a baby swimmer with swim tires. I had some trouble in using django allauth signals to do something.
Django 2.0
Django allauth
All in a virtualenv.
So here is how you use signals trigger some action. Go to https://github.com/pennersr/django-allauth/blob/master/docs/signals.rst to checkout the signal allauth emits at various stages. I am interested in the signal which gets dispatched after the user signs up. I am using email+password to signup and signin.


 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
from django.db import models
import time
from django.contrib.auth.models import User
from allauth.account.signals import user_signed_up, password_set
from django.dispatch import receiver, Signal

# Create your models here.
@receiver(user_signed_up)
def employeeID(sender=User, **kwargs):
    old_username = kwargs['user']
    user = User.objects.get(username = old_username)
    user.username = str(time.time()).split('.')[0]
    user.save()

1-5 import modules.
8. mention the signal that we want to use.
as per the allauth documents mentioned in the https://github.com/pennersr/django-allauth/blob/master/docs/signals.rst this signal contains the following.
user_signed_up = Signal(providing_args=["request", "user"])
So the entire info received from the signal is passed on to the funtion at line 9 as **kwargs.
10. extract the signedup user's username.
11. query the database for the data of that user.
12. change the username to some random number.
13. save the changes to the database.

So, when we are using email+password only to signup and signin, allauth still creates a username='email ID' of the user (without the @email.com, so if i signed up with mail@email.com then my username will be mail.) So I thought I will give my own username and it worked.

Thursday, June 28, 2018

Upgrading your vcenter server made easy with Hybrid Linked Mode

So whenever you upgrade an environment you must start with HCL [hardware compatibility list]. Let us pretend that you want to go from vsphere 6.5 to 6.7 without any downtime. hmm... yes you are asking for a pony and thanks to vmware you have it too.
I always say that the first point of contact should be upgrade first because it will be backward compatible of what lies beneath. Anyway assuming that you made sure that the rest of your stuff like version of your NSX, VRA is all taken care of. You have to then upgrade your
vcenter>esxi>vm tools>vm hardware.
Get a new vcenter 6.7 appliance ready.
Create HLM between the old and the new vcenter.
Decommission the old one.
Yes, that  is it. If you are not satisfied with this high level plan then there are so many bloggers wanting to be at the yearly top 100 virtualization blog list (and its awesome) by http://vsphere-land.com (http://vsphere-land.com/news/top-vblog-2017-full-results.html) and they have detailed posts on how to do hybrid linked mode. It is easy and you dont have to pull out any hair.
I still do recommend the old ways of having a plan B as backup. Yes take a backup of your old vcenter before you get on with this plan/task.
What if you have 2 vcsa already in linked mode and you want to retain the networking information of them?
Let us say that you have vcsa1 linked with vcsa2.

  1. Decommission the vcsa2 from linked mode with vcsa1.
  2. Shutdown the vcsa2, disable the network adapter
  3. get your new vcsa with your newer desired version and assign the vcsa2 networking details (hostname, ip..) and join the linked mode with vcsa1.
  4. Make sure all is well and they are in sync.
  5. Decommission the vcsa1
  6. deploy the newer versioned vcsa and assign vcsa1 networking details to it.
congratulate yourself.

Saturday, May 12, 2018

getting it on with docker

So I use centos for most of experiments, lab work at home. I just have this love hate relationship with it i simply cant explain or resist. Needed get the docker, docker swarm, docker compose to work on cent os.
Optionally please set up vmware tools on your centOS 7. I recommend it.
So here is how I set it up.


1
2
3
yum install epel-release # get the yum repository installed
yum install docker
sudo curl -L https://github.com/docker/compose/releases/download/1.21.2/docker-compose-$(uname -s)-$(uname -m) -o /usr/local/bin/docker-compose # It installs the docker composer of a version which is needed for the docker to work with. 
lines
1. install the epel repository
2. install docker which will also get you the docker swarm
3. installs the docker compose of a version which is needed for the docker version that you installed at line 2.
Get the visual studio code if you do not already have it

sudo rpm --import https://packages.microsoft.com/keys/microsoft.asc
sudo sh -c 'echo -e "[code]\nname=Visual Studio Code\nbaseurl=https://packages.microsoft.com/yumrepos/vscode\nenabled=1\ngpgcheck=1\ngpgkey=https://packages.microsoft.com/keys/microsoft.asc" > /etc/yum.repos.d/vscode.repo'
yum check-update
sudo yum install code
you will notice that you cannot just run docker-compose because you will get permission denied error. So let us do this final bit.
1
2
chown root /usr/local/bin/docker-compose
chmod 777 /usr/local/bin/docker-compose
We are not done yet.

1
2
systemctl enable docker
service docker start
line 1. enable the docker service to start at boot up.
line 2. start the docker service manually for now.

Wednesday, May 2, 2018

Get that damn VMware tools working on centos 7

So I have realized that even though virtualbox is more suited for devops activities like vagrant, docker, container, kubernetes etc., I still somehow like vmware workstation; may be because I just like VMware since I have doing VMware stuff from a very long time. I just liked the grouping of VMs, folder, tabs and more.
I keep hitting a small hurdle and that is getting the VMware tools installed on it. I am currently using centos for jenkins, docker, kubernetes, vagrant, openstack and more. So here is a just a reminder for the future me to just throw these lines at the terminal (preferably as a root user).

UNAME=$(uname -r)
yum install g++ gcc make kernel-headers kernel-devel-${UNAME%.*} -y

  1. Then you can just open the mounted iso in a terminal.
  2. copy the archive to a different system folder.
  3. untar it.
  4. cd into the unarchived folder.
  5. run the perl installer of vmware tools.

Thursday, April 26, 2018

Get set powercli 10

So Powercli 10 is out and powercli 6.5.3+ can only be availed via powershell gallery. Here is what you need to do. I assume that you are one of those who are using windows 10.

  1. Close all commandline windows; cmd, powershell, powercli etc.,
  2. Run powershell (not ISE, just powershell) as administrator
  3. Run the following command in your powershell
    Set-ExecutionPolicy -ExecutionPolicy RemoteSigned
    Accept or click yes to all and close the powershell window.
  4. Now do the step (1 and )2 again.
  5. Run the following to get your powercli 10 installed. Just accept whatever prompt it gives that is choose Y for yes and A for all.
    Install-Module -Name VMware.PowerCLI
  6. Now run the following.
    Import-Module VMware.VimAutomation.Core
  7. The following will opt you out from the customer experience program.
    Set-PowerCLIConfiguration -Scope AllUsers -ParticipateInCEIP $false
    If you wish to opt in then you can change the $false to $true. Here I am using the scope as allusers to make sure all users have this setting.
  8. Now let us set the powercli to ignore the unsigned user certificate error warnings.
    Set-PowerCLIConfiguration -InvalidCertificateAction ignore -Scope AllUsers -Confirm:$false
Now you are good to user the powercli as you are used to. 

Saturday, April 7, 2018

Ansible or Chef ? and Why?

First of all why do you need anything like ansible/chef/puppet/salt which can mainly be classified as configuration management and automation tools.
These are today's devops needs of an IT firm. You want to deploy, configure or manage the configuration of many machines across different platforms (local or cloud) then you need one.
So you have 2 types of CMT (configuration management tools).

ANSIBLE
========

  1. You want/need it to be agentless
    So if your targets are majorly devices and not operating systems or applications then you need this. If you are managing hardware routers , switches or devices where you can have an SSH connection but you cannot install any specific package in it to manage. You can't install your own package or an agent into a cisco nexus switch or any other switch of any other company. The vendors usually have a strict lock on what can be installed on these devices for security reasons. Ansible is most and best known for network automation for this same reason.
  2. Most of your infrastructure is mainly opensource/linux based.
    All ansible requires is SSH and linux systems are mainly managed via ssh.
  3. You like bash or python
    Ansible uses python and python 2.x is present by default on your gnu/linux machines.
  4. You are adventurous and do not mind coming up with your modules (write your own playbook)

CHEF
=====
  1. The need of an agent being present at the target machine/component to be managed isn't a bother.
  2. you want to manage windows, linux, mac seamlessly
  3. you like/know ruby more than you bash/shell/python
  4. you need a more mature product and better documentation
  5. Larger community (which translates to having more ready made modules available for common IT configuration management)
Currently I am fiddling with chef and I am digging it.

Wednesday, April 4, 2018

Deploying instances on gcp (google cloud platform) via powershell

This is that time. That time where I put my hands inside the gcp cookie jar and try to see what I find.
Make sure you have done this https://www.cloudishes.com/2018/04/setting-up-your-machine-for-gcp-google.html first though.
You also have to log into your gcp and enable google compute API. I think is a nice touch by gcp. You decide which of your services should have API access and which shouldnt. May be you can have some people or applications have API access and some don't. In this way you can get this configured. More on that later. May be... Below is a screen grab of my API board.

and ya...wait for a while btw after this, otherwise you will get this.

Add-GceInstance : Google.Apis.Requests.RequestError
Access Not Configured. Compute Engine API has not been used in project 1234567 before or it is disabled. Enable it by visiting 
https://console.developers.google.com/apis/api/compute.googleapis.com/overview?project=1234567 then retry. If you enabled this API 
recently, wait a few minutes for the action to propagate to our systems and retry. [403]
Errors [
 Message[Access Not Configured. Compute Engine API has not been used in project 1234567 before or it is disabled. Enable it by visiting 
https://console.developers.google.com/apis/api/compute.googleapis.com/overview?project=1234567 then retry. If you enabled this API 
recently, wait a few minutes for the action to propagate to our systems and retry.] Location[ - ] Reason[accessNotConfigured] 
Domain[usageLimits]
]
At line:16 char:20
+ ... ta_Config | Add-GceInstance -Project $project -Zone $zone -Region $re ...
+                 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [Add-GceInstance], GoogleApiException
    + FullyQualifiedErrorId : Google.GoogleApiException,Google.PowerShell.ComputeEngine.AddGceInstanceCmdlet
Now let us list out the images that we have there first. We want to deploy a tiny one first since I can't afford to pay for games, cloud costs :|.

PS C:\WINDOWS\system32> Get-GceImage | select Family

Family                          
------                          
centos-6                        
centos-7                        
coreos-alpha                    
coreos-beta                     
coreos-stable                   
debian-8                        
debian-9                        
debian-8                        
debian-9                        
rhel-6                          
rhel-7                          
sles-11                         
sles-12                         
ubuntu-1404-lts                 
ubuntu-1604-lts                 
ubuntu-1710                     
windows-1709-core-for-containers
windows-1709-core               
windows-2008-r2                 
windows-2012-r2-core            
windows-2012-r2                 
windows-2016-core               
windows-2016                    
sql-ent-2012-win-2012-r2        
sql-std-2012-win-2012-r2        
sql-web-2012-win-2012-r2        
sql-ent-2014-win-2012-r2        
sql-ent-2014-win-2016           
sql-std-2014-win-2012-r2        
sql-web-2014-win-2012-r2        
sql-ent-2016-win-2012-r2        
sql-ent-2016-win-2016           
sql-std-2016-win-2012-r2        
sql-std-2016-win-2016           
sql-web-2016-win-2012-r2        
sql-web-2016-win-2016           
sql-ent-2017-win-2016           
sql-exp-2017-win-2012-r2        
sql-exp-2017-win-2016           
sql-std-2017-win-2016           
sql-web-2017-win-2016           
Let us select the one which is highlighted.
I also needed to choose a zone and a region associated with that. Check this out.
https://cloud.google.com/compute/docs/regions-zones/


 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
$vm_name = 'ubuntuDummy' # name of the instance
$machine_type = 'Linux' # Description of the instance
$project = 'dummies' # name of my project which i created from the gcp console

# go here https://cloud.google.com/compute/docs/regions-zones/ and choose
$zone = 'us-east1-b' 
$region = 'us-east1'

# choosing an image from the list of google images
$myImage = Get-GceImage | where Family -Match 'ubuntu-1404-lts'
# create a configuration for our instance
$my_insta_Config = New-GceInstanceConfig $vm_name -MachineType $machine_type -DiskImage $myImage -Region $region

# deploy our instance
$my_insta_Config | Add-GceInstance -Project $project -Zone $zone -Region $region

Good luck.

Resources:
https://cloud.google.com/compute/docs/regions-zones/
https://support.google.com/cloud/answer/6158841?hl=en
http://googlecloudplatform.github.io/google-cloud-powershell/#/

Tuesday, April 3, 2018

Deploying Vagrant VM on AWS

Let us first make sure we have the vagrant aws plugin ready.

vagrant plugin install vagrant-aws
Create a new directory.
get inside that directory
Now let us add a dummy box to AWS made just for this.
Create a Vagrantfile by running vagrant init.

mkdir lab_aws 
cd lab_aws
vagrant box add aws-dummy https://github.com/mitchellh/vagrant-aws/raw/master/dummy.box
vagrant init
Then open the Vagrantfile with any text editor and populate it with the following.

# Install the below plugin with 'vagrant plugin install vagrant-aws'
require 'vagrant-aws'

# VM config
Vagrant.configure('2') do |config|

  # dummy AWS box https://github.com/mitchellh/vagrant-aws/raw/master/dummy.box
  config.vm.box = 'awsDummy'
  # settings from aws
  config.vm.provider :aws do |aws, override|
    # aws credentials
    aws.access_key_id = 'xxxxxxxxxxxxxxxxxxxx'
    aws.secret_access_key = 'yyyyyyyyyyyyyyy'
    aws.keypair_name = "vagrant" # ssh key pair name
    aws.ami = 'ami-5cd4a126'
    aws.region = 'us-east-1'
    aws.instance_type = 't2.micro'
    aws.security_groups = "vagrant" # enabled ssh ports in/out
    
    # the below line will help you avoid asking for username and password for smb if you are doing this from windows
    config.vm.synced_folder ".", "/vagrant", disabled: true
    override.ssh.username = 'vagrant'
    override.ssh.private_key_path = '<path>/vagrant.pem'
  end

end

So the above are the entries for your new Vagrantfile.
I chose the region 'us-east-1'
I also created a security policy group "vagrant" and enabled ssh on it. Make sure this security group is created on your chosen region in the vagrantfile. Also, make sure the ami ID that you have chosen is present in that region. I say to be safe just mimic your aws configuration to mirror this vagrantfile. Also, have your key pair download and saved somewhere. Give that path in the vagrantfille. Now just do a vagrant up and your VM will be deployed on AWS.




Monday, April 2, 2018

Setting up chef + vagrant + virtualbox on windows for aws, azure and google cloud

So spinning up VMs on aws, gcp and azure via GUI, vagrant, SDK and their native cli was okay
but when you want to do configuration management and automation tools like chef then it gets a lot more interesting.
Requirements:

  1. chef sdk https://downloads.chef.io/chefdk
  2. vagrant https://www.vagrantup.com/downloads.html
  3. Virtualbox 5.1 or higher https://www.virtualbox.org/wiki/Downloads

My plan was/is/will be (mostly)---
step 1. deploy workload via GUI on gcp/aws/azure
step 2. deploy workload via SDK on gcp/aws/azure
step 3. deploy workload via vagrant+virtualbox on gcp/aws/azure
step 4. deploy workload via chef --> vagrant+virtualbox on gcp/aws/azure
So now i have to do step 4.
Download and install chef sdk for windows from https://downloads.chef.io/chefdk
Launch your powershell 5.x or 6 if you are brave and run

PS C:\WINDOWS\system32> chef -v
Chef Development Kit Version: 2.5.3
chef-client version: 13.8.5
delivery version: master (73ebb72a6c42b3d2ff5370c476be800fee7e5427)
berks version: 6.3.1
kitchen version: 1.20.0
inspec version: 1.51.21

PS C:\WINDOWS\system32> vagrant --version
Vagrant 2.0.3
and my god takes soooooo much of time on windows just to report the installed sdk info. On any nix systems it is just a second.
PS D:\> mkdir chef


    Directory: D:\


Mode                LastWriteTime         Length Name
----                -------------         ------ ----
d-----       03-04-2018  12.19 AM                chef


PS D:\> cd chef
PS D:\chef> vagrant box add bento/centos-7.2
==> box: Loading metadata for box 'bento/centos-7.2'
    box: URL: https://vagrantcloud.com/bento/centos-7.2
This box can work with multiple providers! The providers that it
can work with are listed below. Please review the list and choose
the provider you will be working with.

1) parallels
2) virtualbox
3) vmware_desktop

Enter your choice: 2
==> box: Adding box 'bento/centos-7.2' (v2.3.1) for provider: virtualbox
    box: Downloading: https://vagrantcloud.com/bento/boxes/centos-7.2/versions/2.3.1/providers/virtualbox.box
    box: Progress: 100% (Rate: 1902k/s, Estimated time remaining: --:--:--)
==> box: Successfully added box 'bento/centos-7.2' (v2.3.1) for 'virtualbox'!
PS D:\chef>
I created a chef directory
added a box to my vagrant from the vagrant cloud. bento/centos-7.2 is the box maintained by chef so I want to play safe and want to start with that. Since I did not mention the provider it gave me 3 choices and I chose virtualbox.
So now vagrant has a box added and ready to deploy on virtualbox.
Now run

vagrant init

vagrant up
 and we have our chef centos box ready to play with.
Let us ssh to our new box with


vagrant ssh
 wget https://packages.chef.io/files/stable/chefdk/2.0.28/el/6/chefdk-2.0.28-1.el6.x86_64.rpm
Then go to /home/vagrant and install the downloaded chef rpm
sudo rpm -ivh chefdk*.rpm
If you get some permission errors then do a chmod 777 on that rpm to set proper permissions.
Run the following to verify what you have
# chef verify
# chef-client -v
# chef --version
# sudo yum install vim, nano -y
The above will install some text editors which we will need.












Setting up your machine for GCP (google cloud platform) with powershell, python and gcloud cli

So I wanted to setup my google cloud platform (gcp) on my desktop. I meant to say I wanted to be able to connect to gcp via powershell, python and gcloud cli. Apparently it seems the competition between google, amazon and microsoft is benefiting us. They have made it easier than ever. I was however disappointed that you cannot get this to work in a virtual environment if you are on windows. Sad face :(.
It is fairly straight forward.

  1. launch your command prompt as administrator
  2. Download and install gcloud SDK from here.
  3. Once the installation is complete run gcloud init (it was preselected for me) to setup the credentials. 
  4. Now let us say yes and a browser opens up asking you for confirmation. 
    I just clicked on allow.
  5. I had a dummy project created. It gives you an option to create a project too if you don't have one created yet. 
  6. I chose 1 and now i can interact with that project with gcloud sdk with the weapon of my choice; python, powershell or gcloud cli.
Now I have my playground ready. So let me play. I will try to update here about my endeavors as much as possible.

Sunday, April 1, 2018

Deploying ubuntu linux VM on Azure via cloud azure cli or powershell

So it is quite interesting. Unlike my previous posts if you don't want to setup your machine for azure by download either python or powershell sdk or any cli locally then there is a quicker and better way.
Here is exactly what it takes to deploy a VM on azure.

  1. Create an azure account if you haven't already with pay as you go model https://signup.azure.com/ you pay for what you use. so no worried. Use it like a lab, where you delete stuff once you are done.
  2. Launch the online azure cli. You can either use powershell or bash. Whichever you love. https://shell.azure.com/?prompt=True
  3. So first I want to create a resource group.
az group create -n linuxVms -l westus
      This btw is azure cli, not azure powershell syntax. Here I am creating a group callled linuxVms and the location of that is at the west us datacenter of azure.
      4. Now let us deploy a llinux vm from an ubuntu image
az vm create -g linuxVms -n dummyUbuntu -i UbuntuLTS --generate-ssh-keys
         So we are creating a VM and the
image is UbuntuLTS 
VM name is dummyUbuntu 
group name is linuxVms 
You are done! :)



Saturday, March 31, 2018

Setting up powershell for microsoft azure cloud

So it is that time of the day, actually night, actually morning 2:15am where I felt like trying out of azure. I most prefer to learn by seeing and doing but i like my words, thank you very much. The way you do with words (command prompt, script, automation) remains same, similar and sometimes identical forever but GUI keeps changing.
Run your powershell 5, 6 or ISE, whichever you like; even your visual studio code if it is configured with powershell.
Install-Module -Name AzureRM -AllowClobber

I also suggest you to go ahead and get the cli tool for powershell from here
close all the powershell windows.
launch the powershell as administrator and enable execution of script by running the following.
Set-ExecutionPolicy -ExecutionPolicy RemoteSigned
Now we want to get the azure credentials.
Get-AzurePublishSettingsFile
It takes you to the azure portal where you can select the payment model, validate and for that you get to download publish settings.
Now we will import them using the downloaded file and delete that file. Once imported that file is no more needed (for security reasons).

PS C:\Users\<username>> Import-AzurePublishSettingsFile 'C:\Users\<username>\Documents\azure\Windows Azure MSDN - 1_4_2018, 04_43_29 - credentials.publishsettings'

Id                                   Name          State ExtendedProperties                        
--                                   ----          ----- ------------------                        
f6ec6bf5-7459-46f5-a274-5a44c60fef0a Pay-As-You-Go       {[Account, 411690FA9A067441A00E1DDBECD7...
Now let us whether it worked by trying to list all the azure environments.

PS C:\Users\<username> Get-AzureEnvironment


Name                                     : AzureGermanCloud
EnableAdfsAuthentication                 : False
ActiveDirectoryServiceEndpointResourceId : https://management.core.cloudapi.de/
AdTenant                                 : Common
GalleryUrl                               : https://gallery.azure.com/
ManagementPortalUrl                      : http://portal.microsoftazure.de/
ServiceManagementUrl                     : https://management.core.cloudapi.de/
PublishSettingsFileUrl                   : https://manage.microsoftazure.de/publishsettings/index
ResourceManagerUrl                       : https://management.microsoftazure.de/
SqlDatabaseDnsSuffix                     : .database.cloudapi.de
StorageEndpointSuffix                    : core.cloudapi.de
ActiveDirectoryAuthority                 : https://login.microsoftonline.de/
GraphUrl                                 : https://graph.cloudapi.de/
TrafficManagerDnsSuffix                  : azuretrafficmanager.de
AzureKeyVaultDnsSuffix                   : vault.microsoftazure.de
AzureKeyVaultServiceEndpointResourceId   : https://vault.microsoftazure.de

Name                                     : AzureCloud
EnableAdfsAuthentication                 : False
ActiveDirectoryServiceEndpointResourceId : https://management.core.windows.net/
AdTenant                                 : Common
GalleryUrl                               : https://gallery.azure.com/
ManagementPortalUrl                      : http://go.microsoft.com/fwlink/?LinkId=254433
ServiceManagementUrl                     : https://management.core.windows.net/
PublishSettingsFileUrl                   : http://go.microsoft.com/fwlink/?LinkID=301775
ResourceManagerUrl                       : https://management.azure.com/
SqlDatabaseDnsSuffix                     : .database.windows.net
StorageEndpointSuffix                    : core.windows.net
ActiveDirectoryAuthority                 : https://login.microsoftonline.com/
GraphUrl                                 : https://graph.windows.net/
TrafficManagerDnsSuffix                  : trafficmanager.net
AzureKeyVaultDnsSuffix                   : vault.azure.net
AzureKeyVaultServiceEndpointResourceId   : https://vault.azure.net

Name                                     : AzureUSGovernment
EnableAdfsAuthentication                 : False
ActiveDirectoryServiceEndpointResourceId : https://management.core.usgovcloudapi.net/
AdTenant                                 : Common
GalleryUrl                               : https://gallery.azure.com/
ManagementPortalUrl                      : https://manage.windowsazure.us
ServiceManagementUrl                     : https://management.core.usgovcloudapi.net/
PublishSettingsFileUrl                   : https://manage.windowsazure.us/publishsettings/index
ResourceManagerUrl                       : https://management.usgovcloudapi.net/
SqlDatabaseDnsSuffix                     : .database.usgovcloudapi.net
StorageEndpointSuffix                    : core.usgovcloudapi.net
ActiveDirectoryAuthority                 : https://login.microsoftonline.us/
GraphUrl                                 : https://graph.windows.net/
TrafficManagerDnsSuffix                  : usgovtrafficmanager.net
AzureKeyVaultDnsSuffix                   : vault.usgovcloudapi.net
AzureKeyVaultServiceEndpointResourceId   : https://vault.usgovcloudapi.net

Name                                     : AzureChinaCloud
EnableAdfsAuthentication                 : False
ActiveDirectoryServiceEndpointResourceId : https://management.core.chinacloudapi.cn/
AdTenant                                 : Common
GalleryUrl                               : https://gallery.azure.com/
ManagementPortalUrl                      : http://go.microsoft.com/fwlink/?LinkId=301902
ServiceManagementUrl                     : https://management.core.chinacloudapi.cn/
PublishSettingsFileUrl                   : http://go.microsoft.com/fwlink/?LinkID=301776
ResourceManagerUrl                       : https://management.chinacloudapi.cn/
SqlDatabaseDnsSuffix                     : .database.chinacloudapi.cn
StorageEndpointSuffix                    : core.chinacloudapi.cn
ActiveDirectoryAuthority                 : https://login.chinacloudapi.cn/
GraphUrl                                 : https://graph.chinacloudapi.cn/
TrafficManagerDnsSuffix                  : trafficmanager.cn
AzureKeyVaultDnsSuffix                   : vault.azure.cn
AzureKeyVaultServiceEndpointResourceId   : https://vault.azure.cn
and yes it did. 

Setting up powershell with AWS

Okay,
So I needed to setup my new powershell 6 with AWS. Didn't want to enter the credentials and all.
Install-Module -Name AWSPowerShell
Run this on your powershell 5 and/or 6 and you will have your aws tools for powershell ready to go.
Also run
notepad $profile
on powershell 5, powershell ISE, powershell 6 and copy paste the following, save them.
Import-Module AWSPowerShell
Also, if you want to be able to run scripts on your system then enable it toot by doing
Set-ExecutionPolicy RemoteSigned
Close all of your powershell windows now, launch powershell and run the following
Get-AWSPowerShellVersion -ListServiceVersionInfo
and you should get

PS C:\WINDOWS\system32> Get-AWSPowerShellVersion -ListServiceVersionInfo

AWS Tools for Windows PowerShell
Version 3.3.208.0
Copyright 2012-2017 Amazon.com, Inc. or its affiliates. All Rights Reserved.

Amazon Web Services SDK for .NET
Core Runtime Version 3.3.21.2
Copyright 2009-2015 Amazon.com, Inc. or its affiliates. All Rights Reserved.

Release notes: https://aws.amazon.com/releasenotes/PowerShell

This software includes third party software subject to the following copyrights:
- Logging from log4net, Apache License
[http://logging.apache.org/log4net/license.html]


Service                               Noun Prefix API Version
-------                               ----------- -----------
AWS AppStream                         APS         2016-12-01 
AWS AppSync                           ASYN        2017-07-25 
AWS Batch                             BAT         2016-08-10 
AWS Budgets                           BGT         2016-10-20 
AWS Certificate Manager               ACM         2015-12-08 
AWS Cloud Directory                   CDIR        2016-05-10 
AWS Cloud HSM                         HSM         2014-05-30 
AWS Cloud HSM V2                      HSM2        2017-04-28 
AWS Cloud9                            C9          2017-09-23 
AWS CloudFormation                    CFN         2010-05-15 
AWS CloudTrail                        CT          2013-11-01 
AWS CodeBuild                         CB          2016-10-06 
AWS CodeCommit                        CC          2015-04-13 
AWS CodeDeploy                        CD          2014-10-06 
AWS CodePipeline                      CP          2015-07-09 
AWS CodeStar                          CST         2017-04-19 
AWS Config                            CFG         2014-11-12 
AWS Cost Explorer                     CE          2017-10-25 
AWS Cost and Usage Report             CUR         2017-01-06 
AWS Data Pipeline                     DP          2012-10-29 
AWS Database Migration Service        DMS         2016-01-01 
AWS Device Farm                       DF          2015-06-23 
AWS Direct Connect                    DC          2012-10-25 
AWS Directory Service                 DS          2015-04-16 
AWS Elastic Beanstalk                 EB          2010-12-01 
AWS Elemental MediaConvert            EMC         2017-08-29 
AWS Elemental MediaLive               EML         2017-10-14 
AWS Elemental MediaPackage            EMP         2017-10-12 
AWS Elemental MediaStore              EMS         2017-09-01 
AWS Elemental MediaStore Data Plane   EMSD        2017-09-01 
AWS Greengrass                        GG          2017-06-07 
AWS Health                            HLTH        2016-08-04 
AWS Identity and Access Management    IAM         2010-05-08 
AWS Import/Export                     IE          2010-06-01 
AWS Import/Export Snowball            SNOW        2016-06-30 
AWS IoT                               IOT         2015-05-28 
AWS IoT Jobs Data Plane               IOTJ        2017-09-29 
AWS Key Management Service            KMS         2014-11-01 
AWS Lambda                            LM          2015-03-31 
AWS Marketplace Commerce Analytics    MCA         2015-07-01 
AWS Marketplace Entitlement Service   MES         2017-01-11 
AWS Marketplace Metering              MM          2016-01-14 
AWS Migration Hub                     MH          2017-05-31 
AWS OpsWorks                          OPS         2013-02-18 
AWS OpsWorksCM                        OWCM        2016-11-01 
AWS Organizations                     ORG         2016-11-28 
AWS Price List Service                PLS         2017-10-15 
AWS Resource Groups                   RG          2017-11-27 
AWS Resource Groups Tagging API       RGT         2017-01-26 
AWS Security Token Service            STS         2011-06-15 
AWS Serverless Application Repository SAR         2017-09-08 
AWS Service Catalog                   SC          2015-12-10 
AWS Shield                            SHLD        2016-06-02 
AWS Storage Gateway                   SG          2013-06-30 
AWS Support API                       ASA         2013-04-15 
AWS WAF                               WAF         2015-08-24 
AWS WAF Regional                      WAFR        2016-11-28 
AWS X-Ray                             XR          2016-04-12 
Alexa For Business                    ALXB        2017-11-09 
Amazon API Gateway                    AG          2015-07-09 
Amazon Athena                         ATH         2017-05-18 
Amazon CloudFront                     CF          2017-03-25 
Amazon CloudSearch                    CS          2013-01-01 
Amazon CloudSearchDomain              CSD         2013-01-01 
Amazon CloudWatch                     CW          2010-08-01 
Amazon CloudWatch Events              CWE         2015-10-07 
Amazon CloudWatch Logs                CWL         2014-03-28 
Amazon Cognito Identity               CGI         2014-06-30 
Amazon Cognito Identity Provider      CGIP        2016-04-18 
Amazon Comprehend                     COMP        2017-11-27 
Amazon DynamoDB                       DDB         2012-08-10 
Amazon DynamoDB Accelerator (DAX)     DAX         2017-04-19 
Amazon EC2 Container Registry         ECR         2015-09-21 
Amazon EC2 Container Service          ECS         2014-11-13 
Amazon ElastiCache                    EC          2015-02-02 
Amazon Elastic Compute Cloud          EC2         2016-11-15 
Amazon Elastic File System            EFS         2015-02-01 
Amazon Elastic MapReduce              EMR         2009-03-31 
Amazon Elastic Transcoder             ETS         2012-09-25 
Amazon Elasticsearch                  ES          2015-01-01 
Amazon GameLift Service               GML         2015-10-01 
Amazon GuardDuty                      GD          2017-11-28 
Amazon Inspector                      INS         2016-02-16 
Amazon Kinesis                        KIN         2013-12-02 
Amazon Kinesis Analytics              KINA        2015-08-14 
Amazon Kinesis Firehose               KINF        2015-08-04 
Amazon Kinesis Video Streams          KV          2017-09-30 
Amazon Kinesis Video Streams Media    KVM         2017-09-30 
Amazon Lex                            LEX         2016-11-28 
Amazon Lex Model Building Service     LMB         2017-04-19 
Amazon Lightsail                      LS          2016-11-28 
Amazon MQ                             MQ          2017-11-27 
Amazon MTurk Service                  MTR         2017-01-17 
Amazon Machine Learning               ML          2014-12-12 
Amazon Pinpoint                       PIN         2016-12-01 
Amazon Polly                          POL         2016-06-10 
Amazon Redshift                       RS          2012-12-01 
Amazon Rekognition                    REK         2016-06-27 
Amazon Relational Database Service    RDS         2014-10-31 
Amazon Route 53                       R53         2013-04-01 
Amazon Route 53 Domains               R53D        2014-05-15 
Amazon SageMaker Runtime              SMR         2017-05-13 
Amazon SageMaker Service              SM          2017-07-24 
Amazon Server Migration Service       SMS         2016-10-24 
Amazon Simple Email Service           SES         2010-12-01 
Amazon Simple Notification Service    SNS         2010-03-31 
Amazon Simple Queue Service           SQS         2012-11-05 
Amazon Simple Storage Service         S3          2006-03-01 
Amazon Simple Systems Management      SSM         2014-11-06 
Amazon Step Functions                 SFN         2016-11-23 
Amazon Translate                      TRN         2017-07-01 
Amazon WorkDocs                       WD          2016-05-01 
Amazon WorkSpaces                     WKS         2015-04-08 
Application Auto Scaling              AAS         2016-02-06 
Application Discovery Service         ADS         2015-11-01 
Auto Scaling                          AS          2011-01-01 
Elastic Load Balancing                ELB         2012-06-01 
Elastic Load Balancing V2             ELB2        2015-12-01 

PS C:\WINDOWS\system32> 

So this means we have the AWS module working. but we want to be able to get this working but we want to set it in a way where it won't ask for credentials so that we can use provisioning tools like vagrant, ansible or chef.
Now we have to create a profile and set aws access to that so that we can perform activities without credentials.

$accessKey = 'ADFASGALKJCUIERHL' # fake example
$secretKey = 'SADFDSGDFH5d6+1P2pmIGW8fdkekdfsneujK14u' # fake example
$profileName = 'myProfileName' # fake example
Set-AWSCredential -AccessKey $accessKey -SecretKey $secretKey -StoreAs $profileName

So now the profile has our credentials stored. Let us now set a default region as Asia Pacific (Singapore)

Initialize-AWSDefaultConfiguration -ProfileName $profileName -Region ap-southeast-1
So we are all done. We can provision stuff from from AWS powershell with ease.

PS C:\WINDOWS\system32> Get-AWSRegion


Region         Name                      IsShellDefault
------         ----                      --------------
ap-northeast-1 Asia Pacific (Tokyo)      False         
ap-northeast-2 Asia Pacific (Seoul)      False         
ap-south-1     Asia Pacific (Mumbai)     False         
ap-southeast-1 Asia Pacific (Singapore)  True          
ap-southeast-2 Asia Pacific (Sydney)     False         
ca-central-1   Canada (Central)          False         
eu-central-1   EU Central (Frankfurt)    False         
eu-west-1      EU West (Ireland)         False         
eu-west-2      EU West (London)          False         
sa-east-1      South America (Sao Paulo) False         
us-east-1      US East (Virginia)        False         
us-east-2      US East (Ohio)            False         
us-west-1      US West (N. California)   False         
us-west-2      US West (Oregon)          False    
As you can see currently I have sen my default region as singapore.
Note: Powershell 6 somehow isn't playing well yet. May be in future it will. Everything howerver worked great with powershell 5.