Azure ARC for Kubernetes Series

Customer Overview

Service Center with multiple stores would like to deploy, configure and manage their multiple K8s containerized applications spread across geography.

Business Requirement

• Struggle to control and govern increasingly complex environments. These environments extend across data centers, multiple clouds, and edge. Each environment and cloud possesses its own set of disjointed management tools that you need to learn and operate.

Azure Arc simplifies governance and management by delivering a consistent multi-cloud and on-premises management platform. Azure Arc enables you to manage your entire environment, with a single pane of glass, by projecting your existing resources into Azure Resource Manager.

azure-arc-control-plane.png (1587×543) (microsoft.com)

architectural-overview.png (2246×1074) (microsoft.com)

Azure Arc for K8s

Below are series of posts to solve business requirement for the customer.

1. (Optional) Deploy EKS Cluster
2. Connect multiple environments
   • Onboard EKS Cluster to Azure Arc
3. Integrate DevOps and Safe Deployment Practices for applications running in stores
   • Deploy GitOps configurations on EKS
4. Monitor the state of applications
5. Apply and monitor at scale governance

Copy secrets between Key Vaults

I had a requirement to setup POC environment for application migration to the cloud. When I looked at the number of secrets in the source POC KV setup I knew I cannot do this manually. So just used a couple of PowerShell commands to move the secrets between the Key Vault.

Pre-requisites:

• Azure PowerShell module installed
• Access to Azure subscription
• Access rights on Key Vault
• Network access to Key Vault (In my case it was a private environment, so needed to make sure VPN setup in place to connect to Key Vault)

# Login to Azure

Connect-AzAccount

# Assign Key Vault  Names

$sourceVault='source-kv-01'
$destVault='target-kv-01'

# Get Source Key Vault Names

$sourceSecretList = (Get-AzKeyVaultSecret -VaultName $sourceVault).Name

# Create secret in target
$sourceSecretList.foreach{
Set-AzKeyVaultSecret -VaultName $destVault -Name $_ `
-SecretValue (Get-AzKeyVaultSecret -VaultName $sourceVault -Name $_).SecretValue
}

Troubleshoot MySQL connectivity from AKS pods

We were stuck in an issue to bring up a Java application dependent on MySQL DB. The application was running on AKS Pod and trying to connect to Azure Database for MySQL. This was a tricky set up as everything was within the virtual Network and public access denied. 

The team thought its connectivity issue, but to isolate that we did 2 quick tests.

• Spin up a Busybox Pod to do nslookup.
  As we were using Private endpoints to connect to DB, it was important to check if name resolution is working.

kubectl run busybox --image=busybox:1.28 --rm -it -- nslookup <MySQL-SererName>.mysql.database.azure.com

• Bring up MySQL client Pod to isolate authentication/authorization issues. Send test queries to MySQL by running a temporary container with the mysql:5.7 image and running the MySQL client binary.

Simple query to check the status.

kubectl run mysql-client --image=mysql:5.7 -i --rm --restart=Never --\

mysql -h <MySQL-SererName>.mysql.database.azure.com -u <Username>@<MySQL-SererName> -p<password> <<EOF
status;
EOF

PS: It is not recommended to pass credentials in the command. Will write a follow up blog using K8s secrets

Certified Kubernetes Administrator (CKA) – Prep

Practice. Practice. Practice.

I have tried to keep the info limited to what worked for me, as we already have enough guidance on the internet for CKA.

I think the key is Practice because based on my experience the exam is challenging but I didn’t find the exam really difficult. In fact, I attempted the questions I knew in 1.5 hours and spent remaining 30 minutes on review and solving Network Policy and Troubleshooting related questions.

Prior experience:

I am new to OSS world, started working on Linux and Kubernetes Instance in last 1 year. You can read about my kickstart to K8s journey here.

My Prep Material:

• Mumshad CKA Course
• Mumshad Kubernetes the hard way Repo and Tutorial (Did this setup on my laptop and used it as my playground for 2 weeks, deploy the cluster, break the cluster, fix the cluster, and again.. )
• Walid Shaari CKA Repo

Tips:

• We got 17 questions in 2 hours, quickly go through all the questions. This is to finish the easy once first. Trust me this helps you to gain confidence that you already passed and now aiming for the dream century (100%)
• Imperative commands are the key to save on time
• kubernetes.io/docs was my only friend present during the exam, know it well before the exam. I bookmarked the pages I needed to save seconds on searching.
• For etcd backup and restore -h (help) in the key. (Practice before the exam)
• Not required but if you are planning to use aliases, make sure you practice them before so your brain is trained to use them. I did set below aliases but hardly used few.
  • Used Kubectl Cheat Sheet for these.
    • source <(kubectl completion bash)
    • echo “source <(kubectl completion bash)” >> ~/.bashrc
    • alias k=kubectl
    • complete -F __start_kubectl k
  • Typed these.
    • alias kgp=’k get pods’
    • alias kgs=’k get service’
    • alias kaf=’k apply -f’
    • alias kgd=’k get deployments’
  • Tweak Vim
    • vim ~/.vimrc
      set nu
      set expandtab
      set shiftwidth=2
      set tabstop=2
• Last but not least, always validate the output. Describe Pods, deployments, services etc to make sure it’s working as expected

What could have gone better:

• More Practice, its never enough
• Practice Network Policy more. So you please don’t miss it and follow this amazing repo. Try to implement all the scenarios in your cluster to get a better understanding.
• More troubleshooting practice to understand the kind of issues kubelet and kube proxy can have.
• Practice with systemctl, journalctl and openssl

And as we have 2 attempts, I would suggest rather fail fast than waiting till you are ready for the exam. Best of luck.