Azure Custom Roles

Working in a secure customer environment, we had requirements of custom roles for Private AKS Cluster Deployment.

  • Create Role Definition JSON as per the requirement
{
  "Name": "Network Reader",
  "IsCustom": true,
  "Description": "Can read Network Properties.",
  "Actions": [
    "Microsoft.Network/virtualNetworks/subnets/read"
  ],
  "NotActions": [],
  "AssignableScopes": [
  "/subscriptions/<subscripion Id>"
  ]
}
  • Deploy Role Definition to azure
az role definition create --role-definition ~/roles/vmoperator.json
Create Role Definition
  • Update Role definition if required
{
  "Name": "Network Reader",
  "IsCustom": true,
  "Description": "Can read and join Network",
  "Actions": [
    "Microsoft.Network/virtualNetworks/subnets/read",
	"Microsoft.Network/virtualNetworks/subnets/join/action"
  ],
  "NotActions": [],
  "AssignableScopes": [
  "/subscriptions/d3819925-7e44-4f5f-8733-1067beaa45ec"
  ]
}
  • Deploy updated Role definition
az role definition update --role-definition amlnetwork.json
Update Role Definition
  • Assign Role Definition
az role assignment create --assignee <client id> --scope "<resourceid>" --role "Network Reader"
Assign Role

AKS Custom Private DNS Zone

We got the below errors when trying to use a pre-created Private DNS Zone for the AKS cluster.

  • Pre-created below resources before deploying AKS
    • vNet and subnets
    • User Assigned Managed identity
    • Private DNS zone is created
  • Deploying Private AKS cluster using Terraform

Issue:

  1. Error: creating Managed Kubernetes Cluster “<aks cluster name>” (Resource Group “<rg name>”): containerservice.ManagedClustersClient#CreateOrUpdate: Failure sending request: StatusCode=400 — Original Error: Code=”CustomPrivateDNSZoneMissingPermissionError” Message=”Service principal or user assigned identity must be given permission to read and write to custom private dns zone /subscriptions/<subid>/resourceGroups/<rg name>/providers/Microsoft.Network/privateDnsZones/privatelink.eastus2.azmk8s.io. Check access result not allowed for action Microsoft.Network/privateDnsZones/read.
  1. Error: waiting for creation of Managed Kubernetes Cluster “<clustername>” (Resource Group “<rgname>”): Code=”ReconcilePrivateDNS” Message=”Reconcile private dns failed. Details: Code=\”LinkedAuthorizationFailed\” Message=\”The client ‘e7fb6da5-7811-4897-a819-c0e442df5fec’ with object id ‘e7fb6da5-7811-4897-a819-c0e442df5fec’ has permission to perform action ‘Microsoft.Network/privateDnsZones/virtualNetworkLinks/write’ on scope ‘/subscriptions/<subscriptionid>/resourceGroups/<rgname>/providers/Microsoft.Network/privateDnsZones/privatelink.eastus2.azmk8s.io/virtualNetworkLinks/<aksclustername>’; however, it does not have permission to perform action ‘Microsoft.Network/virtualNetworks/join/action’ on the linked scope(s) ‘/subscriptions/<subscriptionid>/resourceGroups/<rgname>/providers/Microsoft.Network/virtualNetworks/<vnet name>’ or the linked scope(s) are invalid.\””

Solution:

  • For issue 1: we need to provide Private DNS Contributor role for the AKS user identity to the Private DNS zone.
az role assignment create --assignee <aks user managed identity> --scope "/subscriptions/<subscription id>/resourceGroups/<rg name>/providers/Microsoft.Network/privateDnsZones/privatelink.eastus2.azmk8s.io" --role "private dns zone contributor"

NGINX Ingress Timeout error

Getting error when creating ingress resource

Ingress Error

Need to wait before creating ingress resource. Added this block after NGINX Ingress deployment

# Checking if  Ingress Controller is ready
echo "Waiting for Ingress Controller to be ready"
kubectl wait --namespace $namespace --for=condition=ready pod --selector=app.kubernetes.io/component=controller --timeout=120s

Reference:
https://github.com/kubernetes-sigs/kind/blob/main/site/content/docs/user/ingress.md