|
| 1 | +## Introduction |
| 2 | +This document provides a step-by-step guide to create an Azure Kubernetes Service (AKS) cluster, integrate it with Azure Container Registry (ACR), and set up Azure Key Vault and Nginx ingress with a CA certificate. |
| 3 | + |
| 4 | +## Prerequisites |
| 5 | +- Azure CLI installed |
| 6 | +- Azure subscription |
| 7 | +- Sufficient permissions to create resources in the Azure subscription: |
| 8 | + - **Contributor** - Creates resources and all other Azure resources. |
| 9 | + - **User Access Administrator** - Assign necessary roles. |
| 10 | + |
| 11 | +## Outputs |
| 12 | +- Azure Container Registry (ACR) |
| 13 | +- Azure Kubernetes Service (AKS) connected to ACR |
| 14 | +- Azure Key Vault |
| 15 | +- Nginx ingress with CA certificate |
| 16 | + |
| 17 | +## Steps |
| 18 | + |
| 19 | +### 1. Clone the repo |
| 20 | +Clone the git repo and go to the working folder. |
| 21 | +```bash |
| 22 | +cd acme-fitness-store/azure-kubernetes-service |
| 23 | +``` |
| 24 | + |
| 25 | +### 2. Set Variables |
| 26 | + |
| 27 | +Update `resources/var.sh` and set up the variables for your environment. |
| 28 | +``` |
| 29 | +source resources/var.sh |
| 30 | +az account set -s ${SUBSCRIPTION} |
| 31 | +
|
| 32 | +echo "RESOURCE_GROUP=${RESOURCE_GROUP}" |
| 33 | +echo "AKS_NAME=${AKS_NAME}" |
| 34 | +echo "ACR_NAME=${ACR_NAME}" |
| 35 | +echo "KEYVAULT_NAME=${KEYVAULT_NAME}" |
| 36 | +echo "WORKSPACE_NAME=${WORKSPACE_NAME}" |
| 37 | +``` |
| 38 | + |
| 39 | +### 3. Create Resource Group |
| 40 | +1. Create a resource group to host all the Azure resources. |
| 41 | +```bash |
| 42 | +az group create -n ${RESOURCE_GROUP} -l eastus2 |
| 43 | +``` |
| 44 | + |
| 45 | +### 4. Create Azure Container Registry |
| 46 | +Create Azure Container Registry (ACR). This ACR will be used to: |
| 47 | +- Build application components |
| 48 | +- Store application images built by buildpack |
| 49 | + |
| 50 | +```bash |
| 51 | +az acr create -g ${RESOURCE_GROUP} -n ${ACR_NAME} --sku Premium |
| 52 | +``` |
| 53 | + |
| 54 | +### 5. Create AKS |
| 55 | +1. Enable `EncryptionAtHost`, may take 10+ minutes to finish |
| 56 | +```bash |
| 57 | +az feature register --namespace Microsoft.Compute --name EncryptionAtHost |
| 58 | +``` |
| 59 | + |
| 60 | +Run `az feature register --namespace Microsoft.Compute --name EncryptionAtHost` to wait for its state to be `Registered`. |
| 61 | + |
| 62 | +1. Create workspace |
| 63 | +``` |
| 64 | +az monitor log-analytics workspace create --resource-group ${RESOURCE_GROUP} --workspace-name ${WORKSPACE_NAME} |
| 65 | +``` |
| 66 | + |
| 67 | +1. Create AKS. |
| 68 | + Below commands guide you to create the AKS. For more information on the features enabled in the AKS cluster, refer to the following documentation: |
| 69 | + |
| 70 | + - [Attach Azure Container Registry to AKS](https://learn.microsoft.com/en-us/azure/aks/cluster-container-registry-integration) |
| 71 | + - [Enable Workload Identity](https://learn.microsoft.com/en-us/azure/aks/workload-identity-overview) |
| 72 | + - [Azure Load Balancer SKU](https://learn.microsoft.com/en-us/azure/load-balancer/skus) |
| 73 | + - [Cluster Autoscaler](https://learn.microsoft.com/en-us/azure/aks/cluster-autoscaler) |
| 74 | + - [Network concepts](https://learn.microsoft.com/en-us/azure/aks/concepts-network) |
| 75 | + - [Encryption at Host](https://learn.microsoft.com/en-us/azure/aks/enable-host-encryption) |
| 76 | + - [Outbound Type](https://learn.microsoft.com/en-us/azure/aks/egress-outboundtype) |
| 77 | + - [Node pools](https://learn.microsoft.com/en-us/azure/aks/create-node-pools) |
| 78 | + - [Storage concepts](https://learn.microsoft.com/en-us/azure/aks/concepts-storage) |
| 79 | + - [Monitor AKS](https://learn.microsoft.com/en-us/azure/aks/monitor-aks) |
| 80 | + |
| 81 | + |
| 82 | + ``` |
| 83 | + WORKSPACE_ID=$(az monitor log-analytics workspace show --resource-group ${RESOURCE_GROUP} --workspace-name ${WORKSPACE_NAME} --query id -o tsv) |
| 84 | + ``` |
| 85 | +
|
| 86 | + ``` |
| 87 | + az aks create \ |
| 88 | + -g ${RESOURCE_GROUP} \ |
| 89 | + -n ${AKS_NAME} \ |
| 90 | + --attach-acr ${ACR_NAME} \ |
| 91 | + --enable-workload-identity \ |
| 92 | + --load-balancer-sku standard \ |
| 93 | + --enable-cluster-autoscaler \ |
| 94 | + --max-count 40 \ |
| 95 | + --min-count 1 \ |
| 96 | + --network-plugin azure \ |
| 97 | + --no-ssh-key \ |
| 98 | + --enable-encryption-at-host \ |
| 99 | + --outbound-type loadBalancer \ |
| 100 | + --enable-oidc-issuer \ |
| 101 | + --enable-aad \ |
| 102 | + --vm-set-type VirtualMachineScaleSets \ |
| 103 | + --os-sku Mariner \ |
| 104 | + --node-osdisk-size 100 \ |
| 105 | + --node-osdisk-type Ephemeral \ |
| 106 | + --node-vm-size Standard_D4as_v4 \ |
| 107 | + --enable-azure-monitor-metrics \ |
| 108 | + --enable-addons monitoring \ |
| 109 | + --workspace-resource-id ${WORKSPACE_ID} |
| 110 | + ``` |
| 111 | +
|
| 112 | + > Note: After creating the AKS, it may take some time to update. During this time, the following commands will fail. |
| 113 | +
|
| 114 | + ``` |
| 115 | + az aks nodepool add \ |
| 116 | + --cluster-name ${AKS_NAME} \ |
| 117 | + -g ${RESOURCE_GROUP} \ |
| 118 | + -n nodepool2 \ |
| 119 | + --enable-cluster-autoscaler \ |
| 120 | + --enable-encryption-at-host \ |
| 121 | + --max-count 40 \ |
| 122 | + --min-count 1 \ |
| 123 | + --node-osdisk-size 200 \ |
| 124 | + --node-osdisk-type Ephemeral \ |
| 125 | + --node-vm-size Standard_D8as_v4 \ |
| 126 | + --os-sku Mariner \ |
| 127 | + --os-type Linux \ |
| 128 | + --node-count 1 |
| 129 | + az aks nodepool add \ |
| 130 | + --cluster-name ${AKS_NAME} \ |
| 131 | + -g ${RESOURCE_GROUP} \ |
| 132 | + -n nodepool3 \ |
| 133 | + --enable-cluster-autoscaler \ |
| 134 | + --enable-encryption-at-host \ |
| 135 | + --max-count 40 \ |
| 136 | + --min-count 1 \ |
| 137 | + --node-osdisk-size 200 \ |
| 138 | + --node-osdisk-type Ephemeral \ |
| 139 | + --node-vm-size Standard_D16as_v4 \ |
| 140 | + --os-sku Mariner \ |
| 141 | + --os-type Linux \ |
| 142 | + --node-count 1 |
| 143 | + ``` |
| 144 | +
|
| 145 | +1. Retrieve access token. This command gets the admin access for the AKS cluster. For more access management, see https://learn.microsoft.com/en-us/azure/aks/azure-ad-rbac?tabs=portal |
| 146 | +
|
| 147 | + ``` |
| 148 | + az aks get-credentials --resource-group ${RESOURCE_GROUP} --name ${AKS_NAME} --overwrite-existing --admin |
| 149 | + ``` |
| 150 | +
|
| 151 | +1. Install or update the kubectl CLI |
| 152 | + ``` |
| 153 | + az aks install-cli |
| 154 | + ``` |
| 155 | +
|
| 156 | +1. Verify you can connect to the AKS |
| 157 | +
|
| 158 | + ``` |
| 159 | + kubectl get ns |
| 160 | + ``` |
| 161 | +
|
| 162 | +### 6. Create Azure Keyvault and cert |
| 163 | +
|
| 164 | +1. Get AKS outbound IPs and record these IPs as `<AKS-outbound-ip>` |
| 165 | + ``` |
| 166 | + az aks show -g ${RESOURCE_GROUP} -n ${AKS_NAME} --query networkProfile.loadBalancerProfile.effectiveOutboundIPs[].id |
| 167 | + az resource show --ids <the ID from previous output> --query properties.ipAddress -o tsv |
| 168 | + ``` |
| 169 | +
|
| 170 | +1. Get AKS Vnet IDs |
| 171 | + ``` |
| 172 | + NODE_RESOURCE_GROUP=$(az aks show --resource-group ${RESOURCE_GROUP} --name ${AKS_NAME} --query nodeResourceGroup -o tsv) |
| 173 | + az resource list --resource-type microsoft.network/virtualnetworks -g ${NODE_RESOURCE_GROUP} --query "[?starts_with(name, 'aks-vnet')].name" -o tsv |
| 174 | + ``` |
| 175 | +
|
| 176 | + List all subnets under the vnet, record these ids as `<subnet-ids>` |
| 177 | + ``` |
| 178 | + az network vnet subnet list --resource-group ${NODE_RESOURCE_GROUP} --vnet-name <vnetName> --query "[].id" -o tsv |
| 179 | + ``` |
| 180 | +
|
| 181 | +1. Create Azure KeyVault |
| 182 | +`az keyvault create --resource-group ${RESOURCE_GROUP} --name ${KEYVAULT_NAME} --network-acls-ips <AKS-outbound-ip> --network-acls-vnets <subnet-ids>` |
| 183 | +
|
| 184 | +1. Assign access to yourself |
| 185 | + ``` |
| 186 | + # Get your Azure AD user ID |
| 187 | + USER_ID=$(az ad signed-in-user show --query id --output tsv) |
| 188 | + KEYVUALT_ID=$(az keyvault show --name ${KEYVAULT_NAME} --query id --output tsv) |
| 189 | + # Assign yourself the necessary permissions |
| 190 | + az role assignment create --role "Key Vault Certificates Officer" --assignee ${USER_ID} --scope ${KEYVUALT_ID} |
| 191 | + ``` |
| 192 | +
|
| 193 | +1. Create a self-signed certificate or import your CA cert to the Keyvault, ref: https://learn.microsoft.com/en-us/azure/key-vault/certificates/tutorial-import-certificate?tabs=azure-portal |
| 194 | + > Here suggest to create a wildcard domain cert, like `*.demo.com`. |
| 195 | +
|
| 196 | +### 7. Enable Nginx in Kubernetes |
| 197 | +Below steps guide how to enable the Nginx as add-on in the AKS cluster. For more details can view [Managed NGINX ingress with the application routing add-on](https://learn.microsoft.com/en-us/azure/aks/app-routing). |
| 198 | +
|
| 199 | +1. Enable Nginx |
| 200 | + ``` |
| 201 | + az extension add -n aks-preview --upgrade |
| 202 | + az aks approuting enable --resource-group ${RESOURCE_GROUP} --name ${AKS_NAME} |
| 203 | +
|
| 204 | + KEYVUALT_ID=$(az keyvault show --name ${KEYVAULT_NAME} --query id --output tsv) |
| 205 | + az aks approuting update --resource-group ${RESOURCE_GROUP} --name ${AKS_NAME} --nginx External --enable-kv --attach-kv ${KEYVUALT_ID} |
| 206 | + ``` |
| 207 | +
|
| 208 | +1. Retrieve the Nginx public IP and note the "EXTERNAL-IP" |
| 209 | +
|
| 210 | + ``` |
| 211 | + kubectl get svc nginx -n app-routing-system |
| 212 | + ``` |
| 213 | +
|
| 214 | +1. Go to your DNS zone to add record. |
| 215 | + - Add A record to point the domain in your TLS cert to the external IP you obtained. E.g. `demo.com` points to the IP address. |
| 216 | + - Add a wildcard CName record to the A record. E.g. `*.demo.com` points to the `demo.com` |
| 217 | +
|
| 218 | +## Next Steps |
| 219 | +
|
| 220 | +- Follow [02-create-eureka-server](./02-create-eureka-server.md) to create and deploy a Eureka Server on Azure Kubernetes Service. |
0 commit comments