update

Author: pichuang

README.md

@@ -1,7 +1,7 @@
 # Kubernetes deployment strategies (Azure Edition)
 
 > In Kubernetes there are a few different ways to release an application, you have
-to carefully choose the right strategy to make your infrastructure resilient.
+to carefully choose the right strategy to make Microsoft Azure infrastructure resilient.
 
 - [ ] [recreate](recreate/): terminate the old version and release the new one
   - [x] Application Gateway Ingress Controller
@@ -25,6 +25,9 @@ to carefully choose the right strategy to make your infrastructure resilient.
   response.
   - [x] Azure Load Balancer + Istio service mesh
 
+
+## Deployment strategy Decision Diagram
+
 ![deployment strategy decision diagram](decision-diagram.png)
 
 Before experimenting, checkout the following resources:
@@ -36,17 +39,28 @@ Before experimenting, checkout the following resources:
 - [Canary deployment using Istio and Helm](https://github.com/etiennetremel/istio-cross-namespace-canary-release-demo)
 - [Automated rollback of Helm releases based on logs or metrics](https://container-solutions.com/automated-rollback-helm-releases-based-logs-metrics/)
 
+## Azure Kubernetes Service (AKS) Deployment Strategy Support Matrix
+
+|   Strategy  |      Use AGIC     | Use Istio Service Mesh? |                               Note                               |
+|:-----------:|:-----------------:|:-----------------------:|:----------------------------------------------------------------:|
+| Recreate    |        Yes        |           Yes           | Regardless of whether the Ingress Controller is selected or not. |
+| Ramped      |        Yes        |           Yes           | Regardless of whether the Ingress Controller is selected or not. |
+| Blue/Green  |        Yes        |           Yes           | Regardless of whether the Ingress Controller is selected or not. |
+| Canary      | Yes, but manually |           Yes           | Ingress controller is required to work in conjunction.           |
+| A/B Testing |         No        |           Yes           | Ingress controller is required to work in conjunction.           |
+| Shadow      |         No        |           Yes           | Ingress controller is required to work in conjunction.           |
+
 ## Getting started
 
+### 1. Deploy Azure Kubernetes Service and other resources
+
 These examples were created and tested on
 
 - Azure Kubernetes Service v1.26.3
-- Azure Service Mesh (Istio Service Mesh) v1.17
-- Azure Application Gateway Ingress Controller (AGIC) Standard v2
-- Azure Monitor managed service for Prometheus
-- Azure Monitor managed service for Grafana v9.4.10 (5e7d575327)
-
-## Deploy Azure Kubernetes Service and other resources
+- [Azure Service Mesh (a.k.a Istio Service Mesh)][2] v1.17
+- [Azure Application Gateway Ingress Controller (AGIC)][3] Standard v2
+- [Azure Monitor managed service for Prometheus][1]
+- [Azure Managed Grafana][4] v9.4.10 (5e7d575327)
 
 ```bash
 $ cd ./deploy
@@ -55,37 +69,44 @@ $ kubectl apply -f ama-metrics-prometheus-config.yml
 $ kubectl apply -f ama-metrics-settings-configmap.yml
 ```
 
-## Import Grafana dashboard in Azure Managed Grafana
+### 2. Import existing Grafana dashboard in Azure Managed Grafana
 
 ![](./images/azure-managed-grafana.png)
 
-Create a dashboard with a Time series or import the [JSON export](grafana-dashboard.json). Use the following query:
+Create a dashboard with a Time series or import the [JSON export](grafana-dashboard.json).
 
-```
+Use the following query:
+
+```yaml
 sum(rate(http_requests_total{app="my-app"}[2m])) by (version)
 ```
 
 ![](./images/prometheus-query.png)
 
-Since we installed Prometheus with default settings, it is using the default scrape
-interval of `1m` so the range cannot be lower than that.
+Since we installed [Azure Managed Prometheus][1] with cutomized settings, it is using the short scrape
+interval of `10s` so the range cannot be lower than that.
 
 To have a better overview of the version, add `{{version}}` in the legend field.
 
-## Curl script
+## Test script
+
+### Usage
 
-### Before you begin
+This is a Python script that makes HTTP requests to a web link specified by the provided `AGIC-PUBLIC-IP` address. The script uses the requests library to send GET requests to the specified URL. It has error handling mechanisms to handle different types of exceptions that might occur during the request.
 
 ```bash
+# Install colorama for colorized output if not already installed
 pip3 install colorama
-```
-
-### Usage
 
-This is a Python script that makes HTTP requests to a web link specified by the provided AIGC-PUBLIC-IP address. The script uses the requests library to send GET requests to the specified URL. It has error handling mechanisms to handle different types of exceptions that might occur during the request.
+# Run the script
+# Example:
+# ./curl.py x.x.x.x
+./curl.py $AGIC-PUBLIC-IP
 
-```bash
-./curl.py AIGC-PUBLIC-IP
+# Run the script with a custom header
+# Example:
+# ./curl.py x.x.x.x test.aks.aliez.tw
+./curl.py $AGIC-PUBLIC-IP $HEADER_HOST
 ```
 
 The script continues to run indefinitely, making periodic requests to the web link and monitoring for errors.
@@ -127,4 +148,21 @@ Based on [Troubleshoot collection of Prometheus metrics in Azure Monitor](https:
 kubectl port-forward ama-metrics-* -n kube-system 9090
 ```
 
-![Port Forward prometheus](./images/port-forward-prometheus.png)
+![Port Forward prometheus](./images/port-forward-prometheus.png)
+
+
+### Minimizing Downtime During Deployments with Azure Application Gateway Ingress Controller
+
+Strong recommendation to read [Minimizing Downtime During Deployments](https://azure.github.io/application-gateway-kubernetes-ingress/how-tos/minimize-downtime-during-deployments/)
+
+1. Regarding the `spec.terminationGracePeriodSeconds` parameter, please refer to [ramped/app-v1.yaml#30][5]
+2. Regarding the `spec.containers[0].lifecycle.preStop` parameter, please refer to [ramped/app-v1.yaml#L53-L56][6]
+3. Add connection draining annotation to the Ingress read by AGIC to allow for in-flight connections to complete, please refer to [ramped/app-v1.yaml#L65-L66][7]
+
+[1]: https://learn.microsoft.com/en-Us/azure/azure-monitor/essentials/prometheus-metrics-overview
+[2]: https://learn.microsoft.com/en-us/azure/aks/istio-about
+[3]: https://learn.microsoft.com/en-us/azure/application-gateway/ingress-controller-overview
+[4]: https://learn.microsoft.com/en-us/azure/managed-grafana/
+[5]: https://github.com/pichuang/k8s-deployment-strategies-azure-edition/blob/master/ramped/app-v1.yaml#L30
+[6]: https://github.com/pichuang/k8s-deployment-strategies-azure-edition/blob/master/ramped/app-v1.yaml#L53-L56
+[7]: https://github.com/pichuang/k8s-deployment-strategies-azure-edition/blob/master/ramped/app-v1.yaml#L65-L66

ab-testing/README.md

@@ -1,4 +1,4 @@
-# A/B testing using Istio
+# A/B testing using Istio service mesh
 
 > Version B is released to a subset of users under specific condition.
 
@@ -16,20 +16,19 @@ roll-out the version that converts the most.
 Here is a list of conditions that can be used to distribute traffic amongst the
 versions:
 
-- Weight
-- Cookie value
+- [Weight](https://istio.io/latest/docs/reference/config/networking/virtual-service/#RouteDestination)
+- [Headers](https://istio.io/latest/docs/reference/config/networking/virtual-service/#Headers)
 - Query parameters
 - Geolocalisation
 - Technology support: browser version, screen size, operating system, etc.
 - Language
 
 ## Steps to follow
 
-1. version 1 is serving HTTP traffic using Istio
-1. deploy version 2
-1. wait until all instances are ready
-1. update Istio VirtualService with 90% traffic targetting version 1 and 10%
-   traffic targetting version 2
+1. version 1 is serving HTTP traffic using Istio service mesh
+2. deploy version 2
+3. wait until all instances are ready
+4. update Istio VirtualService with 80% traffic targetting version 1 and 20% traffic targetting version 2
 
 ## In practice
 
@@ -54,15 +53,11 @@ To automatically install sidecar to any new pods, annotate your namespaces:
 
 ```yaml
 # The default istio-injection=enabled labeling doesn't work. Explicit versioning (istio.io/rev=asm-1-17) is required.
-kubectl label namespace/ns-canary-istio istio.io/rev=asm-1-17
+kubectl label namespace/ns-ab-testing istio.io/rev=asm-1-17
 ```
 
 ### Deploy both applications
 
-Back to the a/b testing directory from this repo, deploy both applications using
-the istioctl command to inject the Istio sidecar container which is used to
-proxy requests:
-
 ```bash
 $ kubectl apply -f app-v1.yaml -f app-v2.yaml
 namespace/ns-ab-testing created
@@ -73,8 +68,7 @@ deployment.apps/deployment-my-app-v2 created
 service/my-app-v2 created
 ```
 
-Expose both services via the Istio Gateway and create a VirtualService to match
-requests to the my-app-v1 service:
+Expose both services via the Istio Gateway and create a VirtualService to match requests to the my-app-v1 service:
 
 ```bash
 $ kubectl apply -f ./gateway.yaml -f ./virtualservice-wildcard.yaml

blue-green/multiple-services/README.md

@@ -12,14 +12,14 @@ Ingress controller, this example would also work with the
 ## Steps to follow
 
 1. service a and b are serving traffic
-1. deploy new version of both services
-1. wait for all services to be ready
-1. switch incoming traffic from version 1 to version 2
-1. shutdown version 1
+2. deploy new version of both services
+3. wait for all services to be ready
+4. switch incoming traffic from version 1 to version 2
+5. shutdown version 1
 
 ## In practice
 
-```
+```bash
 # Deploy version 1 of application a and b and the ingress
 $ kubectl apply -f app-a-v1.yaml -f app-b-v1.yaml -f ingress-v1.yaml
 namespace/ns-bluegreen created

blue-green/single-service/README.md

@@ -1,16 +1,15 @@
-Blue/green deployment to release a single service
-=================================================
+# Blue/green deployment to release a single service
 
 > In this example, we release a new version of a single service using the
 blue/green deployment strategy.
 
 ## Steps to follow
 
 1. version 1 is serving traffic
-1. deploy version 2
-1. wait until version 2 is ready
-1. switch incoming traffic from version 1 to version 2
-1. shutdown version 1
+2. deploy version 2
+3. wait until version 2 is ready
+4. switch incoming traffic from version 1 to version 2
+5. shutdown version 1
 
 ## In practice
 
@@ -23,8 +22,7 @@ ingress.networking.k8s.io/ingress-bluegreen created
 service/svc-my-app created
 
 # Test if the deployment was successful
-$ export AGIC-PUBLIC-IP="apgw.aks.aliez.tw"
-$ curl.py $AGIC-PUBLIC-IP
+$ ./curl.py apgw.aks.aliez.tw
 Host: deployment-my-app-v1-657c65694c-jg4hn, Version: v1.0.0
 Host: deployment-my-app-v1-657c65694c-s77mq, Version: v1.0.0
 Host: deployment-my-app-v1-657c65694c-62w74, Version: v1.0.0
@@ -35,7 +33,7 @@ Host: deployment-my-app-v1-657c65694c-jg4hn, Version: v1.0.0
 
 # To see the deployment in action, open a new terminal and run the following
 command:
-$ watch kubectl get po
+$ watch kubectl get pod -n ns-bluegreen
 NAME                                    READY   STATUS    RESTARTS   AGE
 deployment-my-app-v1-657c65694c-62w74   1/1     Running   0          2m27s
 deployment-my-app-v1-657c65694c-jg4hn   1/1     Running   0          2m27s
@@ -84,6 +82,10 @@ service/svc-my-app patched
 $ kubectl delete deployment/deployment-my-app-v1 -n ns-bluegreen
 ```
 
+### Screen Shot
+
+![ScreenShot Bluegreen](screenshot-bluegreen.png)
+
 ### Cleanup
 
 ```bash

blue-green/single-service/screenshot-bluegreen.png

@@ -1,5 +1,4 @@
-Canary deployment
-=================
+# Canary deployment
 
 > Version B is released to a subset of users, then proceed to a full rollout.
 
@@ -17,13 +16,10 @@ platform.
 adjusting the number of replicas or if you use Nginx as Ingress controller you
 can define fine grained traffic splitting via Ingress annotations.**
 
-- [NodePort](NodePort/)
-- [nginx-ingress](nginx-ingress/)
+- [Application Gateway Ingress Controller](application-gateway-ingress-controller/)
+- [Istio service mesh add-on](istio-service-mesh-addon/)
 
-*If you use Helm to deploy applications, the following repository demonstrate
-how to make a [canary deployment using Istio and
-Helm](https://github.com/etiennetremel/istio-cross-namespace-canary-release-demo).*
 
 ## Limitation
 
-- Application Gateway Ingress Controller doesn't support Canary deployment.
+- Application Gateway Ingress Controller doesn't support Traffic Shifting.

canary/README.md

@@ -1,21 +1,22 @@
-Canary deployment using Kubernetes native functionnalities
-==========================================================
+# Canary deployment using Kubernetes native functionnalities and Azure Application Gateway Ingress Controller
 
-> In the following example we apply the poor man's canary using Kubernetes
-native features (replicas). If you want a finer grained control over traffic
-shifting, check the [nginx-ingress](../nginx-ingress) example which use
-[Nginx](http://nginx.org/) to split traffic or the [a/b testing](../../ab-testing)
-example which shift traffic using [Istio](https://istio.io).
+> In the following example we apply the `poor man's` canary using Kubernetes
+native features (replicas) and Azure Application Gateway Ingress Controller.
+
+If you want a finer grained control over traffic shifting, you can use...
+
+- `nginx-ingress ingress controller` which use [Nginx](http://nginx.org/) to split traffic.
+- [`Istio Ingress Gateways`](../istio-service-mesh-addon) which use [Istio](https://istio.io) to split traffic.
+- [a/b testing](../../ab-testing) which shift traffic using [Istio](https://istio.io).
 
 ## Steps to follow
 
 1. 10 replicas of version 1 is serving traffic
-1. deploy 1 replicas version 2 (meaning ~10% of traffic)
-1. wait enought time to confirm that version 2 is stable and not throwing
-   unexpected errors
-1. scale up version 2 replicas to 10
-1. wait until all instances are ready
-1. shutdown version 1
+2. deploy 1 replicas version 2 (meaning ~10% of traffic)
+3. wait enought time to confirm that version 2 is stable and not throwing unexpected errors
+4. scale up version 2 replicas to 10
+5. wait until all instances are ready
+6. shutdown version 1
 
 ## In practice
 
@@ -50,14 +51,13 @@ deployment-my-app-v1-657c65694c-rsckt   1/1     Running   0          68s
 deployment-my-app-v1-657c65694c-vdvd8   1/1     Running   0          68s
 
 # Then deploy version 2 of the application and scale down version 1 to 9 replicas at same time
-version 2 to 1 replicas (meaning ~10% of traffic) + version 1 to 9 replicas (meaning ~90% of traffic)
+# version 2 to 1 replicas (meaning ~10% of traffic) + version 1 to 9 replicas (meaning ~90% of traffic)
 $ kubectl apply -f app-v2.yaml
 deployment.apps/deployment-my-app-v2 created
 
-$ kubectl scale --replicas=9 deployment/deployment-my-app-v1 -n ns-canry
+$ kubectl scale --replicas=9 deployment/deployment-my-app-v1 -n ns-canary
 deployment.apps/deployment-my-app-v1 scaled
 
-
 # Only one pod with the new version should be running.
 # You can test if the second deployment was successful
 $ ./curl.py apgw.aks.aliez.tw
@@ -82,14 +82,14 @@ $ kubectl scale --replicas=5 deployment/deployment-my-app-v1 -n ns-canary
 # version 2 to 10 replicas (meaning 100% of traffic) + version 1 to 0 replicas (meaning ~0% of traffic)
 $ kubectl scale --replicas=10 deployment/deployment-my-app-v2 -n ns-canary
 $ kubectl scale --replicas=0 deployment/deployment-my-app-v1 -n ns-canary
-
-# Then, when all pods are running, you can safely delete the old deployment
-$ kubectl delete deployment/deployment-my-app-v1 -n ns-canary
-$ kubectl delete deployment/deployment-my-app-v2 -n ns-canary
 ```
 
+### Screen Shot
+
+![Screenshot canary-50-50](screenshot-canary-50-50.png)
+
 ### Cleanup
 
 ```bash
-$ kubectl delete all -l app=my-app -n ns-canary
+$ kubectl delete -f .
 ```

canary/application-gateway-ingress-controller/README.md

@@ -1,6 +1,6 @@
 # Canary deployment using Istio service mesh
 
-> In the following example, we will use the Istio service mesh to control
+> In the following example, we will use the Istio service mesh provied by Azure to control
 traffic distribution for a canary deployment using an example application
 This is very similar to the Istio A/B testing example, however instead of
 serving a specific subset of clients based on headers, we are simply dividing
@@ -50,8 +50,6 @@ kubectl label namespace/ns-canary-istio istio.io/rev=asm-1-17
 
 Deploy all of the yaml files in this directory
 
-TODO: Split files and apply/verify seperately
-
 ```bash
 $ kubectl apply -f app-v1.yaml -f app-v2.yaml -f hpa.yaml -f istio.yaml
 namespace/ns-canary-istio configured