In this demonstration, we will deploy Buoyant Enterprise for Linkerd and the Orders application across two k3d clusters deployed in a flat network, and will demonstrate how to deploy the Buoyant Enterprise for Linkerd Multi-Cluster Extension and enable High Availability Zonal Load Balancing (HAZL). We'll then take a look at how HAZL works to keep network traffic in-zone where possible by exploring some different traffic, load and availability situations.
Buoyant Enterprise for Linkerd
Buoyant Enterprise for Linkerd is an enterprise-grade service mesh for Kubernetes. It makes Kubernetes applications reliable, secure, and cost-effective without requiring any changes to application code. Buoyant Enterprise for Linkerd contains all the features of open-source Linkerd, the world's fastest, lightest service mesh, plus additional enterprise-only features such as:
- High-Availability Zonal Load Balancing (HAZL)
- Security Policy Management
- FIPS-140-2/3 Compliance
- Lifecycle Automation
- Buoyant Cloud
- Mesh Expansion
Plus:
- Enterprise-Hardened Images
- Software Bills of Materials (SBOMs)
- Strict SLAs Around CVE Remediation
- 24x7x365 Support With SLAs
- Quarterly Outcomes and Strategy Reviews
We're going to try out Linkerd's Multi-Cluster Expansion and High-Availability Zonal Load Balancing (HAZL) in this Demonstration.
SAY SOMETHING HERE
High Availability Zonal Load Balancing (HAZL) is a dynamic request-level load balancer in Buoyant Enterprise for Linkerd that balances HTTP and gRPC traffic in environments with multiple availability zones. For Kubernetes clusters deployed across multiple zones, HAZL can dramatically reduce cloud spend by minimizing cross-zone traffic.
For more information on HAZL, click here.
In this demonstration, we will deploy Buoyant Enterprise for Linkerd and the Orders application across two k3d clusters deployed in a flat network, and will demonstrate how to enable High Availability Zonal Load Balancing (HAZL). We'll then take a look at how HAZL works to keep network traffic in-zone where possible by exploring some different traffic, load and availability situations.
In this demonstration, we're going to do the following:
- Deploy two
k3dKubernetes clusters that share a common network - Deploy Buoyant Enterprise for Linkerd with HAZL disabled on the clusters
- Deploy the Orders application to the clusters, half on one cluster and half on the other, to generate multi-zonal traffic
- Deploy and configure the Linkerd Multicluster extension
- Monitor traffic from the Orders application, with HAZL disabled
- Enable High Availability Zonal Load Balancing (HAZL)
- Monitor traffic from the Orders application, with HAZL enabled
- Scale traffic in various zones
- Monitor traffic from the Orders application
- Observe effects of HAZL
- Observe interaction with Horizontal Pod Autoscaling
- Add latency to one zone
- Monitor traffic from the Orders application
- Observe effects of HAZL
- Observe interaction with Horizontal Pod Autoscaling
- Simulate an outage in one zone
- Monitor traffic from the Orders application
- Observe effects of HAZL
- Observe interaction with Horizontal Pod Autoscaling
- Restore outage and remove latency
- Monitor traffic from the Orders application
- Observe effects of HAZL
- Observe interaction with Horizontal Pod Autoscaling
- Restore Orders application to initial state
If you'd like to follow along, you're going to need the following:
- Docker
- Helm
- k3d
- step
- The
kubectlcommand must be installed and working - The
watchcommand must be installed and working, if you want to use it - The
kubectxcommand must be installed and working, if you want to use it - Buoyant Enterprise for Linkerd License
- The Demo Assets, from GitHub
All prerequisites must be installed and working properly before proceeding. The instructions in the provided links will get you there. A trial license for Buoyant Enterprise for Linkerd can be obtained from the link above. Instructions on obtaining the demo assets from GitHub are below.
The top-level contents of the repository look like this:
.
├── README.md <-- This README
├── bel-manual-deploy.md <-- Instructions on how to manually deploy BEL
├── certs <-- Directory for the TLS root certificates
├── cluster <-- The k3d cluster configuration files live here
├── deploy-clusters.sh <-- Script to stand up the clusters, install Linkerd and Orders
├── deploy-multicluster.sh <-- Script to deploy the Buoyant Enterprise for Linkerd Multi-Cluster Extension
├── destroy-clusters.sh <-- Script to destroy the cluster environment
├── hazl.md <-- Detailed information on HAZL
├── images <-- Images for the README
├── mc.awk <-- Progfile for the awk command we use when setting up multi-cluster
├── orders -> orders-hpa <-- Soft link
├── orders-hpa <-- The Orders application, with Horizontal Pod Autoscaling
├── orders-nohpa <-- The Orders application, without Horizontal Pod Autoscaling
├── policy.yaml <-- Grants service-mirror access to the core Linkerd control plane
└── traffic_check.sh <-- Script to monitor application trafficThe repository contains the following automation:
deploy-clusters.sh- Script to stand up the clusters, install Linkerd and Orders
deploy-multicluster.sh- Script to deploy the multicluster extension to the clusters
destroy-clusters.sh- Script to destroy the cluster environments and clean up contexts
If you choose to use the deploy-clusters.sh script, make sure you've created the settings.sh file and run source settings.sh to set your environment variables. For more information, see the Obtain Buoyant Enterprise for Linkerd (BEL) Trial Credentials and Log In to Buoyant Cloud instructions.
This repository contains two k3d cluster configuration files:
.
├── cluster
│ ├── orders.yaml
│ └── warehouse.yamlThese will be used to deploy our two clusters.
This repository includes the Orders application, which generates traffic across multiple availability zones in our Kubernetes cluster, allowing us to observe the effect that High Availability Zonal Load Balancing (HAZL) has on traffic.
We're going to deploy the orders-* applications on the orders cluster and the warehouse-* applications on the warehouse cluster.
.
├── orders -> orders-hpa
├── orders-hpa
│ ├── orders
│ │ ├── kustomization.yaml
│ │ ├── ns.yaml
│ │ ├── orders-central.yaml
│ │ ├── orders-east.yaml
│ │ └── orders-west.yaml
│ └── warehouse
│ ├── kustomization.yaml
│ ├── ns.yaml
│ ├── server.yaml
│ ├── warehouse-boston.yaml
│ ├── warehouse-chicago.yaml
│ └── warehouse-oakland.yaml
├── orders-nohpa
│ ├── orders
│ │ ├── kustomization.yaml
│ │ ├── ns.yaml
│ │ ├── orders-central.yaml
│ │ ├── orders-east.yaml
│ │ └── orders-west.yaml
│ └── warehouse
│ ├── kustomization.yaml
│ ├── ns.yaml
│ ├── server.yaml
│ ├── warehouse-boston.yaml
│ ├── warehouse-chicago.yaml
│ └── warehouse-oakland.yamlThe repository contains two copies of the Orders application:
orders-hpa: HAZL version of the orders app with Horizontal Pod Autoscalingorders-nohpa: HAZL version of the orders app without Horizontal Pod Autoscaling
An orders soft link points to the hpa version of the application (orders -> orders-hpa), with Horizontal Pod Autoscaling. We will reference the orders soft link in the steps. If you want to use the nohpa version of the application, without Horizontal Pod Autoscaling, deploy the Orders application from the orders-nohpa directory, or recreate the orders soft link, pointing to the orders-nohpa directory.
Demonstration 1: Deploy a Kubernetes Cluster With Buoyant Enterprise for Linkerd, With HAZL Disabled
First, we'll deploy a Kubernetes cluster using k3d and deploy Buoyant Enterprise for Linkerd (BEL).
We're going to use the provided deploy-clusters.sh script for this. If you'd like to do it by hand, the instructions are here.
From the k3d-multicluster-flat-network-hazl directory, execute the deploy-clusters.sh script:
./deploy-clusters.shThis will create your k3d clusters and shared network, will deploy Buoyant Enterprise for Linkerd on both clusters, and will deploy the Orders application across both clusters.
Let's see how we deploy the Buoyant Enterprise for Linkerd Multi-Cluster Extension!
Explain
kubectl get cm coredns -n kube-system -o yaml --context orders -o yaml | grep -Ev "creationTimestamp|resourceVersion|uid" > coredns.yamlsed -i .orig 's/host.k3d.internal/host.k3d.internal\ kubernetes/g' coredns.yamlmore coredns.yamlkubectl apply -f coredns.yaml -n kube-system --context orderskubectl rollout restart deploy coredns -n kube-system --context orderskubectl get cm coredns -n kube-system -o yaml --context orders -o yaml | grep kubernetesInstalling the Multi-Cluster Extension
source settings.shhelm repo add linkerd-buoyant https://helm.buoyant.cloud
helm repo updatehelm install linkerd-multicluster \
--create-namespace \
--namespace linkerd-multicluster \
--kube-context orders \
--set linkerd-multicluster.gateway.enabled=false \
--set license=$BUOYANT_LICENSE \
linkerd-buoyant/linkerd-enterprise-multiclusterlinkerd --context=orders multicluster checkhelm install linkerd-multicluster \
--create-namespace \
--namespace linkerd-multicluster \
--kube-context warehouse \
--set linkerd-multicluster.gateway.enabled=false \
--set license=$BUOYANT_LICENSE \
linkerd-buoyant/linkerd-enterprise-multiclusterlinkerd --context=warehouse multicluster checkConfigure the Linkerd Multi-Cluster Policy
"As of the Linkerd 2.11 release, the Linkerd multi-cluster extension also includes a policy configuration that prevents unauthorized access to pods running in the linkerd-multicluster namespace. This policy configuration only grants access to the core Linkerd control plane by default. If you’re using the Linkerd multi-cluster extension with Buoyant Cloud, you’ll need to add the following configuration to the linkerd-multicluster namespace."
If you'd like to see the Linkerd Multi-Cluster Policy:
more policy.yamlApply the Linkerd Multi-Cluster Policy to the orders cluster from the included manifest:
kubectl apply -f policy.yaml --context orderslinkerd --context=warehouse multicluster link --cluster-name warehouse --gateway=false > multicluster-link-orig.yamlKC1=`linkerd --context=warehouse multicluster link --cluster-name warehouse --gateway=false | grep kubeconfig: | uniq | awk {'print $2'}` ; KC2=`echo $KC1 | base64 -d | sed 's/0\.0\.0\.0/kubernetes/g' | base64` ; awk -f mc.awk "$KC1" "$KC2" multicluster-link-orig.yaml > multicluster-link.yamlkubectl apply -f multicluster-link.yaml --context orderskubectl get links -A --context=orderskubectl get svc -A --context=orderskubectl get svc -A --context=warehousekubectl --context=warehouse label svc -n orders fulfillment mirror.linkerd.io/exported=remote-discoverykubectl get svc -A --context=ordersNow that BEL is fully deployed, we're going to need some traffic to observe.
In this scenario, we're delving into the operations of an online business specializing in Hacky Sack products. This business relies on a dedicated orders application to manage customer orders efficiently and to ensure that these orders are promptly dispatched to warehouses for shipment. To guarantee high availability and resilience, the system is distributed across three geographical availability zones: zone-east, zone-central, and zone-west. This strategic distribution ensures that the system operates smoothly, maintaining a balanced and steady state across different regions.
For the deployment of the Orders application, the business utilizes a modern infrastructure approach by employing Kubernetes. To further enhance the system's reliability and observability, Buoyant's Enterprise Linkerd service mesh is deployed on our cluster. Remember, Linkerd provides critical features such as dynamic request routing, service discovery, and comprehensive monitoring, which are instrumental for maintaining the health and performance of the Orders application across the clusters. Deploying the Orders application to a fresh Kubernetes cluster, augmented with Buoyant Enterprise Linkerd, signifies a significant step towards achieving robust, scalable, and highly available online business operations, ensuring that customers receive their Hacky Sack products without delays.
We don't know it yet, but our business, and the magic of hacky sack, are about to be featured on an episode of a popular sitcom tonight and orders are going to spike!
With the Orders application deployed across our clusters, we have some traffic to work with.
Let's take a look at traffic flow without HAZL enabled in Buoyant Cloud. This will give us a more visual representation of our baseline traffic. Head over to Buoyant Cloud, and take a look at the contents of the orders namespace in the Topology tab.
We can see that traffic from each orders deployment is flowing to all three warehouse deployments, and that about 2/3 of total traffic is out of zone. Latency is hovering around 80 ms per zone, and Requests By Warehouse has some variation over time. All deployments are currently scaled to one replica.
Let's see what happens when we enable HAZL.
Let's take a look at how quick and easy we can enable High Availability Zonal Load Balancing (HAZL).
Remember, to make adjustments to your BEL deployment simply edit and re-apply the previously-created linkerd-control-plane-config-hazl.yaml manifest. We're going to enable the - -ext-endpoint-zone-weights in the additionalArgs for now, by uncommenting it in the manifest:
Edit the linkerd-control-plane-config-hazl.yaml file:
vi linkerd-control-plane-config-hazl.yamlApply the ControlPlane CRD config to have the Linkerd BEL operator update the Linkerd control plane configuration, and enable HAZL on the hazl cluster only:
kubectl apply -f linkerd-control-plane-config-hazl.yaml --context orderskubectl apply -f linkerd-control-plane-config-hazl.yaml --context warehouseNow, we can see the effect HAZL has on the traffic in our multi-az cluster.
Let's take a look at what traffic looks like with HAZL enabled, using Buoyant Cloud. This will give us a more visual representation of the effect of HAZL on our traffic.
With HAZL enabled, we see traffic stay in zone, at 50 requests per second. Cross-AZ traffic drops to zero, latency by zone drops to around 50 ms and Requests By Warehouse smooths out. Application load stays consistent and all deployments remain at one replica.
A popular sitcom aired an episode in which the characters, a bunch of middle-aged Generation X folks, flash back to their teenage years, and remember the joy they experienced playing hacky sack together. Our characters decide they're going to relive those wonderful hacky sack memories, and go online to order supplies, featuring our website and products. Jackpot!
Let's simulate what that looks like. The first thing we see is an uptick of orders in zone-east, as they're the first to watch the episode.
We can increase traffic in zone-east by scaling the orders-east deployment. Let's scale to 10 replicas.
kubectl scale -n orders deploy orders-east --replicas=10 --context ordersLet's see the results of scaling orders-east:
watch -n 1 kubectl get deploy,hpa -n orders --context warehouseUse CTRL-C to exit the watch command.
Let's take a look at what the increased traffic looks like in Buoyant Cloud. This will give us a more visual representation of the effect of HAZL on our traffic.
After scaling orders-east to 10 replicas, traffic remains in the same AZ and latency holds steady at around 50 ms. Order success remains at 100%.
The middle of the country really liked our hacky sacks! Order volume is running more than double what we saw in zone-east.
We can increase traffic in zone-central by scaling the orders-central deployment. Let's scale to 25 replicas.
kubectl scale -n orders deploy orders-central --replicas=25 --context ordersLet's see the results of scaling orders-central:
watch -n 1 kubectl get deploy,hpa -n orders --context warehouseUse CTRL-C to exit the watch command.
Let's take a look at what the increased traffic looks like in Buoyant Cloud. This will give us a more visual representation of the effect of HAZL on our traffic.
Again, after scaling orders-central to 25 replicas, all traffic remains in the same AZ and latency holds steady at around 50 ms. Order success remains at 100%. Both warehouse-boston and warehouse-chicago have autoscaled to 3 replicas.
By now, word of the episode is all over social media, and when the episode airs in Pacific time, orders in zone-west spike.
We can increase traffic in zone-west by scaling the orders-west deployment. Let's scale to 30 replicas.
kubectl scale -n orders deploy orders-west --replicas=30 --context ordersLet's see the results of scaling orders-west:
watch -n 1 kubectl get deploy,hpa -n orders --context warehouseUse CTRL-C to exit the watch command.
Let's take a look at what the increased traffic looks like in Buoyant Cloud. This will give us a more visual representation of the effect of HAZL on our traffic.
Once more, after scaling orders-west to 30 replicas, all traffic remains in the same AZ and latency holds steady at around 50 ms. Order success remains at 100%. Both warehouse-boston, warehouse-chicago and warehouse-oakland have autoscaled to 3 replicas.
So far, everything has gone right. What about when things go wrong?
Unfortunately, we've had some network issues creep in at our Chicago warehouse!
We can increase latency in zone-central by editing the warehouse-config configmap, which has a setting for latency.
kubectl edit -n orders cm/warehouse-config --context warehouseYou'll see:
data:
blue.yml: |
color: "#0000ff"
averageResponseTime: 0.020
green.yml: |
color: "#00ff00"
averageResponseTime: 0.020
red.yml: |
color: "#ff0000"
averageResponseTime: 0.020The colors map to the warehouses as follows:
- Red: This is the Oakland warehouse (
warehouse-oakland) - Blue: This is the Boston warehouse (
warehouse-boston) - Green: This is the Chicago warehouse (
warehouse-chicago)
Change the value of averageResponseTime under green.yml from 0.020 to 0.920. Save and exit.
We need to restart the warehouse-chicago deployment to pick up the changes:
kubectl rollout restart -n orders deploy warehouse-chicago --context warehouseLet's take a look at what the increased latency looks like in Buoyant Cloud. This will give us a more visual representation of the effect of HAZL on our traffic in response to increased latency.
HAZL steps in and does what it needs to do, redirecting a portion of the traffic from orders-central across AZs to warehouse-oakland to keep success rate at 100%. Latency increases in orders-central, but HAZL adjusts.
More bad news! The latency we've been experiencing is about to turn into an outage!
We can simulate this by scaling the warehouse-chicago deployment. Let's scale to 0 replicas.
kubectl scale -n orders deploy warehouse-chicago --replicas=0 --context warehouseLet's see the results of scaling warehouse-chicago to 0:
watch -n 1 kubectl get deploy,hpa -n orders --context warehouseUse CTRL-C to exit the watch command.
Let's take a look at what the increased traffic looks like in Buoyant Cloud. This will give us a more visual representation of the effect of HAZL on our traffic.
Once again, HAZL steps in and does what it needs to do, redirecting all of the traffic from orders-central across AZs to warehouse-oakland to keep success rate at 100%. Latency drops in orders-central as we are no longer sending traffic to warehouse-chicago, which added latency.
Good news! The latency and outage is about to end!
We can simulate this by scaling the warehouse-chicago deployment. Let's scale to 1 replica. The Horizontal Pod Autoscaler will take over from there.
kubectl scale -n orders deploy warehouse-chicago --replicas=1 --context warehouseWe also need to edit the warehouse-config configmap, and set the latency to match the other warehouse deployments.
kubectl edit -n orders cm/warehouse-config --context warehouseWe need to restart the warehouse-chicago deployment to pick up the changes:
kubectl rollout restart -n orders deploy warehouse-chicago --context warehouseLet's see the results of scaling warehouse-chicago and restarting the deployment:
watch -n 1 kubectl get deploy,hpa -n orders --context warehouseUse CTRL-C to exit the watch command.
Let's take a look at what the service restoration looks like in Buoyant Cloud. This will give us a more visual representation of the effect of HAZL on our traffic.
We can see things have returned to 100% in-zone traffic, with latency back to about 50 ms across the board and success rates at 100%. HPA has 3 replicas of the warehouse deployments per zone.
Now that we're finished, let's reset the Orders application back to its initial state.
kubectl apply -k orders/orders --context orderskubectl apply -k orders/warehouse --context warehouseLet's see the results of the reset:
watch -n 1 kubectl get deploy,hpa -n orders --context warehouseUse CTRL-C to exit the watch command.
If we give things a minute to settle back down, we should see all traffic back in zone and request rates back to 50.
Everything has returned to the initial state with HAZL enabled. All deployments are a single replica, all traffic remains in-zone, and success rates are 100%. Looking good!
You can clean up the Demonstration environment by running the included script:
./destroy-clusters.shChecking our work:
k3d cluster listWe shouldn't see our demo-cluster-orders-hazl cluster.
In this hands-on Demonstration, we deployed Buoyant Enterprise for Linkerd and demonstrated how to enable High Availability Zonal Load Balancing (HAZL). We also took a look at how HAZL works to keep network traffic in-zone where possible by exploring some different traffic, load and availability situations.
Thank you for taking a journey with HAZL and Buoyant!