Incorrect toleration and labels on the sas compute deployment

[Stacey949]

Terraform Version Details

We downloaded this project in February 2025 and used it as is. We deployed into our Azure environment, and it seems to have incorrectly applied the labels and taints for the compute nodes.

Terraform Variable File Details

**************** REQUIRED VARIABLES ****************

These required variables' values MUST be provided by the User

prefix = "XXX-sas"

location = "australiaeast"

subscription_id = "xxxxx-xxxx-4b0e-920e-f1a05da74bd6"

tenant_id = "xxxxx-xxxx-438a-92d5-89eff9d56a6b"

create_jump_vm = false

Bring your own existing networking resources

vnet_resource_group_name = "rg-network-auea"

vnet_name = "vnet-XXX-sas-auea"

subnet_names = {

"aks": "aks",

"misc": "misc",

"netapp": "netapp"

}

also available as BYO

resource_group_name = "rg-terraform-infra"

!NOTE! - Without specifying your CIDR block access rules, ingress traffic

to your cluster will be blocked by default.

**************** REQUIRED VARIABLES ****************

Bring your own existing networking resources

vnet_address_space = "XXX.XX.32.0/20"

aks_cluster_sku_tier = "Premium"

cluster_support_tier = "AKSLongTermSupport"

aks_network_plugin = "azure"

aks_network_plugin_mode = "overlay"

cluster_egress_type = "userDefinedRouting"

!NOTE! - Without specifying your CIDR block access rules, ingress traffic

to your cluster will be blocked by default.

************** RECOMMENDED VARIABLES ***************

default_public_access_cidrs = [] # e.g., ["XXX.XX.6.89/32"]

************** RECOMMENDED VARIABLES ***************

Tags for all taggable items in your cluster.

tags = {"Organisation" = "XXX-XXX-office", "Environment" = "production", "Operational Hours" = "business-hours", "Application" = "sasviya4", "SupportArea" = "XXX", "BackupService" = "infrastructure-as-code", "BackupFileShare": "compliance"} # e.g., { "key1" = "value1", "key2" = "value2" }

Azure Container Registry config

create_container_registry = true

container_registry_sku = "Standard"

container_registry_admin_enabled = false

container_registry_geo_replica_locs = null

Azure monitor

create_aks_azure_monitor = false

AKS config

kubernetes_version = "1.30.7"

default_nodepool_min_nodes = 1

default_nodepool_vm_type = "Standard_D8_v5"

node_pools_proximity_placement = false

default_nodepool_availability_zones = ["1", "2", "3"]

node_pools_availability_zone = "1"

AKS Node Pools config

node_pools = {

cas = {

"machine_type" = "Standard_E16ds_v5"

"os_disk_size" = 256

"min_nodes"    = 2

"max_nodes"    = 2

"max_pods"     = 110

"node_taints"  = ["workload.sas.com/class=cas:NoSchedule"]

"node_labels" = {

  "workload.sas.com/class" = "cas"

}

},

compute = {

"machine_type" = "Standard_E8ds_v5"

"os_disk_size" = 256

"min_nodes"    = 1

"max_nodes"    = 1

"max_pods"     = 110

"node_taints"  = ["workload.sas.com/class=compute:NoSchedule"]

"node_labels" = {

  "workload.sas.com/class"        = "compute"

  "launcher.sas.com/prepullImage" = "sas-programming-environment"

}

},

stateless = {

"machine_type" = "Standard_D8_v5"

"os_disk_size" = 256

"min_nodes"    = 1

"max_nodes"    = 2

"max_pods"     = 110

"node_taints"  = ["workload.sas.com/class=stateless:NoSchedule"]

"node_labels" = {

  "workload.sas.com/class" = "stateless"

}

},

stateful = {

"machine_type" = "Standard_D8_v5"

"os_disk_size" = 256

"min_nodes"    = 4

"max_nodes"    = 4

"max_pods"     = 110

"node_taints"  = ["workload.sas.com/class=stateful:NoSchedule"]

"node_labels" = {

  "workload.sas.com/class" = "stateful"

}

}

}

Storage for SAS Viya CAS/Compute

storage_type = "ha"

netapp_service_level = "Ultra"

netapp_size_in_tb = "4"

Identity values

cluster_api_mode = "private"

aks_identity = "uai"

aks_private_cluster_public_fqdn_enabled = true

RBAC

rbac_aad_enabled = true

rbac_aad_admin_group_object_ids = [ "XXXXXX-25aa-498d-84bb-3aa2f71cb086" ]

Steps to Reproduce

Deploy as is terraform scripts

Expected Behavior

Opening SAS studio and expected it to create a SAS Compute context, and expected to see a host in the Workload Orchestrator

Actual Behavior

SAS Compute in SAS Studio does not start a SAS Compute context

The host should be listed in the Workload Orchestrator

Additional Context

No response

References

No response

Code of Conduct

• I agree to follow this project's Code of Conduct

[Stacey949]

This is an urgent problem that is holding up a delivery timeline for our customer.

[dhoucgitter]

Hi @Stacey949, I see that you downloaded the project in February, could you also let us know which release level of IaC Azure you are using? Thanks for posting your tfvars file for reference. Give us a chance to duplicate the behavior you are seeing with the IaC Azure 10.1.0 release we expect that you are using.

[dhoucgitter]

Hi @Stacey949, myself and another teammate of mine tried to duplicate the incorrect taints and labels for the nodes within the Node Pools you configured using IaC Azure release 10.1.0. We created two different AKS clusters in the useast2 and austrailiaeast Azure locations. I specified kubernetes_version=1.30 which in the useast2 region resulted in a v1.30.10 AKS cluster. We strived to utilize a majority of the configuration values you indicated in your post above with the exception of the BYO entries.

Our results indicated no unexpected taint or labels values for any of the created nodes.
Here's snippets of the kubectl describe info the showing the taints and labels for the compute and stateful nodes that IaC Azure created in useast2 using the 10.1.0 release.

Name:               aks-compute-35298530-vmss000000                                                                         
Roles:              <none>                                                                                                  
Labels:             agentpool=compute                                                                                       
                    beta.kubernetes.io/arch=amd64                                                                           
                    beta.kubernetes.io/instance-type=Standard_E8ds_v5                                                       
                    beta.kubernetes.io/os=linux                                                                             
                    failure-domain.beta.kubernetes.io/region=eastus2                                                        
                    failure-domain.beta.kubernetes.io/zone=eastus2-1                                                        
                    kubernetes.azure.com/agentpool=compute                                                                  
                    kubernetes.azure.com/cluster=MC_******-rg_******-aks_eastus2                                            
                    kubernetes.azure.com/consolidated-additional-properties=7a149412-0fec-11f0-81fb-5e1412734c2e            
                    kubernetes.azure.com/kubelet-identity-client-id=7695c52c-1f8b-430a-a66d-b6bde597225f                    
                    kubernetes.azure.com/mode=user                                                                          
                    kubernetes.azure.com/node-image-version=AKSUbuntu-2204gen2containerd-202503.21.0                        
                    kubernetes.azure.com/nodepool-type=VirtualMachineScaleSets                                              
                    kubernetes.azure.com/os-sku=Ubuntu                                                                      
                    kubernetes.azure.com/role=agent                                                                         
                    kubernetes.azure.com/storageprofile=managed                                                             
                    kubernetes.azure.com/storagetier=Premium_LRS                                                            
                    kubernetes.io/arch=amd64                                                                                
                    kubernetes.io/hostname=aks-compute-35298530-vmss000000                                                  
                    kubernetes.io/os=linux                                                                                  
                    launcher.sas.com/prepullImage=sas-programming-environment                                               
                    node.kubernetes.io/instance-type=Standard_E8ds_v5                                                       
                    storageprofile=managed                                                                                  
                    storagetier=Premium_LRS                                                                                 
                    topology.disk.csi.azure.com/zone=eastus2-1                                                              
                    topology.kubernetes.io/region=eastus2                                                                   
                    topology.kubernetes.io/zone=eastus2-1                                                                   
                    workload.sas.com/class=compute                                                                          
Annotations:        alpha.kubernetes.io/provided-node-ip: 192.168.0.7                                                       
                    csi.volume.kubernetes.io/nodeid:                                                                        
                      {"disk.csi.azure.com":"aks-compute-35298530-vmss000000","file.csi.azure.com":"aks-compute-35298530-vms
s000000"}                                                                                                                   
                    node.alpha.kubernetes.io/ttl: 0                                                                         
                    volumes.kubernetes.io/controller-managed-attach-detach: true                                            
CreationTimestamp:  Wed, 02 Apr 2025 14:03:07 -0400                                                                         
Taints:             workload.sas.com/class=compute:NoSchedule            

Name:               aks-stateful-48152294-vmss000000                                                                        
Roles:              <none>                                                                                                  
Labels:             agentpool=stateful                                                                                      
                    beta.kubernetes.io/arch=amd64                                                                           
                    beta.kubernetes.io/instance-type=Standard_D4s_v5                                                        
                    beta.kubernetes.io/os=linux                                                                             
                    failure-domain.beta.kubernetes.io/region=eastus2                                                        
                    failure-domain.beta.kubernetes.io/zone=eastus2-1                                                        
                    kubernetes.azure.com/agentpool=stateful                                                                 
                    kubernetes.azure.com/cluster=MC_******-rg_******-aks_eastus2                                            
                    kubernetes.azure.com/consolidated-additional-properties=d3a87642-0fec-11f0-9235-428ff36a4fc3            
                    kubernetes.azure.com/kubelet-identity-client-id=7695c52c-1f8b-430a-a66d-b6bde597225f                    
                    kubernetes.azure.com/mode=user                                                                          
                    kubernetes.azure.com/node-image-version=AKSUbuntu-2204gen2containerd-202503.21.0                        
                    kubernetes.azure.com/nodepool-type=VirtualMachineScaleSets                                              
                    kubernetes.azure.com/os-sku=Ubuntu                                                                      
                    kubernetes.azure.com/role=agent                                                                         
                    kubernetes.azure.com/storageprofile=managed                                                             
                    kubernetes.azure.com/storagetier=Premium_LRS                                                            
                    kubernetes.io/arch=amd64                                                                                
                    kubernetes.io/hostname=aks-stateful-48152294-vmss000000                                                 
                    kubernetes.io/os=linux                                                                                  
                    node.kubernetes.io/instance-type=Standard_D4s_v5                                                        
                    storageprofile=managed                                                                                  
                    storagetier=Premium_LRS                                                                                 
                    topology.disk.csi.azure.com/zone=eastus2-1                                                              
                    topology.kubernetes.io/region=eastus2                                                                   
                    topology.kubernetes.io/zone=eastus2-1                                                                   
                    workload.sas.com/class=stateful                                                                         
Annotations:        alpha.kubernetes.io/provided-node-ip: 192.168.0.9                                                       
                    csi.volume.kubernetes.io/nodeid:                                                                        
                      {"disk.csi.azure.com":"aks-stateful-48152294-vmss000000","file.csi.azure.com":"aks-stateful-48152294-v
mss000000"}                                                                                                                 
                    node.alpha.kubernetes.io/ttl: 0                                                                         
                    volumes.kubernetes.io/controller-managed-attach-detach: true                                            
CreationTimestamp:  Wed, 02 Apr 2025 14:05:34 -0400                                                                         
Taints:             workload.sas.com/class=stateful:NoSchedule   

You can see that the "workload.sas.com/class=compute:NoSchedule" taint and "workload.sas.com/class=compute" label are present for the compute node and that the
"workload.sas.com/class=stateful:NoSchedule" taint and "workload.sas.com/class=stateful" label are present on the stateful node. We also checked the cas, and stateless nodes to verify they had the expected taints and labels and they did.

Could I ask you to try a simpler cluster creation for me without deploying Viya yet? If you can, update the sample-input.tfvars file in the examples folder with a unique prefix and the other required/recommended values and run a terraform apply with it to verify that you are getting a consistent set of nodes created with the expected taints and labels without even trying to deploy Viya yet. My expectation is that the taints and labels you noticed as wrong were affected prior to any interaction with deploying Viya, but it's not clear to us how that might have happened with IaC Azure alone. If that attempt creates nodes with the expected taints and labels, please try another terraform apply with a modified version of your byo intput.tfvars to see if you get the same unexpected taints and label results as you initially reported and get back to us with your findings, thanks.

[Carus11]

From the screenshot it appears your compute pods are running on the stateful node, and additionally are labeled with the stateless workload class. There isn't enough information in the screenshot to show that a compute node was mislabeled. I believe this may be a incorrect configuration of the viya deployment yaml files, and suggest you open a support ticket for assistance as this specific repository deals with the terraform deployment of the underlying infrastructure and not the configuration and deployment of the SAS Viya software.

[github-actions]

This issue is stale because it has been open 30 days with no activity.

[ajeffowens]

Isn't this expected? The pod mentioned is likely the actual compute service pod, as opposed to a compute session pod. The session pods are ephemeral, and would have a name like "sas-compute-server-xxx". The relevant tolerations and affinity are defined by a podtemplate such as sas-compute-job-config.

[github-actions]

This issue is stale because it has been open 30 days with no activity.