Parallel resource creation fails because of ServiceAssociationLinkNotReady

[dsteindo]

Terraform Version

1.1.3

Terraform Configuration Files

resource "azurerm_container_group" "jmeter_workers" {
  count               = var.JMETER_WORKERS_COUNT
  name                = "${var.PREFIX}-worker${count.index}"
  location            = azurerm_resource_group.jmeter_rg.location
  resource_group_name = azurerm_resource_group.jmeter_rg.name

  ip_address_type = "private"
  os_type         = "Linux"

  network_profile_id = azurerm_network_profile.jmeter_net_profile.id

  image_registry_credential {
    server   = data.azurerm_container_registry.jmeter_acr.login_server
    username = data.azurerm_container_registry.jmeter_acr.admin_username
    password = data.azurerm_container_registry.jmeter_acr.admin_password
  }

  container {
    name   = "jmeter"
    image  = var.JMETER_DOCKER_IMAGE
    cpu    = var.JMETER_WORKER_CPU
    memory = var.JMETER_WORKER_MEMORY

    ports {
      port     = var.JMETER_DOCKER_PORT
      protocol = "TCP"
    }

    volume {
      name                 = "jmeter"
      mount_path           = "/jmeter"
      read_only            = true
      storage_account_name = azurerm_storage_account.jmeter_storage.name
      storage_account_key  = azurerm_storage_account.jmeter_storage.primary_access_key
      share_name           = azurerm_storage_share.jmeter_share.name
    }

    commands = [
      "/bin/sh",
      "-c",
      "cp -r /jmeter/* .; /entrypoint.sh -s -J server.rmi.ssl.disable=true -Djava.rmi.server.hostname=$(ifconfig eth0 | grep 'inet addr:' | awk '{gsub(\"addr:\", \"\"); print $2}') -Jnode_index=${count.index} -Jusers_per_node=${var.USERS_PER_NODE}",
    ]
  }
}

Debug Output

azurerm_network_profile.jmeter_net_profile: Creating...
azurerm_network_profile.jmeter_net_profile: Creation complete after 1s [id=/subscriptions/HASH/resourceGroups/jmeter-performance-RG/providers/Microsoft.Network/networkProfiles/jmeter3netprofile]
azurerm_container_group.jmeter_workers[0]: Creating...
azurerm_container_group.jmeter_workers[2]: Creating...
azurerm_container_group.jmeter_workers[3]: Creating...
azurerm_container_group.jmeter_workers[1]: Creating...
azurerm_container_group.jmeter_workers[3]: Still creating... [10s elapsed]
azurerm_container_group.jmeter_workers[0]: Still creating... [10s elapsed]
azurerm_container_group.jmeter_workers[3]: Still creating... [20s elapsed]
azurerm_container_group.jmeter_workers[0]: Still creating... [20s elapsed]
azurerm_container_group.jmeter_workers[3]: Still creating... [30s elapsed]
azurerm_container_group.jmeter_workers[0]: Still creating... [30s elapsed]
azurerm_container_group.jmeter_workers[3]: Still creating... [40s elapsed]
azurerm_container_group.jmeter_workers[0]: Still creating... [40s elapsed]
azurerm_container_group.jmeter_workers[3]: Still creating... [50s elapsed]
azurerm_container_group.jmeter_workers[0]: Still creating... [50s elapsed]
azurerm_container_group.jmeter_workers[3]: Still creating... [1m0s elapsed]
azurerm_container_group.jmeter_workers[0]: Still creating... [1m0s elapsed]
azurerm_container_group.jmeter_workers[3]: Still creating... [1m10s elapsed]
azurerm_container_group.jmeter_workers[0]: Still creating... [1m10s elapsed]
azurerm_container_group.jmeter_workers[3]: Still creating... [1m20s elapsed]
azurerm_container_group.jmeter_workers[0]: Still creating... [1m20s elapsed]
azurerm_container_group.jmeter_workers[3]: Still creating... [1m30s elapsed]
azurerm_container_group.jmeter_workers[0]: Still creating... [1m30s elapsed]
azurerm_container_group.jmeter_workers[3]: Creation complete after 1m34s [id=/subscriptions/HASH/resourceGroups/jmeter-performance-RG/providers/Microsoft.ContainerInstance/containerGroups/jmeter3-worker3]
azurerm_container_group.jmeter_workers[0]: Creation complete after 1m34s [id=/subscriptions/HASH/resourceGroups/jmeter-performance-RG/providers/Microsoft.ContainerInstance/containerGroups/jmeter3-worker0]


|
| Error: creating/updating Container Group: (Name "jmeter3-worker1" / Resource Group "jmeter-performance-RG"): containerinstance.ContainerGroupsClient#CreateOrUpdate: Failure sending request: StatusCode=409 -- Original Error: Code="ServiceAssociationLinkNotReady" Message="Network is not ready for container group 'jmeter3-worker1'. Please try again."
| 
|   with azurerm_container_group.jmeter_workers[1],
|   on main.tf line 75, in resource "azurerm_container_group" "jmeter_workers":
|   75: resource "azurerm_container_group" "jmeter_workers" {
| 
|
|
| Error: creating/updating Container Group: (Name "jmeter3-worker2" / Resource Group "jmeter-performance-RG"): containerinstance.ContainerGroupsClient#CreateOrUpdate: Failure sending request: StatusCode=409 -- Original Error: Code="ServiceAssociationLinkNotReady" Message="Network is not ready for container group 'jmeter3-worker2'. Please try again."
| 
|   with azurerm_container_group.jmeter_workers[2],
|   on main.tf line 75, in resource "azurerm_container_group" "jmeter_workers":
|   75: resource "azurerm_container_group" "jmeter_workers" {
| 
|
##[error]Bash exited with code '1'.

Expected Behavior

4 workers for load testing should have been created

Actual Behavior

Sometimes the pipeline breaks because of the "ServiceAssociationLinkNotReady" error

Steps to Reproduce

Use the following project https://github.com/Azure-Samples/jmeter-aci-terraform
Configure that multiple workers are used, see Azure-Samples/jmeter-aci-terraform#81
Also consider that there is still an open PR to raise the terraform version used which we already integrated into our project, see Azure-Samples/jmeter-aci-terraform#84

Additional Context

Variables

TF_VAR_JMETER_DOCKER_IMAGE = justb4/jmeter:5.1.1
TF_VAR_JMETER_WORKERS_COUNT = 4
TF_VAR_PREFIX = jmeter3
TF_VAR_RESOURCE_GROUP_NAME = jmeter-performance-RG
TF_VAR_USERS_PER_NODE = 250

https://github.com/Azure-Samples/jmeter-aci-terraform/blob/main/terraform/variables.tf

References

Azure-Samples/jmeter-aci-terraform#81
hashicorp/terraform#30369

[magodo]

@dsteindo Thank you for submitting this 👍 This seems to be either a MS upstream API issue or a user configuration issue. I've tried following configuration in my local run, which causes timeout (30min) issue: https://gist.github.com/magodo/5dca2240571faa2e232dc8b609e12c56.
This timeout issue is a bit difference than your issue, and it even happens for a single container group. So would you please kindly minimize your setup to the configuration similar as above, so that I can reproduce it locally and analyze further more?

[dsteindo]

Hi @magodo

Thank you for your investigation.
I recreated the issue and pushed the code to a git repository for you to check, I tried to strip away everything that was not related (not needed code that was executed after the error happened)
https://github.com/dsteindo/TerraformRaceCondition

Hope it helps in finding what the potential issue is :)

[tombuildsstuff]

@magodo looks like this resource is missing a lock on the network profile ID during Create/Update/Delete, since this'll cause a NIC to be provisioned and the API doesn't support provisioning them in parallel, by the look of it?

[magodo]

@dsteindo Somehow I can reproduce it with my prior setup, while still thanks a lot for composing this repro repo.
@tombuildsstuff Thank you for the hint 👍 Indeed after adding the lock to serialize the provisioning, the error never occurs.

[github-actions]

This functionality has been released in v2.95.0 of the Terraform Provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template. Thank you!

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.