feat: added new variables import_default_worker_pool_on_create and allow_default_worker_pool_replacement. For more information on these, see https://github.com/terraform-ibm-modules/terraform-ibm-ocp-all-inclusive/tree/main?tab=readme-ov-file#default-worker-pool-management (#325)

Aashiq-J · web-flow · commit df5675325cc0 · 2024-09-18T18:52:33.000+01:00
diff --git a/README.md b/README.md
@@ -20,6 +20,43 @@ This module is a wrapper module that groups the following modules:
 - Make sure that you have a recent version of the [IBM Cloud CLI](https://cloud.ibm.com/docs/cli?topic=cli-getting-started)
 - Make sure that you have a recent version of the [IBM Cloud Kubernetes service CLI](https://cloud.ibm.com/docs/containers?topic=containers-kubernetes-service-cli)
 
+### Default Worker Pool management
+
+You can manage the default worker pool using Terraform, and make changes to it through this module. This option is enabled by default. Under the hood, the default worker pool is imported as a `ibm_container_vpc_worker_pool` resource. Advanced users may opt-out of this option by setting `import_default_worker_pool_on_create` parameter to `false`. For most use cases it is recommended to keep this variable to `true`.
+
+#### Important Considerations for Terraform and Default Worker Pool
+
+**Terraform Destroy**
+
+When using the default behavior of handling the default worker pool as a stand-alone `ibm_container_vpc_worker_pool`, you must manually remove the default worker pool from the Terraform state before running a terraform destroy command on the module. This is due to a [known limitation](https://cloud.ibm.com/docs/containers?topic=containers-faqs#smallest_cluster) in IBM Cloud.
+
+Terraform CLI Example
+
+For a cluster with 2 worker pools, named 'default' and 'secondarypool', follow these steps:
+
+```sh
+      $ terraform state list | grep ibm_container_vpc_worker_pool
+        > module.ocp_all_inclusive.module.ocp_base.data.ibm_container_vpc_worker_pool.all_pools["default"]
+        > module.ocp_all_inclusive.module.ocp_base.data.ibm_container_vpc_worker_pool.all_pools["secondarypool"]
+        > module.ocp_all_inclusive.module.ocp_base.ibm_container_vpc_worker_pool.pool["default"]
+        > module.ocp_all_inclusive.module.ocp_base.ibm_container_vpc_worker_pool.pool["secondarypool"]
+        > ...
+
+      $ terraform state rm "module.ocp_all_inclusive.module.ocp_base.ibm_container_vpc_worker_pool.pool[\"default\"]"
+```
+
+Schematics Example: For a cluster with 2 worker pools, named 'default' and 'secondarypool', follow these steps:
+
+```sh
+        $ ibmcloud schematics workspace state rm --id <workspace_id> --address "module.ocp_all_inclusive.module.ocp_base.ibm_container_vpc_worker_pool.pool[\"default\"]"
+```
+
+**Changes Requiring Re-creation of Default Worker Pool**
+
+If you need to make changes to the default worker pool that require its re-creation (e.g., changing the worker node `operating_system`), you must set the `allow_default_worker_pool_replacement` variable to true, perform the apply, and then set it back to false in the code before the subsequent apply. This is **only** necessary for changes that require the recreation the entire default pool and is **not needed for scenarios that does not require recreating the worker pool such as changing the number of workers in the default worker pool**.
+
+This approach is due to a limitation in the Terraform provider that may be lifted in the future.
+
 <!-- Below content is automatically populated via pre-commit hook -->
 <!-- BEGIN OVERVIEW HOOK -->
 ## Overview
@@ -127,8 +164,8 @@ You need the following permissions to run this module.
 
 | Name | Source | Version |
 |------|--------|---------|
-| <a name="module_observability_agents"></a> [observability\_agents](#module\_observability\_agents) | terraform-ibm-modules/observability-agents/ibm | 1.28.7 |
-| <a name="module_ocp_base"></a> [ocp\_base](#module\_ocp\_base) | terraform-ibm-modules/base-ocp-vpc/ibm | 3.29.3 |
+| <a name="module_observability_agents"></a> [observability\_agents](#module\_observability\_agents) | terraform-ibm-modules/observability-agents/ibm | 1.29.0 |
+| <a name="module_ocp_base"></a> [ocp\_base](#module\_ocp\_base) | terraform-ibm-modules/base-ocp-vpc/ibm | 3.30.1 |
 
 ### Resources
 
@@ -142,6 +179,7 @@ No resources.
 | <a name="input_additional_lb_security_group_ids"></a> [additional\_lb\_security\_group\_ids](#input\_additional\_lb\_security\_group\_ids) | Additional security group IDs to add to the load balancers associated with the cluster. These security groups are in addition to the IBM-maintained security group. | `list(string)` | `[]` | no |
 | <a name="input_additional_vpe_security_group_ids"></a> [additional\_vpe\_security\_group\_ids](#input\_additional\_vpe\_security\_group\_ids) | Additional security groups to add to all the load balancers. This comes in addition to the IBM maintained security group. | <pre>object({<br>    master   = optional(list(string), [])<br>    registry = optional(list(string), [])<br>    api      = optional(list(string), [])<br>  })</pre> | `{}` | no |
 | <a name="input_addons"></a> [addons](#input\_addons) | List of all addons supported by the ocp cluster. | <pre>object({<br>    debug-tool                = optional(string)<br>    image-key-synchronizer    = optional(string)<br>    openshift-data-foundation = optional(string)<br>    vpc-file-csi-driver       = optional(string)<br>    static-route              = optional(string)<br>    cluster-autoscaler        = optional(string)<br>    vpc-block-csi-driver      = optional(string)<br>  })</pre> | `null` | no |
+| <a name="input_allow_default_worker_pool_replacement"></a> [allow\_default\_worker\_pool\_replacement](#input\_allow\_default\_worker\_pool\_replacement) | (Advanced users) Set to true to allow the module to recreate a default worker pool. Only use in the case where you are getting an error indicating that the default worker pool cannot be replaced on apply. Once the default worker pool is handled as a stand-alone ibm\_container\_vpc\_worker\_pool, if you wish to make any change to the default worker pool which requires the re-creation of the default pool set this variable to true. | `bool` | `false` | no |
 | <a name="input_attach_ibm_managed_security_group"></a> [attach\_ibm\_managed\_security\_group](#input\_attach\_ibm\_managed\_security\_group) | Whether to attach the IBM-defined default security group (named `kube-<clusterid>`) to all worker nodes. Applies only if `custom_security_group_ids` is set. | `bool` | `true` | no |
 | <a name="input_cloud_monitoring_access_key"></a> [cloud\_monitoring\_access\_key](#input\_cloud\_monitoring\_access\_key) | Access key for the Cloud Monitoring agent to communicate with the instance. | `string` | `null` | no |
 | <a name="input_cloud_monitoring_add_cluster_name"></a> [cloud\_monitoring\_add\_cluster\_name](#input\_cloud\_monitoring\_add\_cluster\_name) | If true, configure the cloud monitoring agent to attach a tag containing the cluster name to all metric data. | `bool` | `true` | no |
@@ -168,6 +206,7 @@ No resources.
 | <a name="input_existing_kms_root_key_id"></a> [existing\_kms\_root\_key\_id](#input\_existing\_kms\_root\_key\_id) | The Key ID of a root key, existing in the KMS instance passed in var.existing\_kms\_instance\_guid, which will be used to encrypt the data encryption keys (DEKs) which are then used to encrypt the secrets in the cluster. Required if value passed for var.existing\_kms\_instance\_guid. | `string` | `null` | no |
 | <a name="input_force_delete_storage"></a> [force\_delete\_storage](#input\_force\_delete\_storage) | Delete attached storage when destroying the cluster - Default: false | `bool` | `false` | no |
 | <a name="input_ignore_worker_pool_size_changes"></a> [ignore\_worker\_pool\_size\_changes](#input\_ignore\_worker\_pool\_size\_changes) | Enable if using worker autoscaling. Stops Terraform managing worker count | `bool` | `false` | no |
+| <a name="input_import_default_worker_pool_on_create"></a> [import\_default\_worker\_pool\_on\_create](#input\_import\_default\_worker\_pool\_on\_create) | (Advanced users) Whether to handle the default worker pool as a stand-alone ibm\_container\_vpc\_worker\_pool resource on cluster creation. Only set to false if you understand the implications of managing the default worker pool as part of the cluster resource. Set to true to import the default worker pool as a separate resource. Set to false to manage the default worker pool as part of the cluster resource. | `bool` | `true` | no |
 | <a name="input_kms_account_id"></a> [kms\_account\_id](#input\_kms\_account\_id) | Id of the account that owns the KMS instance to encrypt the cluster. It is only required if the KMS instance is in another account. | `string` | `null` | no |
 | <a name="input_kms_use_private_endpoint"></a> [kms\_use\_private\_endpoint](#input\_kms\_use\_private\_endpoint) | Set as true to use the Private endpoint when communicating between cluster and KMS instance. | `bool` | `true` | no |
 | <a name="input_kms_wait_for_apply"></a> [kms\_wait\_for\_apply](#input\_kms\_wait\_for\_apply) | Set true to make terraform wait until KMS is applied to master and it is ready and deployed. Default value is true. | `bool` | `true` | no |
diff --git a/examples/end-to-end-example/main.tf b/examples/end-to-end-example/main.tf
@@ -154,7 +154,7 @@ module "vpc" {
 
 module "observability_instances" {
   source  = "terraform-ibm-modules/observability-instances/ibm"
-  version = "2.14.1"
+  version = "2.18.0"
   providers = {
     logdna.at = logdna.at
     logdna.ld = logdna.ld
@@ -168,7 +168,6 @@ module "observability_instances" {
   cloud_monitoring_plan          = "graduated-tier"
   enable_platform_logs           = false
   enable_platform_metrics        = false
-  cloud_logs_provision           = false
   log_analysis_tags              = var.resource_tags
   cloud_monitoring_tags          = var.resource_tags
 }
@@ -184,7 +183,7 @@ locals {
 
 module "key_protect_all_inclusive" {
   source                    = "terraform-ibm-modules/kms-all-inclusive/ibm"
-  version                   = "4.15.9"
+  version                   = "4.15.13"
   resource_group_id         = module.resource_group.resource_group_id
   region                    = var.region
   key_protect_instance_name = "${var.prefix}-kp"
diff --git a/main.tf b/main.tf
@@ -22,37 +22,39 @@ locals {
 }
 
 module "ocp_base" {
-  source                              = "terraform-ibm-modules/base-ocp-vpc/ibm"
-  version                             = "3.29.3"
-  cluster_name                        = var.cluster_name
-  ocp_version                         = var.ocp_version
-  resource_group_id                   = var.resource_group_id
-  region                              = var.region
-  tags                                = var.cluster_tags
-  access_tags                         = var.access_tags
-  force_delete_storage                = var.force_delete_storage
-  vpc_id                              = var.vpc_id
-  vpc_subnets                         = var.vpc_subnets
-  worker_pools                        = var.worker_pools
-  cluster_ready_when                  = var.cluster_ready_when
-  cos_name                            = var.cos_name
-  existing_cos_id                     = var.existing_cos_id
-  ocp_entitlement                     = var.ocp_entitlement
-  disable_public_endpoint             = var.disable_public_endpoint
-  ignore_worker_pool_size_changes     = var.ignore_worker_pool_size_changes
-  attach_ibm_managed_security_group   = var.attach_ibm_managed_security_group
-  custom_security_group_ids           = var.custom_security_group_ids
-  additional_lb_security_group_ids    = var.additional_lb_security_group_ids
-  number_of_lbs                       = var.number_of_lbs
-  additional_vpe_security_group_ids   = var.additional_vpe_security_group_ids
-  kms_config                          = local.kms_config
-  addons                              = var.addons
-  manage_all_addons                   = var.manage_all_addons
-  verify_worker_network_readiness     = var.verify_worker_network_readiness
-  cluster_config_endpoint_type        = var.cluster_config_endpoint_type
-  enable_registry_storage             = var.enable_registry_storage
-  disable_outbound_traffic_protection = var.disable_outbound_traffic_protection
-  operating_system                    = var.operating_system
+  source                                = "terraform-ibm-modules/base-ocp-vpc/ibm"
+  version                               = "3.30.1"
+  cluster_name                          = var.cluster_name
+  ocp_version                           = var.ocp_version
+  resource_group_id                     = var.resource_group_id
+  region                                = var.region
+  tags                                  = var.cluster_tags
+  access_tags                           = var.access_tags
+  force_delete_storage                  = var.force_delete_storage
+  vpc_id                                = var.vpc_id
+  vpc_subnets                           = var.vpc_subnets
+  worker_pools                          = var.worker_pools
+  cluster_ready_when                    = var.cluster_ready_when
+  cos_name                              = var.cos_name
+  existing_cos_id                       = var.existing_cos_id
+  ocp_entitlement                       = var.ocp_entitlement
+  disable_public_endpoint               = var.disable_public_endpoint
+  ignore_worker_pool_size_changes       = var.ignore_worker_pool_size_changes
+  attach_ibm_managed_security_group     = var.attach_ibm_managed_security_group
+  custom_security_group_ids             = var.custom_security_group_ids
+  additional_lb_security_group_ids      = var.additional_lb_security_group_ids
+  number_of_lbs                         = var.number_of_lbs
+  additional_vpe_security_group_ids     = var.additional_vpe_security_group_ids
+  kms_config                            = local.kms_config
+  addons                                = var.addons
+  manage_all_addons                     = var.manage_all_addons
+  verify_worker_network_readiness       = var.verify_worker_network_readiness
+  cluster_config_endpoint_type          = var.cluster_config_endpoint_type
+  enable_registry_storage               = var.enable_registry_storage
+  disable_outbound_traffic_protection   = var.disable_outbound_traffic_protection
+  operating_system                      = var.operating_system
+  import_default_worker_pool_on_create  = var.import_default_worker_pool_on_create
+  allow_default_worker_pool_replacement = var.allow_default_worker_pool_replacement
 }
 
 ##############################################################################
@@ -62,7 +64,7 @@ module "ocp_base" {
 module "observability_agents" {
   count                                    = var.log_analysis_enabled == true || var.cloud_monitoring_enabled == true ? 1 : 0
   source                                   = "terraform-ibm-modules/observability-agents/ibm"
-  version                                  = "1.28.7"
+  version                                  = "1.29.0"
   cluster_id                               = module.ocp_base.cluster_id
   cluster_resource_group_id                = var.resource_group_id
   cluster_config_endpoint_type             = var.cluster_config_endpoint_type
diff --git a/tests/pr_test.go b/tests/pr_test.go
@@ -46,6 +46,10 @@ func setupOptions(t *testing.T, prefix string, terraformVars map[string]interfac
 		ImplicitDestroy: []string{ // Ignore full destroy to speed up tests
 			"module.ocp_all_inclusive.module.observability_agents",
 			"module.ocp_all_inclusive.module.ocp_base.null_resource.confirm_network_healthy",
+			// workaround for the issue https://github.ibm.com/GoldenEye/issues/issues/10743
+			// when the issue is fixed on IKS, so the destruction of default workers pool is correctly managed on provider/clusters service the next two entries should be removed
+			"'module.ocp_all_inclusive.module.ocp_base.ibm_container_vpc_worker_pool.autoscaling_pool[\"default\"]'",
+			"'module.ocp_all_inclusive.module.ocp_base.ibm_container_vpc_worker_pool.pool[\"default\"]'",
 		},
 		ImplicitRequired: false,
 		TerraformVars:    terraformVars,
@@ -84,7 +88,7 @@ func TestRunCompleteExample(t *testing.T) {
 	t.Parallel()
 
 	// This test should always test the latest and the earliest supported OCP versions.
-	versions := []string{"4.12", "4.13", "4.15"}
+	versions := []string{"4.15"}
 	for _, version := range versions {
 		t.Run(version, func(t *testing.T) { testRunComplete(t, version) })
 	}
diff --git a/variables.tf b/variables.tf
@@ -290,6 +290,20 @@ variable "operating_system" {
   }
 }
 
+variable "import_default_worker_pool_on_create" {
+  type        = bool
+  description = "(Advanced users) Whether to handle the default worker pool as a stand-alone ibm_container_vpc_worker_pool resource on cluster creation. Only set to false if you understand the implications of managing the default worker pool as part of the cluster resource. Set to true to import the default worker pool as a separate resource. Set to false to manage the default worker pool as part of the cluster resource."
+  default     = true
+  nullable    = false
+}
+
+variable "allow_default_worker_pool_replacement" {
+  type        = bool
+  description = "(Advanced users) Set to true to allow the module to recreate a default worker pool. Only use in the case where you are getting an error indicating that the default worker pool cannot be replaced on apply. Once the default worker pool is handled as a stand-alone ibm_container_vpc_worker_pool, if you wish to make any change to the default worker pool which requires the re-creation of the default pool set this variable to true."
+  default     = false
+  nullable    = false
+}
+
 ##############################################################################
 # KMS Variables
 ##############################################################################
