Target plugin for IBMCloud (VSI) Instance Group

[chickenandpork]

This PR brings in a target plugin to scale an IBMCloud "instance group". Consider the following:

(Terraform)

resource "ibm_is_instance_group" "autoscale_group" {
  name = "autoscale-group"

  instance_count    = 1
  instance_template = <some template ID>
  subnets           = [ <subnet.id> <subnet.id> ]
}

NOTE this will create an instance group with an ID such as r006-faceb00c-feed-d00d-beef-123412341234

(nomad-autoscaler config ${NOMAD_TASK_DIR}/config.hcl)

nomad {
  address = "http://{{env "attr.unique.network.ip-address" }}:4646"
}

apm "prometheus" {
  driver = "prometheus"
  config = {
    address = "http://{{ range service "prometheus" }}{{ .Address }}:{{ .Port }}{{ end }}"
  }
}

strategy "target-value" {
  driver = "target-value"
}

target "ibmcloud-ig" {
  driver = "ibmcloud-ig"
  config = {
    api_key = "${ibm_api_key}"  # an IBM API Key: I'm merging mine at deployment
  }
}

(nomad-autoscaler config ${NOMAD_TASK_DIR}/policies/autoscale-group-policy.hcl)

scaling "cluster_policy" {
  enabled = true
  min     = 0
  max     = 3

  policy {
    cooldown            = "2m"
    evaluation_interval = "1m"

    check "utilization" {
      source = "prometheus"
      query = "< a prometheus query yielding a scalar matching the unit of 'strategy.target' below>"
      query_window = "2m"

      strategy "target-value" {
        target = 80
      }
    }

    target "ibmcloud-ig" {
      dry-run             = false
      instance_group_id   = "r006-faceb00c-feed-d00d-beef-123412341234" # instance group ID from above
      node_drain_deadline = "5m"
    }
  }
}

I've tested this using a label as a metric, and witnessed the autoscaler exercising the target to scale an Instance Group up to 20, limited by the max I've set to 3, and then scaling back to 0.

This is currently in test at my employer but I'm contributing this to Hashicorp with supervisor permission. I'm willing to follow this with a contribution to nomad-autoscaler-demos. I can document the process in an article as well.

[hashicorp-cla-app]

All committers have signed the CLA.

[schmichael]

Thanks for the contribution! The error handling doesn't need newlines and could use some rewording in places. Hopefully the comments I left guide you in the right direction.

[chickenandpork]

Thanks @schmichael ! My really bad go code here is a but humbling -- "how did this ever build?" -- thanks for the suggestions. I changed out the identifiers (I was thinking too much like the terraform code) and tried to convert the exception messages similar to your examples with the previous.

Down the road I would like to dedupe some of the code (most of the instanceID, apiKey -> NewVpcV1(...) stuff) but I wanted to share this sooner.

[chickenandpork]

@schmichael changes are made to address all your suggestions -- mostly wholesale accepting your suggestions (thanks) -- is there anything else we need to merge this?

[chickenandpork]

@schmichael while this awaits permission to begin workflows, I'm curious whether you prefer that I rebase locally and force-push, or merge main on top of this PR. Can you advise on your preferred workflow?

[jeffmccollum]

Hello @chickenandpork
I've been testing this out and found the config on the plugin should be api_key not ibm_api_key in your example above.
Otherwise I didn't have any issues scaling up and down in IBM Cloud.

[chickenandpork]

thanks, @jeffmccollum .. I think I originally had ibm_api_key in my version in prod. Nice catch!

I'll fix the description so that it's accurate for the next. [edit: updated description]

Do you know if I should just keep merging updates to this PR until merged, or whether Hashicorp prefers rebase/squash/force-push ? I'd like to keep it as easy to merge as possible when the PR is approved.

[tgross]

Do you know if I should just keep merging updates to this PR until merged, or whether Hashicorp prefers rebase/squash/force-push ? I'd like to keep it as easy to merge as possible when the PR is approved.

We'll squash-merge when we're done anyways. GitHub's review tools are a little easier to use if you don't squash while the review is ongoing.

[jeffmccollum]

@chickenandpork

Looks like plugins/builtin/target/ibmcloud-ig/main.go is missing the copywrite header

// Copyright (c) HashiCorp, Inc.
// SPDX-License-Identifier: MPL-2.0

Also could support for trusted profiles be added as an option?

[chickenandpork]

Hi @jeffmccollum

Looks like plugins/builtin/target/ibmcloud-ig/main.go is missing the copywrite header

// Copyright (c) HashiCorp, Inc.
// SPDX-License-Identifier: MPL-2.0

Strange -- I thought that was checked before, and it was green. taking a look.

Also could support for trusted profiles be added as an option?

Likely, but I'd want to follow this PR with that enhancement. My personal goal is to get this small effort into main so that others can see and leverage, then extend capabilities: minimum-ship before feature-ful.

[chickenandpork]

@jeffmccollum added, but I think the CI needs approval before checking.

[chickenandpork]

@jeffmccollum @schmichael rebased for the 10 commits between my PR and current builds.

fixes https://github.com/hashicorp/nomad-autoscaler/actions/runs/14088268351/job/40258769936

The CI makes any outstanding PR have a recurring cost if we keep up (CI that used to pass can now fail if retried post-dependency update, especially because it enforced tidy-ness, which can become broken by updates while the PR is delayed).

Would you folks be interested in some effort to accelerate the build process?

[chickenandpork]

Easier from the CLI to rebase on main and push. Churn caused due to being blocked on review while #1078 merged

[chickenandpork]

Rebased on CLI to ad8b588 due to churn caused by #1080 merging while this PR is waiting

[chickenandpork]

pushed a change to drop the specific dependency updated in #1042 and trimmed from dependencies in #1069

[chickenandpork]

Rebased on CLI to ce83e63 due to churn caused by #1083, #1084, #1085 merging while this PR is waiting to ensure easy merge on approval.

[Juanadelacuesta]

Hi @chickenandpork! I triggered the test suit to run, but it seems we have a dependency conflict that needs fixing.

[chickenandpork]

Hi @Juanadelacuesta

Thanks for taking a look at this PR :)

Hi @chickenandpork! I triggered the test suit to run, but it seems we have a dependency conflict that needs fixing.

Yeah, looks like #1111 updated a dependency. This has been happening since I created the PR a few months back. merged in main, should be happier.

[Juanadelacuesta]

@chickenandpork The autoscaler is a special case of repo because it includes the codes for 3 internal repos for testing plugins, they are in [nomad-autoscaler/plugins/test/noop-apm](https://github.com/hashicorp/nomad-autoscaler/tree/main/plugins/test/noop-apm), nomad-autoscaler/plugins/test/noop-target andnomad-autoscaler/plugins/test/noop-strategy` respectively. Whenever the high level dependencies are updated, the testing plugins need to be manually updated too and that is why the CI pipeline is failing.

[chickenandpork]

Hi @Juanadelacuesta

The issue was a whitespace error in mitigating the merge against 04f95a9 -- a spacing issue in the go.mod that would have been avoided from the CLI.

@chickenandpork The autoscaler is a special case of repo because it includes the codes for 3 internal repos for testing plugins,

Thanks for the reminder -- I've worked with CI errors in those files on this PR :( . This merge was done through the GitHub UI, and didn't allow my usual shell tricks to catch my human error:

for d in $(find . -name go.mod -exec dirname {} \; )
do
    (cd ${d} && go mod tidy)
done

Sorry :(

[schmichael]

Sorry for the delays! LGTM

[chickenandpork]

Thanks @schmichael :)

I don't seem to have authority to merge, it's possible that @Juanadelacuesta needs to "resolve" the conversation at #1056 (comment) ... maybe that will unblock merge ?

[Edit] Wait, I see they thumbs-upped, I'd take that as resolved, and try to click the "resolve" button myself there.

[Edit] @schmichael I see three other comments that were shown as outdated and I think I addressed in ceb5459 that I have "resolved":

• Target plugin for IBMCloud (VSI) Instance Group #1056 (comment)
• Target plugin for IBMCloud (VSI) Instance Group #1056 (comment)
• Target plugin for IBMCloud (VSI) Instance Group #1056 (comment)

I don't think this enables me to merge though. Maybe waiting on review from hashicorp/github-nomad-core

[tgross]

I don't seem to have authority to merge

Only maintainers do and @schmichael just forgot to push the button I think. It doesn't look like there were any open issues, so I've got this merged and it'll go out in the upcoming release. Thanks!

[schmichael]

Only maintainers do and @schmichael just forgot to push the button I think.
Fact. 🤦 Thanks all.