Variables duplicated on restart of service

[philippreston]

Nomad version

In our production environment:

Nomad v1.9.7
BuildDate 2025-03-11T09:07:15Z
Revision f869597+CHANGES

Also reproducible with latest:

Nomad v1.10.2
BuildDate 2025-06-09T22:00:49Z
Revision df4c764

Operating system and Environment details

Seen running on our Ubuntu 20.04 environment - but also reproducible locally on MacOSX

Issue

Via the UI, when we click Stop Job and then Start Job, in order ro restart a service - we get the following error:

Failed to parse job: unable to parse var content: input.hcl:8,1-12: Attribute redefined; The argument "datacenters" was already set at input.hcl:1,1-12. Each argument may be set only once., and 2 other diagnostic(s)

Reproduction steps

Start nomad:

nomad agent -dev

Submit the example job which has job specification and variables file:

nomad job run -var-file test.vars nginx-restart-test.nomad

Navigate to the job that has been submitted and verify that the job has been running at http://0.0.0.0:4646/ui/jobs/nginx-restart-test@default. Ensure the job is running.

From here:

• Click the Stop Job > Yes, Stop Job to stop the job
• Then click the Start Job > 3, before the job has completed stopping

At this point the following message is displayed:

Clicking Okay and then Start Job lead to the same error. This issue continues until

1. You refresh the page
2. You navigate to the "Definition" Tab, and then back to "Overview" to click Start JOb

Expected Result

Should restart the job

Actual Result

The error is thrown. In reviewing whats happening.

From my analysis this appear to be a bug in the front end handling. On clicking the Start Job button - there are two calls made to the back end:

Firstly a GET /submission?version= which retrieves the following successfully - which is the job and variables:

{
  "Source": "# This is a test for Lume Markets Integration (LMMI)\n#\n# - nginx\n\n######################################################\n# Variable Declarations\n######################################################\n# Datacenters\nvariable \"datacenters\" {\n  type    = list(string)\n  description = \"Datacenters to which this job should be deployed.\"\n}\n\nvariable \"check_restart_limit\" {\n  type = number\n  description = \"Restart task when a health check has failed limit times.\"\n  default = 3\n}\n\nvariable \"check_restart_grace\" {\n  type = string\n  description = \"Duration to wait after a task starts or restarts before checking its health\"\n  default = \"180s\"\n}\n\nvariable \"check_restart_ignore_warnings\" {\n  type = bool\n  description = \"Setting ignore_warnings = true treats a warning status like passing\"\n  default = true\n}\n\n######################################################\n# Service Specific Variable Declarations\n######################################################\nvariable \"nginx_hostname\" {\n  type = string\n  description = \"The hostname on which the UI should be accessed via the load balancer.\"\n  default = \"nginx-dev.prd.lucera.com\"\n}\n\nvariable \"nginx_count\" {\n  type = number\n  description = \"The number of instances of nginx to run\"\n  default = 1\n}\n\nvariable \"nginx_version\" {\n  type = string\n  description = \"The nginx version to run\"\n  default = \"latest\"\n}\n\n######################################################\n# Job Declaration\n######################################################\njob \"nginx-restart-test\" {\n\n  datacenters = var.datacenters\n  type = \"service\"\n\n  group \"nginx\" {\n\n    # Want these redundantly deployed across hosts\n    count = \"${var.nginx_count}\"\n    spread {\n      attribute = \"${attr.unique.hostname}\"\n    }\n\n    network {\n       port \"http\" {\n          static = 8080\n       }\n    }\n\n    task \"nginx\" {\n\n      driver = \"docker\"\n      config {\n        image = \"nginx:${var.nginx_version}\"\n        ports = [\"http\"]\n      }\n      template {\n        data = <<-EOH\nserver {\n    listen       ${NGINX_PORT};\n    server_name  localhost;\n\n    location / {\n        root   /usr/share/nginx/html;\n        index  index.html index.htm;\n    }\n\n    error_page   500 502 503 504  /50x.html;\n    location = /50x.html {\n        root   /usr/share/nginx/html;\n    }\n}\n        EOH\n        destination = \"${NOMAD_TASK_DIR}/templates/default.conf.template\"\n      }\n      env {\n       NGINX_PORT = \"${NOMAD_PORT_http}\"\n       NGINX_ENVSUBST_TEMPLATE_DIR = \"${NOMAD_TASK_DIR}/templates\"\n      }\n    }\n  }\n}\n",
  "Format": "hcl2",
  "VariableFlags": null,
  "Variables": "datacenters = [\n  \"dc1\"\n]\n\nnginx_count = 1\nnginx_version = \"latest\"\n\n",
  "Namespace": "default",
  "JobID": "nginx-restart-test",
  "Version": 3,
  "JobModifyIndex": 85
}

Then there is a parse call which submits the following:

{
  "JobHCL": "# This is a test for Lume Markets Integration (LMMI)\n#\n# - nginx\n\n######################################################\n# Variable Declarations\n######################################################\n# Datacenters\nvariable \"datacenters\" {\n  type    = list(string)\n  description = \"Datacenters to which this job should be deployed.\"\n}\n\nvariable \"check_restart_limit\" {\n  type = number\n  description = \"Restart task when a health check has failed limit times.\"\n  default = 3\n}\n\nvariable \"check_restart_grace\" {\n  type = string\n  description = \"Duration to wait after a task starts or restarts before checking its health\"\n  default = \"180s\"\n}\n\nvariable \"check_restart_ignore_warnings\" {\n  type = bool\n  description = \"Setting ignore_warnings = true treats a warning status like passing\"\n  default = true\n}\n\n######################################################\n# Service Specific Variable Declarations\n######################################################\nvariable \"nginx_hostname\" {\n  type = string\n  description = \"The hostname on which the UI should be accessed via the load balancer.\"\n  default = \"nginx-dev.prd.lucera.com\"\n}\n\nvariable \"nginx_count\" {\n  type = number\n  description = \"The number of instances of nginx to run\"\n  default = 1\n}\n\nvariable \"nginx_version\" {\n  type = string\n  description = \"The nginx version to run\"\n  default = \"latest\"\n}\n\n######################################################\n# Job Declaration\n######################################################\njob \"nginx-restart-test\" {\n\n  datacenters = var.datacenters\n  type = \"service\"\n\n  group \"nginx\" {\n\n    # Want these redundantly deployed across hosts\n    count = \"${var.nginx_count}\"\n    spread {\n      attribute = \"${attr.unique.hostname}\"\n    }\n\n    network {\n       port \"http\" {\n          static = 8080\n       }\n    }\n\n    task \"nginx\" {\n\n      driver = \"docker\"\n      config {\n        image = \"nginx:${var.nginx_version}\"\n        ports = [\"http\"]\n      }\n      template {\n        data = <<-EOH\nserver {\n    listen       ${NGINX_PORT};\n    server_name  localhost;\n\n    location / {\n        root   /usr/share/nginx/html;\n        index  index.html index.htm;\n    }\n\n    error_page   500 502 503 504  /50x.html;\n    location = /50x.html {\n        root   /usr/share/nginx/html;\n    }\n}\n        EOH\n        destination = \"${NOMAD_TASK_DIR}/templates/default.conf.template\"\n      }\n      env {\n       NGINX_PORT = \"${NOMAD_PORT_http}\"\n       NGINX_ENVSUBST_TEMPLATE_DIR = \"${NOMAD_TASK_DIR}/templates\"\n      }\n    }\n  }\n}\n",
  "Variables": "datacenters = [\n  \"dc1\"\n]\n\nnginx_count = 1\nnginx_version = \"latest\"\n\ndatacenters = [\n  \"dc1\"\n]\n\nnginx_count = 1\nnginx_version = \"latest\"\n\ndatacenters = [\n  \"dc1\"\n]\n\nnginx_count = 1\nnginx_version = \"latest\"\n\n",
  "Canonicalize": true
}

You can see there is duplication on the variables that are submitted.

"Variables":"datacenters = [\n \"dc1\"\n]\n\nnginx_count = 1\nnginx_version = \"latest\"\n\ndatacenters = [\n \"dc1\"\n]\n\nnginx_count = 1\nnginx_version = \"latest\"\n\ndatacenters = [\n \"dc1\"\n]\n\nnginx_count = 1\nnginx_version = \"latest\"\n\n"

Job file (if appropriate)

The job file:

variable "datacenters" {
  type    = list(string)
  description = "Datacenters to which this job should be deployed."
}

variable "check_restart_limit" {
  type = number
  description = "Restart task when a health check has failed limit times."
  default = 3
}

variable "check_restart_grace" {
  type = string
  description = "Duration to wait after a task starts or restarts before checking its health"
  default = "180s"
}

variable "check_restart_ignore_warnings" {
  type = bool
  description = "Setting ignore_warnings = true treats a warning status like passing"
  default = true
}

######################################################
# Service Specific Variable Declarations
######################################################
variable "nginx_hostname" {
  type = string
  description = "The hostname on which the UI should be accessed via the load balancer."
  default = "nginx-dev.prd.lucera.com"
}

variable "nginx_count" {
  type = number
  description = "The number of instances of nginx to run"
  default = 1
}

variable "nginx_version" {
  type = string
  description = "The nginx version to run"
  default = "latest"
}

######################################################
# Job Declaration
######################################################
job "nginx-restart-test" {

  datacenters = var.datacenters
  type = "service"

  group "nginx" {

    # Want these redundantly deployed across hosts
    count = "${var.nginx_count}"
    spread {
      attribute = "${attr.unique.hostname}"
    }

    network {
       port "http" {
          static = 8080
       }
    }

    task "nginx" {

      driver = "docker"
      config {
        image = "nginx:${var.nginx_version}"
        ports = ["http"]
      }
      template {
        data = <<-EOH
server {
    listen       ${NGINX_PORT};
    server_name  localhost;

    location / {
        root   /usr/share/nginx/html;
        index  index.html index.htm;
    }

    error_page   500 502 503 504  /50x.html;
    location = /50x.html {
        root   /usr/share/nginx/html;
    }
}
        EOH
        destination = "${NOMAD_TASK_DIR}/templates/default.conf.template"
      }
      env {
       NGINX_PORT = "${NOMAD_PORT_http}"
       NGINX_ENVSUBST_TEMPLATE_DIR = "${NOMAD_TASK_DIR}/templates"
      }
    }
  }
}

The variables file:

datacenters = [
  "dc1"
]

nginx_count = 1
nginx_version = "latest"

[pkazmierczak]

Hi @philippreston, thanks for reporting the issue. It appears to be a problem with the way the frontend handles the API response indeed, but in my testing this only ever comes up if you start the job immediately after stopping it. In fact I had to do it a few times to reproduce the issue, and only managed to reproduce it once, with starcraft-speed clicking.

It's a bug nonetheless and I'll mark it for investigation.