Confusing behavior in declaring variables that are set via variable files

[ChrisDek]

Overview of the Issue

After switching to pkr.hcl I cannot use a vars file of any kind with my template.
Previously I used .json and had separated templates and config I do not want in our repo for security reasons.
To be able to use vshpere iso builders new feature of adding a vTPM during build I could only get the newest version of the builder when adding the version in hcl. So I converted a .json to .pkr.hcl. But I cannot find a way to add a variables.pkr.hcl to the template. -var-file does not like the .pkr.hcl. Adding the variables.json also does not work. Also tried variables.auto.pkr.hcl and variables.pkvars.hcl There were older bug that were closed without it being obvious that this is fixed.

Reproduction Steps

Create a .pkr.hcl template. Also add a variables.pkr.hcl

PACKER_LOG=1 packer build .
with template and variables.pkr.hcl in the current folder

Packer version

1.7.10

Simplified Packer Template

declarative
vcenter_network = "qa.lan"
also tried with variable datatype
variable "vcenter_network" {
type = string
default = "qa.lan"
}

Operating system and Environment details

ubuntu 20.04.4 LTS on amd64
running vsphere iso 1.8.3 against a 7.0.2 vCenter

Log Fragments and crash.log files

Error: Unsupported argument

on variables.auto.pkr.hcl line 8:
8: vcenter_network = "qa.lan"

An argument named "vcenter_network" is not expected here.

[nywilken]

Hi @ChrisDek sorry for the confusion here. The documentation around variable files needs to be made a little clearer. I've add the docs label to this issue. In the mean time, it looks like you were very close with the expected filename.

When using a variables file the file must contain the file extension .pkrvars.hcl and get passed into the build command using the var-file flag. For example packer build -var-file=variables.pkrvars.hcl ..

Alternatively you can add the auto.pkrvars.hcl extension to have the variable file auto-loaded upon running
packer build ..

The variable file in your example would be called variables.auto.pkrvars.hcl. Please let me know if renaming the file works as expected for you.

[ChrisDek]

No time for testing today, but I tried variables.auto.pkrvars.hcl but not -var-file=variables.auto.pkrvars.hcl
Will try that first thing on Monday, thanks!

[ChrisDek]

Both approaches do not for for me.
For the variables.auto.pkrvars.hcl approach I have created a file with declarative syntax with lines like

vcenter_datastore = "esxi-050_ds1_r6_SSD"
A packer init . in this folder gives me

Warning: Undefined variable

A "vcenter_datastore" variable was set but was not found in known variables. To
declare variable "vcenter_datastore", place this block in one of your .pkr
files, such as variables.pkr.hcl

Running a packer build . also gives me

This object does not have an attribute named "vcenter_datastore".

Error: Unsupported attribute

on win11_22000.434_Professional_x64_efi_new.pkr.hcl line 19:
(source code not available)

It does look like the variables that are declared and have values set in the variables.auto.pkrvars.hcl are not considered by packer at all

Both errors are thrown for all lines in my variables.auto.pkrvars.hcl

[ChrisDek]

When passing -vars-file=variables.auto.pkrvars.hcl to packer build I

rundeck@QA-Rundeck:/opt/packer-remote/temp$ PACKER_LOG=1 packer build -var-file=variables.auto.pkrvars.hcl win11_copy.json
2022/03/22 09:08:38 [INFO] Packer version: 1.7.10 [go1.17.6 linux amd64]
2022/03/22 09:08:38 [TRACE] discovering plugins in /usr/bin
2022/03/22 09:08:38 [TRACE] discovering plugins in /var/lib/rundeck/.config/packer/plugins
2022/03/22 09:08:38 [DEBUG] Discovered plugin: vsphere_v1.0.3_x5.0_linux_amd64 = /var/lib/rundeck/.config/packer/plugins/packer-plugin-vsphere_v1.0.3_x5.0_linux_amd64
2022/03/22 09:08:38 [INFO] found external [clone iso] builders from vsphere_v1.0.3_x5.0_linux_amd64 plugin
2022/03/22 09:08:38 [INFO] found external [-packer-default-plugin-name- template] post-processors from vsphere_v1.0.3_x5.0_linux_amd64 plugin
2022/03/22 09:08:38 [TRACE] discovering plugins in .
2022/03/22 09:08:38 [INFO] PACKER_CONFIG env var not set; checking the default config file path
2022/03/22 09:08:38 [INFO] PACKER_CONFIG env var set; attempting to open config file: /var/lib/rundeck/.packerconfig
2022/03/22 09:08:38 [WARN] Config file doesn't exist: /var/lib/rundeck/.packerconfig
2022/03/22 09:08:38 [INFO] Setting cache directory: /var/lib/rundeck/.cache/packer
2022/03/22 09:08:38 [INFO] (telemetry) Finalizing.
Error reading variables in 'variables.auto.pkrvars.hcl': invalid character 'v' looking for beginning of value
2022/03/22 09:08:39 waiting for all plugin processes to complete...
when using the declarative syntax
vcenter_datastore = "esxi-050_ds1_r6_SSD"

When using the other syntax in the documentation my file starts with a p

packer { required_plugins { vsphere = { version = ">= 1.0.3" source = "github.com/hashicorp/vsphere" } } }

causing packer to

rundeck@QA-Rundeck:/opt/packer-remote/temp$ PACKER_LOG=1 packer build -var-file=variables.auto.pkrvars.hcl win11_copy.json
2022/03/22 09:11:25 [INFO] Packer version: 1.7.10 [go1.17.6 linux amd64]
2022/03/22 09:11:25 [TRACE] discovering plugins in /usr/bin
2022/03/22 09:11:25 [TRACE] discovering plugins in /var/lib/rundeck/.config/packer/plugins
2022/03/22 09:11:25 [DEBUG] Discovered plugin: vsphere_v1.0.3_x5.0_linux_amd64 = /var/lib/rundeck/.config/packer/plugins/packer-plugin-vsphere_v1.0.3_x5.0_linux_amd64
2022/03/22 09:11:25 [INFO] found external [clone iso] builders from vsphere_v1.0.3_x5.0_linux_amd64 plugin
2022/03/22 09:11:25 [INFO] found external [-packer-default-plugin-name- template] post-processors from vsphere_v1.0.3_x5.0_linux_amd64 plugin
2022/03/22 09:11:25 [TRACE] discovering plugins in .
2022/03/22 09:11:25 [INFO] PACKER_CONFIG env var not set; checking the default config file path
2022/03/22 09:11:25 [INFO] PACKER_CONFIG env var set; attempting to open config file: /var/lib/rundeck/.packerconfig
2022/03/22 09:11:25 [WARN] Config file doesn't exist: /var/lib/rundeck/.packerconfig
2022/03/22 09:11:25 [INFO] Setting cache directory: /var/lib/rundeck/.cache/packer
Error reading variables in 'variables.auto.pkrvars.hcl': invalid character 'p' looking for beginning of value
2022/03/22 09:11:25 [INFO] (telemetry) Finalizing.
2022/03/22 09:11:26 waiting for all plugin processes to complete...

I´m a bit lost here, splitting template and variables worked fine for me in older packer with JSON. I have now temporary created a restricted repository because the monolithic templates contain sensible information about our infrastructure.

Many thanks in advance for any hints

Chris

[Trezamere]

I'm seeing this exact behavior with packer 1.8.0, the whole variable declaration splitting across files with HC2 seems utterly broken. I've tried a dozen variations but in my ideal case I'm trying to define variables with defaults in a variables.pkr.hcl file, loading a specific template file packer build os-2.0.0.pkr.hcl and it completely ignores the variables.pkr.hcl file.

[ChrisDek]

Nearly 2 months now. Can we still hope for a solution?

[ChrisDek]

Just to make sure this is not autoclosed: any news on this issue?

[nywilken]

@ChrisDek thanks for the follow up. If you could share a simple set of example files I could better assist. I suspect that you might be mixing variable file types which is might be causing the problem. I don't need your exact configuration just a simple source.pkr.hcl variables.pkr.hcl and variables.pkrvars.hcl file to get an understanding. Below is an example on how this is supposed to work.

File set in example directory

user in /tmp/example/
~>  tree
.
├── source.pkr.hcl
├── variables.auto.pkrvars.hcl
└── variables.pkr.hcl

0 directories, 3 files

Running a Packer build will auto load the variable values

~>  packer build .
null.example: output will be in this color.

==> null.example: Running local shell script: /var/folders/_9/_q1cqjtd3358p1ldhy7j399m0000gn/T/packer-shell3266726403
    null.example: foobar
    null.example: 2
Build 'null.example' finished after 144 milliseconds 636 microseconds.

==> Wait completed after 144 milliseconds 680 microseconds

==> Builds finished. The artifacts of successful builds are:
--> null.example: Did not export anything. This is the null builder

Renaming variables.auto.pkrvars.hcl to variables.pkrvars.hcl to prevent auto loading. The default values will be used.

~>  mv variables.auto.pkrvars.hcl variables.pkrvars.hcl

~>  packer build .
null.example: output will be in this color.

==> null.example: Running local shell script: /var/folders/_9/_q1cqjtd3358p1ldhy7j399m0000gn/T/packer-shell3852412498
    null.example:
    null.example: 0
Build 'null.example' finished after 96 milliseconds 979 microseconds.

==> Wait completed after 97 milliseconds 20 microseconds

==> Builds finished. The artifacts of successful builds are:
--> null.example: Did not export anything. This is the null builder

Specifying the var-file at run time

~>  packer build -var-file variables.pkrvars.hcl .
null.example: output will be in this color.

==> null.example: Running local shell script: /var/folders/_9/_q1cqjtd3358p1ldhy7j399m0000gn/T/packer-shell2005700087
    null.example: foobar
    null.example: 2
Build 'null.example' finished after 172 milliseconds 938 microseconds.

==> Wait completed after 172 milliseconds 985 microseconds

==> Builds finished. The artifacts of successful builds are:
--> null.example: Did not export anything. This is the null builder

Below is a zip file containing the example templates

example.zip

[nywilken]

To better understand how to update the documentation can you explain how you understood how variable splitting works.

@Trezamere I would love your input here as well.