Clarify build docs and error messages, remove unnecessary file name restrictions

[pwillis-els]

Community Note

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request.

Description

In short, the problems are:

1. The Packer Build docs do not explain how -var-file works or link to any other documentation on supported arguments
2. Error messages from Packer do not explain how to format .hcl formatted -var-file
3. Packer is requiring specific file names for variables files, rather than detecting them from the file itself, making it impossible to keep configuration DRY (re-using the same format configuration file from Terraform)

Here is an example of the problem:

$ packer build -var-file terraform.sh.tfvars ../../../../modules/root/aws-jenkins-cluster/provisioning/ami.pkr.hcl 
Error: Could not guess format of terraform.sh.tfvars

A var file must be suffixed with `.hcl` or `.json`.

It appears that -var-file in Packer is different than -var-file in Terraform, and this is frustrating.

In Terraform, a -var-file can either be a .tfvars-style configuration file, or a .tfvars.json-style configuration file. I do not believe the Terraform files require a specific file extension to detect the correct file type.

According to the Packer build docs, the -var-file option is to "Set template variables from a file". It does not describe what kind of files this option takes, but according to the above error, it is either an hcl file or a json file.

Using an hcl file here doesn't appear to make sense. HCL is a template language, but -var-file is to set template variables. Does this mean the -var-file option is supposed to be passed an HCL file which only contains variable "name" {} blocks? No idea based on the Build docs.

More searching finally reveals this section on HCL template variables. Apparently you can use a .tfvars-style variables file here. But according to the above error, only if the file name is formatted a specific way. This prevents it from re-using an existing Terraform file, even though it's the same file format. The alternative, using JSON for both with a .json extension, is problematic because editing JSON files by hand leads to unexpected syntax errors, whereas a .tfvars-style key-value file is easier to edit without introducing syntax errors.

Composeable command-line tools should not force users into unnecessary restrictions on their use of the tools. It creates UX headaches, adds use-case complexity, and limits the usability of the tools.

Use Case(s)

Sharing one configuration values file for Packer and Terraform.

Potential configuration

ami.pkr.hcl

source "amazon-ebs" "tio_base_amazonlinux2" {
  ami_name      = "foo"
  instance_type = "t2.micro"
  region        = var.aws_region
  subnet_filter {
    filters = {
      "tag:Name": var.private_subnet_name
    }
  }
}

terraform.tfvars

aws_region = "us-east-1"
vpc_id = "vpc-xxxxxxxxxx"
availability_zone = "us-east-1a"
private_subnet_name = "myvpc.subnet.private"

terraform.tf

data "aws_subnet" "private" {
  vpc_id            = var.vpc_id
  availability_zone = var.availability_zone
  filter {
    name   = "tag:Name"
    values = [var.private_subnet_name]
  }
}

Potential References

[tanmay-hc]

@pwillis-els As I can see, the -var-file flag already mentions which file extensions are supported by it, i.e. HCL and JSON.
The HCL file is supposed to have the same format as that of the .tfvars file you are using. Sample files can be found in the repo as well for the pkrvars.hcl/pkrvars.json.
As for the tfvars support in Packer, we currently have no plan to do this, but you can use the same tfvars file, and rename the extension with hcl to use it in Packer.
I am keeping the ticket open in order to have a look at it sometime later.