Add command-line option to display provided name instead of "Combined"

[mark-rushakoff]

I am validating some YAML with conftest test --combine FILE.... But, I am invoking conftest from a script that is executed independently against multiple collections of files, so my output from this script looks like:

PASS - Combined - data.main.deny_foo
PASS - Combined - data.main.deny_bar
PASS - Combined - data.main.deny_foo
PASS - Combined - data.main.deny_bar

Although I could update my script to print the filename as a header, that wouldn't port to other output formats. It would be better if I could provide something like --combined-filename=x, to instead get output like:

PASS - prod1 - data.main.deny_foo
PASS - prod1 - data.main.deny_bar
PASS - prod2 - data.main.deny_foo
PASS - prod2 - data.main.deny_bar

[xchapter7x]

@mark-rushakoff
I see a lot of value in providing more context in the output. It will def provide a nicer user experience.

There are a few contexts which might require different behaviors and some thought will need to be given to what type of UX we'd like to see.

Context A

GIVEN: a yaml file with multiple documents and
   the combined flag set to true
WHEN: a policy outputs results
THEN: we should see the specific file name the result
 is for

Context B

GIVEN: multiple different files (.tf) and
   the combined flag set to true
WHEN: a policy outputs results
THEN: we should see the specific file name the
   results is for

In Context A, we can output the filename no problem, as all of the things we are combining are from the same file.

In Context B, its a little harder. This might be multiple YAMLs or multiple HCLs. In this context we can't always associate a file to a policy. By design this combine feature allows us to compare values across files. If we are comparing the values of multiple files in a given policy then which file will we output? ill use another Gherkin to clarify my thought :)

Context C

GIVEN: multiple different files (.tf) and
   the combined flag set to true
WHEN: a policy compares values across N files
   in our set of files and outputs results
THEN: we should see the specific file name 
  the result is for

that last part we should see the specific file name the result is for raises the question of, how will we know what file the policy is touching in order to show that information in the output?

we can say the behavior becomes:

Option A

THEN: we should output a list of all of the files
   that have been combined

Option B

THEN: we should output the policy that failed
   so a user can see which files are potentially 
   the cause of the issue

Option N:
I'm sure there are more possible behaviors, open to suggestions :)

As previously stated more information makes for a more delightful user experience, so I agree we should explore some change here.

[jpreese]

Ran into this a couple times as well. This issue is also related to configurations that are passed via standard input (results in - rather than Combined). But it could be applicable to any test run.

That said I would think something like: --name, --title, --indicator, --identifier, etc to be able to set what goes in that slot.

[jpreese]

From offline discussions, we are leaning towards --test-id for this feature.