Verify --data differs from documented and opa behaviour

[mykter]

The docs say that data in a folder "exceptions/a.json" will be exposed as "data.exceptions".

The example, which is the tested behaviour, shows that it is actually available directly under "data".

# check that the original test works
~/tmp/conftest/examples/data(master)$ conftest verify --data exclusions/services.yaml 

1 test, 1 passed, 0 warnings, 0 failures, 0 exceptions

# move the data file, which should change its import path
~/tmp/conftest/examples/data(master)$ mv exclusions/services.yaml a.yaml
# yet the test still works
~/tmp/conftest/examples/data(master)$ conftest verify --data a.yaml

1 test, 1 passed, 0 warnings, 0 failures, 0 exceptions

I think the test logic is inverted - it is checking that running conftest test gives an error, but it should be checking that it doesn't:

~/tmp/conftest(master)$ conftest test -p examples/data/policy -d examples/data/exclusions examples/data/service.yaml
FAIL - examples/data/service.yaml - main - Cannot expose one of the following ports on a LoadBalancer [22]

1 test, 0 passed, 0 warnings, 1 failure, 0 exceptions

Am I right in thinking that opa test --bundle <path> should be equivalent to conftest verify --policy <path> --data <path> ? My tests are working using this opa command at any rate!

[jpreese]

@mykter I think this has to do with 22 not being in quotes. Once the quotes are added, the result fails:

test_services_not_denied {
  deny["Cannot expose one of the following ports on a LoadBalancer [22]"] with input as {"kind": "Service", "metadata": { "name": "sample" }, "spec": { "type": "LoadBalancer", "ports": [{ "port":  "22" }]}}
}

The test does indeed need to be updated

[mykter]

(just in case it got lost in the discussion of the test: the important part of this report is the incorrect behaviour of verify --data)

[jpreese]

I added #536 to hopefully clarify the behavior. Looks like the docs on the website showcase the current behavior: https://www.conftest.dev/options/#-data

At the moment the folder itself doesn't come into play, it's more on the structure of the data. If this differs from OPA behavior it would probably be worth looking into. I personally haven't used the --bundle flag from the OPA binary that much.

Do you have an example OPA and Conftest command that differs so I can compare the behavior? Everytime I try and add multiple bundles I'm seeing a root conflict error.

[mykter]

Here's a fairly minimal example of the difference in behaviour of opa's bundles vs conftest's --data.

$ cat a/dir/data.json
{"v":1}

$ cat a/p.rego

package main

import data.dir
deny[msg] {
        not dir.v == 1
        msg := "couldn't load data via dir"
}

import data.v
deny[msg] {
        not v == 1
        msg := "couldn't load data via v"
}

$ conftest test --policy=a --data=a <(echo 1)
FAIL - /dev/fd/63 - main - couldn't load data via dir

2 tests, 1 passed, 0 warnings, 1 failure, 0 exceptions

$  opa -b a eval data.main.deny
{
  "result": [
    {
      "expressions": [
        {
          "value": [
            "couldn't load data via v"
          ],
          "text": "data.main.deny",
          "location": {
            "row": 1,
            "col": 1
          }
        }
      ]
    }
  ]
}

[jpreese]

To represent what you're trying to do in Conftest, it would be: conftest test a/dir/data.json -p a

Typically you'll pass in the configuration that you're trying to test (data.json) and tell conftest the policy directory. With opa eval you're telling OPA to evaluate data.main.deny which conftest does by default.

[mykter]

Ah sorry I wasn't clear, the data.json is meant to be an example of static data that the policy refers to, as opposed to the input document. I just didn't refer to "input" or have an input doc to conftest for simplicity. The intent was to show that in conftest the first rule matches and the second doesn't, with opa it's the other way round.

[jpreese]

Yeah I see, the --data for Conftest will take the data files passed in and make them accessible based on their keys. So in this case, {"v":1} would be accessible from data.v not data.dir.

If the desire is to have it indexed based on the directory, I'd need to think through it some more as I haven't seen many questions or issues come up about conftest's data parameter.

[jpreese]

@anderseknert are you familiar with OPA and using bundles? There does appear to be a difference with how Conftest and OPA handle external data, but I'm not familiar with them enough to know if the difference is warranted or if making these behaviors consistent is something we should do.

[anderseknert]

Yeah, OPA will load any JSON/YAML data (obviously not policies as those have a package defined) on the (relative) path under the which it was read. The only difference I know of in behavior as far as --bundle is concerned here is that only files named data.json/yaml will be considered in that case.

Whether it makes sense to preserve this behavior for conftest verify you're probably better equipped to tell, but I helped (or well, half-failed to help) another user with this exact issue on Slack the other day, so either the behavior should change or the docs should.

[jpreese]

Thanks! The documentation has been updated to reflect reality. I'll play around with --data and --bundle and see if it makes sense to change. The fear being that I imagine it would be a breaking change, and want to make sure we're actually solving a problem and not a symptom of poor docs 😅

[anderseknert]

Yeah, I guess a sensible middle ground could be to keep current behavoir while introducing the --bundle flag for conftest too, and have that act exactly as it does for OPA? There are already plans/WIP to push more for bundle based policy and data packaging in the OPA docs and so on, so it would make it easier for user to jump seamlessly between the two tools.

[jpreese]

Yeah, I guess a sensible middle ground could be to keep current behavoir while introducing the --bundle flag for conftest too, and have that act exactly as it does for OPA? There are already plans/WIP to push more for bundle based policy and data packaging in the OPA docs and so on, so it would make it easier for user to jump seamlessly between the two tools.

That's honestly not a bad idea either. Love it.

Then come v1.0.0 we could consider removing --data if --bundle does everything we want it to.