Spec discussion: UX for creating Meltano's manifest.json?

[aaronsteers]

We're wanting to build out the Meltano manifest sooner than later, as as an enabling feature for Managed interop:

• Feature proposal: meltano-manifest.json #6876

CLI Proposal

meltano [--environment=ENVIRONMENT] compile [--output=OUTPUT_FILE]

Output proposal

1. If --environment is provided, compile a manifest file that fully describes the target environment. Otherwise, compile a file containing the definition of all environments.
2. If --output is provided, write to the provided file path. Otherwise, save the file to .meltano/compiled/meltano-manifest.json or .meltano/compiles/meltano-{ENVIRONMENT}-manifest.json.

Examples

# create a full manifest containing all environments, in the default location: `.meltano/compiled/meltano-manifest.json`
meltano compile

# create a full manifest containing all environments at the requested path: `~/my-stuff/meltano-manifest.json`
meltano compile --output=~/my-stuff/meltano-manifest.json

# create a manifest.json file containing only the 'dev' environment, in the default location: `.meltano/compiled/meltano-dev-manifest.json`
meltano --environment=dev compile

# create a manifest.json file containing only the 'dev' environment at the requested path: `~/my-stuff/meltano-dev-manifest.json`
meltano --environment=dev compile --output=~/my-stuff/meltano-dev-manifest.json

Note:

• Implementations that require assurances of environment definition fidelity should always use the --environment option, and should only call compile from the branch that maps to that environment definition (prod definition pulls from main branch, staging definition from develop branch, etc.). 6944#discussioncomment-4040768.

cc @tayloramurphy, @WillDaSilva

EDIT (2022-11-22): I've updated the above spec to now use a single-file approach to the manifest. Instead of creating one file per manifest by default, the default builds one file with all environment definitions and the --environment option provides a cleaner and safer definition of just one environment at a time. See my warnings about lack of fidelity for the multi-environment manifest file here: https://github.com/meltano/meltano/discussions

[tayloramurphy]

@aaronsteers at first glance this seems ok but a question comes up for me. Does each file get the global / non-environment compilation with it as well, or is that a separate file?

/.meltano/complied/
  global.json
  dev.json
  prod.json

Something like that?

Also, for this first iteration, does it really need to be separate files or can it just be a single one with separate top-level keys? I'm just not clear on what the separate files get us except slightly quicker lookups. Seems like this is a simpler iteration that would give us a bit more flexibility:

/.meltano/complied/
  meltano.json

Otherwise, the CLI spec and UX works for me 👍

[aaronsteers]

@tayloramurphy - This same question came up also in office hours today.

There are benefits to using a single file, but there would be a lot of redundancy.

More important than the redundancy, though, I'm concerned that not having an option for 'isolated environment' manifest files (pardon the word play 😅 ) creates two risks:

• The interop layer (whereever we send the file to) may get confused and read the wrong data.
• Any parsing issues that exist in other-than-the-target enviornment are likely to crash the process entirely.

Warning
Compounding on the above (but also out of scope to fully solve), I want to call out that we can basically never assume that any single invocation of the manifest creation will be correct for all environments. And this is because most environment are going to be mapped to a specific branch. If I'm on a feature branch, my prod environment manifest will differ from what actually is currently prod, and if I'm working from main, then by definition my print of the manifest will not have the correct staging environment definition that is currently available in the develop branch (for instance). To create an accurate representation of the current state of each environment, you'll need to call meltano compile once from each branch you care about, and then merge into a single file just the paired Meltano environment from each branch.

So, using the examples above, I'd merge the prod environment manifest from main branch with my staging environment manifest from the develop branch, etc. And only after merging all of those together could I accurately describe all environments. Any third party tools that uses the full manifest.json from a single branch's output has to know which environment declarations it has to ignore - or we have to send that tool distinct outputs, one per branch/environment mapping.

As relates to Managed, the managed orchestrator can only trust (for code execution purposes) the environment definition that matches the branch it was paired with. If we are running one environment from the develop branch and one from the main branch, we have to make sure that we don't accidentally let the prod definition from develop leak over into prod, and vice versa for the main branch's definition of staging.

Updated proposal

In order to ensure stability and explicit declarations, I think we have to be able to support manifest files that are truly isolated from each other, and do not contain any artifacts other than those relevant to the selected environment.

That said, upon thinking more on this, there may be an alternative implementation we could chose that still has the benefit of being a 'full project definition' while also having a 'safe' option for interop needs:

1. When run with no --environment input, we build everything as a single file.
2. When run with an explicit --environment= input, we build only what is relevant to the selected environment, again as a single file.

Then, the modified proposal would be something like the below:

CLI Proposal

meltano [--environment=ENVIRONMENT] compile [--output=OUTPUT_FILE]

Output proposal

1. If user provides an --output, we write to the provided file path. Otherwise, the the file would be saved to .meltano/compiled/meltano-manifest.json or .meltano/compiles/meltano-{ENVIRONMENT}-manifest.json.
2. If --environment is provided, compile a file with only that environment. Otherwise, compile one file which contains a definition of all environments.

Examples

# create a full manifest containing all environments, in the default location: `.meltano/compiled/meltano-manifest.json`
meltano compile

# create a full manifest containing all environments at the requested path: `~/my-stuff/meltano-manifest.json`
meltano compile --output=~/my-stuff/meltano-manifest.json

# create a manifest.json file containing only the 'dev' environment, in the default location: `.meltano/compiled/meltano-dev-manifest.json`
meltano --environment=dev compile

# create a manifest.json file containing only the 'dev' environment at the requested path: `~/my-stuff/meltano-dev-manifest.json`
meltano --environment=dev compile --output=~/my-stuff/meltano-dev-manifest.json

Notable changes:

1. We take --output-OUTPUT_FILE instead of --output_dir=OUTPUT_DIR, since there now is always only a single file created.
2. The default behavior creates a file that contains everything.
3. Passing an explicit --environment is an execution choice that intentionally provides more stability for deployment/orchestration, with the tradeoff of not 'fully' describing the entire project in all its iterations.

[aaronsteers]

Spec in the discussion header now incorporates these changes 👍

[tayloramurphy]

@aaronsteers this all makes sense and I like the updated spec!

[WillDaSilva]

@aaronsteers What is the value of having a manifest file with all environments defined in it? You've highlighted a key problem with it, and explained how environment-specific manifests are a safe alternative, but I don't see why we should default to creating an unsafe manifest that complicates matters.

I think we could simplify matters while making it safer by using an output directory, and by default output a separate manifest for each environment. Per my comment on the issue we could do away with the environment key within the manifest file - that information/selection should be elevated outside of the file, to the level of selecting which manifest file will be used.

This has performance benefits too. Going forwards we'll want to automatically compile a manifest file as needed. If per-environment manifest files are all that exist, then when an automatic compilation is needed it can be done for whatever the active environment is. If we support both all-environment manifests and single-environment manifests, then:

• It'll be unclear which should be updated and used, if both are present
• We'll have to re-compile for all environments if the all-environment manifest is present, or else ignore it

[WillDaSilva]

@aaronsteers I think the .meltano/compiled directory should be renamed to .meltano/manifests, since compiled describes what we did to obtain the files within, rather than describing what they are.

The file names within the directory can be <environment name>.json, or manifest.<environment name>.json, or meltano-manifest.<environment name>.json - I'm not sure which I prefer.

I am not a fan of meltano-<environment name>-manifest.json, since breaking up the name in the middle makes it more difficult to work with and identify. Doubly so if we end up supporting all-environment manifests which make the environment name option.

[aaronsteers]

I think the .meltano/compiled directory should be renamed to .meltano/manifests, since compiled describes what we did to obtain the files within, rather than describing what they are.

I like it! 👍

Regarding naming, I think I slightly prefer something with meltano-manifest in the text, specifically because I think dbt already uses a file name called manifest.json and putting "meltano" in the name makes it less likely someone will mix them up.

The file names within the directory can be <environment name>.json, or manifest.<environment name>.json, or meltano-manifest.<environment name>.json - I'm not sure which I prefer.

I like all of those.

I also like meltano-manifest.json for the all-environment one and <environment name>.meltano-manifest.json for the environment-specific one.