Fix: parse runtime-rendered fields, extract python env deps from merge_filter #4905
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
While iterating on a separate task locally, I observed that there are a couple of gaps in our "runtime-rendered" property handling logic:
python_env
, and hence failing to render.merge_filter
for variables. This is true for both Python and SQL models. Again, this means that if a variable is referenced inmerge_filter
, it may not be present in thepython_env
of the model and hence we may fail to render it at runtime.This PR addresses both of the issues above and expands our testing coverage to ensure we properly detect the referenced variables in runtime-rendered properties.
I'm not sure yet if this needs a migration script, because if someone references variables in said properties today, then cadence runs would've failed for them anyway if the variables weren't present in the
python_env
. I'd like some eyes on this to double-check my reasoning.