Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Today, if you try to submit dataflow jobs with a mix of normal pipeline options and dataflow pipeline options, it does not get submitted correctly.
This is because
cfg.command
is quoted, so the final command ends up looking like this after addingdataflow_flags
:Note the leading quote as well as the trailing quote after
output=gs://ttl-30d-us-central2/axlearn/users/remyw/dataflow/wordcount'
This breaks the processing of this command, and all the subsequent
dataflow_flags
are ignored, so it gets run locally instead of on Dataflow.To fix this, we just need to strip the quotes around cfg.command before adding it to our full command.