Consider a provider-specific branching strategy

[t0yv0]

Pulumi providers carry generated files in-source. Consider a dedicated branching strategy that reduces conflicts over generated files and improves contributor experience permitting faster iteration.

          +----------------------+
          | pr/feat-x, pr/feat-y |
          +----------------------+
                |
                |
                | Lightweight fast checks execute immediately on each PR
                v
          +-----------------+
          | merge-candidate |
          +-----------------+
                | Merge Queue checks that derived files can be computed and are correct
                | Slow integration tests only run on merge queue; PRs get kicked out if failing
                |
               \|/
          +-------------+
          |    src      |
          +-------------+
                |
                | Automation recomputes derived files and updates main to be in sync with src
                |
               \|/
          +-------------+
          |    main     |  -- Contains src + derived files (sdk/python, sdk/go, schema etc)
          +-------------+

[t0yv0]

Might also solve #1409 if only maintainers can add PRs to queues, and queues execute tests under a bot user with permission to access our test envs.

[t0yv0]

This assumes that checks executing under a merge queue cannot edit sources as part of said checks. This is the rationale for an extra branch indirection, which is otherwise quite annoying.

[t0yv0]

Summarizing conversation with Bryce.

Objection 1: why not just remove the generared files?

A: Will possibly need a major release so users change location of how Go SDK is consumed; also, for SDKs such as Python and Node it might be beneficial to have them be materialized so GitHub AI trains on the sources.

Objection 2: do you actually see frequent conflicts?

A: indeed repositories even such important ones as AWS remain low-traffic; but we do occasionally see them. We also avoid small changes because of the tooling pain, and are more likely to batch changes instead.

[corymhall]

Objection 1: why not just remove the generared files?

A: Will possibly need a major release so users change location of how Go SDK is consumed; also, for SDKs such as Python and Node it might be beneficial to have them be materialized so GitHub AI trains on the sources.

What if they were generated as part of a release process so that they exist as part of the tag? I think that would also avoid any breaking changes related to Go.

Objection 2: do you actually see frequent conflicts?

Yes I do! I would LOVE to not check in the generated SDKs. Working on the AWS provider v7 was extremely painful when I was trying to stack PRs. Updating one PR in the stack required annoying rebase conflicts in generated SDKs that required re-running the looong build just so I could rebase.

[t0yv0]

What if they were generated as part of a release process so that they exist as part of the tag? I think that would also avoid any breaking changes related to Go.

We could consider doing this. What degrades with this approach is ability to eyeball SDK changes in a PR through the GitHub diff viewer, something that was already quite problematic and did not scale very well. Ideally perhaps we could have custom tooling to understand/summarize these SDK changes. For example, when there are changes of a similar nature to different classes in node, the custom viewer could group them and show only one example.

[t0yv0]

Working on the AWS provider v7 was extremely painful when I was trying to stack PRs

Indeed good observation! Currently PRs do not stack.