Code pointing at branches and forks

[pearce8]

We need to propose/test/document/track a workflow that will enable for us to manage multi-repository merges. Let's consider Laghos as an example (but currently we have a similar issue with Kripke and AMG). Several forks are in play:

original repo: https://github.com/CEED/Laghos

fork 1: @wdhawkins forked https://github.com/CEED/Laghos -> https://github.com/wdhawkins/Laghos, and started a caliper branch to add initial implementation. Develop points at the caliper branch in fork 1 via the following code in benchpark/repo/laghos/package.py::

git = "https://github.com/wdhawkins/Laghos.git"
version("develop", branch="caliper") 

• When the branch in the fork gets merged, our laghos tests in CI should fail. We need to know what the error will be and how easy it will be for someone to know what happened. Ideally, an issue should be created for someone to go fix the issue.
• The same issue can occur if our code points at the main repo, but a branch that gets merged. Can we have a utility to look for branches other than master/develop?
• If a branch gets merged without getting deleted, our CI will not fail. Assuming the merge was a result of a PR process, is there a way for us to track PRs in other repos, and know when they get merged?

fork 2: @gracenansamba forked mfem and instrumented it, which is reflected in benchpark develop repo/mfem/package.py::

version("4.4_comm_cali", branch="comm_cali", submodules=False, git="https://github.com/gracenansamba/mfem.git")

In this case, the PR from fork 3 should have as a task a cleanup procedure for when fork 2 + branch are merged, but I chose to go ahead and remove it from develop instead (#628). Similarly I cleaned up the repo/laghos/package.py in develop as well.

fork 3: @gracenansamba forked https://github.com/wdhawkins/Laghos -> https://github.com/gracenansamba/Laghos, and started a comm_cali branch for her instrumentation. Graces branch in PR #316 points at this fork+branch in repo/laghos/package.py::

depends_on("mfem@4.4_comm_cali+caliper", when="+caliper") 

And, repo/mfem/package.py has a new version::

version("4.4_comm_cali", branch="comm_cali", submodules=False, git="https://github.com/gracenansamba/mfem.git")

If fork 3 were tested in CI, hopefully we would get an error if the merge and delete occurred in both cases. Again a solution may be to add a task in #604 to clean up (which I did).

I think linking external repo PRs in Benchpark PRs/issues would let us see their status by hovering over the link fork has a PR at the moment, but I do not know if there is a way to get a notification of such a change - other than CI failures on our end.

Any suggestions on workflow here? I am in favor of not allowing branches/forks in benchpark develop, but sometimes it is necessary to (i.e., pointing at Daryl's branch of laghos). Any other thoughts on branches vs. forks (to enable CI), linking of issues across repos, and what process we should have going forward on this?