CI/Documentation philosophies

[icweaver]

Hey folks, this is prompted by an interesting discussion (at least to me) going on over in: JuliaAstro/Spectra.jl#38

Not to editorialize too much, but there seems to be a tale of two workflows:

CI.yml and Documentation.yml

Sometimes spelled a different way, and sometimes only one in place of the other, their differences I still think could be worth discussing, at least for the sake of transparency for new folks.

I think I am currently seeing the following three major camps on this topic, but would appreciate some input:

1. All in one: Package tests, doc tests, doc build, and doc deployment are all rolled into a single, monolithic CI.yml file.
2. Separate workflows, mixed intent: Separate CI.yml and Documentation.yml files exist, but doctests are run under the Documentation workflow
3. All in one, separate intent: Still the monolithic CI.yml file, but doctests are disabled in the doc build and enabled in test CI instead.

And of course there are permutations of the above, combined with Aqua.jl and ExplicitImports.jl, which is another reason I wanted to open this discussion.

At least personally, I was originally leaning towards Camp 2 because we could always just include a separate documentation badge in addition to our usual CI badges to signal that doctests are passing. After these insightful comments from Chris though, I am now thinking that a modified version of Camps 2 and 3 may be a good way forward. In other words, Separate workflows, separate intent: separate CI and doc workflows, with doctests disabled in the doc build, and instead run as part of the test suite.

At the risk of going on a tangent, this is all technically "CI" at the end of the day anyway, no? So wouldn't it make sense to rename CI.yml to something like Test.yml as well? Update: Yep, this looks to be the convention that BestieTemplates follows. Glad I'm not going crazy

Do other folks have thoughts about this? Have I been staring at this for too long? Thanks for coming to my TED talk.