feat: add test to detect public names without a docstring

[gdalle]

Fixes #90

Source:

• Add a function test_undocumented_names which checks whether public names are documented or not (only on Julia 1.11 and later). It does so by verifying that Docs.undocumented_names(module) is either empty or contains only the module name itself.

Tests:

• Add a kwarg test_undocumented_names to test_all
• Add tests with one fully documented module and one partially documented module.

Docs:

• Add docs page about test_undocumented_names

Chores:

• Bump version to v0.9.0 (BREAKING CHANGE)
• Update CHANGELOG.

[codecov]

Codecov Report

Attention: Patch coverage is 83.33333% with 3 lines in your changes missing coverage. Please review.

Project coverage is 75.09%. Comparing base (3e90d04) to head (9c067ba).
Report is 1 commits behind head on master.

Files with missing lines	Patch %	Lines
src/Aqua.jl	25.00%	3 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master     #313      +/-   ##
==========================================
+ Coverage   74.90%   75.09%   +0.19%     
==========================================
  Files          11       12       +1     
  Lines         765      783      +18     
==========================================
+ Hits          573      588      +15     
- Misses        192      195       +3     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[KeithWM]

Just some comments. I am not an experienced enough Julia developer to be entirely confident that everything I say is correct, but I hope you value the suggestions/questions.

[KeithWM]

I would prefer seeing this warning in code with an @warn rather than a comment.

[gdalle]

That means adding a warning for every test below Julia 1.11, not sure people will like that. The fact that it does nothing on 1.10 can already be seen from the fact that the relevant @testset is empty

[KeithWM]

I realize now that this implementation relies heavily on the "new" public keyword, which was not actually in the original issue (as it predated the keyword). Given that a main reason for introducing the public keyword was that previously something being documented was used to show it was part of the API, I think the proposed implementation is a good idea. But I'm not quite sure how to deal with the expectations towards <1.11 users. It could help to change the name of the check to something like test_undocumented_api, test_undocumented_public or something else that include the word "public" without being as verbose as test_undocumented_public_names.

[gdalle]

Given that Docs.undocumented_names is not defined on 1.10, I'm not sure we can do anything meaningful before 1.11 (unless that function is added to Compat.jl)

[fingolfin]

Hmm, let me think loudly about this a bit ....

If we are concerned about people accidentally not running this test because they have their CI configured to run Aqua only with, say, Julia 1.10 (as LTS release), then it would indeed make sense to be more aggressive about flagging this. But I am not sure a @warn is that much more likely to be seen, esp. in a CI runner. If we want to ensure that, we'd ought to raise an error that suggests two options: either run Aqua in Julia >= 1.11, or else (for those of us who run tests in multiple Julia versions, like e.g. most projects I am involved in do) use a workaround, like

... pass the option undocumented_names = VERSION >= v"1.11" to Aqua.test_all.

Alas, we have not done that for other tests. In the end, I don't think this inconvenience (be it a warning or an error) for most people is worth it or necessary: the problem will solve itself over time, either by projects eventually switching to newer Julia versions, at which point they'll get the test; or by projects being ended and thus never getting this, but also never needing it.

That leaves the people who desperately try to get undocumented_names = true to do something but somehow missed that their Julia is to old. For them a @warn might be helpful.

To restrict it to those folks, we could also change the default for undocumented_names to something like this: undocumented_names = VERSION >= v"1.11", and then change test_undocumented_names to raise an error (or at least a warning) if it is called in Julia < 1.11.

This way, anyone using test_all with default arguments gets the test in >= 1.11 and otherwise not, without a warning; but someone who desperately tries to enable the test by passing test_undocumented_names = true will get a helpful message as to why the test is not turning on despite their efforts.

Anyway: I am also perfectly fine with the code as it is.

[gdalle]

I think we can merge this as it is, and then possibly add a warning on 1.10 in a later release.

[ericphanson]

the feature and implementation look good to me, but I think it is breaking to add a new default-on check (both on the principle that it would be very annoying to start failing CI "out of nowhere" and precedent, e.g. #174).

I think this could start off-by-default or just be v0.9.0.

[gdalle]

I think this could start off-by-default or just be v0.9.0.

Either is fine by me. I guess it depends whether there are other pending breaking changes which might benefit from tagging v0.9.0 for this one.

Also I'm worried that if we leave this test off at first we'll never remember to turn it on at the next breaking release.

[ericphanson]

Personally I think doing a breaking release whenever there's a new check is alright, and just have all checks opt-out by default. I'm not an Aqua maintainer, but I've used aqua in a lot of repos with varying levels of activity, and I haven't found breaking releases of Aqua particularly annoying. No other packages (should) depend on Aqua so being on an old release doesn't hold back other packages, and it's fine to just stay on an older release until the next maintenance pass. I think mixing opt-in and opt-out is a bit more confusing and requires more attention for users.

[gdalle]

Tentatively bumping the version to v0.9.0, we'll see what the maintainers think

[KeithWM]

Tentatively bumping the version to v0.9.0, we'll see what the maintainers think

I think the whole point of semver falls to bits if packages start cutting corners on what is and what is not breaking. I'm not sure what I usually see happening in practice, what kind of compat (caret, fixed, free) package developers have with their Aqua (test) dependency. A suggestion might be that Aqua warns users of newer versions, without actually breaking anything. Or that the documentation recommends dependents do not use a compat for Aqua, to avoid Aqua-clearance giving a false sense of security that all is well.

[KeithWM]

I'm happy with this, but I wouldn't take my approval as a sufficient proof that the PR is good.

[KeithWM]

I realize now that this implementation relies heavily on the "new" public keyword, which was not actually in the original issue (as it predated the keyword). Given that a main reason for introducing the public keyword was that previously something being documented was used to show it was part of the API, I think the proposed implementation is a good idea. But I'm not quite sure how to deal with the expectations towards <1.11 users. It could help to change the name of the check to something like test_undocumented_api, test_undocumented_public or something else that include the word "public" without being as verbose as test_undocumented_public_names.

[j-fu]

LGTM

[gdalle]

@lgoettgens would you mind taking a look? I got several reviews but none of them from maintainers I think?

[gdalle]

gentle bump on this one, perhaps @fingolfin would be open to reviewing?

[lgoettgens]

I am very busy right now, but I'll try to squeeze looking at this in sometime the upcoming week or the one after. Sorry for not having time earlier for a proper review

[gdalle]

No worries, I completely understand. As a maintainer I know that things sometimes slip under the radar so I thought I'd bump, but there is no rush.

[ufechner7]

Any update?

[gdalle]

@lgoettgens from my end this is good to go

[fingolfin]

Sorry for the long time.

Needs a rebase in the meantime :-/ but hopefully an easy one.

[fingolfin]

Thank you very much for this PR, and so sorry for not tending to it for so long.

Needs a rebase, and a few things should be clarified in the documentation. Otherwise it seems good.

[fingolfin]

Hmm, let me think loudly about this a bit ....

If we are concerned about people accidentally not running this test because they have their CI configured to run Aqua only with, say, Julia 1.10 (as LTS release), then it would indeed make sense to be more aggressive about flagging this. But I am not sure a @warn is that much more likely to be seen, esp. in a CI runner. If we want to ensure that, we'd ought to raise an error that suggests two options: either run Aqua in Julia >= 1.11, or else (for those of us who run tests in multiple Julia versions, like e.g. most projects I am involved in do) use a workaround, like

... pass the option undocumented_names = VERSION >= v"1.11" to Aqua.test_all.

Alas, we have not done that for other tests. In the end, I don't think this inconvenience (be it a warning or an error) for most people is worth it or necessary: the problem will solve itself over time, either by projects eventually switching to newer Julia versions, at which point they'll get the test; or by projects being ended and thus never getting this, but also never needing it.

That leaves the people who desperately try to get undocumented_names = true to do something but somehow missed that their Julia is to old. For them a @warn might be helpful.

To restrict it to those folks, we could also change the default for undocumented_names to something like this: undocumented_names = VERSION >= v"1.11", and then change test_undocumented_names to raise an error (or at least a warning) if it is called in Julia < 1.11.

This way, anyone using test_all with default arguments gets the test in >= 1.11 and otherwise not, without a warning; but someone who desperately tries to enable the test by passing test_undocumented_names = true will get a helpful message as to why the test is not turning on despite their efforts.

Anyway: I am also perfectly fine with the code as it is.

[fingolfin]

Looks good to me

[gdalle]

Anything more needed here?

[lgoettgens]

I would like to release this as a patch release with this test disabled by default to not disrupt the whole ecosystem, as there are many packages that don't declare a compat bound on Aqua. Thus, I'll apply the few suggestions here and will go ahead with the release later today

[lgoettgens]

When having a final look at this, I stumbled over something:

IMO, this should iterate over submodules as well, but only over submodules that are part of the same package. There should be some helper functions to do that already somewhere in Aqua (e.g.

Aqua.jl/src/exports.jl

Lines 26 to 30 in b7d973f

	walkmodules(m) do x
	for n in names(x)
	isdefined(x, n) || push!(undefined, Symbol(join([fullname(x)...; n], '.')))
	end
	end

)

[gdalle]

done in the latest commit

[gdalle]

I would like to release this as a patch release with this test disabled by default to not disrupt the whole ecosystem, as there are many packages that don't declare a compat bound on Aqua.

What is the difference? One day there will be a breaking release switching this on, and people without a compat bound on Aqua will get surprised in exactly the same way.

Maybe we could add a warning telling people to switch to true for this setting, that way at least they catch it if they look at the test output? Otherwise I'm not sure anyone is actively monitoring release notes for Aqua's patch releases.

[lgoettgens]

This should also accept a broken kwarg, with the same semantics as for e.g. test_undefined_exports

[gdalle]

done in the latest commit