`bans` skips maintenance problem at large scale

[Veetaha]

Motivation

In our private repository we have more than 100 duplicate dependencies, and deduplicating them is painful. It is also painful to maintain the deny.toml file with all the bans.skip and bans.skip_tree entries manually.

In fact, even though the bans.skip_tree is intended as a "wildcard skip", but this one hides too many things, and naive developers sometimes put the skip config under skip_tree by mistake therefore making things even worse.

Proposed Solution

cargo deny could generate something like a deny.lock, that locks the list of licences, sources and duplicate crate versions that it found in the repo. That file must be entirely generated by cargo-deny itself thus relieving people from manually maintaining it. Could be generated with something like cargo deny lock update and people could make sure it is up-to-date on CI with cargo deny lock check.

This way, if people add a new dependency that uses some new license or introduces duplicate deps, CI will always fail on cargo deny lock check showing the diff of what changed in the repo's deny.lock - forcing them either to re-review their change to dependencies if an undesired license is added or a new duplicate/source appeared, or accept the diff to the lockfile with cargo deny lock update which will be explicitly visible in the PR's diff.

Comments

We may allow developers to leave comments in the lock file. A crate like toml_edit can parse a TOML file and edit it losslessly while preserving comments.

---

I think that's a brilliant idea. I may try to work on it when I have some spare time (right now I'm a bit busy with some other projects). I know that's a pretty big shift from how licenses/sources/bans are managed now (manually in deny.toml), but I think it's worth it and the lock file feature can be opt-in (at least initially).

@Jake-Shadle I hope to get your approval on this idea. I'm not going to work on it immediately, but just planning for some future (unless someone else would like to work on it before then)

[Jake-Shadle]

I'm fairly skeptical of this. People already don't like having a separate config file, so adding yet another file will just make that perception of clutter/cruft worse, as well as confuse people.

To go back to the motivating reason, it seems like it would be much simpler to just disable the duplicate dependencies check altogether? Over 100 duplicates is pretty severe, and would seem to indicate that there is little value coming from the check, skips are meant to be fairly temporary while waiting on new versions to be release with transitive dependency updates, but that many duplicates would indicate patches lasting months or years.

[Veetaha]

I see what you mean. A separate file may not be then as good as I thought.

indicate patches lasting months or years

In my case - it's just being stuck on the old major versions of some ubiquitous dependencies like syn, structopt (yeah -_-), prost, tonic etc. All of that accumulates many duplicates in the dependency tree. I guess you are right that at this point maintaining this check is more annoying than useful, but that's why I had this idea of auto-maintaining the ignores list (or maybe the idea of something like cargo deny update-ignores that could update the deny.toml directly - not a separate deny.lock)

I think this check is still useful to see an overview of "how fucked we are" in terms of duplicate dependencies. Maybe at the large scale of this problem there could be a more coarse-grained assert like bans.threshold = 100 which tells us that we have at most 100 duplicates in our tree and if we exceed that limit the check starts failing. Or.. maybe the check starts failing if the threshold number differs with the exact amount of duplicates found in the tree - so if a PR somehow influences that amount the developer has to update that number to reflect how the situation changes in the PR diff.

@Jake-Shadle WDYT?