Proposal: Freeze the completion API

[maxlandon]

Disclaimers

First and foremost, I must say that this proposal, any sound solution to it and all the potential
questions and problems raised on the way of finding one, are probably all well above my paygrade.
The enormous reliance and trust of so many projects on cobra can only add weight to my statement.

None of the repositories involved in this proposal are mine nor ones I have made major contributions to,
so I will strive to formulate what I perceive as necessary changes to operate, questions to answer or
debates to open, with the outmost respect and consideration for the authors and contributors of all of
these libraries, for I'm already very grateful to use their code.

Therefore my principal aim and hope about this (as far as I can see it) is at least to gather some
attention around the topic being exposed below. As importantly, I hope that this may at least give
hints or ideas to people, or even answers to some technical aspects, even if these answers are not
favoring any of what follows.

Finally, I will say that I'm still proposing this because in my own and very, very humble view,
the benefits of this proposal ultimately will outweight its costs. But its my guts and my brain.

Introduction

It is pretty much safe to say that cobra is the most widely used and mature CLI library.
As well, that it has been the first library to figure out - and succeedingly implement - how best
to deliver completion code out to users' shell environments: let the tool describe itself, with no
need for more than a couple of shell function builtins to interface with users (in either direction: cobra completion shell snippets don't need many builtins from any shell, and users don't have to do much more than sourcing a script).

The use of __complete has thus proven immensely useful to many, with very few downsides:
The obvious first one is that it has to stick with the initial model and that enhancement in shell completion functionality
and logic would have to be on the shell-side (in other words, that opening the cobra API too much
for the mere sake of completions is very often not a good or acceptable tradeoff). This last point
is one of these cores around which this proposal revolves.

The _git completion file in standard Z-Shell tops 8K LOC. I won't risk myself estimating how much
less it would be had git been written in Go today, and still the benefits would be multiplied by
4 (supported shells for completion). This reminder is not a comparison. On the contrary, it's an
acknowledgement of the fact that things have greatly evolved in the last 10 years in the CLI space,
that some lessons from the past have been learned and new ideas/approaches have succeeded.

More recent iterations

When having to write a console application library for some tool -and after having passed, in this
respect or in others, a significant amount of times burried in completion shell scripts-, I stumbled
upon the carapace completion engine, which some of you may already know.

For those who don't, carapace takes the idea of the __complete command several steps further.
Benefits include - but are not limited to - a support for more shells, more efficient shell scripts
and sourcing, better functionality per line of code (shell and Go), and obviously the luxury of
focusing on one thing (completion), so being free of expanding the related API (all other things
kept equal, obviously).

Having used this engine in several of my projects, across several shell/Go runtimes, I must and will
happily attest of the outmost reliablity of the library, its incredible ease of use and combinatory
power. This engine and its API drastically reduce the feature-set/reliablity to dev-time ratio for
even the most complex completion tasks, turning something that would take (1-hour x 9 shells) shell
script development session into a 5-min job done once and running on all of them.
The laudatum could continue on several other aspects and features. The amount of work put in it is
big with regards to its manpower (1 person).

Not for the sake of advertisement, I must emphasize that I could use both cobra and carapace in
this console closed-loop shell flawlessly, with big a command tree and large functionality, by patching
only ten lines of Go code, which can be removed if one specific issue below is solved.
Two libraries not made for closed loop + a readline shell and a patch = all fun no bugs.

Old programs to new ones: what changed

Consider the following two things:

• There is an entire generation of the-world-running-on-tools with C codebases (git and ffmpeg
  are some examples that I wil use in this section). It is paradoxical that they often have a very
  large and "intimidating" CLI, with various options and arguments whose arguments can be much more
  evolved than what the --help flag suggests (despite all good intents).
• On the other hand and coming decades after this first generation, consider widely used modern CLIs
  like kubectl, helm, etc. We can witness how cobra being an exhaustive CLI library has sometimes
  prompted developers to vastly expand their tools' CLI, because suddenty, building more structured
  and recursive command trees was much easier.

Here again, there is a risk of having an excessively large CLI, doubled by the sometimes not obvious
need to maintain strict backward compatibility for it (especially when invocations are found within
automated, configuration-as-code style scenarios).

My point here is again not to make a side-by-side comparison of tools coming from different eras.
Had cobra existed as a C library, maybe most of the aforementioned tools -or equivalents- would not have been
written the same way. However, I suspect that there is an opportunity not to find oneself in this
paradoxical "discrepancy" between some large programs' toolsets and their capacity to offer this
toolset at various stages of the CLI lifetime with equal power.

Proposal & Suggestions

However scary that might appear, I think my proposal is no more than what this issue title says:
freezing everything in the API that is related to completions, and "pass it into maintenance mode".

That would grossly include:

• No more ShellCompDirective constants added.
• No more shells support added.
• No functions related to flag/arg command completions added.

This does not mean, however:

• Removing support for any of the currently supported shells.
• Removing any of the directives.
• Stopping support for the existing completion shell script code.

Important questions & problems

• Whether or not to integrate carapace code into the cobra module, under a separate directory.
  This is the kind of question that is way, way above my paygrade and stake in this. And I think doing this would have almost no benefit at all, in addition to many people legitimately arguing that they don't want such a dependency. But I might be wrong, and I suspect anyway that some people might ask this to themselves too.
• What mechanism to use so that all calls to the current __complete command can be safely "bridged"
  to carapace if its present/needed, or if the command should be overwritten altogether. This area
  of things will probably expand into a larger set of questions/approaches that I currently don't
  have at mind.
• There are some other questions that I either don't have at mind now, or that I've never had
  (which is a given).

Targeted End State

• Any new comp code going forward for any given software might freely choose between either sticking
  to the legacy cobra completion API because it is enough for their use case/constraints, using the
  carapace library for any new features, or any mix of both with equal support.
• Any user having carapace installed will beneficiate from it whenever he is using a cobra
  program, transparently and without having to know it. Besides, anyone not having carapace
  should see no change from before.

Advantages

• Redirect as many persons interested in completion/hint/whatever support to solve issues on the
  carapace repo, dedicated to this very problem. This is possible not only because the latter is
  quite bug/issue free, but also because using carapace gives an additional layer at which
  problems can be solved. And if really the issue could not be solved at this library level,
  then one could bother the cobra devs for something/hint/fix. See the PR/Issues sections
  below for examples.
• Win-win for everyone: cobra devs can focus on the core library, carapace can take care of at
  least 80% of completions feature/maintenance. Users get both of it at will and at the dose of
  their choosing.
• Obviously, carapace currently has support for 9 shells, and if I'm not mistaken, each and every
  shell supported in cobra does not come close to the quality of carapace. Opiniating again...
• carapace having for core principle of being identical in behavior for all shells to the extent
  that this is possible, it also tends to raise the lowest common denominator for all aspects of
  the program completion/display on all shells. In turn, shells may want to improve themselves
  because they happen to be weighting down on this denominator.
• carapace already has support for exported completions and a restrained but smart set of metadata
  destined to shells for them to consume all this completion output.
• Support for flag types, arguments, positional arguments of all sorts and origins is fantastic.
• carapace enable usage of the OS environment variables, gives access to the entire completion
  context at any point in your code, and much more related to this.

Disadvantages

• carapace is currently a distinct binary, that needs to be installed separately from cobra.
  It currently also requires a sourcing call in users shell for activation, and also naturally one
  for each program being installed that wants to complete itself. 1+(n programs) is still one call
  more than what is currently required. Related to the Important problems section.
  I might balance that disadvantage by opiniating that completion sourcing is much more intuitive
  on the aggregate with carapace, but that's an opinion and nothing else better.

Requirements / Steps

• CompletionOptions might need to be considered/ported/refactored/integrated into carapace.
• Verify compatibility with a representative set of programs: enroll some programs on volunteering ?
• Debug functions/code : maybe not all of it needs to be ported, or could be in more idiomatic way.
• Ensure that the carapace execution workflow is entirely compatible with the current state of
  things, and ensure its API for run/pre-run is sufficiently consistent for long-term compatibility.

Related issues / PRs

Below is a list of the open issues and PRs I have found to cross the proposal at hand at some point.
Some of them raise important concerns that have sometimes not been answered/settled on here, but
where carapace has.

PR/Issues on cobra repo

• Unable to get completion to work in ~/.zshrc #1529: This issue touches on the problem of sourcing shellcomp code into shells, the cost of doing at various points and with various methods, as well as the effects of program binary updates on shells' ability to properly ingest its contents.
• Shell completion does not support the "no space" format of short flags #1629: Note to @rsteube that I've never had this use case so I don't know if this works by default
  in carapace.
• Let the commands store flagComp functions internally (and avoid global state) #2012: This is quite an important one with many regards. First, I'm damn sure there is a way to
  avoid global state for this map, and also pretty sure that the last attempt to do so has made a
  very little mistake that no one seen. What is important here, is that anyone reading this very
  line is very unlikely to realize how much potential is held hostage by this very little problem.
  So. much. potential.
• feat: add getters for flag completions #1943: This is directly related to Let the commands store flagComp functions internally (and avoid global state) #2012, and should be solved conformingly with the latter's fix.
• feat(completion): support no-space completions for short flags #1732: This would be a solved problem.
• Add similar whitespace escape logic to bash v2 completions than in other completions #1743: This one also.
• Implement completion NoMatching #1833: See carapace handling of this with its PrefixMatcher (again, compatible with all shells)
• complete aliases for subcommands #1856: Already solved by carapace on all shells.
• Completion support for Nushell #1857: Completion support for nushell could be redirected at the carapace repo: there is already support for nushell, and it might quicker to port what is missing as compared with this PR.
• [RFC] Redirect bash completion v1 to v2 when possible #1867: Maybe this can be solved within the work of this proposal.
• provide an error message if completion errs in zsh #1922: ActiveHelp is already supported in carapace, and its API for hints/helps is cleaner.
• completers ∀x∈ {cmds, aliases} in powershell_completions.go #1977: Already solved in carapace.

PR/Issues on other repos

• #655: Please see the first answer given by @rsteube to my PR, which pretty much sums up what is
  holding carapace from implementing full "backend/fallback support" for all completions declared
  through the cobra API, which will allow anyone importing carapace in place of / next to the
  completion command to beneficiate from past completion code out of the box.

Conclusion

Go developers are already very lucky to have these existing projects at their disposal, which may
not be said of many other languages and their stacks. All of these projects are going in wonderful
directions and have pretty much all taken smart and rigorous approaches to their own problems,
providing huge impact.

I'm pretty sure that a very little bit of synergy and separation of concerns will enable more
things than most would suspect. In the hope that this will raise some thoughts and initiatives !

[Luap99]

As user and contributor I would not like to see the development stop on the completion current API. I put a lot of time into porting Podman to the cobra completion API and I really do not want to waste time porting hundreds if not thousands of flags to a new API. While you might say it is only deprecated and will stay around what that essentially means is that I will get no new features/bug fixes unless I port all my code to another API.

I don't know how many users carapace has but I am sure there are much more projects using the standard cobra completion API. So IMO the time would be much better spend improving the cobra completion API as this would enable it on much more projects without them having to spend extra time porting code to a different API.

I know it is hard to get stuff review/merged on this repo but I still think that is better then trying to change the thousands of projects using cobra.

[maxlandon]

Thanks for the input @Luap99 !

So as noted above, obviously there is always the possibility to include carapace code in cobra directly.

Deadcode elimination would make most of the code removed anyway for those who don't use it.

Thanks again

[marckhouzam]

Thanks @maxlandon for the detailed proposal. Cobra aims to provide the best user and developer experience and if carapace can help I’m open to discussing it.

I don’t know much about carapace’s usage so I don’t have a clear opinion on your proposal yet. Some questions:

1. Can’t carapace be used with cobra already? Why do we need to do anything different?
2. Why freeze the current API? Is it to avoid maintaining it since carapace is available?
3. Why isn’t this proposal coming from the carapace maintainer?

As a path forward, I would want to avoid requiring any changes in existing programs both for users and developers for the existing feature set. Is it possible, as you hint at, to “bridge” cobra and carapace completely transparently?

[maxlandon]

Hello and thanks for the answer @marckhouzam. Below are the answers to each question.

1 -Can’t carapace be used with cobra already? Why do we need to do anything different?
Carapace only works with Cobra, as its repository description indicates.
There is actually no need to to do anything different, at least in the sense that the whole
"proposal" above would not need any breaking changes. To be clear, I've not said anything
has been done wrong, I'm positing that now that so many pretty flowers have grown, maybe there
is a superb bouquet to do with them. I hope this analogy will speak words...

2 - Why freeze the current API? Is it to avoid maintaining it since carapace is available?
The primary goal is to avoid duplicate maintenance of the same (or partially overlapping) feature set,
thus to avoid not only wasting the time of everyone and ending up with something sub-par on the cobra
repository.

To answer the "why should we freeze the API", again I'm not strictly advocating for freezing it, for
instance if one decides to "merge" carapace code in cobra, the question is not relevant anymore.
On the contrary, if for some reason one decides to leave carapace as a separate repo, then I we are
going back to question 2: how do we avoid duplication and subpar features ?

3 - Why isn’t this proposal coming from the carapace maintainer?
So first, let me link to his answer to this question, which I obviously asked before opening this thread
(I'm educated and don't mind the business of others without their consent). Here is the conversation:
carapace-sh/carapace#655 (comment)

Added to this, I suppose that @rsteube only has so much time a day for this, and since I've had lots
of free time I thought I could give some of it to contribute to the "debate" and bigger picture.

Lastly, and as mentionned in the proposal above, there are a few things that I would be glad to see
solved for the sake of my own libraries, which aim to further enlarge and facilitate the use of cobra
(between others) into closed-loop consoles and various other setups. My selfishness here.
In the end, I must say again that all of these repos are marvelous and immensely useful, smartly maintained.
I'm sure everyone can pick up their flowers to contribute to the bouquet, and everyone will be very happy.

4 - Podman completions, to @Luap99:
I checked your code, and I get why you are one of those who do not want to rewrite a remote path completer
(although you would be very surprised how easy it is to write/port it to carapace). Fortunately, there is
no need for all of that. If you read the proposal a little closer, you will see that it does not require
any breaking changes.

[rsteube]

As a path forward, I would want to avoid requiring any changes
in existing programs both for users and developers for the
existing feature set. Is it possible, as you hint at, to
“bridge” cobra and carapace completely transparently?

With #1943, yes.
I've already got a compatibility layer so carapace completions are also registered in cobra.
Then there is also a bridge to use cobra completions in unsupported shells or for embedding which already contains handling of the directives.
So a mixture of both should be possible with completions being available in both directions.

[maxlandon]

Cool to see you in the loop @rsteube !
So for you and all the others, I will just add that I'm available for anything if needed, just ping ruthlessly.
Looking forward to all of this !

[marckhouzam]

With #1943, yes.

I just merged #1943.

Let's see where this takes us.

[rsteube]

Replaced the --security-opt completion and gave it a spin:

[Luap99]

Replaced the --security-opt completion and gave it a spin:

Thanks, that looks good to me. If it is possible to use both at the same time then I have no problem with it. My concern was basically around having to migrate all at once. If I can selectively port code where I actually want some of your features while keeping the other old API working then I am definitely satisfied.

I haven't yet looked at all the advantages of carapace but it looks promising to me.

[marckhouzam]

The asciinema above does look very nice. I like the colours and the extra shells.

[rsteube]

There's implicit coloring for flags and command groups.
Only elvish, powershell and zsh have support for colored completions though.

Documentation is still a bit lacking so you might want to have a look at carapace-bin for some examples.

[rsteube]

fyi, how the colored highlighting works: https://rsteube.github.io/blog/2023/puking-rainbows.html

[maxlandon]

A couple of things and updates on this:

• Carapace has integrated its cobra compatibility layer, so that all completions registered through carapace will be proposed with the normal cobra completion scripts. The converse is true: any cobra completions will be automatically handled by carapace.
  This should mostly solve the problem of users not having carapace binary installed on their system, which was one of the main caveats of this entire proposal.

• As far as the need for "separation of concerns" can be fulfilled by redirecting most work over on to carapace (and in effect "freezing" work on completions here), how such as strategy could be adopted ?
  It might be as simple as adding a couple paragraphs/hints to the cobra completion documentation/code.

• Finally, regarding the couple of issues that could be solved to this, would you guys appreciate if I try to "merge" or "port" the solutions found in repo "carapace" over to cobra ?
  I think that in order to achieve some freeze (however formal it may be) would be to solve all open issues related to completion. Then, the various questions on the direction to take might be more easily treated.

[rsteube]

Wasn't the plan to continue the development on the internal completions and avoid a premature end of support?

[maxlandon]

Wasn't the plan to continue the development on the internal completions and avoid a premature end of support?

All right I just misunderstood then ! All fine with me obviously, I just felt the need to summarize the conversation a bit !