Feature: Apt install command to install all packages at once

[SubaruArai]

See: #940

This should make the installation faster.
If this is successful, we might need to think about other package managers as well.

[cottsay]

The original reason that rosdep behaves in this way is to ensure that packages will still get installed on a best-effort basis despite an issue with one of them.

If the install operation attempts to install 101 packages and one of them doesn't actually exist, the other 100 packages should still be installed prior to exiting. Some package managers (like dnf) support this behavior, so rosdep invokes the installation in a single command. At the time when apt support was implemented, there didn't appear to be a way to achieve that behavior.

[SubaruArai]

The original reason that rosdep behaves in this way is to ensure that packages will still get installed on a best-effort basis despite an issue with one of them.
...
At the time when apt support was implemented, there didn't appear to be a way to achieve that behavior.

Interesting, thanks for the input!
Although I'm not really sure if I like this decision, I can get it.
(I personally prefer all or nothing (atomic) - which isn't achievable since we're using multiple package managers)

Anyways, we should be able to add such a feature with something like this.

I guess the real question will be - do we want to preserve this behavior, or speed-up the installation instead (especially in CI/CD systems)?
According to @tonynajjar, a 24 apt package + 1 pip package installation was reduced from 58s to 28s - which I think is a big deal in the context of CI systems. I also prefer failing fast (it looks like apt-get fails at the lookup stage) rather than figuring out one package went wrong after a while.

@cottsay What do you say? Keep the best-effort behavior, or ditch it for faster execution?

[tonynajjar]

I would really like to keep both features if possible. In the package.xml I also define dependencies on custom packages that are not released, for the purpose of colcon building. So when running rosdep, it fails to fetch these custom packages but still continues with the rest.
As for faster rosdep, it's obvious why we would want that - imagine all the CI time you would save in total across the world 😬

[tonynajjar]

@cottsay did you have the chance to think about it?

[cottsay]

Keep the best-effort behavior, or ditch it for faster execution?

I do not believe that changing the existing best-effort behavior is a good idea, and I can't see myself approving a PR that changes that only to improve performance.

A change to a single-invocation that preserves the established behavior and keeps apt aligned with the other common package managers supported by rosdep would certainly get my attention.

[tonynajjar]

There is already the flag -r Continue installing despite errors, can't we apply the "one-shot" approach only if this flag is on?

[cottsay]

FWIW, that would still change behavior that was established over a decade ago. I don't feel particularly strong one way or the other - maybe you might be able to convince other maintainers though 😅

Have either of you looked at how we might get apt to do what we want? 30 seconds of googling led me to a --ignore-missing flag that might help, though I haven't tried it.

To be clear, rosdep won't (shouldn't) try to install packages that are already installed. The established behavior mostly revolves around incorrect rules. It's pretty common that we'll have a ubuntu: [foo] rule where foo was available in older Ubuntu versions and then isn't available in a newer one. This behaves differently from an explicitly unavailable rule, where -r allows you to continue the rosdep operation despite lacking resolution for one or more keys.

[SubaruArai]

It's been a while and I can't recall what I tried, but I remember that there wasn't a silver bullet - including the --ignore-missing flag.

[130s]

I reviewed the code. The current state of this PR would change rosdep's behavior as already discussed in the review comments section. I +1 for not changing the default behavior, but also personally +1 for adding an optional feature to faster installation.

I'm not a maintainer.

[Ryanf55]

This is also handy if you develop on a hypervisor (common in Azure). Every package install prompts you to restart services.
This means for a new system, I have a ton of prompts to restart services.

oneshot install would mean only one service restart prompt.

[aaron-riact]

this seems nice. can it be taken out of draft now?

[SubaruArai]

@aaron-riact Sry for the half-assed and abandoned pr.

1. At its current state, I cannot take it out of a draft, nor will @cottsay approve due to aforementioned behavior changes.
   (and I agree with him)
2. I think the middle ground will be:
   2.1. Add a new option (e.g. --fail-fast) to rosdep install
   2.2. Implement a faster install method, (while still half-assed, see Apt install one-shot #940 (comment)) to be triggered by above option
   2.3. Implement relevant test codes

There's one thing that I haven't decided though - what should the option's alias be?
I don't like using -f, my instinct would be --force for -f.
A somewhat better (though still ugly imo) proposal would be: -F --fail-fast

What do you all say?

[krisl]

There's one thing that I haven't decided though - what should the option's alias be?

Given this seems most useful for shaving off those precious CI seconds, probably we can afford an awkward long option. How about --single-install-command=apt or similar, with no short alias?

[SubaruArai]

@krisl afaik, it's only apt that has this behavior, due to lacking lacking proper "continue on error" support. Which I am glad with, because imo it shouldn't even exist.
(Found a su link that has some links explaining why --ignore-missing doens't work)

So basically:

1. We shouldn't need that switching argument
2. I made this pr out of frustration on setting up local instances and CI speed, so I personally want a shorthand.
   I guess -F is good enough.

[krisl]

@SubaruArai cool. Will you make the changes needed to add the optional behaviour?
And can someone comment if that change would be sufficient to get this merged?