remoting API

[hexfusion]

As the popularity of bootc grows a central location for go types/examples would be useful for projects. Speaking with @cgwalters he is willing to consider something similar to rpmostree-client-go[1] for bootc but there is some intersection with [2].

Curious if instead of creating go client as a os exec wrapper, exposing some of the Rust functionality via CGO? Please include me in future conversations on the topic if possible 🙏 Wasmtime CGO examples as ref. [3],[4]. I understand that CGO is not ideal but in terms of interaction with Rust over FFI, I don't believe there is another option.

[1] https://github.com/coreos/rpmostree-client-go/
[2] #518
[3] https://github.com/bytecodealliance/wasmtime/blob/b869b66bce8a57c9e83db4e3bcd6809d88e94a5f/crates/c-api/src/types/extern.rs#L39
[4] https://github.com/bytecodealliance/wasmtime-go/blob/cabc733f95623fb897b2c35266124192ffca94b4/externtype.go#L9C12-L9C29

[cgwalters]

Curious if instead of creating go client as a os exec wrapper, exposing some of the Rust functionality via CGO

My experience with CGO has not been great. We don't have strong high performance needs here right? An IPC framework of any form is almost certainly going to be dramatically safer and easier to maintain.

I would start with a simple "stable CLI with --json output" as the lowest common denominator to start, which is almost what we have - we just need to stabilize the CLI and json schema.

But, if we needed to go to IPC, my preferences would be one of:

• https://capnproto.org/
• https://varlink.org/

[cgwalters]

This issue also intersects with #2

[cgwalters]

I've added this to the 1.2 milestone (next month) as it's pretty important for coreos/rpm-ostree#4994

As of currently...I am slightly leaning towards varlink.

[cgwalters]

OK I'm digging in a bit more to varlink. It does seem like a relatively decent fit conceptually with our current daemonless architecture. But it's pretty obscure now outside of systemd. Doing a code search for Rust users turns up almost nothing, although it is notable that the SerpentOS folks are doing a rewrite in Rust...a lot going on in those repositories, we should probably talk to them at some point.

One hit I dug into there actually ended up in them removing varlink - as podman also did famously, although they had the constraint of needing to support the Docker HTTP API.

I think the main thing we'll want for the install and update phases is basically:

• check-for-update size
• progress reporting (e.g. image download details at a byte level)
• structured errors

This all seems relatively straightforward to do via varlink (though I gotta say while I understand the choice of JSON, not trying to handle the 64 bit integer problem there is just...ugh. Yes in reality, most of our data sizes should be within safeint).

---

Of the other options here (gRPC, capnproto) I guess lately I'd lean more a bit to gRPC just because it's a good bit more widely used and very well maintained.

[cgwalters]

I looked a bit more at users of gRPC over Unix domain socket (what we definitely want here) and came upon hyperium/h2#487 which is also pretty telling (in the opposite direction, most gRPC users are over TCP).

@sashagrunert any opinions on this?

[saschagrunert]

hyperium/h2#487 is still unresolved and I personally think that's an issue. If using TCP is an alternative then gRPC seems more mature than Cap’n Proto. We still made good results using Cap’n Proto in conmon-rs, though. :)

[cgwalters]

We definitely don't want TCP here, it opens up huge questions around authorization, firewalling, lifecycle binding etc. even when operating just on loopback.

I like Cap'n Proto - they promote the speed thing but I think the fact that it's actually capability based (ref "security" on this page https://capnproto.org/rpc.html ) - I don't think that's true of gRPC and it's definitely not true of varlink. But the flip side is capnp is very complex (on the wire format and implementation).

I dunno...

[antheas]

In #921 I introduce progress reporting via jsonl.

Essentially, JSONL is a protocol used in the NLP field to store large amounts of data in text files that can then be streamed.

The protocol consists of JSON objects, separated by new line characters.

In the PR, the caller provides a file descriptor that is used to stream little json packets that contain progress information. Then, those packets can be parsed by the caller using jq or the json parser of their choice to convert it into a progress bar. The format of the packets would be custom for bootc, although due to no tooling required for it it would be a low effort implementation for most callers.

The primary applications for this would be bash scripts and daemons that want to hook into bootc to provide basic update/installer functionality by calling it directly.

The non-ideal part of the PR is that due to the fact that bootc writes to both stdout and stderr indiscriminately, the 0 and 1 file descriptors cannot be used for this. Moreover, rust does not provide a way to change the stdout and stderr fds. This to me suggests that that a formatter lib should be used in lieu of println and eprintln to allow such indirection, but it is too late for that.

I would not consider this little format as the end all be all, but moreso as something to provide stopgap functionality until the bootc implementation matures. Even then, since such format introduces no dependencies and its implementation is tiny, it can stay there forever without introducing a maintenance burden.

[antheas]

I am not sure about embedding HTTP directly to bootc. Although it is something I did for hhd.

For me, it was a strategic decision to allow binding react apps to it directly without a backend glue and to speedup, simplify the implementation of a Decky plugin and a local webapp that later became an appimage that is still used today. We got it running in two months. The main rival application to hhd used dbus, and still does not have a mature frontend application almost a year later. So it paid off for me.

HTTP has introduced some authentication woes for me that are not ideal. To allow frontends to consume it directly I also use a port for this, which requires HTTP auth, and I had to roll a semi-custom http server. As my implementation is more mature now, I have been thinking about offering an alternate IPC API, but I find myself asking the same question as this thread.

I would point to the fact that consuming it as a unix socket pretty much detracts 90% of its value, as you cant bind to it directly from a frontend and would need someone to rebroadcast it.

Something I'd suggest is regardless of the protocol used for this, the protocol itself should be "REST-like" so that it is easy to consume by a rootless process and rebroadcast to a TCP port, so that it can be consumed by javascript based frontends. Rootless being key to security here.

[antheas]

https://github.com/varlink/python/graphs/code-frequency does not inspire much confidence to me. gRPC looks more mature on the surface https://github.com/grpc/grpc but also more complex to use.

[cgwalters]

Right, bootc will never itself serve HTTP. Tools like Cockpit exist and are highly mature ways to bridge from a local Unix to a remote browser frontend. This issue is about how tools exactly like the cockpit bridge talk to bootc, not about any remote networking.

https://github.com/varlink/python/graphs/code-frequency does not inspire much confidence to me. gRPC looks more mature on the surface https://github.com/grpc/grpc but also more complex to use.

Yeah...agreed.

[cgwalters]

FWIW I dug slightly more into the Rust varlink, and hit varlink/rust#106 which isn't a showstopper...and just reading the code it is generally good code and overall well designed just yeah, not very actively maintained or contributed to.

As far as consuming it on the client side from some Python (or bash), what may actually work OK is using the varlinkctl client shipped by systemd and just parsing the output of that maybe?

Anyways...dunno. So short term I guess it's now looking like standardizing JSON output from our CLI and jsonl for progress, and if we feel pressure to do more then we can jump on either varlink or grpc.

[antheas]

I will look a bit more into the PR over this week but I was hit with a tight deadline for Saturday today.

I think it is ok to stick with --json-fd=<fd> for now, but long term it would be better for bootc to start using a logging library and pipe messages over stderr, so that stdout can be dedicated to machine output. After that, --json would be equivalent to --json-fd=0 and that param would be deprecated.

If that happens, bonus points if you pass the logging library through indicatif before stderr, so that progress bars work correctly when logging happens. That's how I did it for rechunk and tqdm if you need an example.

https://docs.rs/indicatif-log-bridge/latest/indicatif_log_bridge/