Project state

[hegerdes]

I'm preparing a comparison for different CRI runtimes with various levels of isolation. I want firecracker to be part of it BUT it seems like I need to run another containerd service next to upstream (main) one in order to use firecracker-containerd.

The upstream containerd project is currently at v2.2.x, a fresh compiled version from main is v1.7.x. The containerd v2 version has overhauled its plugin-system so plugins are independent of the main service. Meanwhile I can run firecracker via kata-containers.

Can I use the upstream containerd v2 somehow with a shim for firecracker?

So I was wondering if this project is still maintained and needed.

BTW: The Slack Link is not aktive anymore :(

[nautalice]

Ah, we meet again! I came here to ask the same thing.

The README hasn't been updated in 5 years, and there are no releases (just a script to build from source). It only mentions running on AWS metal instances, so can one even run this on other platforms? There are active PRs, but is there an active roadmap? Is this project ready for production use?

[hegerdes]

Ah, we meet again! I came here to ask the same thing.

That's the only reason I still like using GH 😅

Actually I think using kata is the better option. Unfortunately the current version has a bug when using fc. The gust kernel image misses a module for firecracker which prevents communication with the guest agent via the vsocket. So its either a older version or compiling your own guest image with the fix.

Do you know by any chance if kata introduces any considerable overhead compared to directly using firecracker

[nautalice]

@hegerdes I haven't done any performance testing yet. Can you link me to a more detailed description of the fc issue? Not sure I've hit that (but also my distro's containerd version is old, maybe that factors into it). I've noticed my firecracker processes hang around after I've exited/killed the container, so maybe that's a symptom of this issue?

I will say I'm also considering kata + cloud-hypervisor and so far it Just Works™️

[hegerdes]

cloud-hypervisor and QEMU also just work for me. Firecracker is a little more challenging. But I really wanna compare them all.

Not sure if you hitting the same issue, I cant even start a firecracker container, VM starts but no communication to the agent in the VM. My issue is kata-containers/kata-containers#12581. There is an aktive PR that probably fixes that.
Cant say anything about hanging containers yet, I recall some problems like this in older containernd versions with some runtimes. I'm using latest upstream containerd v2.2.x

[Jebrano]

I was having the same issue, and posted about it in the slack channel but to no response, the issue is GLIBC version mismatch, the agent is looking for GLIBC_2.32, but following the steps in the getting started guide will install GLIBC_2.34, I haven't gone around to solve it

[nautalice]

@hegerdes ah nice, my distro installs containerd 1.7.28, so maybe that's the issue. I only see firecracker processes hanging around, not cloud-hypervisor ones.

Have either of you taken a look at gVisor?

[hegerdes]

Have either of you taken a look at gVisor?

@nautalice Yes, I did a CRI benchmark a few years ago with runc, crun, youki and gVisor. gVisor is amazing as a drop in replacement (or alternative) OCI runtime for untrusted code. Near native performance for compute but a noticeable overhead if you do lots of IO. For IO heavy tasks a real hypervisor might even be a better choice since I does not have the go overhead for emulating each syscall.

I now want to redo the benchmark but also include kata with different hypervisors.
The old post is on my website in case you are interested

[Jebrano]

For me I'm doing a research paper on firecracker, so containerd integration was nice to have

[nautalice]

@hegerdes thank you for the blog post! I want to run untrusted code with low latency spin up times, and gVisor is a wild card in that space. Appreciate your insight.