Issue when running a spin image in docker/containerd on RISCV64 #128
Comments
|
@matsbror I haven't run the shim on RISCV64 but can you see if bypassing docker and just using Instead of transporting the Spin app in a scratch container, push it to a registry after running a spin registry push ttl.sh/spin-wasm-app:48hThen try sudo ctr image pull ttl.sh/spin-wasm-app:48h
sudo ctr run --rm --net-host --runtime io.containerd.spin.v2 ttl.sh/spin-wasm-app:48h myapp bogus-arg
# output should look similar to a `spin up`You should now be able to interact with the app in the same way as after a curl localhost:80/yo |
|
Thank you @kate-goldenring ! I did not know how to use I would appreciate further pointers on how to debug. |
|
Here is the log from when starting the following command: I do not really know what to look for or how to interpret it. Here is for comparison the log from a successful execution of the hello-world container: |
|
@matsbror I will look into trying to repro this. For starters, I did notice a discrepancy in the platform. The docs seem to indicate it should be |
The Dockerfile should look like this: FROM scratch
COPY spin.toml /spin.toml
COPY target/wasm32-wasi/release/yoyo.wasm /target/wasm32-wasi/release/yoyo.wasm
ENTRYPOINT ["/spin.toml"]I pushed an app with that Dockerfile structure here ttl.sh/spin-yoyo-app:48h if you want to try it out with I have not used |
|
I failed pulling this image: But I replicated this structure in my own example, and got the same result as before:
No, eventually I will use k3s (most likely) but that also needs porting to RISCV64 so I thought docker should be a step on the way. |
I'd recommend going ahead and trying K3s, skipping over docker, since K3s uses containerd. That way you can use the Spin OCI format ( I'd still like to keep debugging this in parallel |
|
We reached out to some of the containerd maintainers on the #spinkube CNCF slack and it sounds like Docker Desktop 4.30 has an issue with containerd that should be resolved in the 4.31 release next week. Another option is to downgrade to 4.29 |
|
Thanks, but I am not using docker desktop. I installed the docker engine manually and built it from source. I will give your suggestion with k3s a try. |
|
With respect to the Wasmtime-specific bits here (the panic here coming from Wasmtime itself) I opened a thread on our Zulip to talk a bit more about that. One hypothesis is that there's a syscall filter in play here (e.g docker filtering syscalls) and Wasmtime is using a new risc-v-specific syscall ( |
|
Afonso on our Zulip found a simliar problem with the JVM. Others might recognize the comment there and various possible hooks to pass to Docker or configuration to specify perhaps as a workaround. |
It was a bit premature that I said this worked. I can run the container this way, and also the docker-built container, but I cannot curl it. In another shell: I thought the Edit: I checked the same on an x86-node, with the same result, so I think that the container is running, but for some reason the http request does not go through. |
|
@matsbror I also was not able to hit the endpoint at first, but with a Not Found: kagold@kagold-ThinkPad-X1-Carbon-6th:~$ curl localhost:80/yo
404 page not foundI think it is due to K3s already binding to port 80. Sadly, the port is not configurable right now, but work is underway on this. After deleting the cluster on the node, the request went through. I'd try adding a RISCV node and deploying a spinapp there and see what happens.
|
|
@kate-golcontainerdenring it does work when I stop k3s agent service. But k3s (and nothing else) does not listen to port 80 according to netstat, so I do not understand why it should make a difference. But how can I make a containerized spin app use another port. I do not understand how it uses port 80 when the default spin app port is 3000. Can you please explain? EDIT: I just looked into the source code of the shim and found that the port 80 is defined here. I will need to figure out how to pass a new IP and port to the shim, or recompile the shim with another port default. |
|
@matsbror I was also a little mystified when i didn't see the port in use by k3s but still saw the outcome of stopping the service. This is due to traefik using port 80 -- see this forum. I confirmed this with Yes, the port is not configurable currently. We have an issue tracking how we could add it through a container arg here #52. Right now, as you mention, you would have to recompile the shim with another port. |
|
@kate-goldenring an update on my part here. I have verified that I can run wasm images directly using containerd (both the bundled with k3s and the latest running as a service). However, when I want to start it from k3s, I get the following error in the logs: I tried to look in the shim code for this error message but did not find it. Any ideas? |
|
@matsbror, it looks like the shim may not be configured properly on that node. How did you install the shim on it? With kwasm? Note, that k3s has it's own separate configuration for containerd that lives at To debug the shim -- if it is even being executed -- we will need to increase the log level of the shim. By default, the log level of the shim is only error messages. To increase the shim log level, you have to set systemctl stop k3s
systemctl edit k3sAt the top of the file, add the environment variable to the service: [Service]
Environment="RUST_LOG=debug"Finally, restart the k3s service: systemctl restart k3s |
|
Thanks! Configuration of containerd is right, it ends with: I get no more information than before with the debug information of the shim so I suspect the shim is never called (although the comment about serde seems to come from a Rust program, but I just learned that Go also has a serde package). I restarted k3s-agent with I think the issue might be in k3s so I will rebuild it for RISCV. Right now I am taking the binaries for it here: https://github.com/CARV-ICS-FORTH/kubernetes-riscv64 |
|
@matsbror just an update that you can now configure the port that the shim runs on by setting Here is an example (you'll want to replace the image ref as it will be expired): kubectl apply -f - <<EOF
apiVersion: apps/v1
kind: Deployment
metadata:
name: spin-test
spec:
replicas: 1
selector:
matchLabels:
app: spin-test
template:
metadata:
labels:
app: spin-test
spec:
runtimeClassName: wasmtime-spin-v2
containers:
- name: spin-test
image: ttl.sh/spin-return-envs:48h
command: ["/"]
ports:
- containerPort: 82
name: http-app
env:
- name: SPIN_HTTP_LISTEN_ADDR
value: "0.0.0.0:82"
EOF
kubectl apply -f - <<EOF
apiVersion: v1
kind: Service
metadata:
name: spin-test
spec:
ports:
- protocol: TCP
port: 82
targetPort: http-app
selector:
app: spin-test
EOF |
I am trying to get containerd (through docker) to run wasm-files on my VisionFive 2 board which has a RISCV64 processor.
Here are the platform details:
I am trying to run the following example: https://github.com/nigelpoulton/dockercon2023-wasm-lab, which is a simple spin application.
I have verified that docker works as I can run natively build docker images.
I have tried the following command, with bad result:
That exact command worked well on both an ARM64 as well as an x86_64 machine.
Docker version 26.1.3, containerd version v2.0.0-rc
I tried running it directly with containerd like this:
This command exited without any error, but the code did not run. It, however left a container that I can see with
sudo ctr c lsThe spin shim was built from a customised fork of the repo where I refer to a local variant of spin where I backported a fix of spin from the master HEAD to v2.5.1 which is referred to in Cargo.toml. I also had to remove trigger-sqs and trigger-command in
engine.rsas it would not build with them on RISCV64.My fork and version of the shim
My spin
Please help me get it to run on RISCV64!
I should note that I can run the program in spin without any issue (with the memory pool fix).
EDIT: On x86, I need to set the platform to
wasi/wasmwhile on RISCV64 and ARM64, if I do that I get the following error:The text was updated successfully, but these errors were encountered: