Extract json-from-wast to its own crate
#2247
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Polish it up a bit for publication to crates.io to get used externally as well, for example in Rust projects that want to consume the AST produced here.
This mostly doesn't change the JSON representation or how things are serialized, but this does change the Rust-level representation to make it easier to consume from a Rust crate (less stringly-typed). A few minor adjustments were made and additionally new structures are supported (e.g. component model types)
alexcrichton
added a commit
to alexcrichton/wasmtime
that referenced
this pull request
Jun 23, 2025
This commit is a refactoring of the `wasmtime-wast` crate to use the `json-from-wast` crate added in bytecodealliance/wasm-tools#2247 instead of the `wast` crate directly. The primary motivation for this PR is to make spec tests more suitable for running inside of Miri. There are two primary factors in how Miri is slow with wast tests today: * Compiling WebAssembly modules. These are all embedded throughout `*.wast` files and basically need to be precompiled to run in Miri. * Parsing the `*.wast` script itself. The scripts are generally quite large in the upstream spec test suite and parsing the script requires parsing all modules as well, so this can take quite some time. The goal of this commit was to leverage `json-from-wast` to perform much of the heavy lifting on the host and then have a "cheap" parse step in Miri which deserializes the JSON and runs precompiled `*.cwasm` files. The main change then required of `wasmtime-wast` was to use the JSON AST as its "IR" instead of `wast` directly. The actual transformation of the `wasmtime-wast` crate isn't too bad, mostly just some refactorings here and there. A new `./ci/miri-wast.sh` script is added which first runs on native with `wasmtime wast --precompile-save` and then runs in Miri with `wasmtime wast --precompile-load`. In this manner I was able to get a number of spec test `*.wast` files running in Miri. Unfortunately, though, Miri is still far too slow to run in CI. One major bottleneck is that the `*.json` files are pretty large and take a nontrivial amount of time to parse. Another issue is that these tests run a fair bit of code which with Miri's ~million-x slowdown. For the first problem I tried using other more-binary serialization formats but the JSON AST is unfortunately not compatible with many of them (e.g. `postcard`, which we use for Wasmtime, is not compatible with the JSON AST's structure in Rust types). For the latter I'm not sure what to do... In the end I figured this might be a reasonable refactoring to go ahead and land anyway. It at least enables running `*.wast` tests in Miri while removing a large part of the runtime slowdown. We can in theory still allow-list some tests to run as they don't all take forever. Some future breakthrough is still going to be required though to run everything through Miri.
pchickey
approved these changes
Jun 23, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This commit extracts the
wasm-tools json-from-wastsubcommand to its own dedicated crate. This namely creates a Rust-typed AST for the JSON encoding which uses types more liberally in various locations instead of stringly-typed things. The intention is to make it easier to consume*.jsonfiles in Rust projects (like Wasmtime using Miri).This started with a goal of not changing the output of any
*.jsontests but in integrating this into Wasmtime I found a few minor mistakes which needed updating anyway so some slight adjustments were made along the way to some JSON bits.