How to generate raw AST from riscv sail files?

[pdn002]

Hello,

I am working on a RISC-V project in a small team and I have been tasked with trying to figure out how to generate an AST from the RISC-V sail files in this repo. Admittedly I am still new to SAIL and learning about what an AST is but I have been told that I should generate an AST file similar to the one in this repo by Derek Tu. I believe the AST file and repo by Derek Tu may have came from this sail-risv repo so I figured this would be a good place to ask. My question is how do I generate a raw AST file using the SAIL compiler?

So far I what I have done is dig through the SAIL documentation and found the flag --ddump-tc-ast for the SAIL compiler. To figure out what sail files need to be compiled, I dug through this repo (sail-riscv) and found the file model/riscv.sail_project which I assume acts like a filelist for all the RISC-V sail files. I also figured that a .sail_project file needs to be run with the --all-modules flag in the SAIL compiler or else none of the modules/files will be parsed. Putting these finding together, my latest attempt to generate an AST was to run the following line.

sail --ddump-tc-ast --all-modules riscv.sail_project > ast.txt

This command ran successfully but the generated file seems to simply be a large concatenated version of all the sail files in riscv.sail_project. It does not look like the AST I am trying to generate.

Is this something that can be done by the SAIL compiler or am I misunderstanding what an AST should be?
I would appreciate some advice on this.

Thank you.

[Timmmm]

There's a little bit of documentation for --ddump-tc-ast here. It seems like it dumps the AST after type checking, and also in Sail source form rather than structured form. So probably not what you want.

There's also --ddump_initial_ast which dumps the AST before type checking but also in pretty-printed Sail source form.

There's also --output-sail which outputs the AST at some other point, but again in pretty-printed Sail source form.

I couldn't actually find anything in Sail that dumps the AST in any structured format, so I'm not sure how they generated that file. It's definitely possible to output the AST in any format you want if you write your own printer for it, but that's quite a lot of work.

@Alasdair would know for sure.

What do you want the AST for just out of interest?

[pdn002]

My group just started on the project but the eventual goal was to parse the generated AST and use in to make modification to the RISC-V GNU toolchain . That way when a programmer makes architecture changes in SAIL, those changes can be easily reflected in some of the GNU tools. This should be similar to the Derek Tu repo that I linked which has functionality to go from SAIL to LLVM. I believe their solution was to modify the SAIL compiler directly in opam but we are trying to avoid that at the moment because no one in my group knows opam well.
Some of my terminology may be wrong since this is new to me but hopefully I got the idea across.

So far from this answer it seems like a possible solution is to write a custom script to make the "raw" AST.

I am also thinking that a possible solution is to just parse the SAIL formatted AST file directly instead of doing this middle step of converting to "raw" AST first. I will have to discuss with my team about this first.

Thank you for the response nonetheless.

[Timmmm]

Ah I see. I think the way most people do stuff like that is to add a new Sail backend. Looks like Derek Tu did that e.g. here for LLVM, and there are also a couple for Pytorch and QEMU. Maybe also worth looking at @ThinkOpenly 's JSON backend. I think it was intended for documentation but maybe it has enough useful stuff for you.

The downsides to making a new Sail backend are that a) you have to learn OCaml (not too hard), and b) you have to learn the Sail compiler internals (really really really hard!).

[Alasdair]

The easiest way to do this is to use some OCaml. I thought it would be easier than it actually is, so I just made a PR to the Sail repository to make it a bit easier, see: rems-project/sail#1497

Once I get that merged, something you can try is dune utop --release from the Sail repository which will open a OCaml REPL with Sail loaded into it (note that utop is a separate package that needs to be installed using opam). Once you are in the REPL, you can do:

open Libsail;;

to open the Sail library, then:

Reporting.opt_warnings := false;;

To disable warnings. That's not strictly necessary, but will cut down on the noise. Finally, you can run:

let ctx, ast, env, effect_info = Frontend.load_project "." ["../repos/sail-riscv/model/riscv.sail_project"];;

to load the sail-riscv model (Changing the directory paths depending on how/where you checked out the repositories). This invokes the compiler frontend to figure out all the module dependencies, load all the source, and parse and type-check everything. From there you can interactively play with the ast object. There are a bunch of helper functions in the Ast_util module. Rather than using the REPL, this can all then be transferred to an OCaml project that imports Libsail and does whatever manipulation/extraction of the syntax tree it wants.

In some ways, I suspect it is easier than you might think, but you will likely run into the issue that we don't have much in the way of documentation so it might be tough going to figure stuff out.

[ThinkOpenly]

As @Timmmm mentioned, I was working on a Sail-to-JSON project with the goal of providing the essential data in the RISC-V in an easily digested format to facilitate downstream uses, including documentation, but also ecosystem projects like toolchains.

For me, at least, the requirements of (1) understanding Sail, (2) understanding the structure of the AST produced by the Sail parser, and (3) having to learn and use OCaml to do it all, were a pretty high bar to clear, especially if you wanted to build an efficient team quickly to do the work. And, there are what I would consider essential pieces of information missing from Sail, and resistance to including them.

I've shifted my efforts from using Sail as the upstream source of information to the RISC-V Unified Database. It's not "perfect", nor complete (yet, but neither is Sail), but it does offer the data as a collection of easily-parsed YAML files. Within the project, there are already backends that use the YAML to generate downstream-consumable artifacts like the "encoding.h" file from the riscv-opcodes project (used in several projects), LLVM content, SystemVerilog content, Go content, a RISC-V simulator, etc.. There is a mentorship underway to add new backends to benefit other projects including (hopefully) more LLVM content, V8, riscv-isa-sim, and hopefully more.

I'm happy to discuss further if you have interest.

[ThinkOpenly]

It may need some work to get caught up with newer Sail and newer RISC-V Sail. It's far from comprehensive. It could serve as a model for walking the AST, though.

If you are only want the JSON artifacts, though, it's going to be easier to start with the YAML in the Unified DB project.

Do you have a good understanding of what data you need? What are your requirements? What do you intend to produce? You mentioned "architecture changes in SAIL [...] can be easily reflected in some of the GNU tools". Do you have any specific targets in mind? binutils? GCC? Any specific content within those projects?

[pdn002]

The title of the project is SAIL to CGEN. For sure we need to use the SAIL compiler to process RISC-V architecture changes. As for what data we need from the output of the SAIL compiler, it does not strictly need to be a specific AST format. I am just exploring AST files at the moment. The target of the project is binutils. Or maybe more accurately the target is to generate new files for CGEN that will update binutils.

[ThinkOpenly]

The title of the project is SAIL to CGEN.

Ah! This is part of the fall RISC-V Mentorship Program, I believe. So, it seems your stuck with Sail. The Sail-to-JSON project, as mentioned, could provide a good model for walking through the Sail-generated AST. It likely already extracts the information you seek if you are targeting binutils, presuming you're looking at encoding/decoding information. At the time (a couple years ago, as Tim said), it was able to extract that information for the vast majority of instructions.

The "latest" JSON artifact can be seen in a downstream project at https://raw.githubusercontent.com/ThinkOpenly/RISC-V_ISA/refs/heads/main/src/ISA.json.

And, this JSON was used as the data source for an ISA reference, which is still visible at https://thinkopenly.github.io/RISC-V_ISA/.

I'm happy to help further if you wish. If you're on the RISC-V International Slack workspace, you can easily find me in channel #team-sailing-downstream.

[pdn002]

Yup, this project is part of the RISC-V Mentorship Program. It seems that the next thing I should try is to explore the SAIL-to-JSON project and see how the extracted information can be used for binutils.

Also just to wrap up this thread, the TLDR for my original question is that the original SAIL compiler does not generate an AST like the "raw" AST found in the Derek Tu's repo. However there may be some alternative for extracting the required information as mentioned above.

If I have any further questions I'll be sure to ask them here, in another thread, or privately.