Explaining Stride's shading compilation and the issues with it

[ykafia]

Stride shader system

Linked issue : #14

Short list of references :

• AST : Abstract syntax tree., a representation of the code as a tree structure that can then be translated into machine code.
• Lexer : A text tokenizer.
• Parser : A tool that will convert a list of tokens into an AST.
• Recursive descent parser : A kind of parser that is implemented with recursive procedures.
• LALR(1) : A kind of parser that will parse a code from left to right, looking 1 token ahead for disambiguation.
• Parser Generators : A kind of parser that takes a grammar as an input and generates a parser, some are as performant as LALR ones, with infinite lookup but the design removes fine grained control over it.
• Irony : C# Parser framework for creating LALR(1) parsers.
• GoldParser : Another library for creating grammars and parsers.
• Spirv : A standard portable shader code representation.
• LLVM : A set of compilers and toolchains for creating frontend for any languages and backend for any architecture

For reference, a comprehensive list of parser libraries in C# can be seen here.

SDSL

SDSL is a language on its own, it is parsed, it has its own grammar and parser (front end).

Lexer/Parser

SDSL is tokenized with GoldParser and parsed with Irony.
Stride includes the full source code of Irony possibly from before 2013.

As a backend, SDSL's AST is translated into other ASTs for either HLSL or GLSL, and after this translation, those ASTs are transformed into shader text files.

In that sense, Shader compilation goes through these 2 routes :

• Desktop : SDSL -> SDSL-AST -> HLSL-AST -> HLSL -> DXIL/SPIRV
• Mobile : SDSL -> SDSL-AST -> HLSL-AST -> HLSL -> GLSL -> DXIL/SPIRV

Shader mixins

SDSL Shaders are organized in mixins that are contained in files and then stored in a ShaderStorage.

When compiling, the shader compiler will go through all the mixins, parse them into their own ASTs and operate/mix them accordingly to create the final AST representation of the shader to compile (as can be seen in this folder of the source code).

Issues and possible Enhancement

On the parser level

Irony is a good fit because we need consistent performances while providing language error handling.
For this purpose, parser generators and recursive descent parsers are either too obscure or not performant enough.

We could probably squeeze some performance out of it by using languages/libraries that are faster in string processing, but i have no benchmarks or enough knowledge to be sure of it.

On the compiler level

Translating from SDSL to HLSL/GLSL in C# is very time consuming, we also have to maintain our own HLSL and GLSL parser.

We can make it more performant by compiling SDSL directly to spirv and using alternative shader translators like spirv-cross or naga.
Those tools are very performant, written in native code but have .NET bindings or can be used as an executable.
With these tools we would take 3 different routes depending our backend :

• Vulkan, OpenGL 4.6+ : SDSL -> SDSL-AST -> SPIRV
• OpenGL prior 4.6 : SDSL -> SDSL-AST -> SPIRV -> GLSL
• Direct3D : SDSL -> SDSL-AST -> SPIRV -> HLSL -> DXIL

Our main issue here is the lack of support for processing SPIRV bytecode, and we have multiple solutions for that :

• Using LLVMSharp for emitting LLVM IR which can then be compiled into SPIRV
• Write the parser in another language (cpp for spirv-cross or rust for naga) to be able to use those libraries
• Write our own SPIRV abstraction by getting some inspiration from some of the work done by the folks who created WGSL (shading language for WGPU, also handled by naga). here's a blog post talking about it, and another project created by google as an abstraction over spirv to help shader translation.

[ykafia]

Updating my comment : More info on LLVM-SPIRV

Going the LLVM way seems more suitable for our needs it seems