Tooling for writing Miden programs, libraries, smart contracts in Rust

[greenhat]

Goals

Provide libraries and tools for developers to write, compile, and test:

• Miden VM programs and libraries;
• Miden rollup accounts and note scripts.

Implementation details

miden-vm-sdk crate

This is the crate the developers writing Miden VM programs and libraries in Rust will use. Provides an API and macros to write and test Miden VM programs and libraries. Contains no rollup-specific features.

Miden VM ops not represented in Wasm (asserts, advice provider ops, hash, mtree) could be implemented as Rust functions (sort of intrinsic) with an extern C functions under the hood. The Miden compiler later transforms these function calls into VM ops. Ops parameters could be exposed as high-level Rust types via adapters.

For example, the high-level RPO hash function might look like this:

pub fn rpo_hash(data: &[u8]) -> Hash {
    // ...
}

and is implemented using a low-level intrinsic hmerge function:

extern "C" fn hmerge(left: Felt, right: Felt) -> Felt;

that is converted to hmerge VM op by the compiler.

The crate also provides an API for running and testing the compiled Miden code. For Miden programs, it should be possible to provide inputs and check outputs. For Miden libraries, it should be possible to call functions from the library and check the result.

To support inter-context procedure calls (call and syscall) we can pass arguments that don't fit in the stack via the advice provider. We can generate the marshaling code for these arguments on the caller and the callee sides with macros (using WIT's ABI). This would make the callee only callable via call and syscall. We should consider marking such functions with a special attribute in MASM to convey that they can be called only via call and syscall.

Public inputs and output for Miden programs are provided via the stack before and after the program execution, which is hard to expose in Rust API. We could load them into some reserved global memory and serve on the Rust side with some read write functions with implementation generated by the Miden IR compiler. @bobbinth @frisitano Do we have any alternative way to provide inputs and capture outputs for a Miden program?

Using Miden libraries in the Rust code

To be able to use Miden libraries in the Rust code, we need to know the high-level type signatures of the functions in the library. We could try to leverage WIT and Wasm component model to provide a high-level interface description for the compiled Miden libraries. There is an actively developed ecosystem for WIT, and we could either adopt it as a whole or take pieces that fit us.
See wit-bindgen and WIT format for more details.

The code for calling functions from the Miden library into Rust code might look like this:

// generate extern "C" Rust functions (`awesome_library::some_hash_func`)
miden_bindgen!("path/to/miden/awesome_library.wit");

// call imported functions
let hash = library::some_hash_func(&data);

Where 'awesome_library.wit' is a WIT file is:

package awesome_library:awesome_library

world awesome_library {
  export some_hash_func: func(bytes: list<u8>) -> list<u8>
}

In the Miden compiler, we import the 'awesome_library' module and convert all calls to some_hash_func to the imported function using either exec or call depending on the special marks we put on inter-context callable functions. In the case of call, we also generate marshaling of the non-integer arguments to the advice provider before the call. If we could not find a way to mark inter-context callable functions in WIT, we could write a call macro to call such functions. So such calls would look like call!(sat::account::add_asset(asset)).

cargo-miden cargo extension

Compiles Rust code written using miden-vm-sdk crate to Miden VM programs and libraries.
The following commands are to be implemented:

• compile to compile Rust code (crate) to Miden VM programs and libraries and to Miden rollup accounts and note scripts.
  Here is an example of a command to compile the current crate's binary target to the Miden VM program - cargo miden compile --output app.masm.

miden-sdk crate

High-level SDK for the smart contract developers. Provides an API and macros to develop and test Miden rollup accounts and note scripts.
A developer should be able to:

• Develop account methods, define and access account storage;
• Develop note script that calls account methods;
• Test accounts and note scripts in various scenarios (state mocking, etc.);

The account definition for a wallet might look like this:

// generate extern "C" Rust functions
miden_bindgen!("path/to/miden/sat.wit");


#[miden_account]
pub struct MyWallet 

#[miden_account]
impl MyWallet {
    pub fn recieve_asset(&mut self, asset: Asset) {
        sat::account::add_asset(asset);
    }

    pub fn send_asset(&mut self, asset: Asset, recipient: Address) {
        sat::account::remove_asset(asset);
        sat::tx::create_note(asset, recipient);
    }
}

In the Miden compiler, we import the 'sat' module and convert all calls to its functions to the imported ones with either exec or call depending on the special marks we put on inter-context callable functions. In the case of call, we also generate marshaling of the non-integer arguments to the advice provider before the call.
[miden_account] attribute generates the WIT file for MyWallet, which can be used in the note scripts like this:

// generate extern "C" Rust functions
miden_bindgen!("path/to/miden/sat.wit");
miden_bindgen!("path/to/miden/my_wallet.wit");

#[miden_note]
fn note_script(target_account_id: AccountId) {
    let account_id = sat::account::get_id();
    assert!(account_id == target_account_id);
    let assets = sat::note::get_assets();
    for asset in assets {
        my_wallet::recieve_asset(asset);
    }
}

This crate could be combined with an SDK that constructs transactions and deploys smart contracts.

[bobbinth]

Great write up! Thank you! A few preliminary comments below.

I wonder if we need to separate miden-vm-sdk and miden-sdk. At least initially, it might make sense to do just the rollup-focused miden-sdk as I think this could help us progress faster specifically on the rollup/smart contract front. In the process, we'll probably also figure out how best to split out rollup-independent components from miden-sdk.

For miden-sdk, I'd probably first focus on accounts, and once we have some initial things for accounts working, we can tackle notes. One reason for this is that for account code, we don't really need to handle call's and syscall's, or worry about how to describe contract ABIs.

For exposing MASM libraries to Rust code, I'd prefer to have Rust wrappers (rather than using miden_bindgen). So, for example, an account code could look like:

use miden::{
    account,
    asset::Asset,
    note::Recipient,
    tx,
};

#[miden_account]
pub struct MyWallet;

#[miden_account]
impl MyWallet {
    pub fn recieve_asset(&mut self, asset: Asset) {
        account::add_asset(asset);
    }

    pub fn send_asset(&mut self, asset: Asset, recipient: Recipient) {
        account::remove_asset(asset);
        tx::create_note(asset, recipient);
    }
}

Where miden would be a library which we write (or maybe auto-generate using convention discussed in #25) which under the hood would use "extern C" functions which the complier would be able to translate into appropriate calls.

In my mind, the first milestone for miden-sdk would be to provide enough functionality to allow users to write something like a basic wallet. If the MASM code in the linked code were to be expanded, it would look something like this:

use.miden::sat::account
use.miden::sat::tx

#! Adds the provided asset to the current account.
export.receive_asset
    exec.account::add_asset
    padw swapw dropw
end

#! Creates a note which sends the specified asset out of the current account 
#! to the specified recipient.
export.send_asset.1
    exec.account::remove_asset
    # => [ASSET, tag, RECIPIENT, ...]

    # insert 8 ZEROs into the stack right after recipient; we temporarily store one of the
    # elements of ASSET in memory to make stack manipulation easier
    push.0 swap loc_store.0 padw push.0.0.0 swapdw loc_load.0
    # => [ASSET, tag, RECIPIENT, ZERO, ZERO, ...]

    exec.tx::create_note
    # => [note_ptr, ZERO, ZERO, ...]
end

#! Authenticate a transaction using Falcon signature scheme.
#!
#! NOTE: this procedure is till WIP.
export.auth_tx_rpo_falcon512
    # update the nonce
    push.1
    exec.account::incr_nonce

    # get commitments to consumed and created notes, new nonce, and ID
    #exec.tx::get_output_notes_hash
    #exec.tx::get_input_notes_hash
    #exec.account::get_nonce push.0.0.0
    #exec.account::get_id push.0.0.0

    # compute the message to be signed
    # M = h(output_notes_hash, h(input_notes_hash, h(0, 0, 0, id, 0, 0, 0, nonce)))
    #hmerge hmerge hmerge

    # get public key from account storage (assuming it is stored at index 0) and verify signature
    #push.0
    #exec.account::get_item
    # [PUB_KEY, M]
    # TODO: sign the message using FALCON
end

Totally understand that the code output by the compiler would not look as nice or be as optimal - but would be great to support same general functionality.

Beyond this initial milestone, there are a few things which we can try to support in the longer term. A couple of examples:

Using self instead of magic function

Ability to use self keyword to refer to the account - e.g., rewriting the above example to look something like:

use miden::{
    asset::Asset,
    note::Recipient,
    tx,
};

#[miden_account]
pub struct MyWallet;

#[miden_account]
impl MyWallet {
    pub fn recieve_asset(&mut self, asset: Asset) {
        self.add_asset(asset);
    }

    pub fn send_asset(&mut self, asset: Asset, recipient: Recipient) {
        self.remove_asset(asset);
        tx::create_note(asset, recipient);
    }
}

The emitted code would still be the same as in the previous example - so, this would be more like syntactic sugar.

Account interface implementations

Ability to implement account interfaces - e.g., something like:

use miden::wallets::BasicWallet;

#[miden_account]
pub struct MyWallet;

#[miden_account_impl]
impl BasicWallet for MyWallet;

With the above code outputting the following (or similar) MASM:

use.miden::wallets::basic->basic_wallet

export.basic_wallet::receive_asset
export.basic_wallet::send_asset

Not sure how doable this is, but one interesting crate from which we might lift some ideas is Supertrait.

Names storage slots

Ability to specify the shape of the storage for an account - e.g., something like this:

use miden::{
    collections::StorageMap,
    crypto::dsa::rpo_falcon512:PubKey,
}

#[miden_account]
pub struct MyWallet {
    pub_key: PubKey,
    my_map: StorageMap
}

#[miden_account]
impl MyWallet {
    pub fn auth_tx_rpo_falcon512(&mut self) {
        self.incr_nonce(1);
        let pub_key = self.pub_key;
        // do something with the key
    }
}

In the above, self.pub_key would be translated by the compiler into account::get_item(0) and self.pub_key = x would be translated into account::set_item(0). Not sure how doable/complicated this is (e.g., may require implementing a custom allocator).

[greenhat]

I wonder if we need to separate miden-vm-sdk and miden-sdk. At least initially, it might make sense to do just the rollup-focused miden-sdk as I think this could help us progress faster specifically on the rollup/smart contract front. In the process, we'll probably also figure out how best to split out rollup-independent components from miden-sdk.

I agree. We can start with miden-sdk and figure out how to split it in the process. The main reason for splitting is decomposition to ease comprehension and maintenance.

For miden-sdk, I'd probably first focus on accounts, and once we have some initial things for accounts working, we can tackle notes. One reason for this is that for account code, we don't really need to handle call's and syscall's, or worry about how to describe contract ABIs.

Agree. Account support is a great first goal for miden-sdk.

For exposing MASM libraries to Rust code, I'd prefer to have Rust wrappers (rather than using miden_bindgen). So, for example, an account code could look like:

use miden::{
    account,
    asset::Asset,
    note::Recipient,
    tx,
};

#[miden_account]
pub struct MyWallet;

#[miden_account]
impl MyWallet {
    pub fn recieve_asset(&mut self, asset: Asset) {
        account::add_asset(asset);
    }

    pub fn send_asset(&mut self, asset: Asset, recipient: Recipient) {
        account::remove_asset(asset);
        tx::create_note(asset, recipient);
    }
}

Where miden would be a library which we write (or maybe auto-generate using convention discussed in #25) which under the hood would use "extern C" functions which the complier would be able to translate into appropriate calls.

In my example, miden_bindgen generates the Rust wrappers (types, extern "C" functions, etc.) from a description file (metadata from #25 or WIT). I don't understand the reason for keeping Rust wrappers in a separate Rust library, though. The types that are written manually would be in some library, but everything that is generated I would not store anywhere to avoid the overhead and getting out of sync with the source.

In my mind, the first milestone for miden-sdk would be to provide enough functionality to allow users to write something like a basic wallet. If the MASM code in the linked code were to be expanded, it would look something like this:

use.miden::sat::account
use.miden::sat::tx

#! Adds the provided asset to the current account.
export.receive_asset
    exec.account::add_asset
    padw swapw dropw
end

#! Creates a note which sends the specified asset out of the current account 
#! to the specified recipient.
export.send_asset.1
    exec.account::remove_asset
    # => [ASSET, tag, RECIPIENT, ...]

    # insert 8 ZEROs into the stack right after recipient; we temporarily store one of the
    # elements of ASSET in memory to make stack manipulation easier
    push.0 swap loc_store.0 padw push.0.0.0 swapdw loc_load.0
    # => [ASSET, tag, RECIPIENT, ZERO, ZERO, ...]

    exec.tx::create_note
    # => [note_ptr, ZERO, ZERO, ...]
end

#! Authenticate a transaction using Falcon signature scheme.
#!
#! NOTE: this procedure is till WIP.
export.auth_tx_rpo_falcon512
    # update the nonce
    push.1
    exec.account::incr_nonce

    # get commitments to consumed and created notes, new nonce, and ID
    #exec.tx::get_output_notes_hash
    #exec.tx::get_input_notes_hash
    #exec.account::get_nonce push.0.0.0
    #exec.account::get_id push.0.0.0

    # compute the message to be signed
    # M = h(output_notes_hash, h(input_notes_hash, h(0, 0, 0, id, 0, 0, 0, nonce)))
    #hmerge hmerge hmerge

    # get public key from account storage (assuming it is stored at index 0) and verify signature
    #push.0
    #exec.account::get_item
    # [PUB_KEY, M]
    # TODO: sign the message using FALCON
end

Totally understand that the code output by the compiler would not look as nice or be as optimal - but would be great to support same general functionality.

Yes, that's how I imagined the MASM would look.

Beyond this initial milestone, there are a few things which we can try to support in the longer term. A couple of examples:

Using self instead of magic function

Ability to use self keyword to refer to the account - e.g., rewriting the above example to look something like:

use miden::{
    asset::Asset,
    note::Recipient,
    tx,
};

#[miden_account]
pub struct MyWallet;

#[miden_account]
impl MyWallet {
    pub fn recieve_asset(&mut self, asset: Asset) {
        self.add_asset(asset);
    }

    pub fn send_asset(&mut self, asset: Asset, recipient: Recipient) {
        self.remove_asset(asset);
        tx::create_note(asset, recipient);
    }
}

The emitted code would still be the same as in the previous example - so, this would be more like syntactic sugar.

Good point! Seems doable.

Account interface implementations

Ability to implement account interfaces - e.g., something like:

use miden::wallets::BasicWallet;

#[miden_account]
pub struct MyWallet;

#[miden_account_impl]
impl BasicWallet for MyWallet;

With the above code outputting the following (or similar) MASM:

use.miden::wallets::basic->basic_wallet

export.basic_wallet::receive_asset
export.basic_wallet::send_asset

Not sure how doable this is, but one interesting crate from which we might lift some ideas is Supertrait.

It seems doable, maybe not in such a clean way. Giving an app developer a way to use some default implementation is great. I'll take a look at the crate.

Names storage slots

Ability to specify the shape of the storage for an account - e.g., something like this:

use miden::{
    collections::StorageMap,
    crypto::dsa::rpo_falcon512:PubKey,
}

#[miden_account]
pub struct MyWallet {
    pub_key: PubKey,
    my_map: StorageMap
}

#[miden_account]
impl MyWallet {
    pub fn auth_tx_rpo_falcon512(&mut self) {
        self.incr_nonce(1);
        let pub_key = self.pub_key;
        // do something with the key
    }
}

In the above, self.pub_key would be translated by the compiler into account::get_item(0) and self.pub_key = x would be translated into account::set_item(0). Not sure how doable/complicated this is (e.g., may require implementing a custom allocator).

Yes, it seems doable. I omitted the storage part in my example because I need to understand how account storage works. I've dug through the relevant miden-base parts but have not yet wrapped my head around it.

[bobbinth]

I don't understand the reason for keeping Rust wrappers in a separate Rust library, though.

This is mostly so that people can just put the library from crates.io and don't need to figure out where to get .wit files. We can still probably support the miden_bindgen approach - but for the main Miden library, I think it is important that users could just add it to the list of dependencies and import it as a regular Rust library (this could also be good for version management).

Yes, it seems doable. I omitted the storage part in my example because I need to understand how account storage works. I've dug through the relevant miden-base parts but have not yet wrapped my head around it.

Yeah - we've been refining the architecture, but I think we now landed on something good. The latest approach is in 0xPolygonMiden/miden-base#239 (comment) and following comments.

[greenhat]

I don't understand the reason for keeping Rust wrappers in a separate Rust library, though.

This is mostly so that people can just put the library from crates.io and don't need to figure out where to get .wit files. We can still probably support the miden_bindgen approach - but for the main Miden library, I think it is important that users could just add it to the list of dependencies and import it as a regular Rust library (this could also be good for version management).

Oh, yes, for the Miden stdlib, it makes total sense. I was thinking about the note code DX and how to use the account's methods. I mistakenly thought anyone could write that note code, so we need a way to pull account method signatures from an arbitrary account. However, since a note can only be created by an account method, the note's code will be written by the account author or a part of the account codebase. In this case, miden_bindgen is not needed. It leaves me with one case I can think of where miden_bindgen might be useful in the rollup - when an account/note/tx_script code wants to call a function from an arbitrary Miden library written manually in MASM. But even in this case, having a separate Rust library for the MASM code published on crates.io might be a good idea.

Yes, it seems doable. I omitted the storage part in my example because I need to understand how account storage works. I've dug through the relevant miden-base parts but have not yet wrapped my head around it.

Yeah - we've been refining the architecture, but I think we now landed on something good. The latest approach is in 0xPolygonMiden/miden-base#239 (comment) and following comments.

Thanks! I think I get it now. So for your example above, self.my_map.get(some_key_word) should compile to account::get_map_value(1, some_key_word) (my_map is in slot 1 since slot 0 holds pub_key). And self.my_map.set(some_key_word, some_value_word) should compile to account::set_map_value(1, some_key_word, some_value_word).

[bobbinth]

I was thinking about the note code DX and how to use the account's methods.

I haven't thought that far yet. So, we might still need it, but there might be alternatives too.

Thanks! I think I get it now. So for your example above, self.my_map.get(some_key_word) should compile to account::get_map_value(1, some_key_word) (my_map is in slot 1 since slot 0 holds pub_key). And self.my_map.set(some_key_word, some_value_word) should compile to account::set_map_value(1, some_key_word, some_value_word).

Yes - exactly correct!

[chainyoda]

i think it makes sense to prioritise smart contract devs need - that will also provide useful information on what api/sdk devs building solely on the vm need

[greenhat]

I made #72 to test how we can compile the above account code.