Implement pallet view function queries #4722

ascjones · 2024-06-06T17:27:02Z

Closes #216.

This PR allows pallets to define a view_functions impl like so:

#[pallet::view_functions]
impl<T: Config> Pallet<T>
where
	T::AccountId: From<SomeType1> + SomeAssociation1,
{
	/// Query value no args.
	pub fn get_value() -> Option<u32> {
		SomeValue::<T>::get()
	}

	/// Query value with args.
	pub fn get_value_with_arg(key: u32) -> Option<u32> {
		SomeMap::<T>::get(key)
	}
}

`QueryId`

Each view function is uniquely identified by a QueryId, which for this implementation is generated by:

twox_128(pallet_name) ++ twox_128("fn_name(fnarg_types) -> return_ty")

The prefix twox_128(pallet_name) is the same as the storage prefix for pallets and take into account multiple instances of the same pallet.

The suffix is generated from the fn type signature so is guaranteed to be unique for that pallet impl. For one of the view fns in the example above it would be twox_128("get_value_with_arg(u32) -> Option<u32>"). It is a known limitation that only the type names themselves are taken into account: in the case of type aliases the signature may have the same underlying types but a different id; for generics the concrete types may be different but the signatures will remain the same.

The existing Runtime Call dispatchables are addressed by their concatenated indices pallet_index ++ call_index, and the dispatching is handled by the SCALE decoding of the RuntimeCallEnum::PalletVariant(PalletCallEnum::dispatchable_variant(payload)). For view_functions the runtime/pallet generated enum structure is replaced by implementing the DispatchQuery trait on the outer (runtime) scope, dispatching to a pallet based on the id prefix, and the inner (pallet) scope dispatching to the specific function based on the id suffix.

Future implementations could also modify/extend this scheme and routing to pallet agnostic queries.

Executing externally

These view functions can be executed externally via the system runtime api:

pub trait ViewFunctionsApi<QueryId, Query, QueryResult, Error> where
	QueryId: codec::Codec,
	Query: codec::Codec,
	QueryResult: codec::Codec,
	Error: codec::Codec,
{
	/// Execute a view function query.
	fn execute_query(query_id: QueryId, query: Query) -> Result<QueryResult, Error>;
}

`XCQ`

Currently there is work going on by @xlc to implement XCQ which may eventually supersede this work.

It may be that we still need the fixed function local query dispatching in addition to XCQ, in the same way that we have chain specific runtime dispatchables and XCM.

I have kept this in mind and the high level query API is agnostic to the underlying query dispatch and execution. I am just providing the implementation for the view_function definition.

Metadata

Currently I am utilizing the custom section of the frame metadata, to avoid modifying the official metadata format until this is standardized.

vs `runtime_api`

There are similarities with runtime_apis, some differences being:

queries can be defined directly on pallets, so no need for boilerplate declarations and implementations
no versioning, the QueryId will change if the signature changes.
possibility for queries to be executed from smart contracts (see below)

Calling from contracts

Future work would be to add weight annotations to the view function queries, and a host function to pallet_contracts to allow executing these queries from contracts.

TODO

athei · 2024-10-08T14:04:54Z

Multiple runtimes importing the same pallets will offer the same semantics. Thats the whole point of bundling them into modules (aka pallets) and not implementing them at runtime level like a generic runtime API.

And yes runtime wide view functions will put a slight wrinkle into this plan. But we can also just package them into a crate that we distribute and that everybody imports and wires them up with their pallets.

Maybe you can view that as some sort of spec.

kianenigma · 2024-10-08T15:45:38Z

Regarding the hashing: no one is strongly backing it, we should set our aim to merge this PR with old school dispatch. This method also has not benefit per-se, but it is at least more consistent.

Some final counterarguments:

There is some engineering effort needed to migrate this PR to enum dispatch. I don't think it is massive.
Hashes will not have this 256 limit. At the RuntimeQuery (RuntimeView), we will only hit the limit when a runtime has more than 255 pallets, so same time as all Runtime* explode. At the pallet level, one might argue that a pallet might normally have waaaaaaaaaaay more view functions than it would have storage items, but I don't see that. A good pallet probably has max a dozen storage, and a dozen view functions.

Having all this conversation is a good encouragement to already start thinking about a migration plan into compact encoded indices for enums to eliminate this time bomb.

jsdw · 2024-10-09T10:48:20Z

And yes runtime wide view functions will put a slight wrinkle into this plan. But we can also just package them into a crate that we distribute and that everybody imports and wires them up with their pallets.

Maybe you can view that as some sort of spec.

This all makes sense, and yup this is the bit that I'm thinking about as a spec, and I agree that the spec can be implemented in the form of code and probably from this we can generate documentation to also describe it to people who want to then use the facade interface at a higher level (client facade implementations for instance)

jsdw · 2024-10-09T15:46:49Z

Just a general thought on the naming in this PR:

How do people feel about calling these "Pallet APIs" rather than "Pallet view functions"? My thought is to be consistent with the term "Runtime APIs", which (from the client perspective at least) act very similar; read-only functions that can be called with some arguments and return some value back.

jsdw · 2024-10-10T15:20:38Z

Regarding the hashing: no one is strongly backing it, we should set our aim to merge this PR with old school dispatch. This method also has not benefit per-se, but it is at least more consistent.

@kianenigma I was thinking what the metadata would look like in order that clients know how to call pallet view functions.

For the case of eg transactions, we provide an enum in the metadata for each pallet, and that works great because each variant name corresponds to the transaction name and each enum field correpsonds to one of the transaction arguments.

For the case of Pallet View Functions (same as rutime APIs), an enum doesn't provide enough information (ie we also need to know the return type for each Pallet View Function. So I suppose the metadata for the ciew functions in a pallet using enum dispatch would look more like:

view_functions: vec![
    PalletViewFunction {
        index: 1u8,
        name: String,
        inputs: vec![12, 34, 8], // u32 type IDs for each argument
        output: 34, // u32 type ID for return type,
        docs: Vec<String>
    },
    // ...
]

The index in the metadata exists only to dispatch to the right function using the enum approach. If we did hash based dispatch, then we could omit the index and describe pallet view functions identically to Runtime APIs, ie like this:

pub struct PalletApiMethodMetadata<T: Form = MetaForm> {
	/// Method name.
	pub name: T::String,
	/// Method parameters.
	pub inputs: Vec<RuntimeApiMethodParamMetadata<T>>,
	/// Method output.
	pub output: T::Type,
	/// Method documentation.
	pub docs: Vec<T::String>,
}

(see https://github.com/paritytech/frame-metadata/blob/main/frame-metadata/src/v15.rs#L132)

Does this change your opinion on enum versus hash based dispatch, or is it actually a non issue?

athei · 2024-10-11T19:30:55Z

How do people feel about calling these "Pallet APIs" rather than "Pallet view functions"? My thought is to be consistent with the term "Runtime APIs",

I don't think we should use bad names for the sake of consistency. Remember that we call host functions "Runtime Interface". It is confusing to use those generic names. There will also be runtime wide view functions eventually.

ascjones · 2024-10-15T09:04:07Z

Regarding the hashing: no one is strongly backing it, we should set our aim to merge this PR with old school dispatch. This method also has not benefit per-se, but it is at least more consistent.

Throwing some thoughts out there:

Inspiration for the hashing is from solidity/ink! selectors, allows defining common standards: a known set of selectors. e.g. NFT wallets/UIs can identify the ERC721 selectors.
Isn't enum based dispatch just a legacy implementation detail? The Outer runtime enum and inner pallet enum are both macro generated, and the derived SCALE decoding is used for the correct dispatching. Hence we inherit the limited u8 + u8 address space. The important part is the Runtime wide guarantee of uniqueness of each dispatchable's prefix

Sadly both pallet name and index are variable per-runtime, so the north star of removing complexity from client libs is not feasible. In all future scenarios, if a client lib wants to provide an abstraction over the same view function in 5 different chains, it would inevitably need to go through some effort and "speculate/hardcode" a few things. I don't think there is any solution for it.

No data to back this up but more often than not the default pallet name is used, so by default most view functions will have the standard hash. In cases where it is not we could allow specifying a custom prefix in construct_runtime to ensure consistency.
Another approach would be to instead of twox_128(pallet_name) ++ twox_128("fn_name(fnarg_types) -> return_ty"), just use the hashed signature with an optional prefix prefix ++ twox_128("fn_name(fnarg_types) -> return_ty"). Prefix provides uniqueness in case of identical fn signatures and could be defined in view fn definition to be pallet agnostic. Downside would be O(n) lookups for view functions, but with a low n that won't be too bad?
Using enum based lookup makes the smart contract use case for querying view functions difficult, since the enum prefixes/definitions will need to be hardcoded. Looking them up would be too much overhead.

Given this, I am also still leaning towards enum based dispatch. It is more unified, and we can pursue someday moving all dispatches to hashes, but not have one foot in each. With the time bomb of reaching the 255 limit, we will have to do it someday :)

I like the idea of things being unified, OTOH if we know we are going there, why not do it here first where the impact is low (nobody is using view functions yet). The cost of migrating in the future should be considered.

kianenigma · 2024-10-17T15:11:40Z

I like the idea of things being unified, OTOH if we know we are going there, why not do it here first where the impact is low (nobody is using view functions yet). The cost of migrating in the future should be considered.

This is a great idea, and a nice bonus: If you think we can already experiment with Enum based dispatch, with compact encoded indices, let's do it here. @ascjones

For the case of Pallet View Functions (same as rutime APIs), an enum doesn't provide enough information (ie we also need to know the return type for each Pallet View Function.

Indeed, the metadata for view functions would need to be more sophisticated than dispatch, so it cannot be purely enum. @jsdw

How do people feel about calling these "Pallet APIs" rather than "Pallet view functions"? My thought is to be consistent with the term "Runtime APIs",
I don't think we should use bad names for the sake of consistency. Remember that we call host functions "Runtime Interface". It is confusing to use those generic names. There will also be runtime wide view functions eventually.

About naming, I would go with view functions for now. We can always rename it later and pre-launch as well, so it is okay.

ascjones · 2024-10-17T15:39:33Z

I like the idea of things being unified, OTOH if we know we are going there, why not do it here first where the impact is low (nobody is using view functions yet). The cost of migrating in the future should be considered.

This is a great idea, and a nice bonus: If you think we can already experiment with Enum based dispatch, with compact encoded indices, let's do it here. @ascjones

I was in fact referring to "we can pursue someday moving all dispatches to hashes", so presenting the case for using hash instead of enum based dispatch 🙈

jsdw · 2024-10-25T10:18:42Z

Me and @kianenigma had a chat about this the other day and the consensus was that, while our opinions differ on enum dispatch versus not, we shouldn't let merging this PR get too held up on this point.

If there are concerns about merging this and then later altering it, then perhaps we could mark it as unstable like we do with new metadata versions and call the attr #[pallet::unstable_view_functions] with appropriate documentation, to give us the ability to merge and experiment with this and then iterate with smaller PRs if we want to update anything to reach consensus.

Does that work for everybody here? Should we mark as "unstable" eg like above or can we review and merge and still tweak it later without this?

athei · 2024-10-25T10:24:06Z

yes

pkhry · 2024-10-30T09:44:13Z

substrate/frame/support/src/traits/query.rs

+	) -> Result<(), QueryDispatchError>;
+}
+
+impl DispatchQuery for () {


dq: but why we need to implement it for unit type?

I'd like somebody else to chime in if wrong, but I believe it's just a convenience impl for tests!

It's the default type for the type RuntimeQuery associated type, as is the convention with the other Runtime* types where there are no implementations.

pkhry · 2024-10-30T09:58:13Z

substrate/frame/support/procedural/src/pallet/expand/view_functions.rs

+	let queries = view_fns.iter().map(|view_fn| {
+		let query_struct_ident = view_fn.query_struct_ident();
+		let name = &view_fn.name;
+		let args: Vec<TokenStream> = vec![]; // todo


is todo still relevant?

Yes, the args still need to be filled in the metadata

pkhry · 2024-10-30T10:02:39Z

substrate/primitives/metadata-ir/src/types.rs

+
+/// Metadata of a runtime method.
+#[derive(Clone, PartialEq, Eq, Encode, Debug)]
+pub struct QueryMetadataIR<T: Form = MetaForm> {


dq: but don't we want to have space for version/deprecation info on the query to allow iteration on already defined apis without explicitly removing them or redefining the type signature?
Or the intention is just o keep them all and fully rely on QueryId for the resolution?

The idea was not to have versioning, a new version would just have a different QueryId. The old version would have to be maintained possibly indefinitely. Perhaps a deprecation flag would be useful, needs some thinking about.

substrate/frame/support/procedural/src/pallet/parse/mod.rs

substrate/primitives/api/src/lib.rs

jsdw · 2024-11-07T16:24:04Z

substrate/primitives/metadata-ir/src/types.rs

+	/// The type implementing the runtime query dispatch.
+	pub ty: T::Type,
+	/// The query interfaces metadata.
+	pub interfaces: Vec<QueryInterfaceIR<T>>,


Aren't pallet view functions scoped to certain pallets? So would it make sense for this information to be inside the PalletMetadataIR?

This PR allows them to be defined only as part of pallets (because it is convenient for the initial implementation), but the requirement is to make them pallet agnostic, so they can be defined independently of pallets.

What's the advantage/reason to replace Runtime APIs in the future with these? My understanding (which might miss something; I haven't read the background docs in a while not) is:

We have "Runtime APIs" as a pallet agnostic set of APIs (called externally via eg state_call by using RuntimeApiTraitName ++ _ ++ method_name and SCALE encoding the args.

We have here "Pallet View Functions", which are like Runtime APIs but are defined at the pallet level instead (and queried via a Runtime API call). These currently are accessed by using twox_128(pallet_name) ++ twox_128("fn_name(fnarg_types) -> return_ty") and SCALE encoding the args.

Both allow you to call functions and get the results back. In the future I'd expect both to become subscribable. The calling convetion of Pallet View Functions (hashing the type sig etc to query them) is different in order to enforce the type signature in the query (which means that changing the type signature of one would lead to old queries being invalid rather than being executed and complaining about wrongly encoded params or whatever).

I guess overall I see them as working alongside Runtime APIs but at the pallet level, which is also why I've tended to push for making them consistent with Runtime APIs in terms of how they are talked about and called etc so far, but I feel like I am missing something because others seem less keen on this so far :D

\cc @kianenigma maybe you also have some insight/thought I'm missing here :)

But all in all, I am happy to go ahead with whatever naming etc and figure it out in followup PRs anyways!

jsdw · 2024-11-07T17:09:36Z

substrate/primitives/metadata-ir/src/types.rs

+
+/// Metadata of a runtime query interface.
+#[derive(Clone, PartialEq, Eq, Encode, Debug)]
+pub struct QueryInterfaceIR<T: Form = MetaForm> {


How much could we align this with Runtime APIs in terms of the naming of structts/fields?

For runtime APIs we have:

RuntimeApiMetadataIR { name, methods, docs, deprecation_info } RuntimeApiMethodMetadataIR { name, inputs, output, docs, deprecation_info } RuntimeApiMethodParamMetadataIR { name, ty }

So here, we could, per pallet, have something like:

struct PalletApiMethodMetadataIR { name, id?, inputs: Vec<PalletApiMethodParamMetadataIR>, output, docs, deprecation_info } struct PalletApiMethodParamMetadataIR { name, ty }

(I went for Api instead of ViewFunction just because I prefer it to viewFunctions and it's shorter, but it's all good!)

I guess having the id field lets us avoid having to construct it, since it's a bit arduous hashing the relevant bits, so makes sense for now! Personally I'd simplify the way we call these things to be more like runtime APIs, but I'm happy to propose that as a separate PR after this merges!

I agree should try to align the metadata with Runtime APIs...in theory we should be able to replace Runtime APIs in the future with "view functions".

As for naming...it would be good to come up with something consistent: as you can see I am using "query" interchangeably with "view_function". API is okay but maybe too generic and could be confused with existing Runtime API? Naming is hard, any suggestions welcome.

I personally prefer Api to align with Runtime APIs, but happy for everything to be named consistently with any of Api, Query, ViewFunc (just to help shorten a touch) or ViewFunction. For the sake of being able to rename it all quite easily in a followup PR if we like, perhaps one of the latter two is ideal (easy to search and replace ViewFunc/ViewFunction!)

jsdw · 2024-11-08T14:40:57Z

substrate/frame/support/procedural/src/pallet/expand/view_functions.rs

+		}
+
+		impl<#type_impl_gen> #query_struct_ident<#type_use_gen> #where_clause {
+			/// Create a new [`#query_struct_ident`] instance.


Nit: this outputs as written here rather than #query_struct_ident being substituted with the ident as you'd want.

Best bet might be to build the doc string beforehand and then use #[doc = #doc_string] to get it in!

jsdw · 2024-11-13T11:15:39Z

This is generally looking good to merge to me now! It's behind an view_functions_experimental which gives us some room to tweak things in followup PRs already. A couple of small outstanding bits from me:

Document the TODOs.
prdoc
Consistify query/view_function related language to all be view_function/ViewFunction. A bit verbose but makes it easy to search and replace if we want to change the name before stabilising things.
MetadataIR: I'd move the view function stuff into pallets rather than haev a new top level section for it. If needed this can be a followup PR though!

Thw top 3 of those would be enough for a tick from me :) It'd be good to also have a FRAME/Runtime dev have a look over since I'm newer to this code.

ascjones · 2024-11-13T15:33:58Z

Okay thanks @jsdw, I will attempt to resolve these by the end of next week.

ascjones added 30 commits May 20, 2024 11:25

Add view functions parsing and definition

264b902

Add to view functions mod definition

987802c

Adding traits and types and wiring up expansion

4f7a513

Use ToTokens for expansion

e8af3c2

fmtl

7038710

Adding generics etc to query struct

9267519

fmt

c6a4e42

example with args

140f7d9

add example for testing

23b9ebf

wire up codegen for view functions

86c5f2f

Generate DispatchQuery impl

f42c5c4

Wire up runtime PalletQuery

ac6643e

Wire up query args

2ea6158

Add get value with args query

8b56a84

decode_all

06f9ffc

Merge branch 'master' into aj/view-functions

9ed181b

Docs

cd52010

Generate suffix from view_fn signature

44707b3

separate suffix id

afa1490

WIP expand runtime level query

f97a40c

Implement DispatchQuery for pallets even without queries

ddff4d0

WIP

538ea8a

Use pallet name hash as query id prefix

db616f5

Commented out code

fd3318b

Inject runtime type

a49e5f3

tidy

dd651ac

Add pallet with instances

c7ff6c9

Pallet instances tests

5a8e559

Refactor tests

e71b9de

Move execute from codegen to trait

22dd94b

Merge branch 'master' into aj/view-functions

fd0896c

kianenigma mentioned this pull request Oct 18, 2024

Vision: Storage Deposits to be Ergonomic and Automatic #328

Open

pkhry reviewed Oct 30, 2024

View reviewed changes

This was referenced Nov 4, 2024

Bounty Pallet: add approve_bounty_with_curator call to bounties pallet #5961

Merged

Nomination pools Runtime API: Get a pool stash and reward address #6358

Closed

[Pools] New runtime api that returns the pot accounts associated with the pool #6357

Merged

re-gius mentioned this pull request Nov 7, 2024

Add pool accounts view_function #6408

Draft

jsdw reviewed Nov 7, 2024

View reviewed changes

substrate/frame/support/procedural/src/pallet/parse/mod.rs Outdated Show resolved Hide resolved

Merge branch 'master' into aj/view-functions

0bf818a

jsdw reviewed Nov 7, 2024

View reviewed changes

substrate/primitives/api/src/lib.rs Show resolved Hide resolved

jsdw reviewed Nov 7, 2024

View reviewed changes

ascjones added 3 commits November 8, 2024 12:16

Rename view_functions to view_functions_experimental

7c71b8a

Remove Vec import

60d78fc

Resore Vec import

19d6f65

jsdw reviewed Nov 8, 2024

View reviewed changes

jsdw mentioned this pull request Nov 11, 2024

[tracking] Facade #6441

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implement pallet view function queries #4722

Implement pallet view function queries #4722

ascjones commented Jun 6, 2024 •

edited

Loading

athei commented Oct 8, 2024

kianenigma commented Oct 8, 2024

jsdw commented Oct 9, 2024

jsdw commented Oct 9, 2024

jsdw commented Oct 10, 2024 •

edited

Loading

athei commented Oct 11, 2024

ascjones commented Oct 15, 2024 •

edited

Loading

kianenigma commented Oct 17, 2024

ascjones commented Oct 17, 2024

jsdw commented Oct 25, 2024 •

edited

Loading

athei commented Oct 25, 2024

pkhry Oct 30, 2024

jsdw Nov 7, 2024

ascjones Nov 8, 2024

pkhry Oct 30, 2024 •

edited

Loading

ascjones Nov 8, 2024

pkhry Oct 30, 2024

ascjones Nov 8, 2024

jsdw Nov 7, 2024

ascjones Nov 8, 2024

jsdw Nov 8, 2024 •

edited

Loading

jsdw Nov 7, 2024 •

edited

Loading

ascjones Nov 8, 2024

jsdw Nov 13, 2024

jsdw Nov 8, 2024 •

edited

Loading

jsdw commented Nov 13, 2024 •

edited

Loading

ascjones commented Nov 13, 2024

Implement pallet view function queries #4722

Are you sure you want to change the base?

Implement pallet view function queries #4722

Conversation

ascjones commented Jun 6, 2024 • edited Loading

QueryId

Executing externally

XCQ

Metadata

vs runtime_api

Calling from contracts

TODO

athei commented Oct 8, 2024

kianenigma commented Oct 8, 2024

jsdw commented Oct 9, 2024

jsdw commented Oct 9, 2024

jsdw commented Oct 10, 2024 • edited Loading

athei commented Oct 11, 2024

ascjones commented Oct 15, 2024 • edited Loading

kianenigma commented Oct 17, 2024

ascjones commented Oct 17, 2024

jsdw commented Oct 25, 2024 • edited Loading

athei commented Oct 25, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pkhry Oct 30, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jsdw Nov 8, 2024 • edited Loading

Choose a reason for hiding this comment

jsdw Nov 7, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jsdw Nov 8, 2024 • edited Loading

Choose a reason for hiding this comment

jsdw commented Nov 13, 2024 • edited Loading

ascjones commented Nov 13, 2024

ascjones commented Jun 6, 2024 •

edited

Loading

`QueryId`

`XCQ`

vs `runtime_api`

jsdw commented Oct 10, 2024 •

edited

Loading

ascjones commented Oct 15, 2024 •

edited

Loading

jsdw commented Oct 25, 2024 •

edited

Loading

pkhry Oct 30, 2024 •

edited

Loading

jsdw Nov 8, 2024 •

edited

Loading

jsdw Nov 7, 2024 •

edited

Loading

jsdw Nov 8, 2024 •

edited

Loading

jsdw commented Nov 13, 2024 •

edited

Loading