Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: implement MPT part of generic extraction (Part 1) #385

Merged
merged 183 commits into from
Mar 5, 2025
Merged
Show file tree
Hide file tree
Changes from 157 commits
Commits
Show all changes
183 commits
Select commit Hold shift + click to select a range
cd191cd
feat: allow for either aggregation-only or tabular-only queries
delehef Sep 10, 2024
719a0c0
Add Merkle-path verification gadget
nicholas-mainardi Sep 11, 2024
3794779
merge MPT extraction
nikkolasg Sep 12, 2024
3793aa1
is_multiplier
nikkolasg Sep 12, 2024
ebdd33a
multiplier for rows
nikkolasg Sep 12, 2024
071a2f0
and api
nikkolasg Sep 12, 2024
ac3443e
Add revelation circuit unproven offset
nicholas-mainardi Sep 12, 2024
3b735cd
refactoring of cell logic
nikkolasg Sep 13, 2024
2b3c289
wip
nikkolasg Sep 13, 2024
d3ae335
row leaf passing
nikkolasg Sep 13, 2024
ff8039c
Add APIs for simple select
nicholas-mainardi Sep 13, 2024
5079597
Update general query APIs
nicholas-mainardi Sep 13, 2024
36673ef
partial fixed
nikkolasg Sep 13, 2024
6713f66
full node fixed
nikkolasg Sep 13, 2024
90f8d1c
row tree test passing
nikkolasg Sep 13, 2024
3fe4b55
Fix build integration test
nicholas-mainardi Sep 13, 2024
8dcdc05
testing merge circuit
nikkolasg Sep 13, 2024
0ea3f41
API for merge
nikkolasg Sep 13, 2024
2d7de6f
adding one more circuit set size
nikkolasg Sep 13, 2024
f8a79d1
Fix build groth-16 crate
nicholas-mainardi Sep 13, 2024
e9c3bcb
wip
nikkolasg Sep 15, 2024
dbb8e7a
Refactor query test cases code
nicholas-mainardi Sep 16, 2024
799a60f
Merge with main
nicholas-mainardi Sep 16, 2024
b26ea94
fix: ORDER BY is not supported
delehef Sep 16, 2024
a36fd13
Merge pull request #355 from Lagrange-Labs/feat/drop-scalar-check
nikkolasg Sep 16, 2024
dfaa7d7
WiP: keep both queries types in a single test
nicholas-mainardi Sep 16, 2024
8fc299f
Merge branch 'main' into feat/tabular-queries
nicholas-mainardi Sep 16, 2024
76a4bdd
wip
nikkolasg Sep 17, 2024
4f7aec5
compiling
nikkolasg Sep 17, 2024
c0bb9a6
Merge branch 'feat/tabular-queries' into feat/unproven-limit-offset-q…
nicholas-mainardi Sep 18, 2024
28a3383
Working intergation test for simple select queries
nicholas-mainardi Sep 18, 2024
c3c199d
fmt
nicholas-mainardi Sep 18, 2024
170ad44
Run also commented test cases
nicholas-mainardi Sep 18, 2024
676dfec
Bind matching row keys and results in a single query
nicholas-mainardi Sep 18, 2024
34d88ec
Remove dead code
nicholas-mainardi Sep 18, 2024
689b929
Add distinct to circuit and Parsil
nicholas-mainardi Sep 19, 2024
56e0dde
Support DISTINCT and SELECT * queries in integration test
nicholas-mainardi Sep 19, 2024
17cd47f
Comment out test cases
nicholas-mainardi Sep 19, 2024
84fe37a
passing test
nikkolasg Sep 19, 2024
cd59ea3
More complex test case with wildcards
nicholas-mainardi Sep 19, 2024
d8bc019
integrated test refactor #1
nikkolasg Sep 19, 2024
1fb227d
refactoring #2 compiling
nikkolasg Sep 20, 2024
735b415
correct size of circuit set
nikkolasg Sep 20, 2024
cebaf1c
full refactoring #3
nikkolasg Sep 20, 2024
a1274ae
multiplier columns
nikkolasg Sep 20, 2024
aee4872
test case for merge
nikkolasg Sep 20, 2024
2ea4e0d
add metadata from contract
nikkolasg Sep 20, 2024
99022b6
merge updates
nikkolasg Sep 21, 2024
1024dbe
default sec cell
nikkolasg Sep 21, 2024
abd9989
debug storage root
nikkolasg Sep 21, 2024
c7bea49
fixing block
nikkolasg Sep 21, 2024
12e083f
ordered cells
nikkolasg Sep 21, 2024
f6a5647
more debug
nikkolasg Sep 21, 2024
4368993
debug
nikkolasg Sep 21, 2024
1d4c10e
fixing vdb row digest
nikkolasg Sep 21, 2024
e0ad8d5
correct update depending on type
nikkolasg Sep 21, 2024
8407145
fixing update of mapping
nikkolasg Sep 21, 2024
6352db1
final test
nikkolasg Sep 21, 2024
84cd5cb
handle deletion for single
nikkolasg Sep 21, 2024
619c950
multiple rows
nikkolasg Sep 22, 2024
ab88009
Update verifiable-db/src/cells_tree/api.rs
nikkolasg Sep 23, 2024
c580c01
Update verifiable-db/src/cells_tree/api.rs
nikkolasg Sep 23, 2024
2c51871
reviews first pass
nikkolasg Sep 23, 2024
8184bd3
correct conditional scalar mul
nikkolasg Sep 23, 2024
8b30e6a
better API
nikkolasg Sep 23, 2024
3b83417
more debug
nikkolasg Sep 23, 2024
e5635d0
more debug
nikkolasg Sep 23, 2024
1d05c47
Optimize SQL queries to get Merkle-path
nicholas-mainardi Sep 23, 2024
f877735
testing split digest
nikkolasg Sep 23, 2024
90e4d42
with correct name
nikkolasg Sep 23, 2024
5e057ed
more tests
nikkolasg Sep 23, 2024
9020e54
fixes
nikkolasg Sep 23, 2024
c90e821
query adaptable
nikkolasg Sep 23, 2024
edad5fa
putting back ignoring query
nikkolasg Sep 23, 2024
71c2ce7
Merge branch 'main' into feat/merge_table
nikkolasg Sep 23, 2024
8ce5ca1
Fix non-existence proven node selection
nicholas-mainardi Sep 25, 2024
185ff79
Comment to clarify non-existence node selection logic
nicholas-mainardi Sep 25, 2024
994a115
Check query upper bounds in Parsil + add LIMIT to query if missing
nicholas-mainardi Sep 26, 2024
5fd774f
Refactor revelation APIs to avoid calling ids_for_placeholder_hash
nicholas-mainardi Sep 26, 2024
c89fc0f
Add comment for cells tree hash construction
nicholas-mainardi Sep 27, 2024
0f706cf
Fix: ensure we always use the appropriate merge flag in rows tre circ…
nicholas-mainardi Oct 1, 2024
06941a7
Implement the MPT part of generic extraction.
silathdiir Sep 30, 2024
3b368f4
Replace `32` with `MAPPING_LEAF_VALUE_LEN` in column gadget.
silathdiir Oct 6, 2024
84143e8
Replace the wrong index `7 - j` with `NUM_LAST_BITS_LOOKUP_TABLES - 1…
silathdiir Oct 6, 2024
69014f4
Re-use `length_bytes` as `real_len`.
silathdiir Oct 6, 2024
22068c4
Replace `less_than_or_equal_to` with `less_than_or_equal_to_unsafe` i…
silathdiir Oct 6, 2024
af66891
Fix to pack `extract_value`.
silathdiir Oct 6, 2024
1fbe615
Delete the wrong addition of `mpt_key` and `offset`.
silathdiir Oct 6, 2024
08f34cd
Replace `UInt256Target::new_from_be_target_limbs` with `new_from_targ…
silathdiir Oct 6, 2024
dce73a4
Delete uesless range check for slot and EVM word, since they're hashe…
silathdiir Oct 6, 2024
9c9519a
Fix to pack before digest computation as `D(key_id || pack(left_pad32…
silathdiir Oct 6, 2024
6640d85
Fix to only add values digest if EVM word is zero in leaf mapping cir…
silathdiir Oct 6, 2024
87809b8
Fix to use one field for the prefix `KEY`.
silathdiir Oct 6, 2024
94c9de8
Fix wrong byte order in `unpack_u32_to_u8_target`.
silathdiir Oct 7, 2024
691b69f
Add mapping of mappings leaf circuit.
silathdiir Oct 7, 2024
4f27919
Update storage key.
silathdiir Oct 7, 2024
08642f3
Update values extraction API.
silathdiir Oct 7, 2024
c7371b0
Add tests for column and metadata gadgets.
silathdiir Oct 8, 2024
6b8c7af
Add tests for storage key.
silathdiir Oct 9, 2024
cdc353b
Add test for leaf single circuit.
silathdiir Oct 10, 2024
7127536
Add test to leaf mapping circuit.
silathdiir Oct 10, 2024
e54449a
Add test for leaf mapping of mappings circuit.
silathdiir Oct 10, 2024
0cd9487
Merge branch 'main' into feat/merge_table
nikkolasg Oct 10, 2024
1da45a1
Delete `aggregation type` (`is_simple_aggregation`) in branch circuit.
silathdiir Oct 10, 2024
b23465d
Replace `31` with `MAPPING_LEAF_VALUE_LEN - 1`.
silathdiir Oct 11, 2024
46eeeb4
Replace wrong `curve_eq` with `connect_curve_points` in `TestMetadata…
silathdiir Oct 11, 2024
65fe5cf
Replace wrong `curve_eq` with `connect_curve_points` in `TestColumnGa…
silathdiir Oct 11, 2024
5b8ad23
Fix a bug in `ColumnGadget`.
silathdiir Oct 11, 2024
23ecf24
Move common fields to `MetadataGadget`.
silathdiir Oct 11, 2024
cd5fa5c
Fix `unpack_u32_to_u8_targets`.
silathdiir Oct 11, 2024
3407665
Fix to only set maximum as `u8::MAX + 8` for `first_bits_5` lookup ta…
silathdiir Oct 11, 2024
bc84083
Fix to generic paramters for `impl CircuitLogicWires`.
silathdiir Oct 11, 2024
380be6b
Add API tests.
silathdiir Oct 13, 2024
02290cd
Add simple struct, mapping struct and mapping of mappings to the test…
silathdiir Oct 13, 2024
ef7d07b
Update bindings.
silathdiir Oct 13, 2024
43791de
Enable single value extraction in integration test.
silathdiir Oct 13, 2024
08ebeea
Enable mapping value extraction in integration test.
silathdiir Oct 14, 2024
3bd0319
Optimize normalize_left to reduce cost of extract values
nicholas-mainardi Oct 14, 2024
7b63f6e
fmt
nicholas-mainardi Oct 14, 2024
9757f60
Fix the assigment to `evm_word`.
silathdiir Oct 14, 2024
6428a2f
Fix `MAX_COLUMNS` and `MAX_FIELD_PER_EVM` to generic parameters.
silathdiir Oct 14, 2024
b7e9ed3
Pass into the extracted column identifier to init `MetadataGadget`.
silathdiir Oct 15, 2024
ad4ddd9
Fix `outer_key_id` and `inner_key_id` to `Option` in `StorageSlotInfo…
silathdiir Oct 15, 2024
20f3933
Add test `test_values_extraction_api_serialization`.
silathdiir Oct 15, 2024
32f9837
Remove `evm_word` for generating mapping key identifier.
silathdiir Oct 15, 2024
d87b8a6
Fix encoded data.
silathdiir Oct 15, 2024
b405f04
Add common functions for metadata and values digest computation.
silathdiir Oct 15, 2024
5916860
Merge branch 'feat/merge_table' into feat/tabular-queries
nicholas-mainardi Oct 15, 2024
7b12a98
Fix to check the expected metadata digest in API test.
silathdiir Oct 15, 2024
1098ce5
Specify `NODE_LEN` to `69` as `MAX_LEAF_NODE_LEN`.
silathdiir Oct 15, 2024
39d522d
Fix build.
silathdiir Oct 15, 2024
635eb56
Clean the current useless code in integration test.
silathdiir Oct 15, 2024
63fc418
Enable rows digest check in integration test.
silathdiir Oct 15, 2024
9002398
Merge branch 'feat/tabular-queries' into feat/unproven-limit-offset-q…
nicholas-mainardi Oct 15, 2024
c87856f
Add LIMIT by default only in simple SELECT queries
nicholas-mainardi Oct 15, 2024
c7bd811
Avoid running wildcard queries on merged tables
nicholas-mainardi Oct 15, 2024
a5a6962
Fix parsil test
nicholas-mainardi Oct 15, 2024
9c325d6
Merge remote-tracking branch 'origin/feat/unproven-limit-offset-queri…
silathdiir Oct 15, 2024
e3292a6
Add a new constant `EVM_WORD_LEN` for the slot value length.
silathdiir Oct 16, 2024
0048e0d
Remove `num_actual_columns` from the arguments of `MetadataGadget::ne…
silathdiir Oct 16, 2024
ef51ed8
Add check for the parent of a Slot mapping entry, it must be type of …
silathdiir Oct 16, 2024
bc90aba
Compute the column information digest in `ColumnInfo` and called in `…
silathdiir Oct 16, 2024
de7e002
Fix the Key ID constants to size of Uint32 or Uint64.
silathdiir Oct 16, 2024
8a85e45
Fix `evm_word` to `usize`, key IDs to `u64` and `extracted_column_ide…
silathdiir Oct 16, 2024
896e85a
Fix build.
silathdiir Oct 16, 2024
765731c
Fix tests.
silathdiir Oct 16, 2024
e36d424
Fix build.
silathdiir Oct 16, 2024
75937e2
Remove `extracted_column_identifiers` and `evm_word` from metadata co…
silathdiir Oct 16, 2024
5249a29
Fix tests.
silathdiir Oct 16, 2024
83a68e3
Pass `MetadataGadget` to the API functions.
silathdiir Oct 16, 2024
ec22c05
Check `values_digest` in `prove_leaf`.
silathdiir Oct 16, 2024
ddd0e9e
Replace `key_id` with `value_id` in `value_metadata` function.
silathdiir Oct 16, 2024
96f2223
Fix to place the value column for mappings of simple variables.
silathdiir Oct 16, 2024
7671861
Support mapping of Struct and mapping of mappings for the value metad…
silathdiir Oct 16, 2024
3043229
Update the metadata hash and integration test.
silathdiir Oct 17, 2024
92c73f4
Add `extracted_column_indentifiers` function to `MetadataGadget`.
silathdiir Oct 17, 2024
e5c21d1
Fix to check MPT `location` bytes without unpacking.
silathdiir Oct 17, 2024
17680fb
Fix the merge circuit metadata to `D(md_a) + D(md_b)`.
silathdiir Oct 17, 2024
1058717
Fix test.
silathdiir Oct 17, 2024
94dca85
fix: MemoryStorage::all_keys_at should not return dead keys
delehef Oct 17, 2024
e58faee
feat: add random_key_at
delehef Oct 17, 2024
3548cc5
Do range-check for the Keccak output which will be converted for Uint…
silathdiir Oct 22, 2024
aea77c6
Refactor the assign functions for the simple and mapping slots.
silathdiir Oct 22, 2024
85cdb54
Merge branch 'feat/merge_table' into feat/tabular-queries
nicholas-mainardi Oct 29, 2024
791ba8a
Merge branch 'feat/tabular-queries' into feat/unproven-limit-offset-q…
nicholas-mainardi Oct 29, 2024
65ff089
Merge branch 'feat/unproven-limit-offset-queries' into generic-mpt-ex…
nicholas-mainardi Oct 29, 2024
5e2df85
Simple Select Queries with Unchecked Offset (#365)
nicholas-mainardi Oct 29, 2024
0ad3977
Merge remote-tracking branch 'origin/feat/tabular-queries' into gener…
silathdiir Oct 30, 2024
5483d50
chore: clippy
delehef Oct 31, 2024
60d3f34
[parsil] correctly handle LIMIT/OFFSET
delehef Nov 5, 2024
35d9623
[mp2] use more explicit names
delehef Nov 5, 2024
5bb0490
fix: imports
delehef Nov 5, 2024
2e32687
Merge branch 'main' into feat/tabular-queries
nicholas-mainardi Nov 6, 2024
3644f30
Fix integration test + fmt
nicholas-mainardi Nov 6, 2024
6a27c73
use u32's for offset & limit
delehef Nov 6, 2024
ec32679
Fix test failing from time to time
nicholas-mainardi Nov 6, 2024
d6cd3ad
Merge remote-tracking branch 'origin/feat/tabular-queries' into gener…
silathdiir Nov 7, 2024
736d790
Merge remote-tracking branch 'origin/main' into generic-mpt-extraction
silathdiir Nov 17, 2024
2c1ee60
Merge remote-tracking branch 'origin/main' into generic-mpt-extraction
silathdiir Dec 13, 2024
be0c0a1
Merge branch 'main' into generic-mpt-extraction
nicholas-mainardi Jan 24, 2025
bedf4be
feat: update DB creation circuits for generic extraction (Part 2) (#393)
silathdiir Mar 5, 2025
59ec982
Merge branch 'feat/new_extraction_features' into generic-mpt-extraction
nicholas-mainardi Mar 5, 2025
adbf314
Fix deterministic serialization test
nicholas-mainardi Mar 5, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
336 changes: 234 additions & 102 deletions Cargo.lock

Large diffs are not rendered by default.

2 changes: 1 addition & 1 deletion Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ members = [
resolver = "2"

[workspace.dependencies]
alloy = { version = "0.2", default-features = false, features = [
alloy = { version = "0.4", default-features = false, features = [
"consensus",
"contract",
"getrandom",
Expand Down
16 changes: 12 additions & 4 deletions groth16-framework/tests/common/context.rs
Original file line number Diff line number Diff line change
Expand Up @@ -8,8 +8,8 @@ use verifiable_db::{
api::WrapCircuitParams,
revelation::api::Parameters as RevelationParameters,
test_utils::{
MAX_NUM_ITEMS_PER_OUTPUT, MAX_NUM_OUTPUTS, MAX_NUM_PLACEHOLDERS, MAX_NUM_PREDICATE_OPS,
MAX_NUM_RESULT_OPS,
INDEX_TREE_MAX_DEPTH, MAX_NUM_COLUMNS, MAX_NUM_ITEMS_PER_OUTPUT, MAX_NUM_OUTPUTS,
MAX_NUM_PLACEHOLDERS, MAX_NUM_PREDICATE_OPS, MAX_NUM_RESULT_OPS, ROW_TREE_MAX_DEPTH,
},
};

Expand All @@ -18,10 +18,14 @@ pub(crate) struct TestContext {
pub(crate) preprocessing_circuits: TestingRecursiveCircuits<F, C, D, NUM_PREPROCESSING_IO>,
pub(crate) query_circuits: TestingRecursiveCircuits<F, C, D, NUM_QUERY_IO>,
pub(crate) revelation_params: RevelationParameters<
ROW_TREE_MAX_DEPTH,
INDEX_TREE_MAX_DEPTH,
MAX_NUM_COLUMNS,
MAX_NUM_PREDICATE_OPS,
MAX_NUM_RESULT_OPS,
MAX_NUM_OUTPUTS,
MAX_NUM_ITEMS_PER_OUTPUT,
MAX_NUM_PLACEHOLDERS,
{ 2 * (MAX_NUM_PREDICATE_OPS + MAX_NUM_RESULT_OPS) },
>,
pub(crate) wrap_circuit:
WrapCircuitParams<MAX_NUM_OUTPUTS, MAX_NUM_ITEMS_PER_OUTPUT, MAX_NUM_PLACEHOLDERS>,
Expand All @@ -39,10 +43,14 @@ impl TestContext {

// Create the revelation parameters.
let revelation_params = RevelationParameters::<
ROW_TREE_MAX_DEPTH,
INDEX_TREE_MAX_DEPTH,
MAX_NUM_COLUMNS,
MAX_NUM_PREDICATE_OPS,
MAX_NUM_RESULT_OPS,
MAX_NUM_OUTPUTS,
MAX_NUM_ITEMS_PER_OUTPUT,
MAX_NUM_PLACEHOLDERS,
{ 2 * (MAX_NUM_PREDICATE_OPS + MAX_NUM_RESULT_OPS) },
>::build(
query_circuits.get_recursive_circuit_set(),
preprocessing_circuits.get_recursive_circuit_set(),
Expand Down
18 changes: 3 additions & 15 deletions groth16-framework/tests/common/query.rs
Original file line number Diff line number Diff line change
Expand Up @@ -30,19 +30,6 @@ impl TestContext {
let max_block_number = 76;
let test_data = TestRevelationData::sample(min_block_number, max_block_number);

let placeholder_hash_ids = QueryInput::<
MAX_NUM_COLUMNS,
MAX_NUM_PREDICATE_OPS,
MAX_NUM_RESULT_OPS,
MAX_NUM_ITEMS_PER_OUTPUT,
>::ids_for_placeholder_hash(
test_data.predicate_operations(),
test_data.results(),
test_data.placeholders(),
test_data.query_bounds(),
)
.unwrap();

// Generate the query proof.
let [query_proof] = self
.query_circuits
Expand All @@ -66,12 +53,13 @@ impl TestContext {
preprocessing_proof,
test_data.query_bounds(),
test_data.placeholders(),
placeholder_hash_ids,
test_data.predicate_operations(),
test_data.results(),
)
.unwrap();
let revelation_proof = self
.revelation_params
.generate_proof(input, self.query_circuits.get_recursive_circuit_set())
.generate_proof(input, self.query_circuits.get_recursive_circuit_set(), None)
.unwrap();
let revelation_proof = ProofWithVK::deserialize(&revelation_proof).unwrap();
let (revelation_proof_with_pi, _) = revelation_proof.clone().into();
Expand Down
11 changes: 8 additions & 3 deletions mp2-common/src/array.rs
Original file line number Diff line number Diff line change
Expand Up @@ -166,16 +166,21 @@ impl<const MAX_LEN: usize> VectorWire<Target, MAX_LEN> {
) -> Array<Target, PAD_LEN> {
let zero = b.zero();
let pad_t = b.constant(F::from_canonical_usize(PAD_LEN));
// initialize is_lt to false
let mut is_lt = b._false();
Array {
arr: create_array(|i| {
// ((pad_len - i) < real_len) * vec[real_len - (pad_len-i)]
// i.e. reading value backwards and inserting in order
let it = b.constant(F::from_canonical_usize(i));
let jt = b.sub(pad_t, it);
let is_lt =
less_than_or_equal_to(b, jt, self.real_len, (PAD_LEN.ilog2() + 1) as usize);
// update is_lt incrementally: it should become true when PAD_LEN-i == self.real_len
let is_eq = b.is_equal(jt, self.real_len);
is_lt = b.or(is_lt, is_eq);
let idx = b.sub(self.real_len, jt);
let val = self.arr.value_at_failover(b, idx);
let idx = b.select(is_lt, idx, zero); // workaround to always have a in-bound access in
// the array
let val = self.arr.value_at(b, idx);
b.select(is_lt, val, zero)
}),
}
Expand Down
Loading
Loading