feat: integrate AuthDecode #634

themighty1 · 2024-10-10T14:03:26Z

This PR integrates AuthDecode into the notarization protocol and also makes accommodating changes to how the plaintext hashes are handled in tlsn-core.
AuthDecode is marked as experimental and is feature-gated.

Depends on #479 being merged into dev first.

(I had trouble basing this PR on PR479, so I'm basing it on dev)

* refactor: selective disclosure api * remove incomplete substring proof API * remove unnecessary type annotation * simplify tests * switch from unit structs to empty structs * skip committing empty strings * fix notary server test * rename RecordKind to MessageKind * update json commit error doc * commits -> commits to * update commit_array doc * function argument doc styling * Update tlsn/tlsn-core/src/proof/substrings.rs Co-authored-by: dan <[email protected]> --------- Co-authored-by: dan <[email protected]>

* Use env var for logging filter, remove otel. * Fix directives. * Revert to using config for logging filter. * Modify default logging strategy and make filter optional. * Revert formatting of other crates. * Update README. * Update notary-server/README.md Co-authored-by: Hendrik Eeckhaut <[email protected]> --------- Co-authored-by: Hendrik Eeckhaut <[email protected]>

* remove dead argument doc * remove another dead argument doc

) * feat(tlsn-formats): default commit to entire http request/response * refactor(tlsn-formats): avoid duplicate HTTP commitments, add test fixtures

* add notary function to examples lib * use hyper 1.1 version in examples * update twitter example to use HTTP prover * use deferred decryption in twitter example

* chore: bump tlsn-utils version * chore: bump mpz version * bump mpz

* bump version to v0.1.0-alpha.4 * set package version for tlsn-formats

feat: basic html info response for notary server's root endpoint Co-authored-by: Christopher Chong <[email protected]>

* feat(tls-mpc): separate transcript size limits * feat: separate transcript limits * feat(tlsn-server-fixture): configurable length byte payload * refactor(tls-mpc): use defaults in ghash setup * fix OT estimates * feat(notary-server): separate transcript limits * remove dep patch * fix notary server test

* Update repo readme. * Doc: Added minor improvements + a link to other repos #353 --------- Co-authored-by: Hendrik Eeckhaut <[email protected]>

#440 Co-authored-by: Christopher Chong <[email protected]>

* feat: automated network benches * Update tlsn/benches/src/metrics.rs Co-authored-by: dan <[email protected]> * remove explicit drops * remove unnecessary sudo --------- Co-authored-by: dan <[email protected]>

* feat: implement record layer preprocessing * fix ke test * fix pa tests * fix aead tests * fix integration test * Apply suggestions from code review Co-authored-by: dan <[email protected]> * add mode sanity check --------- Co-authored-by: dan <[email protected]>

+ Plot runtime vs latency graph and bandwidth

* Remove cargo/bin from PATH * Modify script to run only in nightly env * Modify script to stop the oldest version in stable env * Modify script to support dir preparation for the 3 latest stable versions * Modify script to start service for the 3 latest stable versions * Modify sercice validation script * Create proxy modification script * Add step in workflow to enable ssm execution against proxy + aux script * Add running state filter when fetching InstanceID * Enhancement of validation script * Modify bash behavior * Point tags/deployment to new AWS resources * Change GH owner to production one * Point tags to new EC2 v1 * Move all cd scripts to a new folder * Add comment * Add comment * Add comment * Add comment * Modify scripts to support exit on error * Check if all stable ports are in use and terminate

* Add branches info in readme. * Correct branch links.

* Add hot reload of api key, remove prover ip, move html static text. * Add documentation. * Toggle back config, add comments. * Edit comment and html info. * Edit comment. * Change to sync mutex.

#451

Co-authored-by: sinu.eth <[email protected]>

…ion (#472)

ci: Update rust cache in GitHub action and do not skip draft PRs

* docs: fix style in components (except tls) * Update components/cipher/stream-cipher/src/lib.rs Co-authored-by: Hendrik Eeckhaut <[email protected]> * Update components/universal-hash/src/ghash/ghash_core/mod.rs Co-authored-by: Hendrik Eeckhaut <[email protected]> --------- Co-authored-by: Hendrik Eeckhaut <[email protected]>

* Add tests for signing, index. * Add error scenarios. * Add cert tests, modify previous tests. * Improve cert tests. * Add tests for request. * Fix clippy * Fix clippy. * Change requests test style. * Add attestation unit tests. * Formatting. * Clippy. * make data fixtures optional --------- Co-authored-by: yuroitaki <> Co-authored-by: sinu <[email protected]>

… url path (#614) * (fix: client) Fixed client issue of being able to implement the path for the url * (feat: client) Improved the code to adjust for feedback received as well as extend the path calculation to avoid adding a `/` when already starts with a `/` * (fix: client) Fixed client issue of being able to implement the path for the url * (feat: client) Improved the code to adjust for feedback received as well as extend the path calculation to avoid adding a `/` when already starts with a `/` * Update crates/notary/client/src/client.rs Co-authored-by: yuroitaki <[email protected]> * (fix: client) Renamed `path` to `path_prefix` * (fix: client) Remove condition on the URL * (chore: client) Fix formating --------- Co-authored-by: yuroitaki <[email protected]>

sinui0

Good work!

I think we can make some simplifications, and avoid leaking details into core which should not be aware of the authdecode protocol at all. core should only interact with plaintext data and hashes.

sinui0 · 2024-10-11T22:22:53Z

crates/core/src/hash.rs

+            .map(|bytes| {
+                let mut bytes = bytes.to_vec();
+                // Reverse to little-endian.
+                bytes.reverse();


Why is this necessary? The bytes do not have an endianness, they do not encode an integer. Can this implementation detail be moved behind the authdecode API?

This is necessary for F::from_bytes below which expects LE.

My thinking was that for the devs who will use the hash in their circom circuits, it will be conceptually simpler to think that the 31-byte chunk of plaintext must be viewed as a BE/MSB0 bitstring which represent the field element.
That's why I purposely stick to BE.

This detail should be handled behind the authdecode API. From the perspective of the core lib, it gives a slice of the plaintext and it gets a hash back. It shouldn't be aware

sinui0 · 2024-10-11T22:25:02Z

crates/core/src/hash.rs

@@ -236,6 +244,11 @@ pub trait HashAlgorithm {

    /// Computes the hash of the provided data with a prefix.
    fn hash_prefixed(&self, prefix: &[u8], data: &[u8]) -> Hash;
+
+    /// Computes the hash of the provided blinded data.
+    fn hash_blinded(&self, data: &Blinded<Vec<u8>>) -> Hash {


This is redundant. The hash method should be sufficient

This was needed because currently when hashing the blinded plaintext, it gets serialized first, e.g. here

tlsn/crates/core/src/transcript/hash.rs

Line 65 in 38104bc

if commitment.hash.value != alg.hash_canonical(&self.data) {

This means that some serialization data will be present in the pre-image. Which means that my Poseidon hasher cannot reliably know the last 16 bytes of the pre-image is the blinder.

Recall that the Poseidon hasher places the blinder in a separate field element, so it must be able to separate it from the plaintext. (The need to put the blinder into a separate field element is a limitation of the halo2 circuit. This limitation can potentially be lifted but requires redesigning the circuit.)

Makes sense that we can't use the hash_canonical but instead of adding a hash_blinded method we can just use hash and construct the preimage as needed, ie data | blinder

This trait is a user-facing extension point, and adding this method would force them to be aware of how to serialize Blinded<Vec<u8>> but we decide that. By matter of convention we can say all preimages are data | blinder

it would probably be easier to switch to blinder | data at some point

To summarize, are you suggesting that in this place
if commitment.hash.value != alg.hash_canonical(&self.data) {
we should check if the hash alg requires AuthDecode, and if so, then we use hash(data|blinder).
For all other algs, we keep using hash_canonical(&self.data), correct?

sinui0 · 2024-10-23T00:33:13Z

crates/common/src/config.rs

+    #[cfg(feature = "authdecode_unsafe_common")]
+    /// Maximum number of bytes which can be committed to using a zk-friendly hash.
+    #[builder(default = "0")]
+    max_zk_friendly_hash_data: usize,


Naming is kind of clunky, perhaps max_authdecode?

sinui0 · 2024-10-23T00:34:45Z

crates/components/authdecode/single-range/Cargo.toml

It is overkill to add single-range to the crate name. That can just be communicated via the API and changed in future versions as needed.

Are you suggesting calling it e.g. authdecode-transcript and to only allow it to be instantiated with a single range?

I would just call it authdecode

done, I called it transcript

sinui0 · 2024-10-23T00:36:40Z

crates/components/authdecode/single-range/src/lib.rs

+}
+
+/// An encoder of a TLS transcript.
+pub struct TranscriptEncoder {


I don't think we need this. The encodings can be provided directly via the API

Do you have an approach in mind?
Currently in crates/core/src/transcript/encoding/encoder.rs encode_subsequence returns active encodings and is pub(crate), but I need full encodings.

crates/core/src/transcript/encoding/encoder.rs

sinui0 · 2024-10-23T01:32:51Z

crates/core/src/transcript/hash.rs

@@ -62,7 +59,7 @@ impl PlaintextHashProof {
    ) -> Result<(Direction, Subsequence), PlaintextHashProofError> {
        let alg = provider.get(&commitment.hash.alg)?;

-        if commitment.hash.value != alg.hash_canonical(&self.data) {
+        if commitment.hash.value != alg.hash_blinded(&self.data) {


It is already blinded, hash_canonical is correct. Is this an issue with how the authdecode circuit structures the preimage?

crates/core/src/transcript/proof.rs

sinui0 · 2024-10-23T01:50:31Z

crates/core/src/transcript/proof.rs

@@ -151,7 +170,7 @@ impl<'a> TranscriptProofBuilder<'a> {
    pub(crate) fn new(
        transcript: &'a Transcript,
        encoding_tree: Option<&'a EncodingTree>,
-        plaintext_hashes: &'a Index<PlaintextHashSecret>,
+        plaintext_hashes: &'a Option<Index<PlaintextHashSecret>>,


Why not just accept an empty index?

Seemed more straightforward to reason: "I don't want to send you anything, so Im sending you None".

crates/core/src/transcript/proof.rs

themighty1 · 2024-10-24T12:16:00Z

@sinui0, I'd be happy to not leak authdecode details into core but the prover crate needs access to the fields of the request and the secrets in order to build authdecode inputs.

tlsn/crates/prover/src/notarize.rs

Lines 93 to 99 in e90159f

    
           let mut authdecode_prover = authdecode_prover( 
        
               &request, 
        
               &secrets, 
        
               &*encoding_provider, 
        
               &transcript, 
        
               max, 
        
           )?;

Those fields are marked as pub(crate), so the only solution was to leak authdecode into core.

I couldn't think of a simple solution to this, lmk if you have anything in mind.

sinui0 · 2024-10-24T12:50:39Z

Those fields are marked as pub(crate), so the only solution was to leak authdecode into core.

I couldn't think of a simple solution to this, lmk if you have anything in mind.

Decouple authdecode from the attestation. The prover can instead send as a separate message the ranges they want to commit using authdecode. They run the protocol beforehand, afterwards the Notary holds the hashes and inserts them into the attestation. Remember that we will also want to use this for the P2P case

themighty1 · 2024-10-28T14:14:47Z

@sinui0 , do you have any suggestions what approach you think is best here:

tlsn/crates/prover/src/notarize.rs

Lines 85 to 88 in 30e4e37

    
           vm.finalize().await?; 
        
           let attestation: Attestation = io.expect_next().await?;

we need the prover to get the seed
let seed = vm.finalize().await?;

and then the seed should be passed to the caller who will generate and send AuthDecode proofs in an external context while the prover is awaiting the attestation on line 87.

Also we need a similar change in Verifier<Notarize>::finalize() - the verifier should not send the attestation until some external context signals that it is safe to do so.

sinui0 · 2024-10-28T14:32:34Z

We can add a new message which the Prover sends to the Verifier to signal they want hash commitments (based on the TranscriptCommitConfig).

They start the commit phase of authdecode, finalize the VM, then finalize authdecode. Afterwards the Prover sends the attestation request.

themighty1 · 2024-10-29T07:15:48Z

@sinui0 , but the attestation request contains commitments required before finalization. The attestation request must be sent before finalizing the VM.
We can work around it by having the prover first commit to the attestation request and then the rest of the steps will be like you describe above. wdyt.

sinui0 · 2024-10-29T13:55:50Z

We can work around it by having the prover first commit to the attestation request and then the rest of the steps will be like you describe above. wdyt.

Let's remove the encoding commitment from the request and send them earlier. The Notary can add them when building the attestation

themighty1 · 2024-10-31T13:29:10Z

@sinui0 , what's the best approach for allocating io for the AuthDecode protocol?
I was hoping to add this to Prover<Notarize>

    /// Allocates a new io.
    pub async fn allocate_io(&self, io_name: String) -> Io {
        self.state.mux_ctrl.open_framed(&io_name).await.unwrap()
    }

The problem is that Prover<Notarize>::finalize closes yamux, which will make it impossible for AuthDecode to finalize using the allocated io.
Any good solutions to this?

sinui0 · 2024-10-31T16:10:50Z

I'm not sure I understand. Authdecode can be communicated over the existing io channel, and it should be completed before yamux is shut down

themighty1 · 2024-11-07T08:14:14Z

Ready for review. The biggest changes are:

decouple authdecode from the core crate
field id is sent with the plaintext hash in the attestation request
moved builder logic into crates/core/src/transcript/commit/builder.rs

@sinui0

sinui0 and others added 30 commits February 8, 2024 13:52

fix(tls-core): use non_exhaustive instead of private zst (#428)

29fb409

docs(tlsn-core): update Direction docs (#427)

e19fb00

docs(tlsn-formats): remove dead argument docs (#429)

a439838

* remove dead argument doc * remove another dead argument doc

fix(tlsn-formats): prevent duplicate json array commitment (#432)

82b9582

feat(tlsn-formats): default commit to entire http request/response (#433

98a3c4d

) * feat(tlsn-formats): default commit to entire http request/response * refactor(tlsn-formats): avoid duplicate HTTP commitments, add test fixtures

refactor(tlsn-examples): update hyper and use http prover (#434)

0d269ed

* add notary function to examples lib * use hyper 1.1 version in examples * update twitter example to use HTTP prover * use deferred decryption in twitter example

chore: bump deps (#430)

c7abc8c

* chore: bump tlsn-utils version * chore: bump mpz version * bump mpz

chore: v0.1.0-alpha.4 release prep (#437)

309c37f

* bump version to v0.1.0-alpha.4 * set package version for tlsn-formats

Update notary server README for frequent q&a. (#441)

1e99db8

Show basic html info response for notary server's root endpoint (#439)

be24f58

feat: basic html info response for notary server's root endpoint Co-authored-by: Christopher Chong <[email protected]>

Update repo readme. (#450)

eec9310

* Update repo readme. * Doc: Added minor improvements + a link to other repos #353 --------- Co-authored-by: Hendrik Eeckhaut <[email protected]>

feat: interactive verifier example (#451)

d7bc0e5

#440 Co-authored-by: Christopher Chong <[email protected]>

fix(tlsn-formats): fix commitment error caused by empty headers (#452)

19e9c50

feat: automated network benches (#457)

9a081c6

* feat: automated network benches * Update tlsn/benches/src/metrics.rs Co-authored-by: dan <[email protected]> * remove explicit drops * remove unnecessary sudo --------- Co-authored-by: dan <[email protected]>

perf: Docker container for running benches + manual GitHub action (#460)

aa264a9

+ Plot runtime vs latency graph and bandwidth

Add branches info in readme. (#467)

9e041b8

Correct branch links in readme. (#469)

a4c7760

* Add branches info in readme. * Correct branch links.

Add api key whitelist hot reloading and small touch-up (#458)

5c0a030

* Add hot reload of api key, remove prover ip, move html static text. * Add documentation. * Toggle back config, add comments. * Edit comment and html info. * Edit comment. * Change to sync mutex.

docs: List interative example in the examples README (#471)

d53203e

#451

chore: Bump versions for release alpha.5. (#470)

b4334ad

Co-authored-by: sinu.eth <[email protected]>

fix: drop connection instead of manual close, enable deferred decrypt…

68b9474

…ion (#472)

Update rust cache in github action (#453)

f558d5b

ci: Update rust cache in GitHub action and do not skip draft PRs

docs: misc fixups (#475)

f63a74e

docs: fix style (#476)

9fc0d91

heeckhau and others added 6 commits October 3, 2024 07:18

docs: correct foldername in examples readme (#624)

0596a9a

ci: generate coverage report

2ac9de1

ci: Try codecov.io

61ff3a8

feat: integrate AuthDecode

e90159f

themighty1 requested a review from sinui0 October 10, 2024 14:08

sinui0 requested changes Oct 23, 2024

View reviewed changes

themighty1 changed the base branch from dev to authdecode_2024 October 24, 2024 09:21

themighty1 requested a review from sinui0 November 7, 2024 08:14

themighty1 added 11 commits November 7, 2024 09:19

decouple authdecode from tlsn-core

c70f196

externalize preimage chunking/padding

9b831d0

rename single-range to transcript

a4d498e

give poseidon a better name

2c50136

make builder fn fallible

6317bc4

send field id with plaintext hash in request

bb5701c

change retval for blinder's as_inner

ce85024

make plaintext_hashes return an iter

ad50b39

use let Some instead of match

45c8f90

remove whitespace

f0905c7

fix Cargo.toml

72b2673

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: integrate AuthDecode #634

feat: integrate AuthDecode #634

themighty1 commented Oct 10, 2024 •

edited

Loading

sinui0 left a comment

sinui0 Oct 11, 2024

themighty1 Nov 1, 2024

sinui0 Nov 1, 2024

themighty1 Nov 7, 2024

sinui0 Oct 11, 2024

themighty1 Nov 1, 2024

sinui0 Nov 1, 2024

sinui0 Nov 1, 2024

sinui0 Nov 1, 2024

themighty1 Nov 4, 2024

sinui0 Oct 23, 2024

themighty1 Nov 7, 2024

sinui0 Oct 23, 2024

themighty1 Nov 1, 2024

sinui0 Nov 1, 2024

themighty1 Nov 7, 2024

sinui0 Oct 23, 2024

themighty1 Nov 4, 2024

sinui0 Oct 23, 2024

sinui0 Oct 23, 2024

themighty1 Nov 4, 2024

themighty1 commented Oct 24, 2024

sinui0 commented Oct 24, 2024

themighty1 commented Oct 28, 2024

sinui0 commented Oct 28, 2024

themighty1 commented Oct 29, 2024

sinui0 commented Oct 29, 2024

themighty1 commented Oct 31, 2024 •

edited

Loading

sinui0 commented Oct 31, 2024

themighty1 commented Nov 7, 2024

feat: integrate AuthDecode #634

Are you sure you want to change the base?

feat: integrate AuthDecode #634

Conversation

themighty1 commented Oct 10, 2024 • edited Loading

sinui0 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

themighty1 commented Oct 24, 2024

sinui0 commented Oct 24, 2024

themighty1 commented Oct 28, 2024

sinui0 commented Oct 28, 2024

themighty1 commented Oct 29, 2024

sinui0 commented Oct 29, 2024

themighty1 commented Oct 31, 2024 • edited Loading

sinui0 commented Oct 31, 2024

themighty1 commented Nov 7, 2024

themighty1 commented Oct 10, 2024 •

edited

Loading

themighty1 commented Oct 31, 2024 •

edited

Loading