Batch Amendment #5060

dangell7 · 2024-07-10T15:45:43Z

High Level Overview of Change

Context of Change

Type of Change

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to not work as expected)
Refactor (non-breaking change that only restructures code)
Performance (increase or change in throughput and/or latency)
Tests (you added tests for code that already exists, or your new feature included in this PR)
Documentation update
Chore (no impact to binary, e.g. .gitignore, formatting, dropping support for older tooling)
Release

API Impact

Public API: New feature (new methods and/or new fields)
Public API: Breaking change (in general, breaking changes should only impact the next api_version)
libxrpl change (any change that may affect libxrpl or dependents of libxrpl)
Peer protocol change (must be backward compatible or bump the peer protocol version)

codecov · 2024-07-10T16:10:41Z

Codecov Report

Attention: Patch coverage is 90.02433% with 41 lines in your changes missing coverage. Please review.

Project coverage is 77.9%. Comparing base (7b18006) to head (04519e6).

Files with missing lines	Patch %	Lines
src/xrpld/app/tx/detail/Batch.cpp	94.2%	9 Missing ⚠️
src/xrpld/app/tx/detail/Transactor.cpp	87.1%	9 Missing ⚠️
src/xrpld/overlay/detail/PeerImp.cpp	0.0%	8 Missing ⚠️
src/libxrpl/protocol/STTx.cpp	90.9%	6 Missing ⚠️
src/xrpld/app/misc/NetworkOPs.cpp	66.7%	6 Missing ⚠️
src/libxrpl/protocol/TxMeta.cpp	83.3%	1 Missing ⚠️
src/xrpld/app/ledger/detail/OpenLedger.cpp	50.0%	1 Missing ⚠️
src/xrpld/app/tx/detail/apply.cpp	88.9%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff            @@
##           develop   #5060     +/-   ##
=========================================
+ Coverage     77.9%   77.9%   +0.1%     
=========================================
  Files          782     785      +3     
  Lines        66614   66964    +350     
  Branches      8159    8182     +23     
=========================================
+ Hits         51861   52174    +313     
- Misses       14753   14790     +37

Files with missing lines	Coverage Δ
include/xrpl/protocol/Batch.h	`100.0% <100.0%> (ø)`
include/xrpl/protocol/Feature.h	`100.0% <ø> (ø)`
include/xrpl/protocol/HashPrefix.h	`100.0% <ø> (ø)`
include/xrpl/protocol/STTx.h	`100.0% <ø> (ø)`
include/xrpl/protocol/TER.h	`100.0% <ø> (ø)`
include/xrpl/protocol/TxMeta.h	`93.5% <100.0%> (+1.9%)`	⬆️
include/xrpl/protocol/detail/transactions.macro	`100.0% <100.0%> (ø)`
src/libxrpl/protocol/InnerObjectFormats.cpp	`100.0% <100.0%> (ø)`
src/libxrpl/protocol/TER.cpp	`100.0% <ø> (ø)`
src/xrpld/app/tx/detail/ApplyContext.cpp	`100.0% <100.0%> (ø)`
... and 17 more

... and 5 files with indirect coverage changes

src/libxrpl/protocol/STTx.cpp

This is not finished. Needs to only revert the transactions submitted not the entire open ledger.

src/xrpld/app/tx/detail/Batch.cpp

ximinez · 2024-07-31T16:36:25Z

Also, since it seems this PR is still under active development, could you convert it to a draft?

shawnxie999 · 2024-10-30T20:21:24Z

src/xrpld/app/tx/detail/Batch.cpp

+        if (not3rdParty)
+            ctx_.batchPrevious(avi);
+
+        ctx_.applyOpenView(subView);


what's the reason for this call here?

batchPrevious applies the inner transactions to the open view. Before this, they are not actually applied, or they are applied to the subview but not the BatchOuter OpenView.

batchPrevious is a workaround for the "previousFields issue". The issue is thatwhen the outer batch txn is applied at the end, it applies the old/stale view (from before the inner txns). In the case where the outer batch account has an inner batch txn that adjusts its account root, the inner adjustments are not actually applied. They are on the open view but not applied. So this function applies the open view onto the base view and handles this issue.

This should be thoroughly reviewed. There might be another way. It might be an issue with how the views are applied.

Thanks! I will do some experiments with this. Do you know if this is a known issue that is documented somewhere? Or is it something you found while working on this PR?

Edit: nevermind, this makes sense 👍

Also just curious, does using subView.apply(ctx_.rawView()) not solve this problem? assuming that subView contains all the changes for the inner txns, shouldn't it include the latest account root state of the "non3rdParty" account? Please correct me if I'm wrong.

- allow only one flag - validate batch signers array (max and valid) - move `calculateBaseFee` - move `preflight2` - no duplicate TxIDs - no duplicate BatchSigners

shawnxie999

partial

src/xrpld/app/tx/detail/apply.cpp

src/xrpld/app/tx/detail/Batch.cpp

shawnxie999

partial

src/xrpld/app/tx/detail/Batch.cpp

shawnxie999 · 2024-11-04T15:53:56Z

src/xrpld/app/tx/detail/Batch.cpp

+
+            if (flags & tfUntilFailure)
+            {
+                result = tesSUCCESS;


so it's possible that none of the inner txns succeeds and yet the batch txn is "successful"?

src/xrpld/app/tx/detail/Batch.cpp

src/xrpld/ledger/ApplyViewImpl.h

src/xrpld/app/tx/detail/ApplyContext.cpp

src/xrpld/ledger/ApplyViewImpl.h

src/xrpld/app/tx/detail/Batch.cpp

shawnxie999

partial

src/xrpld/app/tx/detail/Transactor.cpp

src/xrpld/app/tx/detail/ApplyContext.cpp

shawnxie999 · 2024-11-04T18:54:22Z

src/xrpld/app/tx/detail/ApplyContext.cpp

+void
+ApplyContext::applyOpenView(OpenView& open)
+{
+    if (!base_.open())


could you elaborate more on why we are applying when the base is closed, and not when open? perhaps it's helpful to put more into the comment as well. thanks!

src/xrpld/app/tx/detail/Batch.cpp

src/xrpld/app/misc/NetworkOPs.cpp

shawnxie999

have some questions regarding how inner txns states could've been applied

shawnxie999 · 2024-11-05T16:22:13Z

src/xrpld/app/tx/detail/Transactor.cpp

+        if (pkSigner.empty())
+        {
+            STArray const& txSigners(signer.getFieldArray(sfSigners));
+            ret = checkMultiSign(ctx.view, idAccount, txSigners, ctx.j);


Suggested change

ret = checkMultiSign(ctx.view, idAccount, txSigners, ctx.j);

if (ret = checkMultiSign(ctx.view, idAccount, txSigners, ctx.j);

!isTesSuccess(ret))

return ret;

currently, it would return code of the result of the last signature, but shouldn't we be returning the first error that occurs in the loop? If so, we should probably add a test case to exercise the behavior where the last signer signature is correct, but previous ones are bad.

Thanks for the suggestion. I will add a test case as well

shawnxie999 · 2024-11-05T16:22:23Z

src/xrpld/app/tx/detail/Transactor.cpp

+            if (!sleAccount)
+                return tesSUCCESS;
+
+            ret = checkSingleSign(


same as above

shawnxie999 · 2024-11-05T16:31:10Z

src/xrpld/app/tx/detail/Transactor.cpp

+    ApplyViewImpl& avi = dynamic_cast<ApplyViewImpl&>(ctx_.view());
+    std::vector<STObject> executions;
+    avi.copyBatchMetaData(executions);
    ctx_.discard();
+    ApplyViewImpl& avi2 = dynamic_cast<ApplyViewImpl&>(ctx_.view());
+    avi2.setBatchMetaData(std::move(executions));


what is this block of code here for?

This is to apply the batch executions to the metadata even in the event of a tec failure

src/xrpld/app/tx/detail/Transactor.cpp

shawnxie999 · 2024-11-05T16:49:08Z

src/xrpld/app/tx/detail/Batch.cpp

+        if (not3rdParty)
+            ctx_.batchPrevious(avi);
+
+        ctx_.applyOpenView(subView);


Also just curious, does using subView.apply(ctx_.rawView()) not solve this problem? assuming that subView contains all the changes for the inner txns, shouldn't it include the latest account root state of the "non3rdParty" account? Please correct me if I'm wrong.

shawnxie999 · 2024-11-05T16:53:55Z

src/xrpld/app/tx/detail/Transactor.cpp

+        // Only update the account root entry if the batch transaction was
+        // not a 3rd party transaction
+        if (not3rdParty)
+            ctx_.updateAccountRootEntry();


linking to this comment thread

I'm still unclear why exactly we need to explicitly update account root entry here. Does it not work if we apply the state changes from subView inside the Batch transactor?

I get an error when I attempt to apply the subView to the open view. Maybe I'm doing something wrong?

- Update Comments - Update/Change function names - Update/Change argument names - Remove unused preclaim - Fix exit out of signature check (multi & single)

shawnxie999

partial

shawnxie999 · 2024-11-06T21:41:32Z

src/xrpld/ledger/detail/ApplyStateTable.cpp

@@ -147,7 +159,8 @@ ApplyStateTable::apply(
            }
            auto const origNode = to.read(keylet::unchecked(item.first));
            auto curNode = item.second.second;
-            if ((type == &sfModifiedNode) && (*curNode == *origNode))
+            if ((type == &sfModifiedNode) && (*curNode == *origNode) &&
+                !isBatch)


if i'm understanding correctly, when the transaction type is Batch, we are going to add the previous and final fields of the object even though they are the same.

This doesn't sound too correct, please correct me if i'm wrong. TY

We are going to add the batchPrevAcctRootFields. The curNode and origNode are the same on the batch txn because of the way its reapplied in updateAccountRootEntry. I guess handling this as its own case might be preferred for better understanding?

we can just add a comment here to explain the situation. Also it might be worth add an assertion to make sure that nodes are not modified if it is a batch transaction - this might before useful for future debugging

// If it is a batch txn, the curNode and origNode are the same on the batch txn // because of the way its reapplied in updateAccountRootEntry bool const nodesUnmodified = *curNode == *origNode; if ((type == &sfModifiedNode) && nodesUnmodified && !isBatch) continue; assert(isBatch && !nodesUnmodified)

shawnxie999 · 2024-11-06T21:50:47Z

src/xrpld/ledger/detail/ApplyStateTable.cpp

@@ -147,7 +159,8 @@ ApplyStateTable::apply(
            }
            auto const origNode = to.read(keylet::unchecked(item.first));
            auto curNode = item.second.second;
-            if ((type == &sfModifiedNode) && (*curNode == *origNode))
+            if ((type == &sfModifiedNode) && (*curNode == *origNode) &&
+                !isBatch)
                continue;
            std::uint16_t nodeType = curNode


can a batch transaction ever modify an object other than AccountRoot? if not, would be use good to assert that here.

No I dont believe so. I will add that here.

shawnxie999 · 2024-11-06T21:52:33Z

src/xrpld/ledger/detail/ApplyStateTable.cpp

@@ -203,6 +216,13 @@ ApplyStateTable::apply(
                        prevs.emplace_back(obj);
                }

+                if (isBatch && nodeType == ltACCOUNT_ROOT && batchPrev)
+                {
+                    // TODO: This could fail if the fields already exist


is this TODO done? also would love to learn more about this, as the comment itself isn't too comprehensive 😃

- Remove BatchTxn Object - Use Global Flag `tfInnerBatchTxn ` for inner batch txns - Remove Sequence & Ticket workaround - replace sfBatchTxn submit/relay validation with tfInnerBatchTxn - Remove Sequence 0 Requirement

shawnxie999

partial

shawnxie999 · 2024-11-07T20:22:29Z

src/libxrpl/protocol/STTx.cpp

+            Serializer dataStart = startMultiSigningData(*this);
+            finishMultiSigningData(accountID, dataStart);
+            return dataStart.getData();
+        });


what's the rationale for not returning a Slice directly in this lambda?

shawnxie999 · 2024-11-07T20:27:01Z

include/xrpl/protocol/TxFlags.h

+constexpr std::uint32_t tfUniversalV2                      = tfFullyCanonicalSig | tfInnerBatchTxn;
+constexpr std::uint32_t tfUniversalMask                    = ~(tfFullyCanonicalSig | tfInnerBatchTxn);


Suggested change

constexpr std::uint32_t tfUniversalV2 = tfFullyCanonicalSig | tfInnerBatchTxn;

constexpr std::uint32_t tfUniversalMask = ~(tfFullyCanonicalSig | tfInnerBatchTxn);

constexpr std::uint32_t tfUniversal = tfFullyCanonicalSig | tfInnerBatchTxn;

constexpr std::uint32_t tfUniversalMask = ~(tfFullyCanonicalSig | tfInnerBatchTxn);

we can prob add tfInnerBatchTxn into tfUniversal directly, and check in Transactor::preflight1 if tfInnerBatchTxn is usable

shawnxie999 · 2024-11-07T20:27:25Z

include/xrpl/protocol/TxFlags.h

-constexpr std::uint32_t tfBridgeModifyMask = ~(tfUniversal | tfClearAccountCreateAmount);
+constexpr std::uint32_t tfBridgeModifyMask = ~(tfUniversalV2 | tfClearAccountCreateAmount);
+
+// Batch Flags


Suggested change

// Batch Flags

// Batch Flags:

shawnxie999 · 2024-11-07T20:42:30Z

src/xrpld/ledger/ApplyViewImpl.h

+    }
+
+    void
+    addBatchExecutionMetaData(STObject&& batchExecution)


Suggested change

addBatchExecutionMetaData(STObject&& batchExecution)

addBatchExecution(STObject&& batchExecution)

shawnxie999 · 2024-11-07T20:42:41Z

src/xrpld/ledger/ApplyViewImpl.h

+    }
+
+    void
+    setBatchMetaData(std::vector<STObject>&& batch)


Suggested change

setBatchMetaData(std::vector<STObject>&& batch)

setBatchExecutions(std::vector<STObject>&& batchExecutions)

shawnxie999 · 2024-11-07T20:51:17Z

src/xrpld/ledger/detail/ApplyStateTable.cpp

+        {
+            auto array = STArray{sfBatchExecutions};


Suggested change

{

auto array = STArray{sfBatchExecutions};

{

assert(isBatch);

auto array = STArray{sfBatchExecutions};

is this condition only valid if it's a Batch transaction? possibly we should assert that it is non empty only if it is a batch tx.

shawnxie999 · 2024-11-07T20:51:36Z

src/xrpld/ledger/detail/ApplyStateTable.cpp

+        if (!batchExecution.empty())
+        {
+            auto array = STArray{sfBatchExecutions};
+            for (auto element : batchExecution)


Suggested change

for (auto element : batchExecution)

for (auto const& element : batchExecutions)

shawnxie999 · 2024-11-07T20:53:07Z

src/xrpld/ledger/detail/ApplyViewImpl.cpp

@@ -31,7 +31,8 @@ ApplyViewImpl::ApplyViewImpl(ReadView const* base, ApplyFlags flags)
 void
 ApplyViewImpl::apply(OpenView& to, STTx const& tx, TER ter, beast::Journal j)
 {
-    items_.apply(to, tx, ter, deliver_, j);
+    items_.apply(
+        to, tx, ter, deliver_, batchExecution_, batchPrevAcctRootFields_, j);


Suggested change

to, tx, ter, deliver_, batchExecution_, batchPrevAcctRootFields_, j);

to, tx, ter, deliver_, batchExecutions_, batchPrevAcctRootFields_, j);

shawnxie999 · 2024-11-07T20:53:38Z

src/xrpld/ledger/ApplyViewImpl.h

+    void
+    setBatchMetaData(std::vector<STObject>&& batch)
+    {
+        batchExecution_ = std::move(batch);


Suggested change

batchExecution_ = std::move(batch);

batchExecutions_ = std::move(batch);

shawnxie999 · 2024-11-07T20:55:58Z

src/xrpld/ledger/ApplyViewImpl.h

+    void
+    addBatchExecutionMetaData(STObject&& batchExecution)
+    {
+        batchExecution_.push_back(std::move(batchExecution));


Suggested change

batchExecution_.push_back(std::move(batchExecution));

batchExecutions_.push_back(std::move(batchExecution));

mvadari · 2024-11-07T22:17:43Z

src/xrpld/app/tx/detail/apply.cpp

+        if (tx.isFieldPresent(sfTxnSignature))
+            return {Validity::SigBad, "Batch txn contains signature."};


What do you think about also accepting an empty string ""? That's what's used in e.g. multisigning.

dangell7 added 2 commits July 10, 2024 17:44

featureBatch rough draft

879aba5

clang-format

46fd68f

dangell7 added 13 commits July 17, 2024 20:33

add atomic swap

c57ace7

Merge branch 'develop' into feature-batch

91d9909

add batch bsig

f37377d

[fold] clang-format

b3136a2

[fold] remove tapRETRY

5e919eb

[fold] remove comments

1225a61

[fold] fix invariant workaround

304ff3e

[fold] fix applyTransaction workaround

aded948

[fold] remove test template

82d4943

[fold] update logging

eaf7893

[fold] remove log

ca39807

clang-format

dd71073

[fold] remove workaround

26518a5

cindyyan317 reviewed Jul 30, 2024

View reviewed changes

src/libxrpl/protocol/STTx.cpp Outdated Show resolved Hide resolved

dangell7 added 6 commits July 30, 2024 14:05

update atomic

3ceb05c

refactor & OpenView revert()

12324f1

This is not finished. Needs to only revert the transactions submitted not the entire open ledger.

[fold] clang-format

e9f045b

Merge branch 'develop' into feature-batch

51ed234

[fold] add revert

676b9b6

[fold] clang-format

6c6180d

ximinez reviewed Jul 31, 2024

View reviewed changes

src/xrpld/app/tx/detail/Batch.cpp Outdated Show resolved Hide resolved

dangell7 marked this pull request as draft July 31, 2024 19:53

dangell7 added 4 commits August 1, 2024 10:37

[revert] remove rawRevert func

bd03ea3

refactor with stacking views

c0e00b0

clang-format

a3e15a1

Merge branch 'develop' into feature-batch

14eef60

shawnxie999 reviewed Oct 30, 2024

View reviewed changes

dangell7 added 14 commits October 30, 2024 22:08

[fold] remove comment

f185d38

[fold] update headers

767a88e

[fold] change error response text

ee03811

[fold] change flags bit

9d8368d

[fold] fix formatting

b408b80

[fold] normal consequences

27d4685

[fold] refactor single/multi sign

419b14d

[fold] review comments

6c87e70

- allow only one flag - validate batch signers array (max and valid) - move `calculateBaseFee` - move `preflight2` - no duplicate TxIDs - no duplicate BatchSigners

[fold] clang-format

a75cf0e

[fold] tickets are not chronological

e6021bf

[fold] fix fee calculation

fd0acb7

Merge branch 'develop' into feature-batch

8a6d6ce

[fold] fix merge issues

dd77e03

[fold] clang-format

6cb23e3

shawnxie999 reviewed Nov 2, 2024

View reviewed changes

src/xrpld/app/tx/detail/apply.cpp Outdated Show resolved Hide resolved

src/xrpld/app/tx/detail/Batch.cpp Outdated Show resolved Hide resolved

shawnxie999 reviewed Nov 4, 2024

View reviewed changes

shawnxie999 reviewed Nov 5, 2024

View reviewed changes

[fold] addressing review

88c40e4

- Update Comments - Update/Change function names - Update/Change argument names - Remove unused preclaim - Fix exit out of signature check (multi & single)

shawnxie999 reviewed Nov 6, 2024

View reviewed changes

dangell7 added 7 commits November 7, 2024 01:52

[fold] change sequence handling

e4cb784

- Remove BatchTxn Object - Use Global Flag `tfInnerBatchTxn ` for inner batch txns - Remove Sequence & Ticket workaround - replace sfBatchTxn submit/relay validation with tfInnerBatchTxn - Remove Sequence 0 Requirement

[fixup] rename prevFields -> batchPrevAcctRootFields

47120ac

[fold] revert applyFlags change

6ed4ac8

[fold] use unordered_set

52f1ba9

Merge branch 'develop' into feature-batch

1978fd3

[fold] clang-format

6ffe349

[fold] fix bad merge

04519e6

shawnxie999 reviewed Nov 7, 2024

View reviewed changes

mvadari reviewed Nov 7, 2024

View reviewed changes

-            ret = checkMultiSign(ctx.view, idAccount, txSigners, ctx.j);
+            if (ret = checkMultiSign(ctx.view, idAccount, txSigners, ctx.j);
+                !isTesSuccess(ret))
+                return ret;

		constexpr std::uint32_t tfUniversalV2 = tfFullyCanonicalSig \| tfInnerBatchTxn;
		constexpr std::uint32_t tfUniversalMask = ~(tfFullyCanonicalSig \| tfInnerBatchTxn);

	addBatchExecutionMetaData(STObject&& batchExecution)
	addBatchExecution(STObject&& batchExecution)

	setBatchMetaData(std::vector<STObject>&& batch)
	setBatchExecutions(std::vector<STObject>&& batchExecutions)

	for (auto element : batchExecution)
	for (auto const& element : batchExecutions)

	to, tx, ter, deliver_, batchExecution_, batchPrevAcctRootFields_, j);
	to, tx, ter, deliver_, batchExecutions_, batchPrevAcctRootFields_, j);

	batchExecution_ = std::move(batch);
	batchExecutions_ = std::move(batch);

	batchExecution_.push_back(std::move(batchExecution));
	batchExecutions_.push_back(std::move(batchExecution));

		if (tx.isFieldPresent(sfTxnSignature))
		return {Validity::SigBad, "Batch txn contains signature."};

Batch Amendment #5060

Are you sure you want to change the base?

Batch Amendment #5060

Conversation

dangell7 commented Jul 10, 2024 • edited Loading

High Level Overview of Change

Context of Change

Type of Change

API Impact

codecov bot commented Jul 10, 2024 • edited Loading

Codecov Report

ximinez commented Jul 31, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

shawnxie999 Oct 31, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

shawnxie999 left a comment

Choose a reason for hiding this comment

shawnxie999 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

shawnxie999 left a comment

Choose a reason for hiding this comment

shawnxie999 Nov 4, 2024 • edited Loading

Choose a reason for hiding this comment

shawnxie999 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

shawnxie999 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

shawnxie999 Nov 7, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

shawnxie999 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mvadari Nov 7, 2024 • edited Loading

Choose a reason for hiding this comment

dangell7 commented Jul 10, 2024 •

edited

Loading

codecov bot commented Jul 10, 2024 •

edited

Loading

shawnxie999 Oct 31, 2024 •

edited

Loading

shawnxie999 Nov 4, 2024 •

edited

Loading

shawnxie999 Nov 7, 2024 •

edited

Loading

mvadari Nov 7, 2024 •

edited

Loading