[bridge]: finish initial implementation of workflows for XEM:wXEM POC #1492

Jaguar0625 · 2025-07-14T22:31:14Z

[bridge]: refactor and cleanup workflows to support unwrap mode

 problem: most workflows need to support wrap and unwrap modes
solution: restructure the workflows to support dual modes
          extract common helpers into WorkflowUtils

[bridge]: automatically deduct network fee from transfer amount

 problem: network fee is being paid by bridge and could potentially run out
solution: deduct network fee from transfer amount

[bridge]: add unwrap_request database to Databases and missing WrapRequestDatabase functions

 problem: unwrap functionality requires an inverse of the wrap_request database
solution: add unwrap_request database to Databases

 problem: WrapRequestDatabase is missing some functionality
solution: add missing functionality
          * query cumulative wrapped supply at a timestamp
          * mark a wrap request as failed (not just errored/malformed)
          * bidirectional lookups of height and timestamp from block_metadata table

[bridge]: update network facades to enable workflow implementations

 problem: network facades already accept config with bridge address
solution: extract bridge address in init

 problem: is_currency_mosaic_id is only in SymbolNetworkFacade
solution: promote to "interface" and add to NemNetworkFacade

 problem: NEM uses different endpoints for API and rosetta
solution: add kwargs to create_connector to allow selection of appropriate connector

 problem: lookup_account_balance only returns latest (not historical) balance
solution: remove it because it's not needed

[bridge]: add workflow for downloading balance changes of the bridge account

 problem: need to track historical balances of bridge account
solution: add database for tracking inflows and outflows
          add `download_rosetta_block_balance_changes` utility for requesting rosetta block information
          add `download_bridge_balance_changes` workflow for downloading balance changes
            via rosetta and updating the database

codecov · 2025-07-14T22:34:58Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 98.40%. Comparing base (4c4ad12) to head (ed88696).

Additional details and impacted files

@@                Coverage Diff                 @@
##           feature/bridge    #1492      +/-   ##
==================================================
+ Coverage           98.36%   98.40%   +0.04%     
==================================================
  Files                 171      174       +3     
  Lines                7092     7289     +197     
  Branches              143      143              
==================================================
+ Hits                 6976     7173     +197     
  Misses                116      116

Flag	Coverage Δ
bridge	`100.00% <100.00%> (ø)`
lightapi-python	`98.58% <100.00%> (+0.02%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
bridge/bridge/NetworkFacadeLoader.py	`100.00% <100.00%> (ø)`
bridge/bridge/NetworkUtils.py	`100.00% <100.00%> (ø)`
bridge/bridge/WorkflowUtils.py	`100.00% <100.00%> (ø)`
bridge/bridge/db/BalanceChangeDatabase.py	`100.00% <100.00%> (ø)`
bridge/bridge/db/Databases.py	`100.00% <100.00%> (ø)`
bridge/bridge/db/MaxProcessedHeightMixin.py	`100.00% <100.00%> (ø)`
bridge/bridge/db/WrapRequestDatabase.py	`100.00% <100.00%> (ø)`
bridge/bridge/models/WrapRequest.py	`100.00% <100.00%> (ø)`
bridge/bridge/nem/NemNetworkFacade.py	`100.00% <100.00%> (ø)`
bridge/bridge/nem/NemUtils.py	`100.00% <100.00%> (ø)`
... and 3 more

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

…account problem: need to track historical balances of bridge account solution: add database for tracking inflows and outflows add `download_rosetta_block_balance_changes` utility for requesting rosetta block information add `download_bridge_balance_changes` workflow for downloading balance changes via rosetta and updating the database

problem: network facades already accept config with bridge address solution: extract bridge address in init problem: is_currency_mosaic_id is only in SymbolNetworkFacade solution: promote to "interface" and add to NemNetworkFacade problem: NEM uses different endpoints for API and rosetta solution: add kwargs to create_connector to allow selection of appropriate connector problem: lookup_account_balance only returns latest (not historical) balance solution: remove it because it's not needed

…questDatabase functions problem: unwrap functionality requires an inverse of the wrap_request database solution: add unwrap_request database to Databases problem: WrapRequestDatabase is missing some functionality solution: add missing functionality * query cumulative wrapped supply at a timestamp * mark a wrap request as failed (not just errored/malformed) * bidirectional lookups of height and timestamp from block_metadata table

problem: network fee is being paid by bridge and could potentially run out solution: deduct network fee from transfer amount

problem: most workflows need to support wrap and unwrap modes solution: restructure the workflows to support dual modes extract common helpers into WorkflowUtils

…st version problem: upstream logic is complicated for choosing appropriate value of use_version_one solution: change argument to prefer_version_one and choose automatically

problem: address validation is expecting message to exactly equal message some wallets prepend a NUL or add whitespace solution: trim leading and trailing whitespace and NULs

problem: zero amount wrap requests will always fail when being sent solution: mark them as errors during download process

problem: mosaic id is passed around in a clunky manner and inconsistently solution: streamline and fix mosaic id handling

problem: NEM blocks are finalized in 6 hours, which is slow solution: add configuration to lookahead and conceptually speed up finalization

problem: cannot calculate outstanding wrapped tokens directly because of fee deduction solution: store payout_total_fee in database

problem: symbol api does not return aggregates with embedded solution: update extract_wrap_request_from_transaction to accept embedded transaction json

problem: wrapped will grow over time due to deduction of fees paid solution: subtract fees paid from calculation problem: unwrapped changes are processed before native changes solution: calculate unwrapped total from previous block

problem: need to query mosaic fee and supply to determine fee calculation solution: add mosaic_fee_information

problem: NEM calculates transfer fees differently for different mosaics solution: add support for full calculation

…wnloaded data problem: running without first downloading requests and balances can lead to unintended behavior solution: check database contains enough data within balance functions

problem: max_processed_height is calculated from data but it can be set explicitly solution: add database mixin for managing max_processed_height problem: on fault tables might be left in a bad state solution: add reset to database classes to drop all rows beyond max_processed_height

… all required information problem: download_wrap_requests is likely to download one side of the bridge more aggressively due to finalization differences among other factors solution: add is_synced_at_* functions and only process a payout when all evaluate to True

problem: some details of a payout transaction are only logged but not stored solution: store these details in the database

Wayonb

👍🏾

Jaguar0625 requested a review from Wayonb July 14, 2025 22:31

Jaguar0625 force-pushed the task/bridge/wip branch 8 times, most recently from ae56364 to aaf20e3 Compare July 21, 2025 21:50

Jaguar0625 force-pushed the task/bridge/network-facade-arbitrary-mosaic-processing branch 2 times, most recently from 60d2151 to 4c4ad12 Compare July 22, 2025 15:06

Base automatically changed from task/bridge/network-facade-arbitrary-mosaic-processing to feature/bridge July 22, 2025 15:13

Jaguar0625 added 17 commits July 22, 2025 11:14

[bridge]: automatically deduct network fee from transfer amount

e0dbd89

problem: network fee is being paid by bridge and could potentially run out solution: deduct network fee from transfer amount

[bridge]: refactor and cleanup workflows to support unwrap mode

04952bd

problem: most workflows need to support wrap and unwrap modes solution: restructure the workflows to support dual modes extract common helpers into WorkflowUtils

[bridge]: NemNetworkFacade create_transfer_transaction should pick be…

6939a5d

…st version problem: upstream logic is complicated for choosing appropriate value of use_version_one solution: change argument to prefer_version_one and choose automatically

[bridge]: relax validation of destination address

9a51aab

problem: address validation is expecting message to exactly equal message some wallets prepend a NUL or add whitespace solution: trim leading and trailing whitespace and NULs

[bridge]: treat zero amount wrap requests as error

deb6799

problem: zero amount wrap requests will always fail when being sent solution: mark them as errors during download process

[bridge]: simplify passing around of mosaic id

64c3513

problem: mosaic id is passed around in a clunky manner and inconsistently solution: streamline and fix mosaic id handling

[bridge]: add support for 'finalizationLookahead' config property

58131b9

problem: NEM blocks are finalized in 6 hours, which is slow solution: add configuration to lookahead and conceptually speed up finalization

[bridge]: store total fee paid in wrap_request database

dff83e9

problem: cannot calculate outstanding wrapped tokens directly because of fee deduction solution: store payout_total_fee in database

[bridge]: fix processing of Symbol aggregate transactions

5b2b399

problem: symbol api does not return aggregates with embedded solution: update extract_wrap_request_from_transaction to accept embedded transaction json

[bridge]: fix wrapped and unwrapped calculation

cd77e98

problem: wrapped will grow over time due to deduction of fees paid solution: subtract fees paid from calculation problem: unwrapped changes are processed before native changes solution: calculate unwrapped total from previous block

[lightapi/python]: add mosaic_fee_information to NemConnector

7658d26

problem: need to query mosaic fee and supply to determine fee calculation solution: add mosaic_fee_information

[bridge]: add support for mosaic transfer calculations

e8b96f1

problem: NEM calculates transfer fees differently for different mosaics solution: add support for full calculation

[bridge]: add guards in database functions to prevent processing undo…

66127a7

…wnloaded data problem: running without first downloading requests and balances can lead to unintended behavior solution: check database contains enough data within balance functions

Jaguar0625 added 2 commits July 22, 2025 11:14

[bridge]: store net_amount and conversion_rate in database

ed88696

problem: some details of a payout transaction are only logged but not stored solution: store these details in the database

Jaguar0625 force-pushed the task/bridge/wip branch from aaf20e3 to ed88696 Compare July 22, 2025 15:16

Wayonb approved these changes Jul 23, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[bridge]: finish initial implementation of workflows for XEM:wXEM POC #1492

[bridge]: finish initial implementation of workflows for XEM:wXEM POC #1492

Uh oh!

Jaguar0625 commented Jul 14, 2025

Uh oh!

codecov bot commented Jul 14, 2025 •

edited

Loading

Uh oh!

Wayonb left a comment

Uh oh!

Uh oh!

[bridge]: finish initial implementation of workflows for XEM:wXEM POC #1492

Are you sure you want to change the base?

[bridge]: finish initial implementation of workflows for XEM:wXEM POC #1492

Uh oh!

Conversation

Jaguar0625 commented Jul 14, 2025

Uh oh!

codecov bot commented Jul 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Wayonb left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

codecov bot commented Jul 14, 2025 •

edited

Loading