CUR2-1059: extend `stablecoin.balances` (part 2) - base+enrich approach #9103

tomfutago · 2025-12-09T09:34:57Z

Stablecoins Balance Models Refactoring

Problem

Adding new stablecoins to the token list triggers full historical recalculation of incremental balance models, which is computationally expensive.

Solution

Split token lists and balance models into seed (frozen historical) and latest (new additions) streams, with separate enrichment pipelines.

New Architecture

tokens_<chain>_erc20_stablecoins_core     →  stablecoins_<chain>_core_balances     →  stablecoins_<chain>_core_balances_enriched
                                                                                                                              ↘
tokens_<chain>_erc20_stablecoins (view)                                                              stablecoins_<chain>_balances (view)
                                                                                                                              ↗
tokens_<chain>_erc20_stablecoins_extended   →  stablecoins_<chain>_extended_balances   →  stablecoins_<chain>_extended_balances_enriched

Key Changes

Token Lists:

_core.sql - frozen list of existing stablecoins
_extended.sql - empty template for new additions
Union view combines both

Balance Models:

_core_balances - incremental, sources from seed list, original start_date
_extended_balances - incremental, sources from latest list, start_date = 2025-01-01
_core_balances_enriched - incremental, applies enrichment macro
_extended_balances_enriched - incremental, applies enrichment macro
_balances - view unions both enriched models

Adding New Stablecoins

Add token to tokens_<chain>_erc20_stablecoins_extended.sql
Update start_date in stablecoins_<chain>_extended_balances.sql
Full refresh latest_balances and latest_balances_enriched only

Seed models remain untouched.

Chains

With balances (11): arbitrum, avalanche_c, base, bnb, ethereum, kaia, linea, optimism, polygon, scroll, worldchain
Token lists only (6): blast, bob, celo, fantom, gnosis, mantle

tomfutago · 2025-12-17T10:48:29Z

temp excluded arbitrum, bnb, ethereum, optimism and polygon to allow ci tests to complete

tomfutago · 2025-12-17T11:04:58Z

high level dq checks

token coverage -> OK:

-- Check 1: Token coverage
with old_tokens as (
  select distinct token_address, token_symbol
  from stablecoins_linea.balances
),
new_tokens as (
  select distinct token_address, token_symbol
  from test_schema.git_dunesql_04661fe_stablecoins_linea_balances
)
select
  coalesce(o.token_address, n.token_address) as token_address,
  o.token_symbol as old_symbol,
  n.token_symbol as new_symbol,
  case 
    when o.token_address is null then 'EXTRA_IN_NEW'
    when n.token_address is null then 'MISSING_IN_NEW'
    when o.token_symbol != n.token_symbol then 'SYMBOL_MISMATCH'
    else 'OK'
  end as status
from old_tokens o
full outer join new_tokens n on o.token_address = n.token_address
order by status, token_address

spot checks -> OK (expected diffs for mid-day balances in old model)

select
  coalesce(o.day, n.day) as day,
  coalesce(o.token_address, n.token_address) as token_address,
  o.balance as old_balance, n.balance as new_balance,
  coalesce(o.balance, 0) - coalesce(n.balance, 0) as diff
from stablecoins_linea.balances o
full outer join test_schema.git_dunesql_04661fe_stablecoins_linea_balances n
  on o.day = n.day
  and o.address = n.address
  and o.token_address = n.token_address
where coalesce(o.address, n.address) = 0x795facaa76aed7c5f44a053155407199f4075139
order by 1

cursor · 2025-12-17T11:05:12Z

PR Summary

Rearchitects stablecoin balances into per-chain seed/latest base+enrich models with union views, migrates token lists, adds a cross-chain balances view, updates the balances macro to exclude today, and removes legacy models.

Stablecoins architecture (base + enrich):
- Split balances into seed (frozen) and latest (new) incremental streams per chain; enrich in separate models; union into stablecoins_<chain>_balances views.
Per-chain implementations (11 chains):
- Added models and schemas for arbitrum, avalanche_c, base, bnb, ethereum, kaia, linea, optimism, polygon, scroll, worldchain:
  - tokens_<chain>_erc20_stablecoins_seed/latest (+ union view), stablecoins_<chain>_seed_balances/_latest_balances, and corresponding _enriched + balances view.
Cross-chain view:
- New stablecoins.balances view unioning enriched balances across chains.
Macro update:
- balances_incremental_subset_daily: exclude current_date in days to avoid mid-day stale data.
Cleanup/Migration:
- Removed legacy daily_spellbook stablecoin balance models and old tokens_*_erc20_stablecoins files/schemas; migrated to new tokens subproject structure.

^{Written by Cursor Bugbot for commit dea7788. Configure here.}

0xRobin

Structure looks good! Only remark is that we should not use latest and seed in the model name when we are not referring to a model that holds the latest balances (not full history) or models that are seeds. 😅

Some alternatives:
~~seed~~

archive
core
canonical

~~latest~~

extended
dynamic

But open to other options, as long as we can move away from "latest" and "seed" which are already overloaded terms.

tomfutago · 2025-12-17T12:52:22Z

yep, _seed / _latest is probably not the best first choice..

_core / _extended sounds good

or maybe one of these:
_initial / _additions
_frozen / _new

jeff-dude · 2025-12-17T13:46:05Z

yep, _seed / _latest is probably not the best first choice..

_core / _extended sounds good

or maybe one of these: _initial / _additions _frozen / _new

let's do _core / _extended

…ase+avalanche

tomfutago · 2025-12-17T18:02:46Z

updated tokens_<chain>_erc20_stablecoins_core:

list contains only contract_address + symbol in comments
added more tokens per each chain based on: https://dune.com/queries/6371381 (added list separated by 1 extra linebreak)

jeff-dude

stopping review on arbitrum, assuming other chains are similar

jeff-dude · 2025-12-18T01:07:21Z

dbt_subprojects/tokens/models/stablecoins/arbitrum/tokens_arbitrum_erc20_stablecoins_core.sql

+  config(
+    schema = 'tokens_' ~ chain,
+    alias = 'erc20_stablecoins_core',
+    tags = ['static'],


this defaults to view type, we should likely use table in all model configs for static token lists

historically we have found hardcoded lists like this in a view when joined to large datasets can cause trouble on the trino query planner

only commenting on arbitrum as top of list, apply to all chains

jeff-dude · 2025-12-18T01:08:28Z

dbt_subprojects/tokens/models/stablecoins/arbitrum/tokens_arbitrum_erc20_stablecoins.sql

+  config(
+    schema = 'tokens_' ~ chain,
+    alias = 'erc20_stablecoins',
+    tags = ['static'],


this union can likely stay a view, since underlying will be materialized into tables

jeff-dude · 2025-12-18T01:09:41Z

dbt_subprojects/tokens/models/stablecoins/arbitrum/stablecoins_arbitrum_extended_balances.sql

+{% set chain = 'arbitrum' %}
+
+{{
+  config(


should we consider partitions on these tables?
how big are they?
worth looking into the size of each day and consider a partition on that column. if day is too small, then month may make sense.

jeff-dude · 2025-12-18T01:10:26Z

dbt_subprojects/tokens/models/stablecoins/arbitrum/stablecoins_arbitrum_core_balances.sql

+{% set chain = 'arbitrum' %}
+
+{{
+  config(


same question here on partitions

jeff-dude · 2025-12-18T01:11:04Z

...bprojects/tokens/models/stablecoins/arbitrum/stablecoins_arbitrum_core_balances_enriched.sql

+    materialized = 'incremental',
+    file_format = 'delta',
+    incremental_strategy = 'merge',
+    partition_by = ['day'],


i see we partition by day here, may consider same on base level

jeff-dude · 2025-12-18T01:12:20Z

dbt_subprojects/tokens/models/stablecoins/blast/tokens_blast_erc20_stablecoins.sql

@@ -0,0 +1,20 @@
+{% set chain = 'blast' %}


fairly certain blast is deprecated / data is frozen at a point in time?
can you confirm with max(block_time) on the chain?
if so, may make sense to leave out of PR

tomfutago added 2 commits December 9, 2025 09:25

feat: stablecoin balances base+enrich approach

282edb7

chore: move to tokens project

aa82ef2

github-actions bot added WIP work in progress dbt: daily covers the Daily dbt subproject dbt: tokens covers the Tokens dbt subproject labels Dec 9, 2025

tomfutago added 3 commits December 9, 2025 09:51

chore: prep for test run only on linea & scroll

c0a2992

test: add new macros

8772d0c

Merge branch 'main' into CUR2-1059-extend-stablecoin-balances-2

4ceaebf

github-actions bot removed the dbt: daily covers the Daily dbt subproject label Dec 10, 2025

tomfutago added 2 commits December 10, 2025 10:51

fix: correct macro name

fb3f28b

fix: clean up & temp exclude ethereum

3bfc822

github-actions bot added the dbt: daily covers the Daily dbt subproject label Dec 10, 2025

tomfutago and others added 14 commits December 15, 2025 08:40

Merge branch 'main' into CUR2-1059-extend-stablecoin-balances-2

beacbd9

fix: exclude current day balances

087daa6

feat: token-style final model

cc40a0f

fix: just linea & scroll to test

56cf122

test: all chains

54d931d

fix: remove zksync - no erc20_stablecoins for this chain

8d28111

test: temp ignore bnb

58a6d45

fix: eth macro ref

597971d

fix: eth final output

d150cb5

fix: eth cte

e02a46f

test: temp ignore bigger chains

05a1850

chore: move & reformat stablecoin lists

7a625b5

feat: seed + latest split

c604c07

fix: misssing jinja

dea7788

tomfutago marked this pull request as ready for review December 17, 2025 11:05

tomfutago requested review from 0xRobin and jeff-dude December 17, 2025 11:05

github-actions bot added ready-for-review this PR development is complete, please review and removed WIP work in progress labels Dec 17, 2025

0xRobin requested changes Dec 17, 2025

View reviewed changes

tomfutago added 5 commits December 17, 2025 13:57

chore: rename seed+latest to core+extended

bd64f11

feat: simplify stablecoin lists, add more tokens, temp prod_exclude b…

75a414c

…ase+avalanche

fix: drop symbol

b7c4e73

test: temp remove base+avalanche

8aa3668

fix: clean up

0e947cd

tomfutago requested a review from 0xRobin December 17, 2025 18:03

jeff-dude requested changes Dec 18, 2025

View reviewed changes

jeff-dude added in review Assignee is currently reviewing the PR and removed ready-for-review this PR development is complete, please review labels Dec 18, 2025

jeff-dude self-assigned this Dec 18, 2025

CUR2-1059: extend stablecoin.balances (part 2) - base+enrich approach #9103

Are you sure you want to change the base?

CUR2-1059: extend stablecoin.balances (part 2) - base+enrich approach #9103

Uh oh!

Conversation

tomfutago commented Dec 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Stablecoins Balance Models Refactoring

Problem

Solution

New Architecture

Key Changes

Adding New Stablecoins

Chains

Uh oh!

tomfutago commented Dec 17, 2025

Uh oh!

tomfutago commented Dec 17, 2025

Uh oh!

cursor bot commented Dec 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PR Summary

Uh oh!

0xRobin left a comment

Choose a reason for hiding this comment

Uh oh!

tomfutago commented Dec 17, 2025

Uh oh!

jeff-dude commented Dec 17, 2025

Uh oh!

tomfutago commented Dec 17, 2025

Uh oh!

jeff-dude left a comment

Choose a reason for hiding this comment

Uh oh!

jeff-dude Dec 18, 2025

Choose a reason for hiding this comment

Uh oh!

jeff-dude Dec 18, 2025

Choose a reason for hiding this comment

Uh oh!

jeff-dude Dec 18, 2025

Choose a reason for hiding this comment

Uh oh!

jeff-dude Dec 18, 2025

Choose a reason for hiding this comment

Uh oh!

jeff-dude Dec 18, 2025

Choose a reason for hiding this comment

Uh oh!

jeff-dude Dec 18, 2025

Choose a reason for hiding this comment

Uh oh!

jeff-dude Dec 18, 2025

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

CUR2-1059: extend `stablecoin.balances` (part 2) - base+enrich approach #9103

CUR2-1059: extend `stablecoin.balances` (part 2) - base+enrich approach #9103

tomfutago commented Dec 9, 2025 •

edited

Loading

cursor bot commented Dec 17, 2025 •

edited

Loading