Skip to content
Permalink

Comparing changes

Choose two branches to see what’s changed or to start a new pull request. If you need to, you can also or learn more about diff comparisons.

Open a pull request

Create a new pull request by comparing changes across two branches. If you need to, you can also . Learn more about diff comparisons here.
base repository: mozilla-services/syncstorage-rs
Failed to load repositories. Confirm that selected base ref is valid, then try again.
Loading
base: 0.15.3
Choose a base ref
...
head repository: mozilla-services/syncstorage-rs
Failed to load repositories. Confirm that selected head ref is valid, then try again.
Loading
compare: master
Choose a head ref

Commits on Apr 24, 2024

  1. Copy the full SHA
    8098d83 View commit details

Commits on Apr 25, 2024

  1. fix: take keys_changed_at into account w/ migrated records' special c…

    …ase (#1545)
    
    and make the comparison more clear/explicit as the goal is to check if the
    `fxa_kid`s match
    
    Issue SYNC-4225
    pjenvey authored Apr 25, 2024
    Copy the full SHA
    f68fb60 View commit details
  2. Copy the full SHA
    810db6c View commit details
  3. chore: tag 0.15.4

    pjenvey committed Apr 25, 2024
    Copy the full SHA
    9af5bf5 View commit details

Commits on Apr 30, 2024

  1. feat: Allow uid range for purge function (SYNC-4246) (#1547)

    * feat: Allow uid range for purge function
    
    In an attempt to parallelize this script after a very long delay,
    specify a range so that multiple scripts can try to process
    different ranges of the database.
    
    Closes #1548
    jrconlin authored Apr 30, 2024
    Copy the full SHA
    cc16082 View commit details
  2. Copy the full SHA
    3cfc5b0 View commit details
  3. chore: tag 0.15.5

    pjenvey committed Apr 30, 2024
    Copy the full SHA
    1668716 View commit details
  4. chore: tag 0.15.4 (#1544)

    * chore: tag 0.15.3
    
    * chore: fix changelog version anchor
    
    * chore: tag 0.15.4
    
    * chore: tag 0.15.5
    pjenvey authored Apr 30, 2024
    Copy the full SHA
    97389f4 View commit details
  5. Feat/range2 (#1550)

    * bug: validate val names
    
    * not sure why we look for two args when the secret should always be last
    * fix names of the range vals to match opts
    * normalize `range` to `uid_range` everywhere to prevent possible
    problems.
    jrconlin authored Apr 30, 2024
    Copy the full SHA
    5dc53f2 View commit details
  6. Copy the full SHA
    070583c View commit details
  7. chore: tag 0.15.6

    pjenvey committed Apr 30, 2024
    Copy the full SHA
    a7c32de View commit details
  8. Merge pull request #1551 from mozilla-services/release/0.15

    chore: tag 0.15.6
    pjenvey authored Apr 30, 2024
    Copy the full SHA
    aa9ef46 View commit details

Commits on May 2, 2024

  1. Copy the full SHA
    4a145dd View commit details
  2. chore: tag 0.15.7

    pjenvey committed May 2, 2024
    Copy the full SHA
    5ebe45a View commit details
  3. chore: tag 0.15.7 (#1554)

    pjenvey authored May 2, 2024
    Copy the full SHA
    c3d7481 View commit details

Commits on May 8, 2024

  1. Copy the full SHA
    581c250 View commit details
  2. Copy the full SHA
    53ff3ac View commit details
  3. chore: tag 0.15.8

    pjenvey committed May 8, 2024
    Copy the full SHA
    e9f74cb View commit details

Commits on May 9, 2024

  1. chore: tag 0.15.8 (#1558)

    * chore: tag 0.15.8
    pjenvey authored May 9, 2024
    Copy the full SHA
    fa5c9bd View commit details
  2. feat/SYNC-4244_stale (#1546)

    * feat: log bad client state to stderr & metrics
    
    Closes: SYNC-4244
    jrconlin authored May 9, 2024
    Copy the full SHA
    8307955 View commit details

Commits on May 16, 2024

  1. bug: Allow threadpool size to be set. (#1560)

    * bug: Allow threadpool size to be set.
    
    It appears that the replacement for setting ACTIX_THREADPOOL would be to
    call `ServiceBuilder.worker_max_blocking_threads()` This PR introduces
    the ability to set this value by using the
    `worker_max_blocking_threads`configuration variable.
    
    Closes #SYNC-4271
    jrconlin authored May 16, 2024
    Copy the full SHA
    ab7b422 View commit details

Commits on May 20, 2024

  1. feat: Add normalized ReportableError to errors (#1559)

    * feat: Add normalized ReportableError to errors
    
    We want to do things like add tags and other features to sync errors the
    way that we do in other packages. To do so, we're backporting
    ReportableError from Autopush to Syncstorage.
    
    This also addresses some clippy fixes required by 1.78
    
    This continues to use the `Taggable` trait, which we may want to
    port to autopush.
    
    Closes SYNC-4262
    jrconlin authored May 20, 2024
    Copy the full SHA
    7718130 View commit details
  2. Copy the full SHA
    5cdfd03 View commit details
  3. fix: nix-shell: update pkgconfig -> pkg-config build input (#1562)

    Signed-off-by: Christoph Heiss <christoph@c8h4.io>
    Co-authored-by: JR Conlin <jconlin+git@mozilla.com>
    christoph-heiss and jrconlin authored May 20, 2024
    Copy the full SHA
    a55e373 View commit details

Commits on May 23, 2024

  1. feat: Add metrics, gcp logging to tokenserver scripts (#1555)

    * feat: Add metrics, gcp logging to tokenserver scripts
    
    This adds GCP logging JSON output formatting for logs as well as DataDog
    style metric reporting.
    
    New Environs:
    * `METRIC_API_KEY` - Optional Datadog API key
    * `METRIC_APP_KEY` - Optional Datadog APP key
    * `METRIC_HOST` - Collector Host address
    * `METRIC_PORT` - Collector Host port
    
    This also adds the `--human` arg if you want to use the older, human
    readable logs instead.
    
    
    ---------
    
    Co-authored-by: Philip Jenvey <pjenvey@underboss.org>
    jrconlin and pjenvey authored May 23, 2024
    Copy the full SHA
    6537783 View commit details

Commits on May 31, 2024

  1. Copy the full SHA
    2584b97 View commit details
  2. Copy the full SHA
    bca19e4 View commit details
  3. chore: tag 0.15.9

    pjenvey committed May 31, 2024
    Copy the full SHA
    450c378 View commit details

Commits on Jun 3, 2024

  1. Merge pull request #1565 from mozilla-services/release/0.15

    chore: tag 0.15.9
    pjenvey authored Jun 3, 2024
    Copy the full SHA
    6ea9c85 View commit details

Commits on Jun 10, 2024

  1. chore: Update to debian bookworm / Python 3.12 (#1567)

    * chore: Update to debian bookworm / Python 3.12
    jrconlin authored Jun 10, 2024
    Copy the full SHA
    8f9e1c2 View commit details

Commits on Jun 11, 2024

  1. chore: tag 0.16.0

    pjenvey committed Jun 11, 2024
    Copy the full SHA
    3a969f2 View commit details
  2. Merge pull request #1570 from mozilla-services/release/0.16

    chore: tag 0.16.0
    pjenvey authored Jun 11, 2024
    Copy the full SHA
    7004ad0 View commit details

Commits on Jun 14, 2024

  1. feat: Remove support for BrowserID (#1531)

    * feat: Remove support for BrowserID
    * mark test only functions as such. I think we can drop MapAndThenTrait for more recent implementations of rust.
    
    Closes: SYNC-3684
    jrconlin authored Jun 14, 2024
    Copy the full SHA
    dbbdd1d View commit details
  2. bug: Revert the venv configuration for python (#1571)

    use `--system-break-packages` flag to force install to system.
    
    Closes SYNC-4298
    jrconlin authored Jun 14, 2024
    Copy the full SHA
    0f86587 View commit details

Commits on Jun 15, 2024

  1. Copy the full SHA
    6c9b771 View commit details
  2. chore: tag 0.17.0

    pjenvey committed Jun 15, 2024
    Copy the full SHA
    33ffbe0 View commit details

Commits on Jun 17, 2024

  1. Merge pull request #1575 from mozilla-services/release/0.17

    chore: tag 0.17.0
    pjenvey authored Jun 17, 2024
    Copy the full SHA
    2924031 View commit details

Commits on Jun 25, 2024

  1. Chore/update 2406 (#1576)

    chore: Updates for Jun-2024
    
    * f normalize diesel to 1.4
    
    We're currenlty locked to diesel 1.4 due to some significant changes
    in the `diesel_logger` crate and how it expects `Connections` to be
    defined.
    jrconlin authored Jun 25, 2024
    Copy the full SHA
    1713962 View commit details

Commits on Jul 3, 2024

  1. Copy the full SHA
    250ac94 View commit details

Commits on Jul 11, 2024

  1. Copy the full SHA
    1edce04 View commit details
  2. chore: tag 0.17.1

    pjenvey committed Jul 11, 2024
    Copy the full SHA
    1c8e3ef View commit details
  3. Copy the full SHA
    68db54b View commit details

Commits on Jul 16, 2024

  1. Merge pull request #1585 from mozilla-services/release/0.17

    chore: tag 0.17.1
    pjenvey authored Jul 16, 2024
    Copy the full SHA
    0996cb5 View commit details

Commits on Aug 7, 2024

  1. chore: revert back to bullseye (for now) (#1589)

    this also downgrades Oracle's libmysqlclient-dev from 8.0.37-1debian11 to
    8.0.39-1debian12
    
    this is to test if the debian version and or the libmysqlclient-dev version are
    the cause of both db pool initialization failures as well as other seemingly
    mysterious likely network related issues seen in 0.17.x
    
    Issue SYNC-4363
    pjenvey authored Aug 7, 2024
    Copy the full SHA
    4a503f8 View commit details
  2. chore: tag 0.17.2

    pjenvey committed Aug 7, 2024
    Copy the full SHA
    613fcb5 View commit details

Commits on Aug 30, 2024

  1. fix: revert the python3.10 match statement (for now) (#1592)

    so the script can operate under bullseye/python3.9
    
    and ignore RUSTSEC-2024-0365
    
    Issue SYNC-4259
    pjenvey authored Aug 30, 2024
    Copy the full SHA
    dc0d571 View commit details
  2. Copy the full SHA
    a9e503d View commit details
  3. chore: tag 0.17.3

    pjenvey committed Aug 30, 2024
    Copy the full SHA
    ce81db2 View commit details

Commits on Sep 6, 2024

  1. feat: debug "Invalid OAuth token" (verifier returns None) error cases (

    …#1595)
    
    temporarily switch it to a "pyo3 error" so it's logged
    
    Issue SYNC-4363
    pjenvey authored Sep 6, 2024
    Copy the full SHA
    1443b31 View commit details
  2. Copy the full SHA
    d6ef948 View commit details
Showing with 6,899 additions and 7,236 deletions.
  1. +5 −1 .cargo/audit.toml
  2. +274 −71 .circleci/config.yml
  3. +23 −0 .config/nextest.toml
  4. +10 −0 .github/workflows/glean-probe-scraper.yml
  5. +3 −0 .gitignore
  6. +372 −1 CHANGELOG.md
  7. +3 −1 CONTRIBUTING.md
  8. +1,342 −749 Cargo.lock
  9. +29 −11 Cargo.toml
  10. +41 −34 Dockerfile
  11. +77 −13 Makefile
  12. +2 −0 PULL_REQUEST_TEMPLATE.md
  13. +14 −3 README.md
  14. +18 −17 docker-compose.e2e.mysql.yaml
  15. +17 −17 docker-compose.e2e.spanner.yaml
  16. +21 −5 docker-compose.mysql.yaml
  17. +74 −65 docker-compose.spanner.yaml
  18. +118 −0 docs/adr/0001-daily-active-use-server-side-metrics-glean.md
  19. +3 −0 docs/adr/index.md
  20. +94 −0 docs/adr/template.md
  21. +68 −0 docs/tokenserver/tokenserver.md
  22. +155 −0 docs/tokenserver/tokenserver_api.md
  23. +109 −0 docs/tools/process_account_events.md
  24. +135 −0 docs/tools/purge_old_records_tokenserver.md
  25. +143 −0 docs/tools/spanner_purge_ttl.md
  26. +16 −0 glean/Cargo.toml
  27. +109 −0 glean/data-review/request.md
  28. +101 −0 glean/metrics.yaml
  29. +9 −0 glean/pings.yaml
  30. +3 −0 glean/src/lib.rs
  31. +269 −0 glean/src/server_events.rs
  32. +2 −7 requirements.txt
  33. +13 −3 scripts/prepare-spanner.sh
  34. +4 −0 scripts/start_mock_fxa_server.sh
  35. +1 −1 shell.nix
  36. +6 −2 syncserver-common/Cargo.toml
  37. +89 −12 syncserver-common/src/lib.rs
  38. +1 −0 syncserver-common/src/middleware/mod.rs
  39. +268 −0 syncserver-common/src/middleware/sentry.rs
  40. +14 −4 {syncserver/src/server → syncserver-common/src}/tags.rs
  41. +11 −11 syncserver-db-common/Cargo.toml
  42. +37 −18 syncserver-db-common/src/error.rs
  43. +23 −12 syncserver-settings/src/lib.rs
  44. +8 −9 syncserver/Cargo.toml
  45. +11 −9 syncserver/src/error.rs
  46. +70 −17 syncserver/src/server/mod.rs
  47. +30 −8 syncserver/src/server/test.rs
  48. +255 −1 syncserver/src/server/user_agent.rs
  49. +53 −87 syncserver/src/tokenserver/extractors.rs
  50. +39 −13 syncserver/src/tokenserver/handlers.rs
  51. +28 −38 syncserver/src/tokenserver/mod.rs
  52. +16 −9 syncserver/src/web/auth.rs
  53. +28 −30 syncserver/src/web/error.rs
  54. +61 −46 syncserver/src/web/extractors.rs
  55. +39 −12 syncserver/src/web/handlers.rs
  56. +0 −1 syncserver/src/web/middleware/mod.rs
  57. +1 −1 syncserver/src/web/middleware/rejectua.rs
  58. +0 −245 syncserver/src/web/middleware/sentry.rs
  59. +0 −7 syncserver/src/web/mod.rs
  60. +5 −6 syncserver/src/web/transaction.rs
  61. +14 −14 syncstorage-db-common/Cargo.toml
  62. +9 −5 syncstorage-db-common/src/error.rs
  63. +40 −39 syncstorage-db-common/src/lib.rs
  64. +10 −4 syncstorage-db-common/src/params.rs
  65. +1 −1 syncstorage-db-common/src/util.rs
  66. +0 −2 syncstorage-db/Cargo.toml
  67. +3 −1 syncstorage-db/src/tests/support.rs
  68. +15 −14 syncstorage-mysql/Cargo.toml
  69. +0 −54 syncstorage-mysql/src/diesel_ext.rs
  70. +30 −14 syncstorage-mysql/src/error.rs
  71. +1 −0 syncstorage-mysql/src/lib.rs
  72. +2 −4 syncstorage-mysql/src/models.rs
  73. +14 −0 syncstorage-mysql/src/pool.rs
  74. +1 −4 syncstorage-mysql/src/test.rs
  75. +14 −4 syncstorage-settings/src/lib.rs
  76. +4 −13 syncstorage-spanner/Cargo.toml
  77. +5 −20 syncstorage-spanner/src/batch.rs
  78. +0 −399 syncstorage-spanner/src/bin/purge_ttl.rs
  79. +32 −9 syncstorage-spanner/src/error.rs
  80. +1 −0 syncstorage-spanner/src/lib.rs
  81. +11 −6 syncstorage-spanner/src/manager/deadpool.rs
  82. +0 −1 syncstorage-spanner/src/manager/mod.rs
  83. +1 −3 syncstorage-spanner/src/manager/session.rs
  84. +21 −23 syncstorage-spanner/src/models.rs
  85. +41 −11 syncstorage-spanner/src/pool.rs
  86. +260 −0 syncstorage-spanner/src/stream.rs
  87. +3 −224 syncstorage-spanner/src/support.rs
  88. +1 −1 tests.ini
  89. +4 −4 tokenserver-auth/Cargo.toml
  90. +0 −643 tokenserver-auth/src/browserid.rs
  91. +4 −1 tokenserver-auth/src/crypto.rs
  92. +2 −2 tokenserver-auth/src/lib.rs
  93. +17 −24 tokenserver-auth/src/oauth/py.rs
  94. +24 −24 tokenserver-auth/src/token/py.rs
  95. +6 −3 tokenserver-common/Cargo.toml
  96. +87 −24 tokenserver-common/src/error.rs
  97. +1 −1 tokenserver-common/src/lib.rs
  98. +4 −5 tokenserver-db/Cargo.toml
  99. +35 −13 tokenserver-db/src/error.rs
  100. +3 −0 tokenserver-db/src/lib.rs
  101. +24 −7 tokenserver-db/src/models.rs
  102. +8 −0 tokenserver-db/src/pool.rs
  103. +5 −19 tokenserver-settings/src/lib.rs
  104. +0 −2 tools/README.md
  105. +1 −1 tools/hawk/make_hawk_token.py
  106. +162 −0 tools/integration_tests/conftest.py
  107. +3 −3 tools/integration_tests/requirements.txt
  108. +0 −93 tools/integration_tests/run.py
  109. +238 −266 tools/integration_tests/test_storage.py
  110. +3 −90 tools/integration_tests/test_support.py
  111. +0 −30 tools/integration_tests/tokenserver/mock_fxa_server.py
  112. +0 −39 tools/integration_tests/tokenserver/run.py
  113. +48 −58 tools/integration_tests/tokenserver/test_authorization.py
  114. +0 −562 tools/integration_tests/tokenserver/test_browserid.py
  115. +6 −89 tools/integration_tests/tokenserver/test_e2e.py
  116. +3 −1 tools/integration_tests/tokenserver/test_misc.py
  117. +2 −0 tools/integration_tests/tokenserver/test_node_assignment.py
  118. +3 −54 tools/integration_tests/tokenserver/test_support.py
  119. +1 −1 tools/spanner/Dockerfile
  120. +15 −23 tools/spanner/count_expired_rows.py
  121. +17 −20 tools/spanner/count_users.py
  122. +65 −39 tools/spanner/purge_ttl.py
  123. +47 −0 tools/spanner/test_count_expired_rows.py
  124. +147 −0 tools/spanner/test_purge_ttl.py
  125. +46 −0 tools/spanner/test_utils.py
  126. +68 −0 tools/spanner/utils.py
  127. +10 −28 tools/spanner/write_batch.py
  128. +4 −0 tools/tokenserver/conftest.py
  129. +105 −21 tools/tokenserver/database.py
  130. +0 −89 tools/tokenserver/loadtests/locustfile.py
  131. +42 −13 tools/tokenserver/process_account_events.py
  132. +134 −29 tools/tokenserver/purge_old_records.py
  133. +6 −0 tools/tokenserver/pytest.ini
  134. +5 −1 tools/tokenserver/requirements.txt
  135. +0 −25 tools/tokenserver/run_tests.py
  136. +25 −1 tools/tokenserver/test_database.py
  137. +46 −16 tools/tokenserver/test_process_account_events.py
  138. +35 −0 tools/tokenserver/test_purge_old_records.py
  139. +84 −11 tools/tokenserver/util.py
  140. +0 −100 tools/user_migration/README.md
  141. +0 −15 tools/user_migration/fix_collections.sql
  142. +0 −290 tools/user_migration/gen_bso_users.py
  143. +0 −203 tools/user_migration/gen_fxa_users.py
  144. +0 −827 tools/user_migration/migrate_node.py
  145. +0 −111 tools/user_migration/old/dump_avro.py
  146. +0 −312 tools/user_migration/old/dump_mysql.py
  147. +0 −516 tools/user_migration/old/migrate_user.py
  148. +0 −4 tools/user_migration/old/requirements.txt
  149. +0 −13 tools/user_migration/old/sync.avsc
  150. +0 −3 tools/user_migration/requirements.txt
6 changes: 5 additions & 1 deletion .cargo/audit.toml
Original file line number Diff line number Diff line change
@@ -1,2 +1,6 @@
[advisories]
ignore = []
ignore = [
"RUSTSEC-2024-0365", # Bound by diesel 1.4 (4GB limit n/a to tokenserver)
"RUSTSEC-2024-0421", # Bound by diesel 1.4, `idna` < 0.1.5, Upgrade to >=1.0.0
"RUSTSEC-2024-0437", # Bound by grpcio 0.13,
]
345 changes: 274 additions & 71 deletions .circleci/config.yml

Large diffs are not rendered by default.

23 changes: 23 additions & 0 deletions .config/nextest.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
[store]
dir = "target/nextest"

[profile.default]
retries = 0
test-threads = 1
threads-required = 1
status-level = "pass"
final-status-level = "flaky"
failure-output = "immediate"
success-output = "never"
fail-fast = false
slow-timeout = { period = "300s" }

[profile.ci]
fail-fast = false

[profile.ci.junit]
path = "junit.xml"

report-name = "syncstorage-unit-tests"
store-success-output = false
store-failure-output = true
10 changes: 10 additions & 0 deletions .github/workflows/glean-probe-scraper.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
---
name: Glean probe-scraper
on:
push:
branches: [master]
pull_request:
branches: [master]
jobs:
glean-probe-scraper:
uses: mozilla/probe-scraper/.github/workflows/glean.yaml@main
3 changes: 3 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -37,3 +37,6 @@ tools/tokenserver/loadtests/*.pem
tools/tokenserver/loadtests/*.pub
venv
.vscode/settings.json

# circleci
workspace
373 changes: 372 additions & 1 deletion CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,374 @@
<a name="0.18.3"></a>
## 0.18.3 (2025-05-14)


#### Chore

* bump to latest rust ([0148e04d](https://github.com/mozilla-services/syncstorage-rs/commit/0148e04dd2881869ffe52b6ebb93be6929f31a25))
* update python cryptography (#1690) ([e93bb882](https://github.com/mozilla-services/syncstorage-rs/commit/e93bb8821ccdf94e34c184f51ad86f0388333f3d))
* added build-and-push to GAR (#1654) ([cb37e2aa](https://github.com/mozilla-services/syncstorage-rs/commit/cb37e2aa4134d5e8e0c11178e267d3e7565da05d))
* upload test artifacts to gcs ([aeedcf1e](https://github.com/mozilla-services/syncstorage-rs/commit/aeedcf1e19e622b4f0d0e9c813ba4da3c712f125))
* switch back to libmariadb-dev (#1665) ([e0093a88](https://github.com/mozilla-services/syncstorage-rs/commit/e0093a88bfc059a891c1a5d3f74cef068b720861))
* migrate tokenserver tests to pytest with junit output ([15840c5e](https://github.com/mozilla-services/syncstorage-rs/commit/15840c5ecfd1e6fbcd239bed0f50cf3537631775))
* migrate unit tests to nextest and llvm-cov ([8c56cae8](https://github.com/mozilla-services/syncstorage-rs/commit/8c56cae8905325345972a4abe99c12c1fc1b012c))

#### Features

* build docker w/ Oracle's libmysqlclient (#1695) ([569e5100](https://github.com/mozilla-services/syncstorage-rs/commit/569e5100839245cd5869bb12b655b7fe571fbbcf))
* emit oauth verification timeouts as metrics (not sentry) (#1694) ([624eced1](https://github.com/mozilla-services/syncstorage-rs/commit/624eced1e9cad6492a38397c9440b558d263cca0))

#### Bug Fixes

* re-enable tokensever e2e tests ([d0336c88](https://github.com/mozilla-services/syncstorage-rs/commit/d0336c8869e52a48e49fed989b5ac9573a3b1e55))
* avoid underflow of the queued_tasks metric ([10daab06](https://github.com/mozilla-services/syncstorage-rs/commit/10daab06cf35cf5696aa6ed6b790d8115bfeb432))
* Revert "fix: avoid underflow of the queued_tasks metric (#1628)" ([31dda136](https://github.com/mozilla-services/syncstorage-rs/commit/31dda136809879b8e7f91f095bc378bb41b9f304))
* resolve pyo3 vuln deprecations (#1682) ([0675930a](https://github.com/mozilla-services/syncstorage-rs/commit/0675930a155d27bbf2eca2c0abf81d262a9cfb28))
* **infra:** configure gcp utils before upload (#1698) ([5dcfefe2](https://github.com/mozilla-services/syncstorage-rs/commit/5dcfefe2b6a8946f02c7bfac2fd641b0a6a3356b))

#### Test

* **e2e:** run integration and e2e tests with pytest (#1697) ([6f15ad54](https://github.com/mozilla-services/syncstorage-rs/commit/6f15ad546d3c5234986db09fec485fb911624e5f))

#### Doc

* add tokenserver documentation to sync (#1681) ([dadbcea3](https://github.com/mozilla-services/syncstorage-rs/commit/dadbcea3f7428ad7f0a5ae6f0c2ad966c331660a))
* update purge script's status now that it's running (#1679) ([6f0c7b28](https://github.com/mozilla-services/syncstorage-rs/commit/6f0c7b28db3f8a2701c4af4dfe7a2d691fc079ef))
* document pruning scripts (#1645) ([7c9bc008](https://github.com/mozilla-services/syncstorage-rs/commit/7c9bc0089dd73a9ecaba8b33e26634b2a69b5ff0))

#### Refactor

* kill tokenserver's TokenType now that it's solely oauth ([a26ff490](https://github.com/mozilla-services/syncstorage-rs/commit/a26ff490b8086ce3c12b837ca00cc757caa54169))
* simplify metric_label to return a &str ([0ca435fb](https://github.com/mozilla-services/syncstorage-rs/commit/0ca435fb1a05f073d1e78ed420d953a00c8d0d53))



<a name="0.18.2"></a>
## 0.18.2 (2024-12-05)


#### Chore

* bump to latest sentry (#1639) ([bc79ccb9](https://github.com/mozilla-services/syncstorage-rs/commit/bc79ccb97243f946c1abb436f07a1be8b63f6ba6))



<a name="0.18.1"></a>
## 0.18.1 (2024-11-27)


#### Features

* Enable Glean probe-scraper task (#1636) ([8363f82d](https://github.com/mozilla-services/syncstorage-rs/commit/8363f82d4197923e8ee1062de849d2c61e467db4))



<a name="0.18.0"></a>
## 0.18.0 (2024-11-26)


#### Doc

* sync DAU server side metrics adr (#1608) ([7e211542](https://github.com/mozilla-services/syncstorage-rs/commit/7e21154203411e98200e7af60e2e7199050e9fb7))

#### Features

* glean metrics logic (#1626) ([9e9869ee](https://github.com/mozilla-services/syncstorage-rs/commit/9e9869ee0605d0610d6c94bf6185eb1eabd6b6a2))



<a name="0.17.15"></a>
## 0.17.15 (2024-11-21)


#### Bug Fixes

* upgrade to latest deadpool (#1631) ([9a97b6ce](https://github.com/mozilla-services/syncstorage-rs/commit/9a97b6ce1ae8295ea45ba017d8b0ef81ec1cf694))



<a name="0.17.14"></a>
## 0.17.14 (2024-11-19)


#### Bug Fixes

* don't add extra prefixes to middleware emitted metrics (#1630) ([9b033edc](https://github.com/mozilla-services/syncstorage-rs/commit/9b033edcb0a6479bdb7fe02e50602f85bf41cf8f))
* avoid underflow of the queued_tasks metric (#1628) ([3ed6d607](https://github.com/mozilla-services/syncstorage-rs/commit/3ed6d6077cf987f31d35e3ff772cfbb5f81f5b73))

#### Features

* add metric values to get_collections (#1616) ([98ccc954](https://github.com/mozilla-services/syncstorage-rs/commit/98ccc95482e79ed038abcdb87f6ef5cacaee0bf2))



<a name="0.17.13"></a>
## 0.17.13 (2024-10-30)


#### Features

* namespace the db error labels (#1625) ([bab5e1fe](https://github.com/mozilla-services/syncstorage-rs/commit/bab5e1fe51ef13fb36810cde93347d61372ae57c))



<a name="0.17.12"></a>
## 0.17.12 (2024-10-29)


#### Bug Fixes

* upgrade sentry w/ a fix for the blocking curl Transport (#1621) ([b8641a6c](https://github.com/mozilla-services/syncstorage-rs/commit/b8641a6cabd8ad043956fa8cb478dd6db25ca58a))

#### Features

* glean metrics data review (#1609) ([c8ec7cab](https://github.com/mozilla-services/syncstorage-rs/commit/c8ec7cab68d132a8d2a3230c49627db5da62db63))
* add hashed_device_id to HawkIdentifier (#1615) ([cc6dd137](https://github.com/mozilla-services/syncstorage-rs/commit/cc6dd13749a61793a715ab4775074090588c75a1))



<a name="0.17.11"></a>
## 0.17.11 (2024-10-22)


#### Features

* Revert "feat: Revert "fix: revert the python3.10 match statement (for now) (#1592)"" ([1b13123a](https://github.com/mozilla-services/syncstorage-rs/commit/1b13123a2b9a61d53f03c7f89672c6fbb7568f2d))
* revert "feat: Revert "chore: revert back to bullseye (for now) (#1589)"" ([e170518c](https://github.com/mozilla-services/syncstorage-rs/commit/e170518c0f5696ed51478fecafc1a59eca176053))
* add hashed_fxa_uid to HawkPayload (#1613) ([715cf950](https://github.com/mozilla-services/syncstorage-rs/commit/715cf950ba22d25d85264ecb6360305b29ec70eb))
* user agent parsing (#1607) ([7f2ef062](https://github.com/mozilla-services/syncstorage-rs/commit/7f2ef062fc71e749a00f4d960e70380c4fe44ea1))



<a name="0.17.10"></a>
## 0.17.10 (2024-10-19)


#### Features

* wire MysqlError's ReportableError impl into TokenserverError (#1611) ([c535e5ae](https://github.com/mozilla-services/syncstorage-rs/commit/c535e5ae52d03ee1c2df287c3bbed6c2321f377b))
* create DAU glean schema and configs (#1606) ([d2313310](https://github.com/mozilla-services/syncstorage-rs/commit/d23133101f5367e2070a0cc5b711e756f36f5b72))
* track the pool's queued vs actually active tasks (#1605) ([1f0e28d7](https://github.com/mozilla-services/syncstorage-rs/commit/1f0e28d7af9c6f9aea38073a099699897464ceac))



<a name="0.17.9"></a>
## 0.17.9 (2024-09-26)


#### Bug Fixes

* ensure the pool counter's always decremented via scopeguard (#1604) ([4259183a](https://github.com/mozilla-services/syncstorage-rs/commit/4259183ae4ef71efb7cd77db9b9d0e1637ca8dc2))



<a name="0.17.8"></a>
## 0.17.8 (2024-09-24)


#### Chore

* **deps:** bump cryptography in /tools/integration_tests (#1594) ([be23e391](https://github.com/mozilla-services/syncstorage-rs/commit/be23e39135d58ecaee917c49bf14aa52a406ccea))

#### Bug Fixes

* correctly read the SYNC_STATSD_HOST/PORT settings (#1601) ([3675c938](https://github.com/mozilla-services/syncstorage-rs/commit/3675c9387b8418a1a67dd13d95b338e12ca5dae3))



<a name="0.17.7"></a>
## 0.17.7 (2024-09-19)


#### Bug Fixes

* correct TokenserverError's sentry "type"/"value" fields ([bbd5abac](https://github.com/mozilla-services/syncstorage-rs/commit/bbd5abac8e060d0083aaec3c3d8f88c374d44828))

#### Refactor

* move sentry middlware and Taggable to syncserver-common ([5d9d203c](https://github.com/mozilla-services/syncstorage-rs/commit/5d9d203c62aa1f4df7c627c37eb0bc6c47ddae0b))

#### Features

* Revert "fix: revert the python3.10 match statement (for now) (#1592)" ([f3bdda91](https://github.com/mozilla-services/syncstorage-rs/commit/f3bdda91660a6777b715b59253234c4d7ba4a520))
* Revert "chore: revert back to bullseye (for now) (#1589)" ([bbdfb193](https://github.com/mozilla-services/syncstorage-rs/commit/bbdfb1933dc557ae23fabcb87eb5a22e4478a069))



<a name="0.17.6"></a>
## 0.17.6 (2024-09-17)


#### Features

* pickup the syncserver metrics settings (#1598) ([b52e44ab](https://github.com/mozilla-services/syncstorage-rs/commit/b52e44ab52796b30bf94f39d7db54ae3981c6437))



<a name="0.17.5"></a>
## 0.17.5 (2024-09-12)


#### Bug Fixes

* downcast to tokenserver's actual error type (#1596) ([2b8b1f5d](https://github.com/mozilla-services/syncstorage-rs/commit/2b8b1f5dde7fbb5717ad2d7c292f9dbf69b0d271))



<a name="0.17.4"></a>
## 0.17.4 (2024-09-06)


#### Features

* debug "Invalid OAuth token" (verifier returns None) error cases (#1595) ([1443b31e](https://github.com/mozilla-services/syncstorage-rs/commit/1443b31e5af1f10f8a52bf1bb91dc817ce0b75f2))



<a name="0.17.3"></a>
## 0.17.3 (2024-08-30)


#### Bug Fixes

* revert the python3.10 match statement (for now) (#1592) ([dc0d571c](https://github.com/mozilla-services/syncstorage-rs/commit/dc0d571c055741297a77dd47c70b7ef55b552530))



<a name="0.17.2"></a>
## 0.17.2 (2024-08-07)


#### Chore

* revert back to bullseye (for now) (#1589) ([4a503f8c](https://github.com/mozilla-services/syncstorage-rs/commit/4a503f8c36fe070e11df43a8ce0b3c71358e983c))

#### Doc

* add missing changelog for dep updates ([68db54b5](https://github.com/mozilla-services/syncstorage-rs/commit/68db54b5ce226d96da449d501a08d15392a35122))



<a name="0.17.1"></a>
## 0.17.1 (2024-07-11)


#### Chore

* Updates for Jun-2024 (#1576) ([1713962c](https://github.com/mozilla-services/syncstorage-rs/commit/1713962c6a48ca5d2a0efd4fac739482649b650c))

#### Doc

* clarify the handling of existing expired bsos in writes (#1581) ([250ac943](https://github.com/mozilla-services/syncstorage-rs/commit/250ac94353d0fdd0c387bb69f5ab90aa28a4689d), closes [#619](https://github.com/mozilla-services/syncstorage-rs/issues/619))

#### Bug Fixes

* don't hide TokenserverPool initialization errors on startup (#1584) ([1edce041](https://github.com/mozilla-services/syncstorage-rs/commit/1edce04154d354e78994621a0b88ddf42fb7ff66))



<a name="0.17.0"></a>
## 0.17.0 (2024-06-15)


#### Chore

* bump crytography/pyramid to quiet a number of security alerts (#1574) ([6c9b771b](https://github.com/mozilla-services/syncstorage-rs/commit/6c9b771bc576207d642f91bf69c4fce21a98e4c3))

#### Bug Fixes

* Revert the venv configuration for python (#1571) ([0f86587e](https://github.com/mozilla-services/syncstorage-rs/commit/0f86587edd5cf35263558e7e72e808e11f2612fd))

#### Features

* Remove support for BrowserID (#1531) ([dbbdd1df](https://github.com/mozilla-services/syncstorage-rs/commit/dbbdd1dfc3a130be46d4586133daa36c67378e7a))



<a name="0.16.0"></a>
## 0.16.0 (2024-06-11)


#### Chore

* Update to debian bookworm / Python 3.12 (#1567) ([8f9e1c27](https://github.com/mozilla-services/syncstorage-rs/commit/8f9e1c27cf8dc9e6bc176a98cc049e9735330e43))



<a name="0.15.9"></a>
## 0.15.9 (2024-05-31)


#### Features

* Add timeouts for tokenserver database calls. (#1561) ([2584b977](https://github.com/mozilla-services/syncstorage-rs/commit/2584b977b8a315a571066c0a417e76401b14bdfd))
* Add metrics, gcp logging to tokenserver scripts (#1555) ([6537783a](https://github.com/mozilla-services/syncstorage-rs/commit/6537783a9c3781802fd16478867e912868f7f8d7))
* Add normalized ReportableError to errors (#1559) ([77181308](https://github.com/mozilla-services/syncstorage-rs/commit/771813087c8eccc448530cea2d323f8de8ee81a3))

#### Bug Fixes

* nix-shell: update `pkgconfig` -> `pkg-config` build input (#1562) ([a55e3738](https://github.com/mozilla-services/syncstorage-rs/commit/a55e373823ac2c54280a9633f67143ff29ec828b))
* Allow threadpool size to be set. (#1560) ([ab7b4221](https://github.com/mozilla-services/syncstorage-rs/commit/ab7b4221fd664e23604a77041746f6f12a0a7d7e))

#### Doc

* Remove commented code, unneeded TODO, unneeded collision tracking (#1563) ([5cdfd034](https://github.com/mozilla-services/syncstorage-rs/commit/5cdfd03498055865fc27a53e263303355ac5fdb0))



<a name="0.15.8"></a>
## 0.15.8 (2024-05-08)


#### Features

* Ignore non-spanner nodes for scripts (#1557) ([581c2507](https://github.com/mozilla-services/syncstorage-rs/commit/581c250739f0f51f392dc5dc5984924395545791))



<a name="0.15.7"></a>
## 0.15.7 (2024-05-02)


#### Features

* optionally force the spanner node via get_best_node (#1553) ([4a145dd1](https://github.com/mozilla-services/syncstorage-rs/commit/4a145dd18bc13345179dbaedf6a0ae2d31ad4281))



<a name="0.15.6"></a>
## 0.15.6 (2024-04-30)


#### Bug Fixes

* validate val names (#1550) ([5dc53f22](https://github.com/mozilla-services/syncstorage-rs/commit/5dc53f2282d1d97c3b5baf730bb4b8165f06d8a1))



<a name="0.15.5"></a>
## 0.15.5 (2024-04-30)


#### Features

* Allow uid range for purge function (SYNC-4246) (#1547) ([cc160822](https://github.com/mozilla-services/syncstorage-rs/commit/cc160822419cd56646d15d425812cf36a19d89a2), closes [#1548](https://github.com/mozilla-services/syncstorage-rs/issues/1548))



<a name="0.15.4"></a>
## 0.15.4 (2024-04-25)


#### Bug Fixes

* take keys_changed_at into account w/ migrated records' special case (#1545) ([f68fb607](https://github.com/mozilla-services/syncstorage-rs/commit/f68fb607fe0284f74c77faa4eb1de14ed95e3d3e))

#### Chore

* fix changelog version anchor ([8098d839](https://github.com/mozilla-services/syncstorage-rs/commit/8098d839b6987bfa0731f876162672bb21e8fded))



<a name="0.15.3"></a>
## 0.15.3 (2024-04-24)

@@ -8,7 +379,7 @@



<a name="0.14.5"></a>
<a name="0.15.2"></a>
## 0.15.2 (2024-04-16)


4 changes: 3 additions & 1 deletion CONTRIBUTING.md
Original file line number Diff line number Diff line change
@@ -24,11 +24,13 @@ Before submitting a PR:
your reviewer's responsibility to ensure your patch includes adequate tests.

When submitting a PR:
- **[Sign all your git commits](https://docs.github.com/en/authentication/managing-commit-signature-verification/about-commit-signature-verification#ssh-commit-verification)**.
We cannot accept any PR that does not have all commits signed. This is a policy
put in place by our Security Operations team and is enforced by our CI processes.
- You agree to license your code under the project's open source license
([MPL 2.0](/LICENSE)).
- Base your branch off the current `master`.
- Add both your code and new tests if relevant.
- Sign your git commit.
- Run the test suite to make sure your code passes linting and tests.
- Ensure your changes do not reduce code coverage of the test suite.
- Please do not include merge commits in pull requests; include only commits
2,091 changes: 1,342 additions & 749 deletions Cargo.lock

Large diffs are not rendered by default.

40 changes: 29 additions & 11 deletions Cargo.toml
Original file line number Diff line number Diff line change
@@ -18,50 +18,67 @@ members = [
default-members = ["syncserver"]

[workspace.package]
version = "0.15.3"
version = "0.18.3"
authors = [
"Ben Bangert <ben@groovie.org>",
"Phil Jenvey <pjenvey@underboss.org>",
"Mozilla Services Engineering <services-engineering+code@mozilla.com>",
]
edition = "2021"
rust-version = "1.86"
license = "MPL-2.0"

[workspace.dependencies]
actix-web = "4"
actix-web = { version = "4", default-features = false, features = ["compat", "http2", "macros"] }

base64 = "0.21"
cadence = "0.29"
docopt = "1.1"
base64 = "0.22"

# Updating to 2.* requires changes to the Connection code for logging.
# (Adding an `instrumentation()` and `set_instrumentation()` method.)
# More investigation required.
diesel = "1.4"
diesel_migrations = "1.4"
diesel_logger = "0.1"

cadence = "1.3"
backtrace = "0.3"
chrono = "0.4"
docopt = "1.1"
env_logger = "0.10"
deadpool = { version = "0.12", features = ["rt_tokio_1"] }
env_logger = "0.11"
futures = { version = "0.3", features = ["compat"] }
futures-util = { version = "0.3", features = [
"async-await",
"compat",
"sink",
"io",
] }
hex = "0.4"
hostname = "0.4"
hkdf = "0.12"
hmac = "0.12"
http = "0.2"
http = "1.1"
jsonwebtoken = { version = "9.2", default-features = false }
lazy_static = "1.4"
protobuf = "=2.25.2" # pin to 2.25.2 to prevent side updating
rand = "0.8"
regex = "1.4"
reqwest = { version = "0.11", default-features = false, features = [
reqwest = { version = "0.12", default-features = false, features = [
"rustls-tls",
] }
sentry = { version = "0.31", default-features = false, features = [
sentry = { version = "0.35", default-features = false, features = [
"curl",
"backtrace",
"contexts",
"debug-images",
] }
sentry-backtrace = "0.31"
sentry-backtrace = "0.35"
serde = "1.0"
serde_derive = "1.0"
serde_json = { version = "1.0", features = ["arbitrary_precision"] }
sha2 = "0.10"
slog = { version = "2.5", features = [
"max_level_info",
"max_level_trace",
"release_max_level_info",
"dynamic-keys",
] }
@@ -73,6 +90,7 @@ slog-stdlog = "4.1"
slog-term = "2.6"
tokio = "1"
thiserror = "1.0.26"
uuid = { version = "1.11", features = ["serde", "v4"] }

[profile.release]
# Enables line numbers in Sentry reporting
75 changes: 41 additions & 34 deletions Dockerfile
Original file line number Diff line number Diff line change
@@ -1,43 +1,53 @@
ARG DATABASE_BACKEND=spanner
# Alternatively MYSQLCLIENT_PKG=libmysqlclient-dev for the Oracle/MySQL official client
ARG MYSQLCLIENT_PKG=libmariadb-dev-compat

# NOTE: Ensure builder's Rust version matches CI's in .circleci/config.yml
FROM docker.io/lukemathwalker/cargo-chef:0.1.62-rust-1.75-bullseye as chef
# RUST_VER
FROM docker.io/lukemathwalker/cargo-chef:0.1.71-rust-1.86-bullseye AS chef
WORKDIR /app

FROM chef AS planner
COPY . .
RUN cargo chef prepare --recipe-path recipe.json

FROM chef AS cacher
ARG DATABASE_BACKEND=spanner
ARG DATABASE_BACKEND
ARG MYSQLCLIENT_PKG

# cmake is required to build grpcio-sys for Spanner builds
RUN \
# Fetch and load the MySQL public key. We need to install libmysqlclient-dev to build syncstorage-rs
# which wants the mariadb
wget -qO- https://repo.mysql.com/RPM-GPG-KEY-mysql-2023 > /etc/apt/trusted.gpg.d/mysql.asc && \
echo "deb https://repo.mysql.com/apt/debian/ bullseye mysql-8.0" >> /etc/apt/sources.list && \
if [ "$MYSQLCLIENT_PKG" = libmysqlclient-dev ] ; then \
# Fetch and load the MySQL public key.
wget -qO- https://repo.mysql.com/RPM-GPG-KEY-mysql-2023 > /etc/apt/trusted.gpg.d/mysql.asc && \
echo "deb https://repo.mysql.com/apt/debian/ bullseye mysql-8.0" >> /etc/apt/sources.list ; \
fi && \
apt-get -q update && \
apt-get -q install -y --no-install-recommends libmysqlclient-dev cmake
apt-get -q install -y --no-install-recommends $MYSQLCLIENT_PKG cmake

COPY --from=planner /app/recipe.json recipe.json
RUN cargo chef cook --release --no-default-features --features=syncstorage-db/$DATABASE_BACKEND --features=py_verifier --recipe-path recipe.json

FROM chef as builder
ARG DATABASE_BACKEND=spanner
FROM chef AS builder
ARG DATABASE_BACKEND
ARG MYSQLCLIENT_PKG

COPY . /app
COPY --from=cacher /app/target /app/target
COPY --from=cacher $CARGO_HOME /app/$CARGO_HOME

RUN \
# Fetch and load the MySQL public key
wget -qO- https://repo.mysql.com/RPM-GPG-KEY-mysql-2023 > /etc/apt/trusted.gpg.d/mysql.asc && \
echo "deb https://repo.mysql.com/apt/debian/ bullseye mysql-8.0" >> /etc/apt/sources.list && \
# mysql_pubkey.asc from:
# https://dev.mysql.com/doc/refman/8.0/en/checking-gpg-signature.html
# related:
# https://dev.mysql.com/doc/mysql-apt-repo-quick-guide/en/#repo-qg-apt-repo-manual-setup
if [ "$MYSQLCLIENT_PKG" = libmysqlclient-dev ] ; then \
# Fetch and load the MySQL public key.
# mysql_pubkey.asc from:
# https://dev.mysql.com/doc/refman/8.0/en/checking-gpg-signature.html
# related:
# https://dev.mysql.com/doc/mysql-apt-repo-quick-guide/en/#repo-qg-apt-repo-manual-setup
wget -qO- https://repo.mysql.com/RPM-GPG-KEY-mysql-2023 > /etc/apt/trusted.gpg.d/mysql.asc && \
echo "deb https://repo.mysql.com/apt/debian/ bullseye mysql-8.0" >> /etc/apt/sources.list ; \
fi && \
apt-get -q update && \
apt-get -q install -y --no-install-recommends libmysqlclient-dev cmake golang-go python3-dev python3-pip python3-setuptools python3-wheel && \
apt-get -q install -y --no-install-recommends $MYSQLCLIENT_PKG cmake golang-go python3-dev python3-pip python3-setuptools python3-wheel && \
pip3 install -r requirements.txt && \
rm -rf /var/lib/apt/lists/*

@@ -46,35 +56,31 @@ ENV PATH=$PATH:/root/.cargo/bin
RUN \
cargo --version && \
rustc --version && \
cargo install --path ./syncserver --no-default-features --features=syncstorage-db/$DATABASE_BACKEND --features=py_verifier --locked --root /app && \
if [ "$DATABASE_BACKEND" = "spanner" ] ; then cargo install --path ./syncstorage-spanner --locked --root /app --bin purge_ttl ; fi
cargo install --path ./syncserver --no-default-features --features=syncstorage-db/$DATABASE_BACKEND --features=py_verifier --locked --root /app

FROM docker.io/library/debian:bullseye-slim
ARG MYSQLCLIENT_PKG

WORKDIR /app
COPY --from=builder /app/requirements.txt /app
# Due to a build error that occurs with the Python cryptography package, we
# have to set this env var to prevent the cryptography package from building
# with Rust. See this link for more information:
# https://pythonshowcase.com/question/problem-installing-cryptography-on-raspberry-pi
ENV CRYPTOGRAPHY_DONT_BUILD_RUST=1

RUN \
apt-get -q update && apt-get -qy install wget


RUN \
groupadd --gid 10001 app && \
useradd --uid 10001 --gid 10001 --home /app --create-home app && \
# first, an apt-get update is required for gnupg, which is required for apt-key adv
apt-get -q update && \
# and ca-certificates needed for https://repo.mysql.com
apt-get install -y gnupg ca-certificates wget && \
echo "deb https://repo.mysql.com/apt/debian/ bullseye mysql-8.0" >> /etc/apt/sources.list && \
wget -qO- https://repo.mysql.com/RPM-GPG-KEY-mysql-2023 > /etc/apt/trusted.gpg.d/mysql.asc && \
if [ "$MYSQLCLIENT_PKG" = libmysqlclient-dev ] ; then \
# first, an apt-get update is required for gnupg, which is required for apt-key adv
apt-get -q update && \
# and ca-certificates needed for https://repo.mysql.com
apt-get install -y gnupg ca-certificates wget && \
# Fetch and load the MySQL public key
echo "deb https://repo.mysql.com/apt/debian/ bullseye mysql-8.0" >> /etc/apt/sources.list && \
wget -qO- https://repo.mysql.com/RPM-GPG-KEY-mysql-2023 > /etc/apt/trusted.gpg.d/mysql.asc ; \
fi && \
# update again now that we trust repo.mysql.com
apt-get -q update && \
# Fetch and load the MySQL public key
apt-get -q install -y build-essential libmysqlclient-dev libssl-dev libffi-dev libcurl4 python3-dev python3-pip python3-setuptools python3-wheel cargo curl jq && \
apt-get -q install -y build-essential $MYSQLCLIENT_PKG libssl-dev libffi-dev libcurl4 python3-dev python3-pip python3-setuptools python3-wheel cargo curl jq && \
# The python3-cryptography debian package installs version 2.6.1, but we
# we want to use the version specified in requirements.txt. To do this,
# we have to remove the python3-cryptography package here.
@@ -88,6 +94,7 @@ COPY --from=builder /app/tools/spanner /app/tools/spanner
COPY --from=builder /app/tools/integration_tests /app/tools/integration_tests
COPY --from=builder /app/tools/tokenserver /app/tools/tokenserver
COPY --from=builder /app/scripts/prepare-spanner.sh /app/scripts/prepare-spanner.sh
COPY --from=builder /app/scripts/start_mock_fxa_server.sh /app/scripts/start_mock_fxa_server.sh
COPY --from=builder /app/syncstorage-spanner/src/schema.ddl /app/schema.ddl

RUN chmod +x /app/scripts/prepare-spanner.sh
90 changes: 77 additions & 13 deletions Makefile
Original file line number Diff line number Diff line change
@@ -10,44 +10,89 @@ PATH_TO_SYNC_SPANNER_KEYS = `pwd`/service-account.json
# https://github.com/mozilla-services/server-syncstorage
PATH_TO_GRPC_CERT = ../server-syncstorage/local/lib/python2.7/site-packages/grpc/_cython/_credentials/roots.pem

# In order to be consumed by the ETE Test Metric Pipeline, files need to follow a strict naming
# convention: {job_number}__{utc_epoch_datetime}__{workflow}__{test_suite}__results{-index}.xml
# TODO: update workflow name appropriately
WORKFLOW := build-deploy
EPOCH_TIME := $(shell date +"%s")
TEST_RESULTS_DIR ?= workflow/test-results
TEST_PROFILE := $(if $(CIRCLECI),ci,default)
TEST_FILE_PREFIX := $(if $(CIRCLECI),$(CIRCLE_BUILD_NUM)__$(EPOCH_TIME)__$(CIRCLE_PROJECT_REPONAME)__$(WORKFLOW)__)
UNIT_JUNIT_XML := $(TEST_RESULTS_DIR)/$(TEST_FILE_PREFIX)unit__results.xml
UNIT_COVERAGE_JSON := $(TEST_RESULTS_DIR)/$(TEST_FILE_PREFIX)unit__coverage.json

SPANNER_INT_JUNIT_XML := $(TEST_RESULTS_DIR)/$(TEST_FILE_PREFIX)spanner_integration__results.xml
SPANNER_NO_JWK_INT_JUNIT_XML := $(TEST_RESULTS_DIR)/$(TEST_FILE_PREFIX)spanner_no_oauth_integration__results.xml
MYSQL_INT_JUNIT_XML := $(TEST_RESULTS_DIR)/$(TEST_FILE_PREFIX)mysql_integration__results.xml
MYSQL_NO_JWK_INT_JUNIT_XML := $(TEST_RESULTS_DIR)/$(TEST_FILE_PREFIX)mysql_no_oauth_integration__results.xml

LOCAL_INTEGRATION_JUNIT_XML := $(TEST_RESULTS_DIR)/$(TEST_FILE_PREFIX)local_integration__results.xml
SYNC_SYNCSTORAGE__DATABASE_URL ?= mysql://sample_user:sample_password@localhost/syncstorage_rs
SYNC_TOKENSERVER__DATABASE_URL ?= mysql://sample_user:sample_password@localhost/tokenserver_rs

SRC_ROOT = $(shell pwd)
PYTHON_SITE_PACKGES = $(shell $(SRC_ROOT)/venv/bin/python -c "from distutils.sysconfig import get_python_lib; print(get_python_lib())")

clippy_mysql:
# Matches what's run in circleci
cargo clippy --workspace --all-targets --no-default-features --features=syncstorage-db/mysql --features=py_verifier -- -D warnings
cargo clippy --workspace --all-targets --no-default-features --features=syncstorage-db/mysql --features=py_verifier -- -D clippy::dbg_macro -D warnings

clippy_spanner:
# Matches what's run in circleci
cargo clippy --workspace --all-targets --no-default-features --features=syncstorage-db/spanner --features=py_verifier -- -D warnings
cargo clippy --workspace --all-targets --no-default-features --features=syncstorage-db/spanner --features=py_verifier -- -D clippy::dbg_macro -D warnings

clean:
cargo clean
rm -r venv

docker_start_mysql:
docker-compose -f docker-compose.mysql.yaml up -d
docker compose -f docker-compose.mysql.yaml up -d

docker_start_mysql_rebuild:
docker-compose -f docker-compose.mysql.yaml up --build -d
docker compose -f docker-compose.mysql.yaml up --build -d

docker_stop_mysql:
docker-compose -f docker-compose.mysql.yaml down
docker compose -f docker-compose.mysql.yaml down

docker_start_spanner:
docker-compose -f docker-compose.spanner.yaml up -d
docker compose -f docker-compose.spanner.yaml up -d

docker_start_spanner_rebuild:
docker-compose -f docker-compose.spanner.yaml up --build -d
docker compose -f docker-compose.spanner.yaml up --build -d

docker_stop_spanner:
docker-compose -f docker-compose.spanner.yaml down
docker compose -f docker-compose.spanner.yaml down

.ONESHELL:
docker_run_mysql_e2e_tests:
docker compose \
-f docker-compose.mysql.yaml \
-f docker-compose.e2e.mysql.yaml \
up \
--exit-code-from mysql-e2e-tests \
--abort-on-container-exit;
exit_code=$$?;
docker cp mysql-e2e-tests:/mysql_integration_results.xml ${MYSQL_INT_JUNIT_XML};
docker cp mysql-e2e-tests:/mysql_no_jwk_integration_results.xml ${MYSQL_NO_JWK_INT_JUNIT_XML};
exit $$exit_code;

.ONESHELL:
docker_run_spanner_e2e_tests:
docker compose \
-f docker-compose.spanner.yaml \
-f docker-compose.e2e.spanner.yaml \
up \
--exit-code-from spanner-e2e-tests \
--abort-on-container-exit;
exit_code=$$?;
docker cp spanner-e2e-tests:/spanner_integration_results.xml ${SPANNER_INT_JUNIT_XML};
docker cp spanner-e2e-tests:/spanner_no_jwk_integration_results.xml ${SPANNER_NO_JWK_INT_JUNIT_XML};
exit $$exit_code;

python:
python3 -m venv venv
venv/bin/python -m pip install -r requirements.txt


run_mysql: python
PATH="./venv/bin:$(PATH)" \
# See https://github.com/PyO3/pyo3/issues/1741 for discussion re: why we need to set the
@@ -68,8 +113,27 @@ run_spanner: python
RUST_BACKTRACE=full \
cargo run --no-default-features --features=syncstorage-db/spanner --features=py_verifier -- --config config/local.toml

.ONESHELL:
test:
SYNC_SYNCSTORAGE__DATABASE_URL=mysql://sample_user:sample_password@localhost/syncstorage_rs \
SYNC_TOKENSERVER__DATABASE_URL=mysql://sample_user:sample_password@localhost/tokenserver_rs \
RUST_TEST_THREADS=1 \
cargo test --workspace
SYNC_SYNCSTORAGE__DATABASE_URL=${SYNC_SYNCSTORAGE__DATABASE_URL} \
SYNC_TOKENSERVER__DATABASE_URL=${SYNC_TOKENSERVER__DATABASE_URL} \
RUST_TEST_THREADS=1 \
cargo nextest run --workspace --profile ${TEST_PROFILE} $(ARGS)

.ONESHELL:
test_with_coverage:
SYNC_SYNCSTORAGE__DATABASE_URL=${SYNC_SYNCSTORAGE__DATABASE_URL} \
SYNC_TOKENSERVER__DATABASE_URL=${SYNC_TOKENSERVER__DATABASE_URL} \
RUST_TEST_THREADS=1 \
cargo llvm-cov --no-report --summary-only \
nextest --workspace --profile ${TEST_PROFILE}; exit_code=$$?
mv target/nextest/${TEST_PROFILE}/junit.xml ${UNIT_JUNIT_XML}
exit $$exit_code

merge_coverage_results:
cargo llvm-cov report --summary-only --json --output-path ${UNIT_COVERAGE_JSON}

.ONESHELL:
run_token_server_integration_tests:
pip3 install -r tools/tokenserver/requirements.txt
pytest tools/tokenserver --junit-xml=${INTEGRATION_JUNIT_XML}
2 changes: 2 additions & 0 deletions PULL_REQUEST_TEMPLATE.md
Original file line number Diff line number Diff line change
@@ -2,6 +2,8 @@

Describe these changes.

> **NOTE:** We can only accept PRS with all commits [signed](https://docs.github.com/en/authentication/managing-commit-signature-verification/about-commit-signature-verification#ssh-commit-verification). PRs that contain _any_ unsigned commits will not be accepted and the PR _must_ be resubmitted. If this is something you cannot provide, please disclose and a team member _may_ duplicate the PR as signed for you (depending on availablity and priority. Thank you for your understanding and cooperation.)
## Testing

How should reviewers test?
17 changes: 14 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
@@ -37,7 +37,7 @@ Mozilla Sync Storage built with [Rust](https://rust-lang.org).
- pkg-config
- [Rust stable](https://rustup.rs)
- python 3.9+
- MySQL 5.7 (or compatible)
- MySQL 8.0 (or compatible)
* libmysqlclient (`brew install mysql` on macOS, `apt install libmysqlclient-dev` on Ubuntu, `apt install libmariadb-dev-compat` on Debian)

Depending on your OS, you may also need to install `libgrpcdev`,
@@ -241,7 +241,18 @@ We use [env_logger](https://crates.io/crates/env_logger): set the `RUST_LOG` env

### Unit tests

`make test` - open the Makefile to adjust your `SYNC_SYNCSTORAGE__DATABASE_URL` as needed.
You'll need [`nextest`](https://nexte.st/docs/installation/from-source/) and [`llvm-cov`](https://github.com/taiki-e/cargo-llvm-cov?tab=readme-ov-file#installation) installed for full unittest and test coverage.

$ cargo install cargo-nextest --locked
$ cargo install cargo-llvm-cov --locked

- `make test` - Runs all tests
- `make test_with_coverage` - This will use `llvm-cov` to run tests and generate [source-based code coverage](https://clang.llvm.org/docs/SourceBasedCodeCoverage.html)

If you need to override `SYNC_SYNCSTORAGE__DATABASE_URL` or `SYNC_TOKENSERVER__DATABASE_URL` variables, you can modify them in the `Makefile` or by setting them in your shell

$ echo 'export SYNC_SYNCSTORAGE__DATABASE_URL="mysql://sample_user:sample_password@localhost/syncstorage_rs"' >> ~/.zshrc
$ echo 'export SYNC_TOKENSERVER__DATABASE_URL="mysql://sample_user:sample_password@localhost/tokenserver?rs"' >> ~/.zshrc

#### Debugging unit test state

@@ -250,7 +261,7 @@ default, we use the diesel test_transaction functionality to ensure test data
is not committed to the database. Therefore, there is an environment variable
which can be used to turn off test_transaction.

SYNC_SYNCSTORAGE__DATABASE_USE_TEST_TRANSACTIONS=false cargo test [testname]
SYNC_SYNCSTORAGE__DATABASE_USE_TEST_TRANSACTIONS=false make test ARGS="[testname]"

Note that you will almost certainly want to pass a single test name. When running
the entire test suite, data from previous tests will cause future tests to fail.
35 changes: 18 additions & 17 deletions docker-compose.e2e.mysql.yaml
Original file line number Diff line number Diff line change
@@ -1,27 +1,24 @@
version: "3"
services:
sync-db:
tokenserver-db:
syncserver:
depends_on:
- sync-db
- tokenserver-db
# TODO: either syncserver should retry the db connection
# itself a few times or should include a wait-for-it.sh script
# inside its docker that would do this for us.
entrypoint: >
/bin/sh -c "
sleep 15;
/app/bin/syncserver;
"
mysql-e2e-tests:
container_name: mysql-e2e-tests
depends_on:
- mock-fxa-server
- syncserver
sync-db:
condition: service_healthy
mock-fxa-server:
condition: service_started
tokenserver-db:
condition: service_healthy
# this depend is to avoid migration collisions.
# the syncserver isn't actually used for the tests,
# but collisions can happen particularly in CI.
syncserver:
condition: service_started
image: app:build
privileged: true
user: root
environment:
JWK_CACHE_DISABLED: false
MOCK_FXA_SERVER_URL: http://mock-fxa-server:6000
SYNC_HOST: 0.0.0.0
SYNC_MASTER_SECRET: secret0
@@ -43,5 +40,9 @@ services:
TOKENSERVER_HOST: http://localhost:8000
entrypoint: >
/bin/sh -c "
sleep 28; python3 /app/tools/integration_tests/run.py 'http://localhost:8000#secret0'
exit_code=0;
pytest /app/tools/integration_tests/ --junit-xml=/mysql_integration_results.xml || exit_code=$$?;
export JWK_CACHE_DISABLED=true;
pytest /app/tools/integration_tests/ --junit-xml=/mysql_no_jwk_integration_results.xml || exit_code=$$?;
exit $$exit_code;
"
34 changes: 17 additions & 17 deletions docker-compose.e2e.spanner.yaml
Original file line number Diff line number Diff line change
@@ -1,27 +1,23 @@
version: "3"
services:
sync-db:
sync-db-setup:
tokenserver-db:
syncserver:
depends_on:
- sync-db-setup
# TODO: either syncserver should retry the db connection
# itself a few times or should include a wait-for-it.sh script
# inside its docker that would do this for us.
entrypoint: >
/bin/sh -c "
sleep 15;
/app/bin/syncserver;
"
spanner-e2e-tests:
container_name: spanner-e2e-tests
depends_on:
- mock-fxa-server
- syncserver
mock-fxa-server:
condition: service_started
syncserver:
condition: service_started
tokenserver-db:
condition: service_healthy
image: app:build
privileged: true
user: root
environment:
# Some tests can run without the `FXA_OAUTH...` vars.
# Setting this to false will delete any of those keys before starting
# the syncserver and startging the test. This can be set/passed
# in from CircleCI when calling `docker-compose -f docker-compose.e2e.spanner.yaml`
JWK_CACHE_DISABLED: false
MOCK_FXA_SERVER_URL: http://mock-fxa-server:6000
SYNC_HOST: 0.0.0.0
SYNC_MASTER_SECRET: secret0
@@ -44,5 +40,9 @@ services:
TOKENSERVER_HOST: http://localhost:8000
entrypoint: >
/bin/sh -c "
sleep 28; python3 /app/tools/integration_tests/run.py 'http://localhost:8000#secret0'
exit_code=0;
pytest /app/tools/integration_tests/ --junit-xml=/spanner_integration_results.xml || exit_code=$$?;
export JWK_CACHE_DISABLED=true;
pytest /app/tools/integration_tests/ --junit-xml=/spanner_no_jwk_integration_results.xml || exit_code=$$?;
exit $$exit_code;
"
26 changes: 21 additions & 5 deletions docker-compose.mysql.yaml
Original file line number Diff line number Diff line change
@@ -11,37 +11,51 @@
version: "3"
services:
sync-db:
image: docker.io/library/mysql:5.7
image: docker.io/library/mysql:8.0
volumes:
- sync_db_data:/var/lib/mysql
restart: always
ports:
- "3306"
command: --explicit_defaults_for_timestamp
environment:
#MYSQL_RANDOM_ROOT_PASSWORD: yes
MYSQL_ROOT_PASSWORD: random
MYSQL_DATABASE: syncstorage
MYSQL_USER: test
MYSQL_PASSWORD: test
healthcheck:
test: ["CMD-SHELL", "mysqladmin -uroot -p$${MYSQL_ROOT_PASSWORD} version"]
interval: 2s
retries: 10
start_period: 20s
timeout: 2s

tokenserver-db:
image: docker.io/library/mysql:5.7
image: docker.io/library/mysql:8.0
volumes:
- tokenserver_db_data:/var/lib/mysql
restart: always
ports:
- "3306"
command: --explicit_defaults_for_timestamp
environment:
#MYSQL_RANDOM_ROOT_PASSWORD: yes
MYSQL_ROOT_PASSWORD: random
MYSQL_DATABASE: tokenserver
MYSQL_USER: test
MYSQL_PASSWORD: test
healthcheck:
test: ["CMD-SHELL", "mysqladmin -uroot -p$${MYSQL_ROOT_PASSWORD} version"]
interval: 2s
retries: 10
start_period: 20s
timeout: 2s

mock-fxa-server:
image: app:build
restart: "no"
entrypoint: python3 /app/tools/integration_tests/tokenserver/mock_fxa_server.py
entrypoint: "sh scripts/start_mock_fxa_server.sh"
environment:
MOCK_FXA_SERVER_HOST: 0.0.0.0
MOCK_FXA_SERVER_PORT: 6000
@@ -56,8 +70,10 @@ services:
ports:
- "8000:8000"
depends_on:
- sync-db
- tokenserver-db
sync-db:
condition: service_healthy
tokenserver-db:
condition: service_healthy
environment:
SYNC_HOST: 0.0.0.0
SYNC_MASTER_SECRET: secret0
139 changes: 74 additions & 65 deletions docker-compose.spanner.yaml
Original file line number Diff line number Diff line change
@@ -1,69 +1,78 @@
version: '3'
version: "3"
services:
sync-db:
# Getting sporadic errors in spanner.
# These errors are "INTERNAL: ZETASQL_RET_CHECK failure"
# in the `backend/query/query_engine.cc` file for spanner.
# These result in a 500 error, which causes the test to fail.
# I believe come from the SQL parser, but am not sure. I am
# unable to produce these errors locally, and am using the cited
# version.
image: gcr.io/cloud-spanner-emulator/emulator:1.5.13
ports:
- "9010:9010"
- "9020:9020"
environment:
PATH: /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
sync-db-setup:
image: app:build
depends_on:
- sync-db
restart: "no"
entrypoint: "/app/scripts/prepare-spanner.sh"
environment:
SYNC_SYNCSTORAGE__SPANNER_EMULATOR_HOST: sync-db:9020
tokenserver-db:
image: docker.io/library/mysql:5.7
volumes:
- tokenserver_db_data:/var/lib/mysql
restart: always
ports:
- "3306"
environment:
#MYSQL_RANDOM_ROOT_PASSWORD: yes
MYSQL_ROOT_PASSWORD: random
MYSQL_DATABASE: tokenserver
MYSQL_USER: test
MYSQL_PASSWORD: test
mock-fxa-server:
image: app:build
restart: "no"
entrypoint: "python3 /app/tools/integration_tests/tokenserver/mock_fxa_server.py"
environment:
MOCK_FXA_SERVER_HOST: 0.0.0.0
MOCK_FXA_SERVER_PORT: 6000
syncserver:
# NOTE: The naming in the rest of this repository has been updated to reflect the fact
# that Syncstorage and Tokenserver are now part of one repository/server called
# "Syncserver" (updated from "syncstorage-rs"). We keep the legacy naming below for
# backwards compatibility with previous Docker images.
image: ${SYNCSTORAGE_RS_IMAGE:-syncstorage-rs:latest}
restart: always
ports:
- "8000:8000"
depends_on:
- sync-db-setup
environment:
SYNC_HOST: 0.0.0.0
SYNC_MASTER_SECRET: secret0
SYNC_SYNCSTORAGE__DATABASE_URL: spanner://projects/test-project/instances/test-instance/databases/test-database
SYNC_SYNCSTORAGE__SPANNER_EMULATOR_HOST: sync-db:9010
SYNC_TOKENSERVER__DATABASE_URL: mysql://test:test@tokenserver-db:3306/tokenserver
SYNC_TOKENSERVER__RUN_MIGRATIONS: "true"
sync-db:
# Getting sporadic errors in spanner.
# These errors are "INTERNAL: ZETASQL_RET_CHECK failure"
# in the `backend/query/query_engine.cc` file for spanner.
# These result in a 500 error, which causes the test to fail.
# I believe come from the SQL parser, but am not sure. I am
# unable to produce these errors locally, and am using the cited
# version.
image: gcr.io/cloud-spanner-emulator/emulator:1.5.13
ports:
- "9010:9010"
- "9020:9020"
environment:
PATH: /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
sync-db-setup:
image: app:build
depends_on:
- sync-db
restart: "no"
entrypoint:
- /bin/sh
- -c
- /app/scripts/prepare-spanner.sh && sleep infinity
environment:
SYNC_SYNCSTORAGE__SPANNER_EMULATOR_HOST: sync-db:9020
tokenserver-db:
image: docker.io/library/mysql:8.0
volumes:
- tokenserver_db_data:/var/lib/mysql
restart: always
ports:
- "3306"
environment:
#MYSQL_RANDOM_ROOT_PASSWORD: yes
MYSQL_ROOT_PASSWORD: random
MYSQL_DATABASE: tokenserver
MYSQL_USER: test
MYSQL_PASSWORD: test
healthcheck:
test: ["CMD-SHELL", "mysqladmin -uroot -p$${MYSQL_ROOT_PASSWORD} version"]
interval: 2s
retries: 10
start_period: 20s
timeout: 2s
mock-fxa-server:
image: app:build
restart: "no"
entrypoint: "sh /app/scripts/start_mock_fxa_server.sh"
environment:
MOCK_FXA_SERVER_HOST: 0.0.0.0
MOCK_FXA_SERVER_PORT: 6000
syncserver:
# NOTE: The naming in the rest of this repository has been updated to reflect the fact
# that Syncstorage and Tokenserver are now part of one repository/server called
# "Syncserver" (updated from "syncstorage-rs"). We keep the legacy naming below for
# backwards compatibility with previous Docker images.
image: ${SYNCSTORAGE_RS_IMAGE:-syncstorage-rs:latest}
restart: always
ports:
- "8000:8000"
depends_on:
- sync-db-setup
environment:
SYNC_HOST: 0.0.0.0
SYNC_MASTER_SECRET: secret0
SYNC_SYNCSTORAGE__DATABASE_URL: spanner://projects/test-project/instances/test-instance/databases/test-database
SYNC_SYNCSTORAGE__SPANNER_EMULATOR_HOST: sync-db:9010
SYNC_TOKENSERVER__DATABASE_URL: mysql://test:test@tokenserver-db:3306/tokenserver
SYNC_TOKENSERVER__RUN_MIGRATIONS: "true"

volumes:
tokenserver_db_data:
tokenserver_db_data:

# Application runs off of port 8000.
# you can test if it's available with
# curl "http://localhost:8000/__heartbeat__"
# Application runs off of port 8000.
# you can test if it's available with
# curl "http://localhost:8000/__heartbeat__"
118 changes: 118 additions & 0 deletions docs/adr/0001-daily-active-use-server-side-metrics-glean.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,118 @@
# Measuring Server-Side Daily Active Use (DAU) With Glean

* Status: approved
* Deciders: Taddes Korris, David Durst, JR Conlin, Phillip Jenvey
* Date: 2024-10-09

Technical Story:
[Jira Epic Url](https://mozilla-hub.atlassian.net/browse/SYNC-4389)


## Context and Problem Statement

There is an organizational requirement for each service to be able to measure its own DAU (Daily Active Users). Sync historically measured DAU via FxA through browser login. With the addition of Relay, device backups/migrations, etc. this will no longer an be accurate reflection of Sync usage, nor a direct measurement originating within Sync. This is due to increased browser sign-ins with the potential of overcounting, or not counting those logging into the browser without Sync enabled. See [Decision Brief - Server-Side Sync Usage Attribution from Mozilla Accounts](https://docs.google.com/document/d/1zD-ia3fP43o-dYpwavDgH5Hb6Xo_fgQzzoWqTiX_wR8/edit?tab=t.0#heading=h.mdoaoiyvqgfo).

The goal is to measure DAU (and subsequently WAU & MAU) by emitting metrics from syncstorage-rs itself. This requires the following data:
* User identifier (hashed_fxa_uid)
* Timestamp
* Platform (Desktop, Fenix, iOS, Other)
* Device Family (Desktop, Mobioe, Tablet, Other)
* Device ID (hashed_device_id) for opt-out/deletion

## Decision Drivers

1. Simplicity of implementation, not only for internal metric emission, but for processing and querying.
2. Reduction of complexity and load on team to process and make sense of data. Ideally there is existing infrastructure to reduce the possible significant work to calculate DAU.
3. Glean is an internal tool successfully used by many teams at Mozilla.
4. The Glean team has extensive experience and can provide considerable support in establishing application metrics.
5. Extensibility: ability to expand scope of metrics to other measurements and apply methodology to other services in Sync ecosystem.

## Considered Options

* A. Glean Parser for Rust - Contribute to glean team repo by implementing Rust server
* B. Custom Glean Implementation - Our own custom implementation, internal only to Sync
* C. StatsD and Grafana

## Decision Outcome

Chosen option:

* A. "Glean Parser for Rust: for server-side measurement of DAU"

In researching possible implementation methods, it became clear that many options did not offer us the ease and flexibility to reconcile the data after emission. This is why Glean is recommended as a clear frontrunner, due to its rich tooling in aggregating, querying, and visualizing data.

However, this left an additional decision to either 1. implement our own custom Glean implementation to emit "Glean-compliant" output, or 2. to contribute to the Glean team's `glean_parser` repository for server-side metrics. The `glean_parser` is used for all server implementations of Glean, since the SDK is only available for client-side applications. Currently, Rust is not supported in the server-side.

In consultation with the Glean team, we determined that avoiding our custom implementation and opting for the general-purpose `glean_parser` for Rust was the ideal solution. The Glean team does/did not have capacity to implement the Rust `glean_parser` feature, so we had to decide if we were willing to take on the effort.

Agreeing to implement this feature gave us the benefits of a general purpose solution that solved our immediate challenge but also solved for future possible challenges, all while making this feature available to other Mozilla teams.

## Pros and Cons of the Options

### A. Glean Parser for Rust

Glean is a widely used tool at Mozilla and provides us with a solution to the given issue with possible extensibility in the future. There would be some challenges related to the required development effort and coordination with the Glean team. However, the potential for positive impact within our team and the organization is significant: all Rust server applications will be able to use Glean with full server support, our possible intention to integrate Glean into Push is made easier, and this is done in partnership with the Glean team.

#### Pros

* Makes Glean compatible for all Rust server applications going forward.
* Preferred option of the Glean team.
* Glean team believes, based on FxA metrics volume, that our volume will not be a problem (180-190K per minute).
* A collection of metrics, emitted as a single "Ping Event" make querying of related data simpler.
* Core of Glean's purpose is to measure user interactions and the rich metadata that accompanies it.
* Capacity for future expansion of application metrics within Sync, beyond DAU.
* Easier to query, have data team support to set up queries.
* Use of standardized Mozilla tooling.
* Establishment of team knowledge of using Glean.
* Establishment of server-side Rust best practices, leading to easier development for backend Rust applications.
* Transparency in data review process with consideration made to minimum collection of data. This is as a result of defining `metrics.yaml` files that require data steward review to implement.

#### Cons

* We have to write the Rust-compatible code. The `glean_parser` tool used by other server-side Glean applications is currently not supported for Rust.

### B. Custom Glean Implementation

This was originally the desired approach to measuring DAU via Glean. It was predicated on the ease of prototyping a custom implementation that imitated the Glean team's logic to create "Glean-compliant" output. However, in consulting with the Glean team and evaluating the pros and cons of this approach, it became clear this approach had considerably more drawbacks than implementing the `glean_parser` for Rust. These drawbacks were a lack of testing, validation, less support from the Glean team, and the potential problems with maintenance and adding Glean metrics in the future.

#### Pros
* Gives team control over implementation and allows us to customize the Glean logging as we see fit.
* Effort is smaller because it doesn't involve our integrating with an existing repo we are not familiar with. Ex. the templating logic and libraries in the `glean_parser`.
* Easy to prototype and make changes.


#### Cons
* There is no built-in testing suite or validation, so this would put a larger development burden on us and require the Glean team's review.
* Lack of testing and validation means higher likelihood of bugs.
* If we decide to add new Glean metrics in the future, this may break the custom implementation and impose a greater maintenance overhead.
* Time required to understand the Glean team's implementations in order to replicate behavior and data structures.
* Likely won't have same support from Glean team as it is not related to their implementation.

### C. StatsD, Grafana, InfluxDB

StatsD and Grafana offer us core application metrics and service health. While we use this frequently, it doesn't neatly fit the measurement requirements we have for DAU and would likely be very difficult to process via queries. This is because such application metrics are geared towards increment counters, response codes, and timers. Given DAU is a user-initiated interaction, and we need to query unique events based on the `hashed_fxa_id`, this is not suitable for InfluxDB/StatsD. Figuring out how to query this data poses challenges as it has not been implemented for such a use case.

#### Pros

* Already utilized for core service metrics (status codes, API endpoint counts, cluster health, etc).
* Well understood and used.
* Possible for SRE for more complex queries.

#### Cons

* StatsD is not a good format for measuring something like user interactions.
* Somewhat opaque and complicated query logic required.
* Significant difficulty in aggregation and reconciliation logic.
* May not scale well given number of events.
* Likely a considerable overhead in understanding how to make sense of data.
* Heavier load for team to manage data processing, as this approach has not been tried.


## Links

* [Decision Brief - Server-Side Sync Usage Attribution from Mozilla Accounts](https://docs.google.com/document/d/1zD-ia3fP43o-dYpwavDgH5Hb6Xo_fgQzzoWqTiX_wR8/edit?tab=t.0#heading=h.mdoaoiyvqgfo)
* [FxA DAU Metric in Redash](https://sql.telemetry.mozilla.org/queries/101007/source?p_end%20date=2024-06-26&p_start%20date=2024-05-01#248905)
* [Working Document](https://docs.google.com/document/d/1Tk4VIuQZcn8IG-UI38kziZn5e-FMOI0Z-VrvaYTI1SM/edit#heading=h.b0mqx1fng4wa)
* [Sync Ecosystem Infrastructure and Metrics: KPI Metrics](https://mozilla-hub.atlassian.net/wiki/spaces/CLOUDSERVICES/pages/969834589/Establish+KPI+metrics+DAU+Retention)
* [Integrating Glean](https://mozilla.github.io/glean/book/user/adding-glean-to-your-project/rust.html)
* [Glean Metrics](https://mozilla.github.io/glean/book/reference/metrics/index.html)
3 changes: 3 additions & 0 deletions docs/adr/index.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
# Synsctorage-rs ADRs

This directory archives all the Architectural Decision Records (ADRs) for Syncstorage-rs.
94 changes: 94 additions & 0 deletions docs/adr/template.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,94 @@
# [short title of solved problem and solution]

* Status: [proposed | rejected | accepted | deprecated | … | superseded by [ADR-0005](0005-example.md)] <!-- optional -->
* Deciders: [list everyone involved in the decision] <!-- optional -->
* Date: [YYYY-MM-DD when the decision was last updated] <!-- optional -->

Technical Story: [description | ticket/issue URL] <!-- optional -->

## Context and Problem Statement

[Describe the context and problem statement, e.g., in free form using two to three sentences. You may want to articulate the problem in form of a question.]

## Decision Drivers <!-- optional -->

1. [primary driver, e.g., a force, facing concern, …]
2. [secondary driver, e.g., a force, facing concern, …]
3.<!-- numbers of drivers can vary, ranked in order of importance -->

## Considered Options

* A. [option A]
* B. [option B]
* C. [option C]
* D. … <!-- numbers of options can vary -->

## Decision Outcome

Chosen option:

* A. "[option A]"

[justification. e.g., only option, which meets primary decision driver | which resolves a force or facing concern | … | comes out best (see below)].

### Positive Consequences <!-- optional -->

* [e.g., improvement of quality attribute satisfaction, follow-up decisions required, …]
*

### Negative Consequences <!-- optional -->

* [e.g., compromising quality attribute, follow-up decisions required, …]
*

## Pros and Cons of the Options <!-- optional -->

### [option A]

[example | description | pointer to more information | …] <!-- optional -->

#### Pros

* [argument for]
* [argument for]
*<!-- numbers of pros can vary -->

#### Cons

* [argument against]
*<!-- numbers of cons can vary -->

### [option B]

[example | description | pointer to more information | …] <!-- optional -->

#### Pros

* [argument for]
* [argument for]
*<!-- numbers of pros can vary -->

#### Cons

* [argument against]
*<!-- numbers of cons can vary -->

### [option C]

[example | description | pointer to more information | …] <!-- optional -->

#### Pros

* [argument for]
* [argument for]
*<!-- numbers of pros can vary -->

#### Cons

* [argument against]
*<!-- numbers of cons can vary -->

## Links <!-- optional -->

* [Link type] [Link to ADR] <!-- example: Refined by [ADR-0005](0005-example.md) -->
*<!-- numbers of links can vary -->
68 changes: 68 additions & 0 deletions docs/tokenserver/tokenserver.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,68 @@
# Tokenserver

## What is Tokenserver?
Tokenserver is responsible for allocating Firefox Sync users to Sync Storage nodes hosted in our Spanner GCP Backend.
Tokenserver provides the "glue" between [Firefox Accounts](https://github.com/mozilla/fxa/) and the
[SyncStorage API](https://mozilla-services.readthedocs.io/en/latest/storage/apis-1.5.html).

Broadly, Tokenserver is responsible for:

* Checking the user's credentials as provided by FxA.
* Sharding users across storage nodes in a way that evenly distributes server load.
* Re-assigning the user to a new storage node if their FxA encryption key changes.
* Cleaning up old data from deleted accounts.

The service was originally conceived to be a general-purpose mechanism for connecting users
to multiple different Mozilla-run services, and you can see some of the historical context
for that original design [here](https://wiki.mozilla.org/Services/Sagrada/TokenServer)
and [here](https://mozilla-services.readthedocs.io/en/latest/token/index.html).

In practice today, it is only used for connecting to Sync.

## Tokenserver Crates & Their Purpose

### `tokenserver-auth`
Handles authentication logic, including:
- Token generation and validation.
- Ensuring clients are authorized before accessing Sync services.

### `tokenserver-common`
Provides shared functionality and types used across the Tokenserver ecosystem:
- Common utility functions.
- Structs and traits reused in other Tokenserver modules.

### `tokenserver-db`
Responsible for persisting and retrieving authentication/session-related data securely and efficiently.
Manages all database interactions for Tokenserver:
- Database schema definitions.
- Connection pooling and querying logic.

### `tokenserver-settings`
Handles configuration management:
- Loads and validates settings for Tokenserver.
- Supports integration with different deployment environments.

## How Tokenserver Handles Failure Cases

### Token Expiry
When a Tokenserver token expires, Sync Storage returns a 401 code, requiring clients to get a new token. Then, clients would use their FxA OAuth Access tokens to generate a new token, if the FxA Access Token is itself expired, then Tokenserver returns a 401 itself.

### User revoking access token
The user could revoke the access token by signing out using the Mozilla Account’s Manage Account settings. In that case, clients continue to sync up to the expiry time, which is one hour. To mitigate against this case, Firefox clients currently receive push notifications from FxA instructing them to disconnect. Additionally, any requests done against FxA itself (for example to get the user’s profile data, connected devices, etc) will also trigger the client to disconnect.

### User Changes Their Password
This is similar to the case where users revoke their access tokens. Any devices with a not-expired access token will continue to sync until expiry, but clients will likely disconnect those clients faster than the 1 hour - however, a malicious user might be able to sync upwards of 1 hour.

### User Forgetting Their Password (without a recovery key)
When a user forgets and resets their password without a recovery key, their Sync keys change. The Tokenserver request includes the key ID (which is a hash of the sync key). Thus, on the next sync, Tokenserver recognizes that the password changed, and ensures that the tokens it issues point users to a new location on Sync Storage. In practice, it does that by including the Key ID itself in the Tokenserver token, which is then sent to Sync Storage.

### User Forgetting Their Password (with a recovery key)
When a user forgets and resets their password, but has their recovery key, the behavior is similar to the password change and user revoking token cases.


## Utilities
Tokenserver has two regular running utility scripts:
1 - [Process Account Events](../tools/process_account_events.md)
2 - [Purge Old Records](../tools/purge_old_records_tokenserver.md)

For context on these processes, their purpose, and how to run them, please review their documentation pages.
155 changes: 155 additions & 0 deletions docs/tokenserver/tokenserver_api.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,155 @@
# Token Server API v1.0

Unless stated otherwise, all APIs are using application/json for the requests
and responses content types.


**GET** **/1.0/<app_name>/<app_version>**

Asks for new token given some credentials in the Authorization header.

By default, the authentication scheme is Mozilla Accounts OAuth 2.0
but other schemes can
potentially be used if supported by the login server.

- **app_name** is the name of the application to access, like **sync**.
- **app_version** is the specific version number of the api that you want
to access.

The first /1.0/ in the URL defines the version of the authentication
token itself.

Example for Mozilla Account OAuth 2.0::
```
GET /1.0/sync/1.5
Host: token.services.mozilla.com
Authorization: bearer <assertion>
```

This API returns several values in a json mapping:

- **id** -- a signed authorization token, containing the
user's id for the application and the node.
- **key** -- a secret derived from the shared secret
- **uid** -- the user id for this service
- **api_endpoint** -- the root URL for the user for the service.
- **duration** -- the validity duration of the issued token, in seconds.

Example::
```
HTTP/1.1 200 OK
Content-Type: application/json
{'id': <token>,
'key': <derived-secret>,
'uid': 12345,
'api_endpoint': 'https://db42.sync.services.mozilla.com/1.5/12345',
'duration': 300,
}
```

If the **X-Client-State** header is included in the request, the
server will compare the submitted value to any previously-seen value.
If it has changed then a new uid and api_endpoint are generated, in
effect "resetting" the node allocation for this user.


## Request Headers


**X-Client-State**

An optional string that can be sent to identify a unique configuration
of client-side state. It may be up to 32 characters long, and must
contain only characters from the urlsafe-base64 alphabet (i.e.
alphanumeric characters, underscore and hyphen) and the period.

A change in the value of this header may cause the user's node
allocation to be reset, keeping in mind Sync currently has a single node.
Clients should include any client-side state
that is necessary for accessing the selected app. For example, clients
accessing :ref:`server_syncstorage_api_15` would include a hex-encoded
hash of the encryption key in this header, since a change in the encryption
key will make any existing data unreadable.

Updated values of the **X-Client-State** will be rejected with an error
status of **"invalid-client-state"** if:

* The proposed new value is in the server's list of previously-seen
client-state values for that user.
* The client-state is missing or empty, but the server has previously
seen a non-empty client-state for that user.
* The user's IdP provides generation numbers in their identity
certificates, and the changed client-state value does not correspond
to an increase in generation number.


## Response Headers

**Retry-After**

When sent together with an HTTP 503 status code, this header signifies that
the server is undergoing maintenance. The client should not attempt any
further requests to the server for the number of seconds specified in
the header value.

**X-Backoff**

This header may be sent to indicate that the server is under heavy load
but is still capable of servicing requests. Unlike the **Retry-After**
header, **X-Backoff** may be included with any type of response, including
a **200 OK**.

Clients should avoid unnecessary requests to the server for the number of seconds
specified in the header value. For example, clients may avoid pre-emptively
refreshing token if an X-Backoff header was recently seen.

**X-Timestamp**

This header will be included with all "200" and "401" responses, giving
the current POSIX timestamp as seen by the server, in seconds. It may
be useful for client to adjust their local clock when generating authorization
assertions.


## Error Responses
===============

All errors are also returned, wherever possible, as json responses following the
structure `described in Cornice
<https://cornice.readthedocs.io/en/latest/validation.html#dealing-with-errors>`_.

In cases where generating such a response is not possible (e.g. when a request
if so malformed as to be unparsable) then the resulting error response will
have a *Content-Type* that is not **application/json**.

The top-level JSON object in the response will always contain a key named
`status`, which maps to a string identifying the cause of the error. Unexpected
errors will have a `status` string of "error"; errors expected as part of
the protocol flow will have a specific `status` string as detailed below.

Error status codes and their corresponding output are:

- **404** : unknown URL, or unsupported application.
- **400** : malformed request. Possible causes include a missing
option, bad values or malformed json.
- **401** : authentication failed or protocol not supported.
The response in that case will contain WWW-Authenticate headers
(one per supported scheme) and may report the following `status`
strings:

- **"invalid-credentials"**: authentication failed due to invalid
credentials e.g. a bad signature on the Authorization assertion.
- **"invalid-timestamp"**: authentication failed because the included
timestamp differed too greatly from the server's current time.
- **"invalid-generation"**: authentication failed because the server
has seen credentials with a more recent generation number.
- **"invalid-client-state"**: authentication failed because the server
has seen an updated value of the *X-Client-State* header.
- **"new-users-disabled"**: authentication failed because the user has
not been seen previously on this server, and new user accounts have
been disabled in the application config.

- **405** : unsupported method
- **406** : unacceptable - the client asked for an Accept we don't support
- **503** : service unavailable (ldap or snode backends may be down)
109 changes: 109 additions & 0 deletions docs/tools/process_account_events.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,109 @@
# Documentation for `process_account_events.py`

## Summary

The `process_account_events.py` is a Python utility designed to handle account-related events for Tokenserver. It connects to an Amazon Simple Queue Service (SQS) queue to poll for events indicating activity on an upstream account, related to user account activities, such as account deletions, password resets, and password changes. These events are processed to maintain synchronization between upstream account actions and Tokenserver's database.

The script is intended for internal use within Mozilla's Firefox Accounts (FxA)-supported deployments and provides a mechanism for administrative tasks like disconnecting devices or marking accounts for cleanup.

Note that this is a purely optional administrative task, highly specific to
Mozilla's internal Firefox-Accounts-supported deployment.

See [FxA Auth Server Docs](https://github.com/mozilla/fxa-auth-server/blob/master/docs/service_notifications.md) for more information on these events.

---

## Status
- Running as Kubernetes Workload as part of deployment in `sync-prod` as: `tokenserver-prod-sync-app-1-process-account-events`.
- See YAML configuration when editing configuration.
- See Kubernetes Engine Workload Panel in [`sync-prod`](https://console.cloud.google.com/kubernetes/workload/overview?inv=1&invt=AbmJeQ&project=moz-fx-sync-prod-3f0c) for more information.

---

## Supported Event Types

The script processes the following event types:

1. **Delete**
- **Event:** Account was deleted.
- **Description:** Marks user accounts as "retired" to flag them for garbage collection.
- **Purpose:** Ensures that deleted accounts are appropriately flagged for eventual cleanup.
- **Implementation:** Calls `database.retire_user(email)`.

2. **Reset**
- **Event:** Account password was reset.
- **Description:** Handles password reset events by updating the generation number in the database.
- **Purpose:** Disconnects other devices associated with the account.
- **Implementation:** Calls `update_generation_number()` with a decremented generation number.

3. **PasswordChange**
- **Event:** Account password was changed.
- **Description:** Processes password change events similarly to reset events by updating the generation number.
- **Purpose:** Disconnects other devices to reflect the password change.
- **Implementation:** Calls `update_generation_number()` with a decremented generation number.

---

## How It Works

1. **Connects to the SQS Queue:**
- Automatically determines the AWS region if not provided.
- Connects to the specified queue and sets up polling.

2. **Polls for Events:**
- Polls indefinitely, waiting for messages on the queue.
- Processes each event based on its type, using the `process_account_event()` function.

3. **Handles Event Logic:**
- Parses the event JSON.
- Identifies the event type and processes it using specialized logic for each supported event type.

4. **Updates Database:**
- Performs necessary database updates, such as retiring accounts or adjusting generation numbers.

5. **Logs and Metrics:**
- Logs actions for debugging and administrative purposes.
- Tracks metrics for processed events using the `metrics` utility.

---

## Notes

- **Optional Administrative Task:** This script is a utility for administrative purposes and is not required for the core functionality of the Syncstorage service.
- **Error Handling:** The script is designed to handle unexpected errors gracefully, logging invalid messages and continuing with the next event.
- **Event Backlog:** Unrecognized event types are logged as warnings and removed from the queue to avoid backlog.

---

## Instructions for Running the Script

### Prerequisites

1. **Python Environment:** Ensure you have Python installed along with the required libraries (`boto`, `json`, and others mentioned in the script).
2. **AWS Credentials:** The script needs access to AWS credentials to connect to the SQS queue. These credentials can be provided via environment variables, AWS CLI configurations, or IAM roles.
3. **Database Configuration:** The script relies on a database connection for processing account events. Ensure the `Database` class in the script is correctly configured to interact with your database.
4. **Logging:** The script uses a custom logging utility (`util.configure_script_logging()`). Ensure the `util` module is available and properly configured.

#### Command-Line Arguments
- **`queue_name`** (Required): Name of the SQS queue to poll for events.
- **Options:**
- `--aws-region`: Specify the AWS region of the queue (e.g., `us-west-2`). Defaults to the instance's AWS region.
- `--queue-wait-time`: Number of seconds to wait for jobs on the queue (default: `20`).
- `--verbose` (`-v`): Increase verbosity of log messages. Use multiple `-v` flags for higher verbosity levels.
- `--human_logs`: Format logs for easier human readability.

### Usage

Run the script using the following command:

```bash
python process_account_events.py [options] queue_name
```

#### Example

To process events from an SQS queue named `account-events-queue` in the `us-west-2` region:

```bash
python process_account_events.py --aws-region us-west-2 account-events-queue
```
135 changes: 135 additions & 0 deletions docs/tools/purge_old_records_tokenserver.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,135 @@
# Documentation for `purge_old_records.py`

## Summary

The `purge_old_records.py` script is an administrative utility for managing obsolete user records in Tokenserver. It removes outdated user records from the database and deletes associated data from storage nodes. This process helps reduce storage overhead, improve database performance, and maintain the health of the Tokenserver system.

Obsolete records are those replaced by newer records for the same user or marked for deletion if the user has deleted their account. The script can run in batch mode for periodic cleanup and includes options for dry-run testing and forced purging when nodes are down.

---

## Status
- Running as Kubernetes Workload as part of deployment in `sync-prod` as: `tokenserver-prod-sync-app-1-purge-old-records-0`
- See YAML configuration when editing each job.
- See Kubernetes Engine Workload Panel in [`sync-prod`](https://console.cloud.google.com/kubernetes/workload/overview?inv=1&invt=AbmJeQ&project=moz-fx-sync-prod-3f0c) for more information.

---

## Specifics

- **Primary Functionality**:
- Deletes obsolete user records.
- Issues delete requests to user storage nodes to purge related data.
- **Optional Administrative Task**:
- The script complements internal record replacement handled by the backend but is not mandatory for system operation.
- **Batch Processing**:
- Operates in loops, processing records in batches of a configurable size.
- **Grace Period**:
- Provides a grace period to avoid prematurely deleting recently replaced records.
- **Dry Run**:
- Offers a non-destructive mode for testing.

---

## Notes
- **Regular Use**:
- Running this script regularly can help maintain system performance and reduce storage overhead.
- **Concurrency**:
- When running multiple instances of this script, use the `--max-offset` option to reduce collisions.
- **Forced Deletion**:
- Use the `--force` option cautiously, especially if storage nodes are down.

---

## Instructions for Running the Script

### Prerequisites

1. **Python Environment**: Ensure Python 3.7+ is installed.
2. **Dependencies**:
- Install required Python packages:
```
pip install requests hawkauthlib backoff
```
3. **Configuration**:
- Set up access to the Tokenserver database.
- Provide necessary metrics and logging configurations.
### Usage
Run the script using the following command:
```bash
python purge_old_records.py [options] secret
```


### Options

| Option | Description | Default |
|-------------------------|-------------------------------------------------------------------------------------------------|--------------|
| `--purge-interval` | Interval in seconds to sleep between purging runs. | `3600` |
| `--grace-period` | Grace period in seconds before deleting records. | `86400` |
| `--max-per-loop` | Maximum number of items to process in each batch. | `10` |
| `--max-offset` | Random offset to select records for purging, reducing collisions in concurrent tasks. | `0` |
| `--max-records` | Maximum number of records to purge before exiting. | `0` (no limit) |
| `--request-timeout` | Timeout in seconds for delete requests to storage nodes. | `60` |
| `--oneshot` | Perform a single purge run and exit. | Disabled |
| `--dryrun` | Test the script without making destructive changes. | Disabled |
| `--force` | Force purging even if the user's storage node is marked as down. | Disabled |
| `--override_node` | Specify a node to override for deletions if data is copied. | None |
| `--range_start` | Start of UID range to process. | None |
| `--range_end` | End of UID range to process. | None |
| `--human_logs` | Enable human-readable logs. | Disabled |

### Examples

#### Example 1: Basic Purge
Perform a basic purge of obsolete user records:
```bash
python purge_old_records.py secret_key
```
#### Example 2: Grace Period and Dry Run
Purge records with a 48-hour grace period in dry-run mode:
```bash
python purge_old_records.py --grace-period 172800 --dryrun secret_key
```

#### Example 3: Specify Range and Offset
Purge records within a UID range with a random offset:

```bash
python purge_old_records.py --range_start uid_start --range_end uid_end --max-offset 50 secret_key
```

#### Example 4: Force Purge on Downed Nodes
Force the deletion of data on downed nodes:
```bash
python purge_old_records.py --force secret_key
```

---

## Detailed Usage

1. **Batch Processing**:
- The script processes records in batches defined by the `--max-per-loop` option.
- Each batch is fetched from the database using random offsets to avoid overlapping with concurrent runs.

2. **Grace Period**:
- The grace period ensures that recently replaced records are not prematurely deleted.

3. **Storage Node Cleanup**:
- For each user, the script sends a delete request to their storage node to remove associated data.

4. **Metrics Tracking**:
- Tracks operations like user record deletions, service data deletions, and errors using metrics integration.

5. **Error Handling**:
- Uses exponential backoff to retry failed HTTP requests.
- Detects loops in batch processing and raises exceptions.

6. **Dry Run Mode**:
- Simulates deletions without modifying the database or storage nodes, useful for testing.

---

143 changes: 143 additions & 0 deletions docs/tools/spanner_purge_ttl.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,143 @@
# Documentation for `purge_ttl.py`

## Summary

The `purge_ttl.py` script is a utility for purging expired Time-To-Live (TTL) records from a Google Spanner database. This script is designed to manage and clean up old data from specific database tables, ensuring efficient use of storage and maintaining database performance. It offers flexible options for targeting specific collections, user ID prefixes, and modes of operation, with optional dry-run functionality for testing changes without affecting the database.

---

## Status
- Running as Kubernetes Workload cron job in `sync-prod`.
- Runs at 10 minutes past every 2nd hour.
- Runs per-collection and is configured for each of the following:
- batches
- clients
- crypto
- forms
- meta
- tabs
- See YAML configuration when editing each job.
- See Kubernetes Engine Workload Panel in [`sync-prod`](https://console.cloud.google.com/kubernetes/workload/overview?inv=1&invt=AbmJeQ&project=moz-fx-sync-prod-3f0c) for more information.

## Specifics

- **Database**: Google Spanner.
- **Tables**:
- `batches`: Contains batch entries, with cascading deletes for child `batch_bsos`.
- `bsos`: Stores Sync Basic Storage Objects (BSO).
- **Supported Modes**:
- `batches`: Purges expired entries in the `batches` table.
- `bsos`: Purges expired entries in the `bsos` table.
- `both`: Performs purges on both tables.
- **Expiry Modes**:
- `now`: Purges entries with `expiry < CURRENT_TIMESTAMP()`.
- `midnight`: Purges entries with `expiry < TIMESTAMP_TRUNC(CURRENT_TIMESTAMP(), DAY, "UTC")`.

The script uses parameters like collection IDs, user ID prefixes, and auto-splitting for fine-grained control over the purging process. It tracks execution duration and results using StatsD metrics for performance monitoring.

---

## Notes

- Ensure proper access to the Spanner instance and database through IAM permissions.
- Use the `--dryrun` option to verify query logic before actual purging.
- Consider setting up automated monitoring for long-running operations or performance issues.

---

## Instructions for Running the Script

### Prerequisites

1. **Python Environment**: Ensure Python 3.7+ is installed.
2. **Google Cloud SDK**: Install and authenticate with Google Cloud.
3. **Dependencies**: Install required Python packages:
```bash
pip install google-cloud-spanner statsd
```
4. **Environment Variables:**
`INSTANCE_ID`: Spanner instance ID (default: spanner-test).
`DATABASE_ID`: Database ID (default: sync_schema3).
`SYNC_SYNCSTORAGE__DATABASE_URL`: Database connection URL (e.g., spanner://instance/database).

### Usage

Run the script using the following command:
```bash
python purge_ttl.py [options]
```

#### Options

| Option | Description | Default |
|---------------------------------|-----------------------------------------------------------------------------------------------------------------|------------------------------|
| `-i`, `--instance_id` | Spanner instance ID. | `spanner-test` |
| `-d`, `--database_id` | Spanner database ID. | `sync_schema3` |
| `-u`, `--sync_database_url` | Spanner DSN connection URL (overrides `instance_id` and `database_id`). | `SYNC_SYNCSTORAGE__DATABASE_URL` |
| `--collection_ids`, `--ids` | Comma-separated list of collection IDs to purge. | `[]` |
| `--uid_prefixes`, `--prefix` | Comma-separated list of UID prefixes to filter purges. | `[]` |
| `--auto_split` | Automatically generate UID prefixes for the specified number of hexadecimal digits. | None |
| `--mode` | Purge mode: `batches`, `bsos`, or `both`. | `both` |
| `--expiry_mode` | Expiry mode: `now` (current timestamp) or `midnight` (start of current day, UTC). | `midnight` |
| `--dryrun` | Perform a dry run without making changes to the database. | `False` |

#### Examples

##### Example 1: Basic Purge
Purge expired entries from both `batches` and `bsos` tables using default configurations:
```bash
python purge_ttl.py
```

##### Example 2: Specify Instance and Database
Purge expired entries in a specific instance and database:
```bash
python purge_ttl.py -i my-instance -d my-database
```
##### Example 3: Filter by Collection IDs
Purge only for specific collection IDs:
```bash
python purge_ttl.py --collection_ids [123,456,789]
```
##### Example 4: Filter by UID Prefixes
Limit purging to specific UID prefixes:
```bash
python purge_ttl.py --uid_prefixes [abc,def,123]
```
##### Example 5: Auto-Generated Prefixes
Generate prefixes automatically for a 2-digit hexadecimal range:
```bash
python purge_ttl.py --auto_split 2
```
##### Example 6: Perform a Dry Run
Test the script without making actual changes:
```bash
python purge_ttl.py --dryrun
```

### Detailed Usage

1. **Connecting to Spanner**:
- The script connects to Google Spanner using either explicitly provided `instance_id` and `database_id` or a DSN URL.

2. **Purge Modes**:
- `batches`: Deletes expired entries from the `batches` table, which cascades deletions for `batch_bsos` via Spanner's `ON DELETE CASCADE`.
- `bsos`: Deletes expired Binary Sync Objects (BSOs).
- `both`: Executes purges on both `batches` and `bsos`.

3. **Expiry Conditions**:
- `now`: Purge entries that have already expired at the current timestamp.
- `midnight`: Purge entries that expired at or before the start of the current UTC day.

4. **Query Customization**:
- Filters can be added based on collection IDs or UID prefixes.
- Queries are dynamically constructed using helper functions (`add_conditions`, `get_expiry_condition`).

5. **Performance Monitoring**:
- Metrics for execution duration and rows affected are logged and sent to StatsD for monitoring.

6. **Error Handling**:
- The script validates input parameters, raises exceptions for invalid configurations, and logs details for troubleshooting.

7. **Dry Run**:
- Enabling the `--dryrun` flag ensures that the queries are constructed and logged without executing them on the database.
16 changes: 16 additions & 0 deletions glean/Cargo.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
[package]
name = "glean"
version.workspace = true
license.workspace = true
authors.workspace = true
edition.workspace = true

[dependencies]
chrono.workspace = true
serde.workspace = true
serde_json.workspace = true
uuid.workspace = true

[lib]
name = "glean"
path = "src/lib.rs"
109 changes: 109 additions & 0 deletions glean/data-review/request.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,109 @@

# Request for data collection review form
# Syncstorage: Daily Active User Count

**All questions are mandatory. You must receive review from a data steward peer on your responses to these questions before shipping new data collection.**

1) What questions will you answer with this data?

The objective is to measure Daily Active Users for Sync (DAU). This is to be an internal metric, emitted from the server of the application. We want to understand the usage of the service per user interaction. This is important for understanding how many users have Sync enabled and at a more granular level, whether the usage is on Firefox Desktop, Fenix or iOS. Multiple actions across devices or on the same device will be reconciled to count as a single Active User.

2) Why does Mozilla need to answer these questions? Are there benefits for users? Do we need this information to address product or business requirements? Some example responses:

* We need to understand usage patterns of Sync from the server-side.
* Would provide the most granular measurement of service interaction from the server-side.
* This is necessary to establish a baseline measurement of Daily, Weekly, and Monthly usage of Sync. Essential KPI metric for Sync.
* Helps us understand usage trends related to feature changes.

3) What alternative methods did you consider to answer these questions? Why were they not sufficient?

* Previously, this metric was measured by FxA/Mozilla Accounts. Logins were associated with an OAuth Client ID. However, as Relay and other account recovery tools make use of Firefox accounts for their auth, this will alter the reliability of this metric. A user could also make use of the aforementioned services without Sync being enabled, resulting in further inaccuracies.
* A decision was made to move away from attributing active use through auth tokens towards a metric that counts each time Sync collections are accessed. Links to these decision briefs are cited at the bottom of the data review document.

4) Can current instrumentation answer these questions?

* It does at the current moment, but in a short time will no longer be able to. As described in #3, we are needing to move the metric measurement to the service itself and from the server.

5) List all proposed measurements and indicate the category of data collection for each measurement, using the [Firefox data collection categories](https://wiki.mozilla.org/Data_Collection) found on the Mozilla wiki.

**Note that the data steward reviewing your request will characterize your data collection based on the highest (and most sensitive) category.**

<table>
<tr>
<td>Measurement Name</td>
<td>Measurement Description</td>
<td>Data Collection Category</td>
<td>Tracking Bug #</td>
</tr>
<tr>
<td>sync_event</td>
<td>Event to record an instance of sync backend activity initiated by client.</td>
<td>Cat 2: Interaction Data</td>
<td><a href="https://bugzilla.mozilla.org/show_bug.cgi?id=1923967">https://bugzilla.mozilla.org/show_bug.cgi?id=1923967</a></td>
</tr>
<tr>
<td>hashed_fxa_uid</td>
<td>Sync user identifier. Uses `hashed_fxa_uid` for accurate count of sync actions. This is the Firefox Accounts (FxA) User Identifier (UID) value passed through a SHA-256 hash to render a value that is unique, but ensures the privacy of the original UID. A single user could make numerous sync actions in a given time and this id is required to ensure only a single count of daily active use is made, given a number of actions.</td>
<td>Cat 2: Interaction Data</td>
<td><a href="https://bugzilla.mozilla.org/show_bug.cgi?id=1923967">https://bugzilla.mozilla.org/show_bug.cgi?id=1923967</a></td>
</tr>
<tr>
<td>hashed_device_id</td>
<td>Hashed identifier of device. This is necessary to correlate postential users that do not want telemetry tracking of this sort. The "deletion-request" ping within clint-side sync is the most likely mechanism to ensure associative removal of data.</td>
<td>Cat 2: Interaction Data</td>
<td><a href="https://bugzilla.mozilla.org/show_bug.cgi?id=1923967">https://bugzilla.mozilla.org/show_bug.cgi?id=1923967</a></td>
</tr>
<tr>
<td>platform</td>
<td>Platform from which sync action was initiated: Firefox Desktop, Fenix (Android), or Firefox iOS. </td>
<td>Cat 1: Technical Data</td>
<td><a href="https://bugzilla.mozilla.org/show_bug.cgi?id=1923967">https://bugzilla.mozilla.org/show_bug.cgi?id=1923967</a></td>
</tr>
<tr>
<td>device_family</td>
<td>Type of device being used to make Sync action. Desktop PC, Tablet, Mobile.</td>
<td>Cat 1: Technical Data</td>
<td><a href="https://bugzilla.mozilla.org/show_bug.cgi?id=1923967">https://bugzilla.mozilla.org/show_bug.cgi?id=1923967</a></td>
</tr>
<tr>
<td>submission_timestamp</td>
<td>Glean internal submission timestamp of metric ping.</td>
<td>Cat 1: Technical Data</td>
<td><a href="https://bugzilla.mozilla.org/show_bug.cgi?id=1923967">https://bugzilla.mozilla.org/show_bug.cgi?id=1923967</a></td>
</tr>
</table>

6) Please provide a link to the documentation for this data collection which describes the ultimate data set in a public, complete, and accurate way.

* This schema (which matches the table above) is defined within the service repository in the [`/glean`](https://github.com/mozilla-services/syncstorage-rs/blob/master/glean/metrics.yaml) directory. Documentation for metrics collection of DAU will be within this directory and all related server-side code. Will add data to [Glean Dictionary](https://dictionary.telemetry.mozilla.org/) and [Probe Dictionary](https://probes.telemetry.mozilla.org) as development process proceeds.

* [Decision Brief - Server-Side Sync Usage Attribution from Mozilla Accounts](https://docs.google.com/document/d/1zD-ia3fP43o-dYpwavDgH5Hb6Xo_fgQzzoWqTiX_wR8/edit)

7) How long will this data be collected? Choose one of the following:

* We will retain the data at present for 1 year, as defaulted within Glean, however we can modify the [Probe Scraper configuration](https://github.com/mozilla/probe-scraper/blob/58040f058c55cc375c1fd6f4460bccee50b3fa8e/repositories.yaml#L446) should we deem the collection period to be shorter.
* We will want to at least retain the trends and counts of usage for at least one year.

8) What populations will you measure?

* We will measure all active use of Sync (attributed from the server-side). This includes all countries, locales and release channels which Sync is used.

9) If this data collection is default on, what is the opt-out mechanism for users?
* Our initial plan for opting out of collection would invovle the client side "deletion-request" ping. This uses the `hashed_device_id` value referenced above to remove entries that are liked to the user's Firefox Account.
* Alternatively, opting out would involve a user not being signed into Sync as a service. Since there is no active attribution that could identify a user, especially since we are using a hash value and generalized collection representation, there is not an immediate user concern.

10) Please provide a general description of how you will analyze this data.

* Measurement of Daily Active Use:
- Based on the individual unique user_id (`hashed_fxa_uid`), we will count a single active user when, during a 24-hour period, the user makes a request to any collection within their Sync account (including bookmarks, tabs, prefs, etc.). Note we are using a value derived from the `user_id`, not the `user_id` itself. This value is passed through a SHA-256 hash to render a value that is unique, but ensures the privacy of the original user_id.

- A user that initiates multiple requests to the same collection or other collections will only be counted once. This is why the unique `hashed_fxa_uid` is required to derive uniqueness. Active use can also be across several devices, so an accompanying `platform` key defines whether the user initiated the action on Desktop, iOS or Android. Device family, whether desktop, tablet, or mobile will also be of interest. Actions taken across multiple platforms for the same user should only count as an active user once, within a 24-hour period.

11) Where do you intend to share the results of your analysis?

* The Sync Ecosystem Team, related product partners, and the broader PXI organization will use this data.
* We will be using Glean for our telemetry emission, aggregation and analysis. Queries will be defined through Google BigQuery and visualized using Glean's provided tools.

12) Is there a third-party tool (i.e. not Glean or Telemetry) that you are proposing to use for this data collection? If so:

* No, since we are using Glean, this is internal.
101 changes: 101 additions & 0 deletions glean/metrics.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,101 @@
## This file describes the syncserver-rs daily active user (DAU) metrics.
## This defines the various allowed metrics that are to be captured.
## Each metric is written as a JSON blob to the default logger output.

---
# Schema
$schema: moz://mozilla.org/schemas/glean/metrics/2-0-0

# Category
syncstorage:
get_collections:
type: event
description: |
Event to record an instance of sync backend activity initiated by client.
notification_emails:
- sync-backend@mozilla.com
bugs:
- https://github.com/mozilla-services/syncstorage-rs/issues
data_reviews:
- https://bugzilla.mozilla.org/show_bug.cgi?id=1923967
expires: never

hashed_fxa_uid:
type: string
# yamllint disable
description: >
User identifier. Uses `hashed_fxa_uid` for accurate count of sync actions.
Used to determine which user has initiated sync activity.
This is the Firefox Accounts (FxA) User Identifier (UID) value passed through
a SHA-256 hash to render a value that is unique, but ensures the privacy of the original UID.
A single user could make numerous sync actions in a given time
and this id is required to ensure only a single count of daily active use
is made, given a number of actions. Sync_id is not used due to possibility
of new keys being generated during resets or timeouts, whenever encryption
keys change.
# yamllint enable
lifetime: application
send_in_pings:
- events
notification_emails:
- sync-backend@mozilla.com
bugs:
- https://github.com/mozilla-services/syncstorage-rs/issues
data_reviews:
- https://bugzilla.mozilla.org/show_bug.cgi?id=1923967
expires: never

platform:
type: string
# yamllint disable
description: |
Platform from which sync action was initiated.
Firefox Desktop, Fenix, or Firefox iOS.
# yamllint enable
lifetime: application
send_in_pings:
- events
notification_emails:
- sync-backend@mozilla.com
bugs:
- https://github.com/mozilla-services/syncstorage-rs/issues
data_reviews:
- https://bugzilla.mozilla.org/show_bug.cgi?id=1923967
expires: never

device_family:
type: string
# yamllint disable
description: |
Device family from which sync action was initiated.
Desktop PC, Tablet, Mobile, and Other.
# yamllint enable
lifetime: application
send_in_pings:
- events
notification_emails:
- sync-backend@mozilla.com
bugs:
- https://github.com/mozilla-services/syncstorage-rs/issues
data_reviews:
- https://bugzilla.mozilla.org/show_bug.cgi?id=1923967
expires: never

hashed_device_id:
type: string
# yamllint disable
description: |
Hashed device id that is associated with a given account. This is used
entirely to associate opt-out or removal requests, as they make use of
the "deletion-request" ping associated with the client side of Sync.
# yamllint enable
lifetime: application
send_in_pings:
- events
notification_emails:
- sync-backend@mozilla.com
bugs:
- https://github.com/mozilla-services/syncstorage-rs/issues
data_reviews:
- https://bugzilla.mozilla.org/show_bug.cgi?id=1923967
expires: never
9 changes: 9 additions & 0 deletions glean/pings.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
# This Source Code Form is subject to the terms of the Mozilla Public
# License, v. 2.0. If a copy of the MPL was not distributed with this
# file, You can obtain one at http://mozilla.org/MPL/2.0/.

---
## Describes the pings being sent out to Glean.

# Schema
$schema: moz://mozilla.org/schemas/glean/pings/2-0-0
3 changes: 3 additions & 0 deletions glean/src/lib.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
//! Module related to Glean server metrics.
pub mod server_events;
Loading