Skip to content

feat: blob sliver data existence check #1997

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 8 commits into from
May 8, 2025
Merged

Conversation

halfprice
Copy link
Contributor

@halfprice halfprice commented Apr 18, 2025

Description

This PR implements the basic sliver data existence check conducted along with blob info consistency check
at beginning of every epoch. It exposes a metric which represents the percentage of fully synced blobs that
are actually fully stored. Note that here it has already taken blobs that are being synced into consideration.

Contribute to WAL-701

Test plan

How did you test the new or updated feature?


Release notes

Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required.
For each box you select, include information after the relevant heading that describes the impact of your changes that
a user might notice and any actions they must take to implement updates. (Add release notes after the colon for each item)

  • Storage node:
  • Aggregator:
  • Publisher:
  • CLI:

Copy link
Contributor

Warning: This PR modifies one of the example config files. Please consider the
following:

  • Make sure the changes are backwards compatible with the current configuration.
  • Make sure any added parameters follow the conventions of the existing parameters; in
    particular, durations should take seconds or milliseconds using the naming convention
    _secs or _millis, respectively.
  • If there are added optional parameter sections, it should be possible to specify them
    partially. A useful pattern there is to implement Default for the struct and derive
    #[serde(default)] on it, see BlobRecoveryConfig as an example.
  • You may need to update the documentation to reflect the changes.

@halfprice halfprice force-pushed the zhewu/sliver-existance-check branch from 4557d80 to 0f2ccfa Compare April 18, 2025 09:18
@halfprice halfprice requested a review from jpcsmith April 18, 2025 09:25
@halfprice halfprice marked this pull request as ready for review April 18, 2025 09:33
@mlegner mlegner removed their request for review April 23, 2025 11:14
@halfprice halfprice force-pushed the zhewu/sliver-existance-check branch from 2bb613f to e2cb416 Compare May 6, 2025 23:08
@halfprice halfprice requested review from jpcsmith and sadhansood May 6, 2025 23:24
@halfprice halfprice requested a review from sadhansood May 7, 2025 20:39
@halfprice halfprice requested a review from liquid-helium May 8, 2025 04:53
Copy link
Contributor

@sadhansood sadhansood left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this looks good to me, i hope @liquid-helium can also once take a look and ensure his comments are fully addressed.

Copy link
Contributor

@liquid-helium liquid-helium left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks good to me, I left some comments but we don't need to resolve them in this PR.

@halfprice halfprice merged commit 748f3b6 into main May 8, 2025
24 checks passed
@halfprice halfprice deleted the zhewu/sliver-existance-check branch May 8, 2025 22:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants