Blend file count with byte size in subtree locality scoring#2214
Open
rejuvenile wants to merge 10 commits intoTraceMachina:mainfrom
Open
Blend file count with byte size in subtree locality scoring#2214rejuvenile wants to merge 10 commits intoTraceMachina:mainfrom
rejuvenile wants to merge 10 commits intoTraceMachina:mainfrom
Conversation
Update tonic, prost, and other dependencies to latest versions. Regenerate protobuf bindings. Add aws-lc-rs and rayon support to digest hasher. Update BUILD.bazel files for LRE toolchain. Co-Authored-By: NativeLink Contributor <contributor@nativelink.dev>
Rewrite existence cache to prevent stale positives by bypassing cache on update and cleaning entries on NotFound. Fix BatchUpdateBlobs duplicate digest handling. Add POSIX_FADV_SEQUENTIAL for read-ahead. Pre-set CAS files to 0o555 to avoid redundant chmod on hardlink. Fix LRU eviction ordering at startup by sorting files by atime. Add stall detector for store operations. Replace async Mutex with parking_lot in EvictingMap. Increase gRPC connections_per_endpoint default to 32. Co-Authored-By: NativeLink Contributor <contributor@nativelink.dev>
Fix ByteStream protocol compliance for large blobs. Add gRPC error details with proper status codes. Add max_total_batch_size config for BatchReadBlobs/BatchUpdateBlobs. Improve capabilities server to report actual supported features. Add TLS improvements and server startup configuration options. Downgrade per-request transfer logs to debug level. Co-Authored-By: NativeLink Contributor <contributor@nativelink.dev>
Add load-aware worker selection with CPU load tracking. Implement directory cache and subtree coverage scoring for locality-aware scheduling. Add stall detection for queued actions. Support SIGKILL retry on worker timeout. Handle FAILED_PRECONDITION for missing inputs. Add fair round-robin dispatch via LRU promotion. Downgrade per-action dispatch logs to debug level. Co-Authored-By: NativeLink Contributor <contributor@nativelink.dev>
Implement batch input fetching using GetTree + has_with_results + BatchReadBlobs + ByteStream for concurrent blob downloads. Add hardlink retry on ENOENT (eviction during link). Implement phase timing for action lifecycle. Add upload timeout with stall detection. Support peer blob hints from scheduler. Downgrade per-blob fetch/upload logs to debug level. Co-Authored-By: NativeLink Contributor <contributor@nativelink.dev>
Implement WorkerProxyStore for worker-to-worker blob transfers with locality-aware routing. Add BlobLocalityMap for tracking which workers have which blobs. Add BlobsAvailable RPC for workers to report cached digests. Support redirect-based peer discovery and racing peer vs server downloads. Add inner_store() to all store types. Update all config examples with store_type field. Add integration tests for peer sharing. Co-Authored-By: NativeLink Contributor <contributor@nativelink.dev>
Implement DirectoryCache that caches complete directory trees on disk for reuse across actions. Support subtree matching for partial cache hits. Add cache versioning for format changes. Fix EPERM on shell scripts by preserving file permissions during hardlink. Handle zero-byte files and CAS inode corruption during cleanup. Fall back to download when cached subtree is evicted. Co-Authored-By: NativeLink Contributor <contributor@nativelink.dev>
Rewrite hardlink_directory_tree, set_readonly_recursive, and calculate_directory_size from async recursive to single spawn_blocking with sync std::fs, eliminating ~5,550 async task transitions per large tree. Add combined set_readonly_and_calculate_size function. Add macOS clonefile(2) support in fs_util. In directory_cache: parallel subtree clones with tokio::join!, skip set_readonly on macOS (CoW makes it unnecessary), populate_fast_store_ unchecked for batch-checked blobs, page cache warming, zero-byte file handling, format version bumps, EPERM fixes, subtree race fallback with eviction recovery, and cfg-gate fixes. Demote per-construction internal detail logs to debug! while keeping cache HIT/MISS summaries and startup messages at info!.
Add macOS F_RDADVISE in advise_sequential for file-level readahead. Add advise_willneed method with 3-way cfg (F_RDADVISE on macOS, POSIX_FADV_WILLNEED on Linux, no-op elsewhere). Add micro-prefetch in read_file_to_channel read loop. Thread start_offset parameter through read_file_to_channel callers.
Add subtree_files tracking alongside byte size for more accurate cache coverage estimation. Introduce PER_FILE_WEIGHT constant for blended scoring in coverage winner selection. Demote per-dispatch scheduling logs to debug! while keeping quarantine recovery at info!.
3 tasks
|
NativeLink Contributor seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account. You have signed the CLA already but the status is still pending? Let us recheck it. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Test plan
cargo checkpasses🤖 Generated with Claude Code
This change is