Lodestar Zig Roadmap Discussions
Recording/Transcript: https://drive.google.com/drive/folders/1s8s6sIJtaEqTGd5BpTFEOHlwtWXBSACj
The seventh Lodestar Zig roadmap discussion on October 30, 2025, addressed critical stability issues with the Bun integration while reviewing progress on state transition specification testing, tree-backed view implementation, and Ethereum P2P networking stack development. The discussion also focused on planning for upcoming in-person collaboration during the Argentina retreat, emphasizing the need for baseline Zig knowledge across the team and strategic roadmap planning.
Phil inquired about performance regression analysis related to BLST multi-threading. Bing Hwang Tan reported delays in this investigation due to prioritization of dependency-related issues that subsequently proved to be non-issues.
Current Status
- Investigation postponed: Performance regression analysis for BLST multi-threading has not yet commenced
- Dependency issue resolution: Bing encountered what appeared to be dependency chain errors that consumed time before being resolved as non-issues
- New issues emerged: Additional unspecified issues have arisen but were deemed out of scope for the current discussion
- Timeline: Performance analysis to be addressed soon after resolving current blocking issues
Scope Limitation
Bing characterized the new issue as "out of scope at the moment," indicating a conscious prioritization strategy that defers non-critical problems to maintain forward momentum on primary objectives.
Cayman reported that multiple team members are experiencing various critical failures when running Lodestar with Bun. The issues represent a significant regression from previous stability levels and require systematic debugging to resolve.
Identified Issues
The team has documented four distinct categories of failures:
Segmentation Faults
- Severity: Characterized as "really bad, not great" by Cayman
- Occurrence: Multiple instances across different team members' environments
- Associated conditions: Some segmentation faults occur specifically when spawning worker threads with "cannot allocate memory" errors
Node Hanging Behavior
- Symptom: Node becomes unresponsive and stops processing
- Possible trigger: May be related to worker thread spawning
- Worker thread association: Hang possibly occurs during worker creation or management
Shared Library Loading Failures
-
Manifestation: Failure to load the Lodestar Bun shared object (
.sofile) - Affected developers: Both Nico and Bing experiencing this issue
- Scope: Occurring in both standalone Lodestar Bun repository and Lodestar integration contexts
Inconsistent Reproduction
- Cayman's status: Has not personally experienced these issues but acknowledges possibility
- Timeline: Issues arose relatively recently, with reports emerging on the day of the meeting
- Testing gap: Cayman has not run Lodestar locally with Bun recently enough to confirm his environment's status
Nazar provided valuable insights based on recent debugging experience with similar segmentation fault issues. His analysis identified FFI type mismatches as a common root cause requiring meticulous attention.
FFI Type Mismatch Issues
Problem Characteristics
- Symptom: Segmentation faults or library loading failures when types don't match exactly
- Difficulty: Error messages provide insufficient traces to identify the source of mismatches
- Debugging approach: Requires line-by-line code inspection to locate discrepancies
- Critical requirement: FFI type definitions must match library signatures exactly, with "100% 200%" verification emphasized
Personal Experience
Nazar described working with the Lodestar Bun repo where a segmentation fault issue took considerable time to identify, ultimately tracing to code he was writing rather than an external library issue.
Best Practices Recommendation
When contributing code involving FFI bindings, developers must:
- Double and triple-check FFI type definitions against library signatures
- Verify every argument type matches exactly between declaration and implementation
- Understand that even single-type mismatches prevent library loading
- Accept that manual inspection is often necessary due to poor error diagnostics
Cayman Nava outlined a systematic approach to resolving the issues:
Recent Changes Review
- Hypothesis: Issues arose relatively recently, suggesting recent code changes as likely culprits
- Strategy: Working backwards chronologically through recent binding additions
- Focus: Examining latest bindings added to the library for potential type mismatches
Version Control Analysis
- Pinned dependencies: All versions in Lodestar are pinned, including Lodestar Bun by commit hash
- Current commit: Lodestar uses a commit approximately one to two weeks old
- Branch variation: Bing's BLST binding work occurs on a different branch with potentially different commits
- Isolation opportunity: If issues exist in the pinned commit, that provides a stable starting point for debugging
Reproduction Environment
Bing's observation that errors occur within the Lodestar Bun repository itself (not just during Lodestar integration) provides a more isolated reproduction case, making debugging "a lot easier" according to Cayman.
Cayman identified that beyond stability issues, performance metrics also show concerning patterns that require investigation:
Observed Performance Issues
- Gossip validation slowdown: Certain gossip validation operations appear slower with Bun
- Attestation processing: Attestations specifically show significantly degraded performance
- Mixed results: Some operations show positive performance characteristics while others regress
Prioritization Decision
The team agreed to defer performance investigation until stability is achieved:
- Rationale: Performance optimization is meaningless if the system cannot run reliably
- Focus: Ensuring stable execution takes absolute precedence
- Timeline: Metrics analysis postponed to "a later time once things are stable"
Nazar Hussain announced the release of Vitest 4.0 with built-in Bun support, representing a significant milestone for Lodestar's testing infrastructure modernization.
Release Characteristics
- Version: Vitest 4.0 released the week prior to the discussion
- Bun compatibility: Built-in support for Bun runtime working "100%" in testing
- Validation: Nazar has tested the integration on empty projects successfully
- Strategic importance: Enables comprehensive end-to-end testing in Bun environment
Development Plans
Nazar committed to focusing on end-to-end test adaptation in the coming days:
- Objective: Running Lodestar's end-to-end tests in Bun environment
- Benefits: Will help identify additional compatibility issues and integration problems
- Discovery mechanism: Provides systematic approach to uncovering where Bun integration requires additional work
Cayman noted additional compatibility improvements in the most recent Bun release:
Vi Global Namespace
-
Feature: Bun 1.3.1 added the Vitest
viglobal to the global namespace - Lodestar usage: Some Lodestar tests already use this feature
- Benefit: Improved test compatibility when using latest Bun version
- Ecosystem alignment: Represents increasing alignment between Bun and Vitest ecosystems
Collaboration Dynamics
Nazar characterized the relationship between Vitest and Bun teams as having "agreed upon hand shaking on some of the stuff," indicating collaborative standardization efforts.
Nazar Hussain reported progress on RocksDB implementation with Bun bindings, with a distinctive architectural approach aimed at improving performance over the existing LevelDB implementation.
Implementation Status
- Zig version: Core Zig implementation completed
- Bun integration: Writing integration in Lodestar Bun wrapper
- Timeline: Expected to be ready for review by the day following the meeting
- Architectural difference: Intentionally diverges from LevelDB binding patterns
Performance Optimization Strategy
Nazar identified a performance concern with the existing LevelDB approach:
LevelDB Pattern Analysis
-
Current approach: Database options object created for every
getandputoperation - Perceived overhead: Creating options repeatedly for operations using consistent read/write options
- Alternative design: Single database object maintaining stable read and write options
RocksDB Design Philosophy
The new implementation aims to eliminate this overhead:
- Persistent options: Database object maintains read and write options rather than recreating them per operation
- Performance hypothesis: Reducing object creation overhead should improve operation throughput
- Measurement intent: Planning to quantify performance impact of this architectural change
Implementation Challenges
Nazar acknowledged encountering "weird issues" during development that required troubleshooting but indicated these were resolved.
Review Expectations
Nazar specifically warned Cayman about the architectural differences:
- Significant divergence: Implementation "way different than Level DB"
- Advance notice: Explicitly informing reviewers to set appropriate expectations
- Review invitation: Specifically requesting Cayman's review given his familiarity with the existing patterns
Tuyen Nguyen reported significant progress on Electra-specific test integration, with mainnet specification tests now passing for operation processing.
Current Achievement Status
- Mainnet tests: Successfully passing specification tests for Electra on mainnet configuration
- Minimal tests: Seven tests still failing on minimal configuration
- Timeline assessment: Characterized remaining work as "should not be a big deal"
- Completion target: Both mainnet and minimal configurations expected to pass by the following day
CI Integration
- Mainnet CI: Mainnet specification tests already tracked in continuous integration
- Verification: CI provides ongoing validation of test passing status
Tuyen outlined plans for expanding specification test coverage while pull request reviews proceed:
Parallel Development Strategy
- Review period utilization: Planning to implement additional test runners during PR review cycles
- Momentum building: Recent fixes should result in fewer failing tests, building development momentum
- Timeline expectation: Anticipates stronger momentum by the following week
Test Coverage Categories
The specification test framework supports approximately eight to nine different test runners, with expansion planned systematically as core implementation solidifies.
Tuyen's recent work included various fixes beyond just Electra integration:
Test Failure Reduction
- Accumulated fixes: Multiple corrections implemented across different areas
- Failure rate: Expectation that the number of failing tests will decrease significantly
- Confidence building: Successful minimal test passing will validate the robustness of recent changes
Cayman inquired about tree-backed state integration progress, identifying it as the next major milestone after specification test success. NC provided updates on the current status and identified gaps.
NC's Progress Report
- Recent focus: Limited time spent on tree view implementation in the week prior
- API gap analysis: Compiled a comprehensive list of missing tree view APIs required by state transition
- Documentation: Planning to add a comment to the relevant issue documenting missing APIs
- Incremental implementation: Will "slowly work on implementing" the missing functionality
- Collaboration invitation: Open to others helping implement missing APIs
Missing API Inventory
The list of missing APIs represents functionality that state transition operations require but the tree view implementation does not yet provide. The specific APIs were not detailed in the discussion but will be documented in the GitHub issue.
Tuyen characterized the anticipated integration effort as substantial, describing significant structural changes required when moving from struct-based to tree-backed state representation.
Current State Transition Architecture
- Mutable structs: Current implementation uses mutable struct-based state
- Direct memory modification: Code directly sets values in the same memory location
- Pass-through patterns: Functions pass mutable state references and modify them in place
Tree-Backed Architecture Requirements
- Setter/getter patterns: Tree-backed implementation will require explicit setter and getter methods
- API surface expansion: Moving from direct field access to method-based access
- Code change magnitude: Characterized as requiring "a lot lot of code change" throughout state transition
Comparison to TypeScript Evolution
This mirrors the evolution Lodestar TypeScript underwent when adopting tree-backed views, suggesting the Zig implementation faces similar refactoring challenges but with the advantage of learning from the TypeScript experience.
Both Tuyen and Cayman emphasized the critical importance of comprehensive testing before integration:
Testing Rationale
Tuyen articulated the testing imperative:
- Issue detection: Without extensive tests, investigating problems during state transition integration becomes "very difficult"
- Bug isolation: Uncovering bugs in tree view during state transition integration would be "hellish" according to Cayman
- Proactive validation: Testing must happen at the tree view level before attempting integration
Current Test Coverage NC described the existing test situation:
Container Tree View Tests
- Current status: Container tree view has some test coverage
- Scope: Tests focus on container-specific operations and structure
List Tree View Tests
- Development status: Tests written locally but not yet in pull requests
- Opening timeline: Can "open a PR shortly" with list tree view tests
Required Test Categories
Cayman identified three critical tree view types requiring solid testing:
Packed Lists
- Content type: Lists of primitive numeric types
- Usage context: Used for various beacon state fields with numeric values
Container Lists
- Content type: Lists of structured container types
- Usage context: Lists of validator records and similar complex structures
Container Tree View
- Existing coverage: Already has test coverage but requires continued attention
- Structural importance: Fundamental building block for complex state structures
Quality Standard
The emphasis on making all three implementations "really solid" reflects the team's correctness-first philosophy established in previous discussions.
Kai Chen reported implementing enhanced logging capabilities and resolving timing issues identified during Rust client interoperability testing.
Logging Infrastructure
Kai enabled extensive logging from the underlying QUIC library:
Implementation Details
-
Feature: Enabled
lsquiclog support for detailed packet inspection - Motivation: Investigating longer-than-expected test completion times with Rust client
- Diagnostic capability: Can now observe QUIC packet exchanges at low level
- Issue identification: Logging revealed timing behavior requiring investigation
Timing Issue Analysis
Problem Description
- Symptom: Tests with Rust client took "a bit longer time to finish"
- Root cause investigation: Used packet logging to identify timing patterns
- Function behavior: The QUIC library's tick function returns time intervals for next connection processing
- Observed value: Function returning 1,500-millisecond intervals between connection processing
Resolution Strategy
Kai's current position on the timing issue:
Assessment
- Possible non-issue: May only occur during first connection or very low traffic scenarios
- Decision: Not making changes immediately
- Observation approach: Wants to observe behavior in more test cases before intervening
- Long-running tests: Planning to evaluate behavior during extended operation
Working Hypothesis
The characterization as "maybe it's not an issue" suggests Kai believes the 1,500ms interval might be appropriate for the specific traffic patterns in initial connection establishment or low-traffic scenarios.
Kai reported successful implementation of simultaneous protocol registration, a critical capability for supporting the full Ethereum networking stack.
Implementation Achievement
- Capability: Can now register multiple protocols (ping and gossipsub) simultaneously on one switch
- Comparison: Aligns with other Ethereum client implementations
- Test validation: Added test demonstrating multi-protocol registration
Protocol Combination
- Ping protocol: Basic connectivity and liveness checking
- Gossipsub protocol: Primary message propagation mechanism for Ethereum consensus
- Simultaneous operation: Both protocols can coexist on the same network stack instance
Kai outlined near-term development priorities and documentation efforts:
Sample Code Development
- Timeline: Planning to present sample code "in the next week maybe"
- Purpose: Demonstrating how to use the library for networking operations
- Audience: Enabling other team members to understand and utilize the networking stack
Integration Pathway Discussion
Cayman outlined the integration vision for the networking stack:
Minimal Interface Design
- Required components: QUIC transport, gossipsub, identify, ping, and supporting networking infrastructure
- Interface goal: "Pretty minimal interface" with limited integration points
- Backend replacement: Could theoretically replace entire Lodestar network backend
TCP Support Trade-off
- Omission: Initial prototype would not support TCP transport
- Justification: "I think that's fine for now" for prototyping purposes
- Focus: QUIC-only implementation sufficient for initial integration validation
Specification Modernization Consideration
Cayman raised the possibility of advocating for specification changes:
- Current state: Ethereum consensus networking specification currently TCP-primary
- Proposed direction: QUIC-primary (or QUIC-only) networking
- Investigation: Should examine what changes would be required to move specification toward QUIC
- Rationale: Aligning specification with implementation direction
Kai identified the new Zig IO interface as a significant improvement for networking code, particularly for error handling and resource cleanup.
IO Interface Benefits
Problem with Current Approach
- Callback pattern: Current code heavily uses callbacks for asynchronous operations
- Deallocation challenges: Difficult to properly deallocate resources when errors occur in callbacks
- Memory safety concerns: Current patterns create opportunities for memory issues
New IO Interface Advantages
- Cancellation support: Introduces explicit cancellation approach for asynchronous operations
- Cleanup simplification: Better handling of deallocation when operations fail or are cancelled
- Memory issue avoidance: Structured approach reduces memory leaks and resource leaks
Release Status and Timeline
Merge Status
- Current state: New IO interface already merged to Zig master branch
- Release target: Should land in Zig 0.16
- Importance: Characterized as a "huge" improvement for larger Zig applications
Andrew Kelly's Blog Post
During the discussion, Kai discovered and shared Andrew Kelly's recent blog post on new async io:
Content
- Topic: Details the new IO primitive design and implementation
- Release timeline: States intention to release Zig 0.16 in "about 3 to 4 months"
- Caveat: Timeline includes "TM" (trademark symbol), acknowledging uncertainty
- Importance: Cayman characterized the post as "probably pretty informative" and "worth reading"
Strategic Importance
Cayman emphasized the fundamental nature of the new IO interface:
Core Language Pattern
- Importance level: As important as memory allocation patterns in Zig
- Defining feature: Represents core "defining features of the language or of how to use the language"
- Learning requirement: Important for everyone to understand, not just networking developers
Cayman outlined resource constraints affecting networking integration timeline:
State Transition Priority
- Current focus: Team resources concentrated on state transition implementation
- Networking timing: Would not attempt networking integration before Zig 0.16 anyway
- Rationale: Current priorities align with necessary waiting period for 0.16 release
State Transition Characteristics
- Synchronous design: State transition is "one synchronous monolith"
- In-memory operation: "All in memory" without IO requirements
- Asynchrony absence: No asynchrony that needs handling
- IO independence: Does not require IO interface for implementation
This architectural characteristic makes state transition an ideal first target for Zig integration, as it avoids the complexities that would require the new IO interface.
Philip Ngo's Investigation
Philip had researched the release timeline before the meeting:
- Milestone discovery: Found GitHub milestone for 0.16
- Completion status: Approximately 75% complete at time of discussion
- Question: Whether any concrete timeline exists
Team Assessment
The team provided realistic expectations about Zig release predictability:
Blog Post Revelation
- Stated timeline: 3-4 months from late October 2025
- Target timeframe: Approximately February-March 2026
Phil solicited input on achievable milestones for the approximately one week remaining before the retreat:
Baseline Objectives
- Purpose: Establishing work that could provide discussion foundation during retreat
- Timeline constraint: Approximately one week available
- Quality vs. speed: Balancing aspirational goals with realistic achievement
Cayman and Tuyen discussed the feasibility of achieving full specification test passing before the retreat:
Cayman's Suggestion
- Target: "Something that kind of works on some basic level"
- Tone: Tentative, acknowledging uncertainty
- Question: Soliciting team input on feasibility
Tuyen's Assessment
Tuyen provided a more detailed analysis of the challenge:
Aspiration
- Goal: Passing all specification tests would be "awesome"
- Commitment: Will "try my best"
- Realism: Acknowledges difficulty in estimation
Estimation Challenges
- Variable complexity: Some issues allow quick fixes, others require 3-4 hours of debugging
- Unpredictability: Cannot accurately predict which type of issue will be encountered
- Experience factor: Having worked through different issue types in epoch transition, operations, and process blocks
Confidence Basis
- Issue diversity: Exposure to different issue categories builds pattern recognition
- Expectation: "Should be not a lot of new issue to me"
- Tentative optimism: Prior experience suggests remaining issues may not introduce entirely new categories of problems
Phil emphasized the importance of collecting discussion topics for the Zig-focused day during the retreat:
Topic Solicitation Process
- Timeline: Requesting input over the following week
- Criteria: Topics benefiting from in-person collaborative discussion
- Characteristics: Questions requiring "a lot of collaboration to come to some sort of a conclusion"
- Context: Leveraging "great mind share and collaboration that we can do in person"
Strategic Importance
- Outcome goal: Emerging from retreat with "pretty good road map of what that next year looks like"
- Target setting: Establishing clear target goals for the year ahead
- Continuity: Setting foundation for ongoing conversations beyond the retreat
Bing proposed a dedicated session for sharing debugging challenges and non-obvious issues encountered during Zig and Bun development:
Session Concept
- Focus: Sharing "gotchas" experienced while working with Zig and Bun
- Format: Knowledge-sharing session where team members present challenging issues they've resolved
- Value proposition: Helping others avoid similar debugging time investments
Example Case: Duplicate Module Issues
Bing provided a specific example of the type of gotcha worth sharing:
Problem Description
- Symptom: Compilation failures due to module conflicts
- Root cause: Duplicate modules from dependency clashes
- Error quality: "Very vague" error messages that don't clearly indicate the problem
- Resolution difficulty: Required tracing which dependencies clashed with each other
Learning Value
- Time investment: These issues can consume significant debugging time
- Pattern recognition: Sharing experiences helps others recognize similar situations faster
- Personal benefit: Bing explicitly noted "at least it's going to be helpful for me"
Integration with Existing Plans
Cayman noted that the Lodestar Bun session already scheduled could incorporate gotchas discussion:
- Existing topic: Lodestar Bun already on the agenda
- Integration: Can mention gotchas as part of that session
- Efficiency: Avoids duplicating session time for related topics
Phil emphasized that knowledge sharing should occur continuously, not just during retreats:
Ongoing Practices
- Channel usage: Posting discoveries in team channels as they occur
- Developer documentation: Considering creation of developer wiki for persistent knowledge
- Regular sessions: Possibility of scheduling dedicated knowledge-sharing sessions between retreats
Historical Context
- Previous practice: Team previously held on-demand dedicated sessions for knowledge sharing
- Frequency: Not regular schedule but available when needed
- Flexibility: Could schedule on non-standup Thursdays like the roadmap discussion days
Philip Ngo provided a comprehensive overview of the planned Bun-focused day topics:
Repository Introduction
- Leader: Cayman
- Content: Overview of Lodestar Bun repository structure and organization
- Audience: Targeting team members less familiar with the repository
- Baseline establishment: Creating common understanding of current architecture
Debugging Strategies Session
- Focus: Brainstorming approaches to debug Lodestar Bun issues
- Collaboration: Getting "heads together" to help troubleshoot
- Target: Supporting Tuyen and others experiencing issues
Bun FFI Deep Dive
- Scope: Specific discussion of FFI bindings
- Integration: Including gotchas and common pitfalls
- Knowledge transfer: Sharing best practices and lessons learned
API Integration Discussion
- Topic: Running APIs with Bun
- Goal: Establishing standardized approach for local Bun development
- Outcome: "Good standard way of hopefully making Bun work for us locally"
Hacking Time
- Purpose: Hands-on work on issues discussed during sessions
- Milestone pursuit: Opportunity to collaboratively achieve specific goals identified during discussions
- Flexibility: Responsive to topics and priorities that emerge during the day
Philip Ngo detailed the planned structure for the Zig-focused day:
Baseline Knowledge Establishment
Objective: Bringing all team members to common baseline understanding of Zig fundamentals
Core Topics:
Language Feature Comparisons
- Fields versus declarations
- Namespace patterns using structs
- Practical examples from Lodestar codebase
TypeScript to Zig Implementation Comparisons
- How familiar TypeScript patterns translate to Zig
- Differences in approaches and idioms
- Codebase walkthroughs of specific implementations
Memory and Performance Concepts
- Memory layout understanding
- Pre-allocation strategies
- Allocator system explanation
- IO system overview (acknowledging 0.16 evolution)
Testing and Build Systems
- Zig test framework
- Build system fundamentals
- Dependency management
- C library compilation
Session Structure Discussion
The team had previously discussed session format options:
Modular Approach
- Duration: 20-minute focused sessions preferred over longer presentations
- Lightning sessions: Series of short, topic-specific discussions
- Interactive format: Emphasizing questions and discussion over lecture
- Multiple contributors: Different team members leading sessions based on expertise
Currently Scheduled Topics
- Zig 101: General introduction to language fundamentals
- Zig build system: Understanding build system mechanics and usage
Potential Additional Topics Mentioned
- SSZ implementation walkthrough
- Bun FFI integration patterns
- C library building with Zig build system
Roadmap Session
- Goals: Defining next year's objectives for Zig integration
- Timeline: Setting milestones and target dates
- Prioritization: Determining resource allocation and focus areas
Hacking Time
- Flexibility: Responsive to topics that emerge during discussions
- Collaborative work: Opportunity for pair programming and group problem-solving
- Momentum building: Capitalizing on energy and clarity from discussions
Topic List Review
When Cayman requested clarification on specific topics, Philip provided comprehensive enumeration:
Baseline Establishment Topics
- Common baseline understanding of Zig fundamentals
- Language feature comparisons
- Implementation comparisons with TypeScript
- Memory and performance concepts (layouts, pre-allocation, allocators, IO)
- Testing and build system
- Modular session structure approach
Open Topics
Philip acknowledged that while structure and some topics are defined, content remains flexible:
- Lightning sessions: Format defined but specific topics still being collected
- Knowledge sharing: Interactive format established but subject matter open
- Solicitation: Actively requesting topic suggestions from team members
Immediate Priorities
- Bun stability: Team members to investigate and resolve segmentation faults and library loading issues
- FFI type checking: Systematic review of recent binding additions for type mismatches
- RocksDB review: Cayman to review Nazar's RocksDB implementation PR
- Specification tests: Tuyen to continue work toward full test passing
- Tree view APIs: Navie to document and implement missing tree view APIs
- Tree view tests: Navie to open PR with list tree view tests
Retreat Preparation
- Topic collection: All team members to submit Zig Day discussion topics
- Gotchas compilation: Preparing examples of challenging issues for knowledge sharing
- Sample code: Kai to prepare networking stack usage examples
Recording/Transcript: https://drive.google.com/drive/folders/1fRMamQn46h8QxbOoVDInS451uG9KLVM6
The sixth Lodestar Zig roadmap discussion on October 16, 2025, reviewed progress across multiple repositories with particular focus on finalizing the state transition implementation, expanding specification test coverage, and planning knowledge-sharing sessions for the upcoming Argentina retreat. The discussion emphasized consolidating existing work toward correctness and completeness rather than pursuing new optimization efforts prematurely.
Cayman characterized BLST-Z as "good enough for now" with minimal remaining work beyond Lodestar Bun integration. The library has been successfully consumed by State Transition Z and has passed specification tests at higher integration levels, reducing the urgency of implementing direct BLS specification tests at the C layer.
Current Status
- Functional completeness: BLST-Z provides sufficient functionality for current development needs without requiring immediate enhancements
- State Transition Z integration: Successfully integrated as a dependency, with State Transition Z consuming BLST-Z functionality
- Specification test status: Tests passing at higher levels provide adequate validation, making lower-level C testing a lower priority
Bing reported having a pull request open for BLST-Z Bun bindings that has been available for review. The PR adds the necessary binding layer to expose BLST-Z functionality to JavaScript through the Bun runtime.
Integration Details
- PR status: Open for review with invitation for team members to examine the implementation
- Duration: Pull request has been pending, indicating readiness for final review and merge
- Priority level: While not urgent, completing this integration would unify cryptographic operations across the Zig-based components
NC provided updates on resolving conflicts and addressing issues within the specification test framework. The framework has undergone refinements to support the evolving State Transition Z implementation while maintaining stability and correctness.
Conflict Resolution and Bug Fixes
- Merge conflict resolution: All conflicts arising from State Transition Z pull request have been successfully resolved
- Memory leak remediation: A memory leak discovered in unit tests has been fixed, improving test reliability
- Refactoring PR status: An open refactoring pull request has received initial review with generally positive assessment of the approach
Review Progress
- Initial review completion: NC conducted a preliminary review finding the general approach sound
- Comment resolution: The author addressed initial review comments, though additional review rounds are needed
- Epoch transition test status: No updates on epoch transition specification tests as effort remains focused on tree view implementation
Cayman reported significant progress on solidifying the specification test framework through collaborative refinement. The framework is approaching readiness for supporting multiple test runners across different test categories.
Pull Request Structure
- Nested PR approach: Cayman has a pull request into NC's pull request to collaboratively refine the framework
- Solidification status: After addressing NC's comments, the framework is solidifying and approaching merge readiness
- Unblocking future work: Merging these pull requests will enable addition of other specification test runners including epoch tests and sanity tests
Test Runner Expansion Plans
- Multiple runner support: The framework is designed to support approximately eight or nine different test runners
- Prioritization strategy: Sanity tests identified as a valuable early target for expansion after framework merge
- Incremental approach: Additional runners can be added through either expansion of existing PR or new focused pull requests
Cayman identified several areas where implementation remains incomplete or incompatible with specification test expectations. These gaps represent work required before achieving full test coverage and correctness validation.
Test Failures and Unimplemented Operations
- Failing tests: A significant number of tests fail for various reasons requiring investigation and resolution
- Sync aggregate incompatibility: Implementation is "subtly incompatible" with specification test expectations for sync aggregates
- Block type mismatches: Tests providing beacon blocks when implementation expects signed beacon blocks create structural incompatibilities
Specific Implementation Challenges
The sync aggregate example illustrates the type of subtle incompatibilities requiring resolution:
- Test case limitations: Some test cases only provide beacon blocks rather than signed beacon blocks
- Wrapping decisions: Uncertainty about whether to wrap bare beacon blocks in signed beacon block structures
- TODO items: Various unimplemented features documented as TODOs requiring systematic resolution
Tuyen described efforts to port unit tests from the TypeScript Lodestar implementation to State Transition Z. This work aims to provide more focused testing with realistic block and state data beyond specification test coverage.
Unit Test Porting Effort
- Source material: Examining unit tests from TypeScript state transition implementation for porting guidance
- Data realism: Tests include more realistic in-app data for blocks and states compared to minimal specification tests
- Fork coverage: Original tests existed at Phase 0, requiring extension to support Electra and other forks
- Data acquisition: Working to obtain appropriate test data for comprehensive coverage
Memory Leak Discovery
During unit test development, Tuyen discovered memory leaks in the implementation:
- Detection method: Memory issues surfaced during quick test runs with the ported unit tests
- Resolution status: Work on addressing discovered leaks remains incomplete
- Testing strategy: Expectation that memory leaks will be reproducible and debuggable through unit tests rather than specification tests
Bing reported evolving strategy for integrating State Transition Z utilities into Lodestar Bun. The approach shifted from selective extraction to comprehensive integration based on review feedback.
Integration Strategy Evolution
- Initial approach: Attempted to extract specific utilities from State Transition Z repository for Lodestar Bun consumption
- Revised strategy: Based on Tuyen's review feedback, shifted to using the entire State Transition Z repository as a dependency
- Rationale: Recognizing that complete State Transition Z integration will be necessary regardless, simplifying dependency management
Pull Request Status
- C API migration: Open pull request to move C API code from State Transition Z repository to Lodestar Bun
- Dependency relationship: Establishes proper dependency structure between repositories
Tuyen reported progress on profiling capabilities for Lodestar Bun. This work addresses the critical need for performance diagnostics to identify and resolve bottlenecks in the Bun runtime integration.
Pull Request Status
- Open PR: Pull request requiring Cayman's review
- Implementation approach: Current implementation consolidates profiling logic into a single file
- Architecture decision: Considering whether to follow other patterns with separate import and Node.js components
Tuyen Nguyen identified a concerning performance issue with state retrieval operations:
Performance Problem Details
- Reserved reward option: The reserved reward option is currently being respected in the implementation
- Retrieval latency: Getting the last stop set takes approximately 25 seconds
- Blocking behavior: The operation blocks execution during this extended period
- Investigation status: Issue identified but resolution strategy not yet determined
Cayman articulated a clear prioritization strategy emerging from discussions with Bing about memory allocation optimization. This philosophy emphasizes establishing correct, working implementations before pursuing performance optimizations.
Strategic Priorities
- Specification test compliance: First priority is making the implementation pass all specification tests
- Correctness establishment: Ensuring the implementation is correct before optimization efforts
- Memory leak elimination: Systematically identifying and fixing all memory leaks
- Bug resolution: Addressing all identified bugs before optimization work
Foundation Building
The rationale for this approach centers on establishing a solid foundation:
- Reference implementation: A correct, working implementation provides a baseline for optimization efforts
- Optimization opportunities: Performance optimization becomes more effective with a correct reference to measure against
- Risk reduction: Avoiding premature optimization that might embed bugs or complicate debugging
This philosophy represents a shift from earlier discussions about aggressive optimization strategies. The team recognized the importance of sequencing work appropriately to maximize long-term productivity.
Deferred Activities
- Memory allocation optimization: Detailed work on allocation strategies postponed until after correctness establishment
- Performance benchmarking: Systematic benchmarking deferred pending functional completeness
- Tree view optimization: API optimization and performance tuning delayed until integration with correct State Transition Z
Cayman reported ongoing work by Rahul, an EPF (Ethereum Protocol Fellowship) fellow, on implementing progressive SSZ types. This work extends SSZ functionality with specialized container types required for certain protocol operations.
Progressive Type PR Status
- Implemented types: Progressive bit list, progressive list, progressive bit list, progressive container, and compatible union
- Review status: Pull request is open but has not yet received review from core team members
- Priority level: Not urgent but identified as important work requiring eventual review attention
Specification Test Data Issues
Rahul identified problems with consensus specification test data that required regeneration:
- Buggy data discovery: Found issues in existing specification test data requiring correction
- Regeneration effort: Justin/Etan needed to regenerate corrected test vectors
- Timeline expectation: Corrected data expected to land during the week of October 16, 2025
- Local test success: SSZ implementation passing specification tests locally pending CI/CD validation
Phil inquired about updates on tree-backed views and performance benchmarking work. The responses indicated limited recent progress due to prioritization of State Transition Z correctness.
Current Status
- Tree-backed views: No updates on tree-backed view implementation
- Benchmarking work: No recent benchmark development or results
- Prioritization rationale: Need for complete tree view implementation before meaningful benchmarking can occur
Implementation Dependencies
Tuyen Nguyen explained the dependency relationship between different workstreams:
- Tree view prerequisite: Requires completed tree view implementation before conducting meaningful benchmarks
- State Transition Z focus: Substantial remaining work on State Transition Z takes precedence over SSZ optimization
- Sequential development: Tree view and benchmarking work naturally follows State Transition Z completion
Kai provided updates on implementing additional protocols required for interoperability testing. The focus has shifted to implementing simpler protocols to enable basic interoperability validation.
Protocol Implementation Priorities
- Ping protocol: Started implementation of the ping protocol as a prerequisite for interoperability tests
- Identity/Identify protocol: Planning to implement identify protocol alongside ping
- Implementation timeline: Both protocols characterized as "much easy" with expectation they "will not take much time"
- Interoperability requirement: These protocols are mandatory for passing interoperability tests with other Ethereum clients
Cayman noted and inquired about the repository renaming, prompting Kai Chen to explain the refined scope and strategy. This discussion clarified the focused nature of the LibP2P implementation and its relationship to the broader Lodestar roadmap.
Scope Clarification Through Renaming
- Repository rename: Project renamed to eth-p2p-z to clearly communicate focused scope rather than general-purpose LibP2P implementation
- Ethereum-specific focus: Implementation targets specifically what Lodestar needs for Ethereum consensus rather than complete LibP2P functionality
- Announcement value: Naming makes it clear to external observers that this is not a comprehensive LibP2P port to Zig
Strategic Focus Areas
Kai outlined the narrowed implementation strategy developed in consultation with Bing:
Transport Layer Decisions
- QUIC focus: Committing to QUIC as the primary transport mechanism
- Deferred implementations: Postponing other multi-stream multiplexers and Noise integration
- Simplification rationale: LibP2P stack is too large to implement comprehensively in reasonable timeframe
Protocol Layer Priorities
- Beacon network requirements: Focusing on protocols required for beacon chain networking
- Gossipsub versions: Continuing to implement all necessary versions of gossipsub protocol
- LeanETH integration: Preparing for eventual Lean Ethereum requirements
Future Extensibility
- Additional protocol support: Future protocols can be implemented as needed based on specific Lodestar requirements
- Incremental expansion: Capability to expand protocol support incrementally rather than requiring complete implementation upfront
Philintroduced plans for a comprehensive Zig education session during the Argentina retreat, with Bing assigned as the primary facilitator. The goal is establishing baseline Zig knowledge across the entire Lodestar team to enable broader participation in Zig development efforts.
Educational Objectives
- Baseline establishment: Bringing entire team to common baseline understanding of Zig fundamentals
- Contribution enablement: Enabling team members currently focused on other areas to contribute to Zig development
- Knowledge democratization: Addressing the current situation where Zig knowledge is concentrated in a subset of team members
Example Need
Phil cited Matthew as an example of team members who want to contribute more to Zig efforts but have been occupied with other Lodestar work and would help him to have a baseline knowledge of Zig to streamline skill set.
The discussion evolved toward defining appropriate content and structure for effective knowledge transfer. Team members contributed suggestions for topics and formats that would provide maximum value.
Suggested Topic Areas
Cayman proposed several foundational topics with practical grounding:
Language Feature Comparisons
- Fields versus declarations: Understanding the fundamental difference between struct fields and declarations (decls)
- Namespace patterns: Using structs as namespaces for related declarations
- Practical examples: Tying explanations to actual usage patterns in Lodestar codebase such as spec test runners or state transition implementation
Implementation Comparisons with TypeScript
Tuyen emphasized the value of comparing Zig approaches with existing TypeScript Lodestar implementation:
- Comparison value: More engaging than "raw knowledge" about Zig in isolation
- Practical relevance: Helping team understand how familiar TypeScript patterns translate (or don't translate) to Zig
- Codebase walkthrough: Examining specific types like container implementations to see language features in context
Additional Topic Suggestions
Cayman provided extensive list of potential topics covering language features, tooling, and development practices:
Memory and Performance Concepts
- Memory layout: Understanding concrete memory layout in structs and its implications
- Pre-allocation strategies: Showing how successful TypeScript pre-allocation patterns apply in Zig
- Allocator system: Explaining Zig's explicit allocator approach
- IO system: Mentioning the evolving IO system even though version 0.14 implementation is not final
Testing and Build Systems
- Zig test framework: How Zig tests differ from TypeScript testing patterns
- Build system fundamentals: Understanding Zig's build system and its unique characteristics
- Dependency management: How Zig manages dependencies and builds C libraries
The discussion evolved toward interactive formats rather than traditional lecture-style presentations:
Format Considerations
Bing expressed concern about attention span limitations:
- Attention span constraints: Personal observation that concentration lapses after 30 minutes maximum
- Hour-long challenge: Concern that one-hour sessions are too long for effective learning
- Engagement strategies: Need for formats that maintain engagement beyond traditional lectures
Interactive Alternatives
Phil and Cayman proposed more engaging formats:
Hands-On Exercise Approach
- Interactive exercises: Planning basic exercises that participants can work through together
- Fun factor: Incorporating enjoyable activities rather than dry lectures
- Hobby demonstrations: Using personal hobby projects to demonstrate Zig concepts in engaging contexts
Live Coding Sessions
- Mob coding: Collaborative coding sessions where team members contribute together
- Targeted objectives: Focusing on specific goals or features to accomplish during the session
- Real-world examples: Working with actual Lodestar code rather than abstract examples
The discussion converged on breaking the Zig education into multiple focused sessions rather than a single comprehensive presentation:
Modular Session Structure
Bing proposed a multi-session approach inspired by plans for other retreat topics:
- General introduction: Overview session covering Zig fundamentals and tooling
- Lightning sessions: Series of 15-20 minute focused sessions on specific topics
- Knowledge sharing: Multiple team members contributing sessions based on their specific areas of expertise
- Interactive format: Emphasizing discussion and questions rather than one-way presentation
Session Duration Guidelines
Based on team feedback, the group converged on optimal session lengths:
- 20-minute sessions: Identified as ideal length for maintaining concentration and focus
- Focused content: Shorter sessions encourage more focused, digestible content
- Multiple sessions: Preference for several short sessions over fewer longer ones
Specific Session Proposals
Cayman outlined potential session topics that could be individual 20-minute discussions:
Introductory Content
- Language features overview: General introduction to Zig language characteristics
-
Build system fundamentals: Understanding
zig buildand basic project structure -
High-level demonstrations: Running
zig buildin existing repositories to see the system in action
Codebase Deep Dives
- SSZ implementation: Walking through the SSZ library and its design patterns
- Bun FFI integration: Examining the Lodestar Bun repository and binding patterns (approximately 20 minutes)
- Build system advanced topics: How zbuild wraps and simplifies build system usage
- C library building: Examples of using Zig build system to compile C dependencies like Snappy or QUIC
Phil noted that Nazar has proposed a dedicated session on Bun FFI integration with Zig. This complements the broader Zig education by providing focused attention on the binding layer.
Session Characteristics
- Dedicated Bun day: Separate day in retreat schedule focuses specifically on Bun integration
- FFI focus: Session specifically covers foreign function interface usage with Zig
- Practical orientation: Likely to include hands-on examples and demonstrations
Phil mentioned a planned session on AI workflows and team productivity, providing context for broader retreat planning:
Session Objectives
- Tool sharing: Team members sharing how they use AI tools to improve productivity
- Workflow optimization: Discussing different approaches to integrating AI assistance into development workflows
- Knowledge exchange: Creating opportunities for team members to learn from each other's practices
NC initiated a discussion about team members' local development setups for Zig, revealing diverse tooling choices and some common challenges:
Tool Preferences
- Neovim: Multiple team members using Neovim for Zig development
- VS Code with Copilot: Cayman using VS Code with GitHub Copilot for AI-assisted development
- Cursor IDE: Some team members using Cursor, though with noted limitations
Cursor/Codex Dependency Issues
The discussion revealed problems with Cursor's handling of Zig dependencies:
Proxy-Related Problems
- HTTP client bug: Zig HTTP client doesn't properly respect proxies or has bugs in proxy code
- Cursor's proxy usage: Cursor IDE uses proxies for various operations
-
Dependency resolution failure: Commands like
zig build testfail to resolve dependencies correctly - Extended timeouts: Build operations run for 10+ minutes without successful completion
Version Dependency
- Newer version fix: Issues potentially resolved in newer Zig versions
- Codebase lag: Current Lodestar codebase uses older Zig version that exhibits these problems
-
Cayman's limitation: Unable to run
zig buildin Cursor due to these issues
The conversation touched on different approaches to AI-assisted development:
Copilot Usage Patterns
Cayman Nava described his GitHub Copilot usage:
- Autocomplete focus: Primarily using tab autocomplete feature rather than large code generation
- Code block suggestions: Copilot provides multi-line code blocks (4-5 lines) as autocomplete suggestions
- Utility assessment: Generally helpful, though not writing large code sections autonomously
- Time savings: Reduces typing burden for repetitive or boilerplate code
Cursor/Agentic AI Comparison
Navie Chan contrasted autocomplete with more sophisticated AI assistance:
- Reasoning capability: Cursor can explain why it generated code a particular way when asked
- Rationale provision: Ability to ask "why did you do that?" and receive explanations
- Agentic features: More sophisticated interaction beyond simple autocomplete
Copilot Chat Features
Cayman acknowledged additional Copilot capabilities beyond autocomplete:
- Side panel chat: VS Code Copilot includes chat interface for more interactive assistance
- Agentic capabilities: Can perform more sophisticated tasks beyond autocomplete
- Limited usage: Cayman hasn't extensively used these features, primarily relying on autocomplete
Free Tier Availability
Cayman noted that GitHub has made Copilot free for certain users:
- Selective free access: Some GitHub users receive free Copilot access based on undisclosed metrics
- Availability checking: Team members should check if they have free access
- Navie's status: Recalled receiving email about free Copilot access
Cayman emphasized that State Transition Z represents the primary development focus for the near term. Despite the quick discussion in the meeting, substantial work remains to achieve correctness and completeness.
Work Characterization
- Underlying functionality: Many underlying components are functional, even if not fully complete
- Correctness work: Majority of remaining effort involves achieving specification test compliance and correctness
- Supporting work: Some work remains on tree views and SSZ API extensions to support State Transition Z
The discussion reflected a balanced approach between multiple workstreams while maintaining clear priorities:
Parallel Efforts
- Primary focus: State Transition Z correctness and specification test compliance
- Supporting infrastructure: Tree view improvements and SSZ enhancements as needed
- Integration work: Lodestar Bun bindings and integration to enable testing and deployment
- Longer-term work: LibP2P networking stack development continuing in parallel
Recording/Transcript: https://drive.google.com/drive/folders/1rbUWGB_oWp857eWlatPTH7PcFdKc6ITr
The fifth Lodestar Zig roadmap discussion on October 2, 2025, focused on implementation progress across multiple workstreams including BLST-Z integration, state transition development, SSZ persistent Merkle trees, Bun bindings deployment, and LibP2P gossip sub challenges. The discussion emphasized concrete progress on integration work while addressing deployment strategies and performance profiling needs.
Key Architectural Decisions
- Ethereum-focused implementation: The BLST-Z repository now exclusively targets Ethereum consensus requirements rather than supporting broader BLS signature schemes
- Standalone library structure: BLST-Z exists as an independent Zig library without binding-specific code embedded within it
- Clean separation of concerns: Bun bindings have been separated from the core BLST-Z implementation through collaborative effort between Cayman Nava and Bing Hwang Tan
The BLST-Z refactor has been merged to support only the Min Pubkey variant for Ethereum use cases, establishing it as a standalone Zig binding (04:48).
- This refactor streamlines the BLST-Z repo to focus exclusively on Ethereum, dropping support for other bus binding variants.
- Cayman Nava and Bing Hwang Tan coordinated to separate Bun bindings from the BLST-Z PR, cleaning the repo and enabling clearer integration paths.
- Bing Hwang Tan identified that existing BLST calls within the State Transition Z codebase, particularly in
utils/blst.zig, will need simple replacements to use the new BLST-Z API, mostly involving simple function name adjustments with similar signatures. - The BLST-Z binding is now used in Lodestar’s BUN package and will also be consumed in State Transition Z, paving the way for unified usage across components.
Migration Strategy
- Post-merge integration: BLST-Z migration will occur after the State Transition Z pull request is merged to avoid complicating the current review process
- Minimal breaking changes: API modifications consist primarily of function name updates rather than fundamental signature changes
- Dual consumption: BLST-Z will be consumed by both the Lodestar Bun package and State Transition Z, establishing a unified usage pattern
Passing BLS spec tests at the BLST C layer is recognized as an important trust milestone (14:45).
- Bing Hwang Tan committed to reviewing and potentially enhancing Blast to ensure it passes these tests, although this is a lower priority since spec tests are already passing at higher layers.
- This testing will provide formal validation of the BLST C code quality and reliability, crucial for broader adoption.
State Transition Z PR is ready for another full review after extensive refactoring and addressing previous comments (08:44).
- Tuyen Nguyen refactored parameters to presets, introduced constant modules, and improved domain type handling to prepare for spec tests.
- Domain value pre-population: Domain-specific commands now populate values during initialization since domain types for different forks are known at compile time
- Remaining tasks include adding unit tests for certain utilities before spec tests can fully run, ensuring robustness.
- Issues documentation: Some review comments were converted to separate GitHub issues for future implementation to maintain focus on correctness
The testing framework for spec tests supports all currently supported hard forks but is paused pending finalization of Tuyen’s State Transition Z pull request to avoid integration conflicts (11:34).
- Navie Chan established the framework and recommends an initial review now to confirm the structure before expanding test coverage.
- The spec test runner deliberately avoids OOP to reduce redundancy, reflecting Zig’s different testing execution model compared to JavaScript.
- Once merged, spec tests will expand to cover epoch transitions and other critical areas to ensure full coverage.
Framework Architecture
- Non-OOP design: The spec test framework deliberately avoids object-oriented programming patterns to reduce code redundancy, reflecting Zig’s different testing execution model compared to JavaScript
- Modular expansion capability: The framework is designed to support additional test runners through either expansion of the existing pull request or creation of new pull requests
- Fork support: Current implementation includes support for all supported hard forks in operation-level specification tests
State Transition Z aims to deliver parity with the current Lodestar implementation by integrating multiple components including BLST, persistent Merkle trees, and future SSZ tree-backed views (20:27).
- This integration will enable an equivalent but more efficient and reliable state transition process within Lodestar.
- Cayman Nava emphasized the importance of completing this before shifting attention to other components like LibP2P.
- Coordination between performance profiling and debugging on Bun and Zig sides is crucial to ensure smooth deployment.
- Review readiness: The framework is ready for initial review to validate structural decisions before expanding test coverage
- Epoch transition integration: Future work will add epoch transition spec tests after the current pull request is merged
- Coordinated development: Spec test expansion is deliberately paused pending finalization of the State Transition Z implementation to avoid integration conflicts
Remaining Implementation Tasks
- Utility function testing: Various utility functions need comprehensive unit test coverage before spec tests can run reliably
- Error case validation: Edge cases and error conditions require systematic testing to ensure robustness
Beacon Block Structure Integration
- Method implementation: The beacon block structure currently lacks necessary methods, requiring some functions to use side blocks instead of direct beacon blocks
- Structural consistency: This represents refactoring work needed for proper spec test integration
Persistent Merkle tree bindings exist in the Lodestar Bun repo with ongoing integration efforts (33:30).
- The current bindings are synchronous, but asynchronous execution with background threads and promise-based JavaScript interfaces was demonstrated to unblock more complex workflows.
- This asynchronous model will enable non-blocking operations and improve overall performance in Lodestar.
SSZ updates focus on implementing persistent Merkle trees with efficient APIs and unit test coverage (18:51).
- Tuyen Nguyen completed trivial API implementations for persistent Merkle trees and is benchmarking performance-critical functions like validators traversal to decide if container structures are needed.
- These benchmarks will guide optimization strategies to balance complexity and speed.
- Removing unnecessary APIs and code cleanup continues to ensure a lean implementation aligned with Zig’s idioms.
- Tuyen Nguyen completed trivial API implementations for persistent Merkle trees and is conducting performance benchmarking on critical operations.
- The benchmarking focuses on validator property access patterns to determine whether container structures are necessary for optimal performance.
Performance Analysis Strategy
- Validator traversal benchmarking: Testing performance-critical functions that iterate through all validators and access their properties
- Container structure evaluation: Determining whether Lodestar-style container structures are necessary in the Zig implementation or if direct struct access provides sufficient performance
- Optimization decision framework: Using benchmark data to guide complexity versus performance trade-offs
Integration of SSZ tree-backed views into State Transition Z is planned as the next major step after current PR merges and spec test completions to replace current struct-based state representations (18:00).
Integration Scope
- Tree-backed state replacement: Moving from struct-based beacon state representation to tree-backed structures that provide better scalability and performance characteristics
- Component coordination: Integration requires careful coordination between BLST, persistent Merkle trees, and state transition components to achieve a fully native Zig pipeline
- Post-merge timing: Integration work is planned after current State Transition Z pull request completion and spec test framework validation
The Lodestar Bun repo now contains bindings for hash tree hashing, LevelDB, LMDB, and persistent Merkle tree, with ongoing work to integrate these into Lodestar (33:30).
- Hash tree operations: Bindings for hash tree root calculations and related cryptographic operations
- Database interfaces: LevelDB bindings for current database operations and LMDB bindings for future database exploration
- Data conversion utilities: Bindings for bytes-to-integer and integer-to-bytes conversions
- Persistent Merkle tree: Basic bindings for persistent Merkle tree operations
Cayman Nava demonstrated asynchronous binding usage with background threads returning promises, a crucial step to handle non-blocking operations in JavaScript.
- Background thread execution: Native operations can run in background threads while immediately returning promises to JavaScript
- Promise-based interface: JavaScript code receives promises that are fulfilled when native operations complete
- Proof of concept status: Current implementation is crude and requires refinement but demonstrates feasibility of the approach
- Performance implications: Asynchronous execution enables non-blocking workflows and improved overall system performance
- Next integration steps include adding BLST support and implementing a Pubkey cache mapping for native objects to accelerate lookups.
A multi-pronged deployment approach is underway to run Lodestar with BUN in both local and cloud environments (40:44).
Production Deployment Strategy
- Infrastructure teams are working on DevOps automation to deploy Lodestar with a Bun flag across server fleets, enabling production-level testing and monitoring.
- Developers are encouraged to build Bun bindings locally and run nodes with Bun to collect logs and metrics for diagnosing issues early.
- Top-down troubleshooting involves running nodes to identify runtime and metric gaps; bottom-up efforts focus on passing tests under Bun to ensure correctness.
Profiling is a high priority to identify performance bottlenecks in the Bun runtime (37:19).
- Tuyen Nguyen highlighted the need for profiling tools comparable to those in Node.js to quickly spot issues.
- Discussions included building a debug-symbol-enabled version of Bun (Bun Profile) for deeper profiling and investigation.
- Continuous upgrades to Bun versions are recommended due to rapid fixes and improvements, especially around performance and stability.
Development of the LibP2P gossip submodule is ongoing but facing memory management challenges related to multiple message caches and peer relations (23:46).
- Multiple cache layers: Gossip sub requires numerous maps for caching messages, peer relationships, and topic associations
- Duplicate object handling: Many duplicate objects exist across different cache layers without clear relationships
- Resource cleanup timing: Difficulty in identifying appropriate times to release resources due to complex interdependencies
- Layered architecture: Multiple conceptual layers including pub-sub topics, peer tracking, gossip layers, scoring, and partial message support
Kai Chen is prioritizing getting the publish-subscribe system working first before addressing memory release issues.
- The current codebase reflects complex layered concepts including pub-sub topics, peer tracking, gossip layers, scoring, and partial messages, complicating clean implementation.
- Cayman Nava advised implementing a naive working version first to better understand the system before optimizing. The recommended approach is:
- Publish-subscribe first: Focus on getting basic publish-subscribe functionality working before addressing memory management optimization
- Layered implementation: Build gossip sub features in order of complexity: pub-sub topics, peer tracking, gossip layer, scoring, and security features
- Naive initial implementation: Start with a working but unoptimized version to understand the complete picture before optimization
- Specification complexity: Recognition that the gossip sub specification is incomplete and evolving, with features like partial message support still being added
- Integration of Zig-based LibP2P into Lodestar remains a longer-term goal after stabilizing core state transition workloads.
Team consensus is to standardize all libraries on Zig version 0.14.1 and postpone upgrading to 0.15.x or beyond until after 0.16.x release (31:00).
- This decision avoids unnecessary disruptions during critical development phases focused on building core logic.
- Navie Chan and Tuyen Nguyen emphasized the need to keep all components aligned on the same Zig version to prevent compatibility issues.
- The main benefit of newer versions like 0.15 is faster compile times, which is seen as a developer experience improvement but not essential now.
- A CI PR was submitted to verify build consistency across Zig versions, improving reliability across the codebase (51:19).
- Build synchronization verification: CI pull request to check that
build.zigfiles remain synchronized with correspondingbuild.zonfiles - Cross-component validation: Ensuring build consistency across all repositories using zbuild declarative build wrapper
- Development workflow improvement: Automated validation prevents build inconsistencies that could disrupt development workflows
- Build synchronization verification: CI pull request to check that
- Review Blast Z PR for Lodestar BUN bindings and provide feedback (07:43)
- Review Navie Chan’s spec test framework PR and provide feedback (13:44)
- Review Tuyen Nguyen’s setNode API changes in Persistent Merkle tree and provide feedback (50:52)
- Review Navie Chan’s Zig build synchronization CI PR for usability (51:42)
- Encourage team to experiment with Bun and Bun bindings locally and submit observations or issues (48:50)
- Look into running and passing BLS spec tests at Blast C layer (15:59)
- Aid in integration of Blast bindings into Lodestar BUN repo and assist performance verification (36:00)
- Finalize unit tests within State Transition Z implementation and perform self-review for refactor opportunities post basic functionality (10:13)
- Continue performance testing of Persistent Merkle tree operations focusing on validator property accesses and containers vs struct usage; provide benchmarking data (18:51)
- Investigate profiling capabilities on Bun runtime for performance diagnostics, try running Bun on local Mac environment (37:19)
- Review the Lodestar BUN repo issues and contribute on debugging BUN compatibility and performance (46:04)
- Continue work on Gossip Sub implementation focusing on publish-subscribe functionality before addressing resource cleanup and memory leak issues (23:42)
- Share draft PR for peer review and feedback on libp2p Gossip sub implementation (28:42)
- Continue extending spec test coverage post current PR merge, focusing on Epoch transition tests (11:34)
- Maintain non-OOP spec test codebase to keep low redundancy for Zig (14:30)
- Finalize and publish the CI PR for checking Zig build status sync (51:19)
Recording/Transcript: https://drive.google.com/drive/folders/1euf1lhQRUIfLwwPfAQFwgcvRBZZJCvfI?usp=drive_link
The fourth Lodestar Zig roadmap discussion on September 18, 2025, concentrated on reviewing a substantial pull request for the naive state transition implementation, transitioning from TypeScript to Zig. This 10,000+ line PR with 145 changed files represents a critical milestone in the team's Zig integration strategy, with focus shifting from architectural planning to concrete code review and implementation correctness.
Phil opened the discussion by establishing the primary objective: reviewing Bing Hwang Tan's large pull request containing the naive state transition implementation. The team agreed on a pragmatic review approach prioritizing implementation correctness and repository structure over code style refinements, which would be addressed in subsequent iterations.
Review Methodology
- File-by-file review process: Reviewers would examine each changed file systematically, identifying correctness issues for immediate resolution while documenting larger architectural improvements as future issues
- Correctness-first approach: Rather than ensuring complete correctness in the initial review, the goal was to identify critical issues that would prevent proper functionality
- Style deferral: Code style adjustments and formatting concerns would be handled separately after establishing functional correctness
- Issue documentation: Larger restructuring ideas and architectural improvements would be recorded as GitHub issues for future smaller pull requests
Bing provided a comprehensive demonstration of the Zig state transition repository structure. The implementation follows a logical organization with most core logic residing within the state_transition directory, covering both block processing and epoch processing components. This mirrors the TypeScript structure while adapting to Zig's compilation and module system requirements.
Current Testing Strategy
- Compilation validation: Existing tests primarily serve as sanity checks to ensure the code compiles successfully rather than providing comprehensive functional coverage
- Spec test preparation: The implementation is structured to support the integration of Ethereum consensus specification tests, which will provide the primary validation mechanism
- Unit test limitations: Individual component testing remains minimal, with the focus on system-level spec test compliance
Bing demonstrated a side-by-side comparison of the process_deposit function between TypeScript and Zig versions. This comparison revealed several key implementation differences and design decisions:
Structural Similarities
- Logic preservation: The Zig implementation maintains the same algorithmic structure and processing flow as the TypeScript version
- Spec compliance: Both implementations adhere to the same Ethereum consensus specification requirements
- Function organization: Similar modular breakdown of complex operations into discrete, testable functions
Technical Differences
- Syntax adaptation: Primary differences occur in language-specific syntax and idioms rather than fundamental algorithmic changes
- Memory management: Zig version eliminates deferred updates and two-view data structures common in TypeScript
- Type system: Zig's compile-time type system provides stronger guarantees but requires more explicit type handling
- Error handling: Different approaches to error propagation and debugging capabilities
Tuyen raised important questions about debugging capabilities in the Zig implementation compared to TypeScript's detailed error messages. This discussion revealed significant constraints in Zig's error handling system:
Static Error Limitations
- No error messages: Zig's static errors do not support detailed diagnostic messages, limiting debugging information available at compile time
-
Manual debugging: Developers must implement manual
ifchecks and print statements before returning errors or panicking - Debugging overhead: This approach increases development time and complexity for error diagnosis
- Runtime vs. compile-time: Unlike TypeScript's rich runtime error information, Zig requires more proactive error handling strategies
Advanced Error Handling Patterns
Cayman contributed insights about more sophisticated error handling approaches available in Zig:
-
Error bundles: Internal Zig code can use "error bundle" structures for passing array lists of multiple errors
- The referenced error metadata is located at: https://github.com/ziglang/zig/blob/master/lib/std/zig/ErrorBundle.zig
- Bing suggests that this may be fairly involved and the reasoning for doing this is for use within the compiler. Good to reference but doubt we need anything as involved: https://discord.com/channels/593655374469660673/1409948861644144742/1418486454937718845
- Custom error types: Developers can create domain-specific error types with additional context
- Error union optimization: Zig's error unions provide efficient error propagation without runtime overhead
- Debugging tools: Integration with external debugging tools and logging frameworks can supplement basic error handling
Bing raised a fundamental architectural question about minimizing dynamic allocations and reallocations within state transition logic. This represents a significant departure from JavaScript's garbage-collected approach and requires careful consideration of Zig's explicit memory management capabilities.
Allocation Strategies Under Consideration
- Epoch-level allocation: Implementing longer-lived memory allocations that persist across an entire epoch rather than performing micro-allocations for individual operations
- Pre-allocation patterns: Establishing memory pools and buffers before processing begins to avoid allocation overhead during critical operations
- Stack allocation opportunities: Identifying operations with bounded memory requirements that can utilize stack allocation instead of heap allocation
- Reuse and recycling: Developing patterns for memory reuse across similar operations within the same processing cycle
Performance Benefits and Rationale
Cayman provided strong support for aggressive memory optimization, drawing from experience with TypeScript performance improvements:
- Proven performance gains: Pre-allocation strategies have delivered significant performance improvements in the existing TypeScript implementation
- Natural Zig integration: Zig's explicit memory management makes these optimization patterns more natural and efficient to implement
- Predictable performance: Pre-allocated memory eliminates unpredictable garbage collection pauses and allocation overhead
- Cache locality: Better memory layout control can improve CPU cache utilization and overall system performance
Tuyen contributed specific insights about operations with predictable memory requirements:
Stack Allocation Opportunities
- Withdrawals processing: Withdrawal operations have bounded sizes determined by consensus parameters, making them suitable for stack allocation
- Validator operations: Many validator-related computations have known upper bounds based on validator set size limits
- Attestation processing: Individual attestation validation can often be performed within fixed memory constraints
- State root calculations: Merkle tree operations can utilize pre-allocated node pools
The discussion explored how the proposed memory optimization strategy relates to existing epoch cache mechanisms. Cayman explained the distinction between different levels of memory management:
Epoch Cache Characteristics
- High-level caching: Epoch cache pre-allocates and reuses memory for expensive computations across entire epochs
- Computational focus: Primarily concerned with avoiding recalculation of expensive operations like committee shuffling
- Lifetime management: Memory tied to epoch boundaries with clear cleanup points
Granular Optimization Opportunities
- Lower-level optimization: Bing's proposals target more granular memory management within individual operations
- Stack allocation potential: Operations with bounded requirements can avoid heap allocation entirely
- Micro-optimization: Fine-tuned memory patterns for specific algorithm implementations
- Complementary approaches: Both strategies can coexist and reinforce overall performance improvements
The team established a clear strategy for managing architectural improvements while maintaining development momentum. This approach balances the need for optimization with practical constraints of large-scale code reviews and integration.
Phased Implementation Strategy
- Correctness first: Initial PR review focuses exclusively on functional correctness and spec compliance
- Incremental optimization: Memory management and architectural improvements will be implemented in subsequent, smaller pull requests
- Issue documentation: Larger improvements will be documented as GitHub issues with detailed specifications for future implementation
- Parallel development: Different team members can work on complementary improvements without blocking the main integration path
Risk Management Considerations
For a 10,000+ line pull request, the team recognized the need to balance ambition with practicality:
- Review complexity: Attempting to optimize everything simultaneously would make the PR virtually unreviewable
- Integration risk: Large-scale architectural changes increase the risk of introducing subtle bugs
- Development velocity: Focusing on correctness first allows faster integration and parallel optimization work
- Testing strategy: Establishing correct behavior enables more effective testing of subsequent optimizations
Bing inquired about opportunities to address existing technical debt from the TypeScript Lodestar implementation during the Zig rewrite. This question prompted discussion about the constraints imposed by consensus specification requirements.
Technical Debt Limitations
- Specification constraints: The state transition function is heavily constrained by the Ethereum consensus specification, limiting opportunities for major architectural changes
- Spec test validation: Any deviations from expected behavior would be caught by specification tests, preventing fundamental algorithm modifications
- Straightforward implementation: The state transition logic is relatively straightforward, with limited scope for architectural debt reduction
Improvement Opportunities Within Constraints
Despite specification constraints, some improvement opportunities were identified:
- Data structure optimization: Internal data structures can be optimized for performance without changing external behavior
- Documentation improvements: Better internal documentation and code organization can reduce maintenance burden
- Memory layout optimization: How data is stored and accessed internally can be improved while maintaining spec compliance
- Error handling enhancement: Better error reporting and debugging capabilities within specification requirements
Tuyen Nguyen and Cayman agreed that major debt reduction opportunities are limited, with potential improvements focusing on implementation quality rather than fundamental algorithmic changes.
The team established a comprehensive approach to reviewing the large pull request:
Review Structure
- File-by-file examination: Systematic review of each changed file to ensure comprehensive coverage
- Correctness prioritization: Focus on identifying functional issues that could cause spec test failures
- Issue categorization: Distinguishing between immediate correctness issues and longer-term architectural improvements
- Documentation standards: Recording findings in a structured format for tracking and resolution
Issue Management Strategy
- Immediate fixes: Small correctness issues would be addressed within the current PR
- Future improvements: Larger restructuring ideas would be documented as separate GitHub issues
- Branch management: Critical spec test fixes might be implemented in separate branches to avoid disrupting the main review process
NC provided valuable insights into practical challenges encountered during the TypeScript to Zig migration:
Common Migration Issues
- Integer overflow/underflow: Zig's stricter integer handling revealed edge cases not apparent in TypeScript
- Memory management errors: Double-free problems and shallow copy issues required careful attention to Zig's ownership model
- Type system differences: Adapting to Zig's compile-time type system from TypeScript's runtime typing
- Spec test stability: Ensuring the Zig implementation passes the same specification tests as the TypeScript version
Resolution Strategies
- Incremental testing: Running spec tests continuously during migration to catch issues early
- Memory debugging: Utilizing Zig's built-in memory safety features to identify ownership and lifecycle issues
- Type annotation: Explicit type annotations to help with Zig's type inference and error messages
- Spec compliance validation: Regular comparison with reference implementations to ensure correctness
The team agreed on a disciplined approach to managing changes during the review process:
Change Freeze Guidelines
- Critical fixes only: Only changes essential for making spec tests run would be permitted
- Separate branch option: Non-critical changes might be implemented in separate branches to avoid disrupting the review
- Review focus: Maintaining reviewer attention on correctness rather than continuous integration of new changes
- Documentation emphasis: Recording improvement opportunities rather than implementing them immediately
Automated Style Enforcement
Phil mentioned plans for integrating automated code style enforcement:
- Gemini bot integration: Custom bot configuration with team-specific style guide
- Style guide foundation: Starting with Tigerbeetle's style guide as a baseline with team-specific adjustments
- Automation benefits: Reducing manual style review burden to focus on functional correctness
- Consistency enforcement: Ensuring consistent code style across the growing Zig codebase
Style Guide Considerations
- Naming conventions: Decisions about camel case versus snake case for different code elements. Currently sticking to camelCase for functions and variable naming. snake_case for filenames
- Formatting standards: Consistent indentation, spacing, and line length requirements
- Documentation standards: Requirements for function and module documentation
- Error handling patterns: Consistent approaches to error propagation and handling
The discussion briefly addressed the BLST (BLS signature) library integration strategy:
Implementation Progress
- Test functionality: Bing is working on making tests function properly with the BLST integration
- Bun binding familiarity: Learning the bun binding system and TypeScript integration patterns
- Review preparation: Preparing the implementation for team review and feedback
Integration Architecture
- Dependency management: The current Lodestar bun repository does not yet include state transition or BLST as dependencies
- Incremental integration: BLST integration will be phased in after the state transition implementation is stable
- Testing strategy: Comprehensive testing of cryptographic operations before production deployment
The team established clear next steps for the following two-week period:
Review and Integration Tasks
- PR review completion: Team members will complete the systematic file-by-file review of the large pull request
- Issue documentation: Recording improvement opportunities and architectural enhancements as GitHub issues
- Correctness verification: Ensuring the Zig state transition function passes all relevant specification tests
- Integration preparation: Preparing for integration with existing Lodestar components
Process Improvements
- Review methodology: Establishing standardized approaches to reviewing large Zig pull requests
- Issue tracking: Implementing systematic tracking of architectural improvements and technical debt
- Testing integration: Developing processes for continuous specification test validation
- Code quality automation: Deploying automated style checking and quality assurance tools
Long-term Development Strategy
- Incremental optimization: Planning for systematic implementation of identified performance improvements
- Testing expansion: Developing comprehensive unit and integration testing beyond specification tests
- Documentation enhancement: Improving code documentation and architectural decision recording
- Team knowledge sharing: Establishing patterns for knowledge transfer and collaborative development
Recording/Transcript: https://drive.google.com/drive/folders/1tvfb4o3VDl0zwH1cPEXhBd2yaGDnqx_U?usp=drive_link
The third Lodestar Zig roadmap discussion on September 4, 2025, focused on architectural and coding standards decisions for the ongoing transition from JavaScript/TypeScript to Zig implementation of the Ethereum consensus client. The discussion centered on adopting Tiger Style programming principles, managing technical debt, and establishing development workflows for core libraries.
The team extensively discussed adopting Tiger Style, a programming philosophy inspired by TigerBeetle that emphasizes three core principles:
- Safety: Writing code that works in all situations with predictable control flow and bounded system resources
- Performance: Using resources efficiently through early design considerations and data-oriented approaches
- Developer Experience: Creating maintainable, readable code through clear naming, logical organization, and consistent practices
Key Tiger Style principles discussed include:
- Simple and explicit control flow to avoid complexity and unpredictable execution
- Fixed limits on all operations including loops, queues, and data structures to prevent infinite loops and resource exhaustion
- Extensive use of assertions for pre-conditions, post-conditions, and invariants
- Static memory allocation to avoid unpredictable runtime behavior
- Zero technical debt policy to maintain long-term productivity
Bing introduced concepts from data-oriented programming (DOP), which focuses on data transformations rather than object-oriented design. The discussion highlighted:
- Data plane vs. control plane separation: Lower-level performance-oriented operations on data with higher-level control logic feeding data to operators
- Elimination of object-oriented patterns: Moving away from getter/setter methods and encapsulation toward explicit data transformation functions
- Performance benefits: Better CPU cache utilization and clearer understanding of actual problem domains through data-focused design
Cayman noted that data-oriented design programming provided "the first glimpse of something that could take me to the next level" as a programmer, indicating its potential for advancing the team's technical capabilities.
Bing reported that the state transition Z implementation is nearly complete:
- Code compiles successfully with unit tests for individual process block components
- Ready for review and integration into Tuyen's branch within 3-4 days to one week
- Focus on merging working code rather than perfect code to unblock parallel development
NC is implementing the spec test framework, starting with operations spec tests as the smallest unit rather than full state transition tests. The approach leverages existing code patterns from the SSC Z repository for directory traversal, file parsing, and snappy-encoded SSZ payload handling.
Tuyen Nguyen identified tree view (Merkle tree implementation) as requiring additional testing and bug fixes before production use. Cayman emphasized the need for correctness in error cases to support advanced features like saving Merkle node pools to disk for faster startup.
The team discussed the BLST-Z library integration approach[4]:
- BLST-Z exists as a Zig wrapper for the supranational/blst library
- Bing had previously opened a PR to add bindings directly to BLST-Z
- Missing comprehensive BLS spec test coverage
Cayman advocated for centralizing all bindings in the Lodestar bun repository rather than embedding them in individual libraries:
- Keep BLST-Z as a pure Zig library without binding-specific code
- Build bindings layer separately in the Lodestar bun repository
- Focus on an "Ethereum BLST-Z library" rather than supporting all BLS variants (e.g., min_sig may not be necessary)
Bing committed to reviewing and advancing the BLST-Z library after completing the state transition work.
The team discussed their approach to benchmarking binding layer performance:
- Microbenchmarks: Comparing individual operations between JavaScript and Zig implementations via bun bindings
- Real-world testing: Using ChainSafe benchmark library to measure actual performance differences
- Avoiding synthetic data: Only using real data from sources, never simulating representative data
Current methodology involves:
- Building Zig dynamic library
- JavaScript code consuming and re-exporting functions
- Benchmark library measuring both new bun-based and existing JavaScript implementations
- Comparing operations like hash tree root calculation and SSZ deserialization
Cayman reported working on treebacked state bindings to expose getters for beacon state data through the binding layer. This represents a natural boundary where the JavaScript application can access Zig-managed state data without excessive cross-boundary calls.
The team identified that the binding layer architecture will evolve once the state transition function is complete, allowing for more comprehensive integration testing.
The team agreed on several process improvements:
- Break large PRs into smaller components to avoid year-long integration branches
- Create design document outlining coding standards and architectural patterns
- Establish contribution guidelines that incorporate Tiger Style principles
- Increase code review coverage to develop shared values and culture
Phil suggested persisting these guidelines in the Lodestar monorepo as part of contribution documentation.
To avoid blocking dependencies, the team outlined parallel work streams:
- Bing: Complete state transition Z implementation and BLSTZ review
- Navie: Develop spec test framework for operations
- Tuyen: Enhance tree view library testing and reliability
- Cayman: Continue bun binding development for treebacked state
Phil committed to maintaining detailed notes from roadmap discussions, building on previous meeting documentation available in the GitHub wiki.
The team embraced Tiger Style's zero technical debt policy:
- "Do it right the first time": Take time for proper design and implementation rather than rushing features
- Proactive problem-solving: Anticipate issues and fix them before escalation
- Build momentum: Deliver solid, reliable code that builds confidence and enables faster future development
Cayman emphasized that current Lodestar challenges with the block input refactor and sync strategy stem from either initial design issues or accumulated technical debt from "bolting things on" over time.
The discussion established expectations for high code quality:
- Move slowly initially, then quickly: Invest in proper setup and design to enable rapid future development
- Code should stand on its own: Zig libraries should be excellent independent of their integration context
- Extensive testing: Prioritize correctness through comprehensive unit and integration testing
- Pride in craftsmanship: Write code that developers can be proud of as high-quality software
- State transition Z completion: Bing to finalize implementation for review and merge
- Spec test development: Navie to get first operational tests working
- Tree view testing: Tuyen to add comprehensive test coverage
- Design document creation: Team to collaborate on coding standards documentation
- BLSTZ advancement: Bing to review and enhance BLS library implementation
- Binding integration: Cayman to expand treebacked state accessibility
- Workflow establishment: Implement agreed-upon code review and documentation practices
- Performance benchmarking: Expand measurement of binding layer overhead
The team scheduled the next roadmap discussion for September 18, 2025, maintaining bi-weekly intervals to track progress and address emerging architectural decisions.
Recording/Transcript: https://drive.google.com/drive/folders/1EpLxzKetQuMr8klth1z5ZAoZmtnUacGu?usp=sharing
- Cayman reported significant progress on the napi-z library, which is being developed to provide Zig-based native bindings similar to their existing bun-ffi-z library.
- Napi-rs vs C++ napi is that you build your native binding when you do your npm install versus napi-rs which publishes a separate package with a pre-built binary for each target.
- The key advancement involves implementing platform-specific binary publishing capabilities, mirroring the approach used by napi-rs where pre-built binaries are published for each target platform rather than compiling native bindings during npm install.
- A pull request has been opened against the hashtree-z library for testing, though it currently fails on ARM64 Mac builds due to lack of access to that platform.
- The library maintains a principled foundation by mapping the C library with a more user-friendly Zig interface, including wrappers for function creation and type conversion between NAPI values and Zig types.
- Nazar revealed he had been independently working on a similar native binding project before discovering the existing napi-z effort.
- Additionally, he developed CPU features bindings that support Bun, Deno, and Node.js, replacing their previous single-runtime dependency on Google's C implementation.
- The team discussed potentially moving this CPU features binding to the ChainSafe organization, though they noted that the underlying need may be eliminated since hashtree now includes fallback implementations, reducing dependency on CPU feature detection.
- Current State: Using Google’s C implementation of CPU features via native bindings
- Intermediate Step: Multi-runtime bindings while maintaining C dependency
- Future Vision: Pure Zig implementation leveraging Zig’s built-in CPU feature detection (
builtin.cpu)
- Tuyen Nguyen presented concerning performance findings regarding native beacon state property access, describing it as "really really expensive".
- This challenges their initial assumption of seamless interaction between runtime and Zig, suggesting they need to minimize beacon state pinging and instead implement separate caches in the beacon chain, similar to Lighthouse's architecture.
- Lodestar’s Current Approach: Heavy reliance on beacon state for data access and storage
- Lighthouse’s Approach: Minimal beacon state usage with separate caches in the beacon chain
- This comparison suggests that Lodestar’s current architecture may be inherently inefficient for their Zig integration goals.
- Cayman provided perspective on when binding overhead becomes problematic, noting that it typically only matters for operations occurring thousands of times per second, such as gossip validation with attestations.
- For less frequent operations like API access, the binding overhead should not be significant.
- Is bun-ffi really a true bottleneck?
- Tuyen’s hashtree benchmark represented a “worst case possible” scenario because individual hash operations are close to no-ops, meaning the benchmark primarily measured binding overhead rather than realistic workload performance.
- Quantitative Thresholds: Cayman established specific performance criteria, noting that binding overhead only becomes problematic at “thousands of times per second” with “several hundred nanoseconds” overhead per operation.
- Cayman identified specific high-risk scenarios where binding overhead could become problematic:
- High-Risk: Gossip validation with attestations processing thousands per second Low-Risk: API access operations that occur infrequently
- This differentiation suggests a selective optimization strategy rather than wholesale architectural changes.
- Tuyen outlined a concrete optimization approach for attestation validation:
- Direct beacon chain access for beacon committee data
- Configuration and indexing lookups without beacon state traversal
- Leveraging existing shuffling cache in the beacon chain
- This represents a surgical approach to reducing binding overhead by minimizing cross-boundary calls.
-
Hop Reduction Strategy
- Reducing “three hops or four hops” to “one hop”, indicating they can achieve significant performance improvements through call path optimization without fundamental architectural changes.
-
Memory Management and Lifecycle Challenges
- Cayman identified a critical architectural dependency in their current approach:
- Current System: All cache items attached to beacon state or epoch cache objects for automatic lifetime management
- Advantage: Simplified memory cleanup through reference counting tied to parent objects
- Mechanism: Cache lifetimes bound together through parent object disposal
- The proposed shift to separate beacon chain caches introduces significant complexity:
- New Challenge: Individual cache lifetime management without parent object coordination
- Risk: More complex memory management patterns
- Benefit: Improved performance and reduced memory footprint
- Cayman identified a critical architectural dependency in their current approach:
-
Storage Capacity Considerations
- Tuyen referenced discussions with Lion about fundamental capacity constraints:
- Problem: Beacon state objects are “too big” limiting stored object quantity
- Current Constraint: Limited historical data storage due to beacon state size
- Alternative Approach: Beacon chain cache layouts enabling “a lot of shuffling in the past” storage
- The beacon chain cache approach offers superior historical data retention:
- Higher storage capacity for historical shuffling data
- Better scalability for long-term cache requirements
- Reduced memory pressure from oversized beacon state objects
- Tuyen referenced discussions with Lion about fundamental capacity constraints:
- Cayman revealed that much of the required infrastructure already exists:
- Available Caches:
- Pubkey index and index pubkey caches
- Shuffling cache
- Various other specialized caches
- Missing Components:
- Decision routes in fork choice
- Proposer cache
- Available Caches:
-
Timeline and Implementation Concerns
- Tuyen emphasized the urgency of architectural decisions, noting that previous small changes like shuffling took “one month”.
- This timeline constraint suggests that early architectural decisions will significantly impact development velocity.
- Risk Assessment - The conversation reveals two competing risks:
- Performance Risk: Continuing with current beacon state-heavy approach
- Complexity Risk: Implementing individual cache lifetime management
- Optimization Strategy - Rather than wholesale architectural changes, the discussion points toward selective optimization:
- Target high-frequency operations like gossip validation
- Maintain current architecture for low-frequency operations
- Leverage existing cache infrastructure more efficiently
- Matthew Keil raises the fundamental question:
- If the caches are moved off beacon state, should they be centrally stored in a global singleton object passed everywhere (a “global context,” like the chain object)?
- He questions if that is any different from passing a pointer to a cache-enabled beacon state—since in either case, one ends up with a “large object with lots of pointers,” but all are passed by reference, so memory footprint is minimal and passing cost is negligible.
- Tuyen Nguyen and Cayman Nava elaborate:
- Keeping everything inside beacon state offers maintainability and convenient access, which can make maintainers default to storing too much there, possibly at the expense of architectural soundness.
- If everything is passed as part of a global context, it’s explicit but can become unwieldy if functions only use a small subset of what’s inside.
- Cayman notes at the JS layer, such a pattern (object with pointers/globals) is almost unavoidable, but at the native/Zig layer, there’s a choice: only pass explicit references needed.
- Matthew (drawing on C language experience) and Cayman agree:
- The cost in Zig (or C) of passing large structs by reference is minimal—just a pointer copy, not moving the data.
- In traditional C/low-level design, it’s normal to keep everything in a singleton context struct and pass a reference.
- Nazar objects strongly:
- Putting “everything” in a single global context is a security and maintainability liability: functions that shouldn’t have access to some internals do have access, leading to potential bugs and side effects.
- Instead, each function should only be given what it needs: explicit references as parameters, not access to the whole global state.
- Ekaterina Riazantseva notes that languages (like Rust) often clone objects for safety—to avoid pointers that allow unwanted mutations.
- Cloning can be expensive if objects are large.
- But passing by
const ref(Zig’s equivalent of read-only reference) can provide safety guarantees with low overhead. - Cayman agrees: functions can use
const refand need not rely on mutation or access to global state.
- Cayman explains that currently for large caches (e.g., pub key index cache), the codebase already acts like these are singletons/globals—variables at the top level, referenced everywhere, especially exposed to JS via bindings.
- Matthew says: this is mirrored on the JS side too—JS code refers to the same handle pointing at the same native singleton, so the pattern persists across boundaries.
- Nazar remains opposed to using native globals, advocating for pure functions relying only on explicit references, with no direct or implicit access to globals (citing historical challenges with side effects in impure/implicit code).
- The group discusses what happens if the context/singleton also needs to store a memory allocator (common for caches needing dynamic memory):
- Nazar questions if it’s necessary to include an allocator (behavior) when sometimes a pointer to the data alone should be sufficient.
- Cayman/Bing: If a cache or function needs an allocator for updates, and global state is discouraged, the allocator reference must also be passed explicitly.
- Nazar questions if it’s necessary to include an allocator (behavior) when sometimes a pointer to the data alone should be sufficient.
Resolution and Next Steps: Focus on Concrete Design
- Cayman proposes the group move from broad theoretical discussion to specifics:
- Create a design document or skeleton outlining the exact interfaces/patterns to be supported (e.g., state transition + caches in v1).
- Decide, for each case, if a global pattern, explicit parameter passing, or hybrid is best—then make these choices explicit in written designs.
| Pattern | Pros/Arguments | Cons/Arguments |
|---|---|---|
| Singleton/global context | Simplifies access, like C style, easy for JS | Security, hidden dependencies, possible side effects |
| Explicit reference-passing | Pure functions, explicit deps, safer | Boilerplate, possible verbosity |
| Mixing allocators in context | Direct allocation management | Blurs data vs. behavior, explicitness questioned |
- Cayman Nava emphasized the need for a concrete design document outlining the binding interface for their first iteration, focusing on state transition plus caches as the initial implementation target. He suggested studying well-architected Zig codebases like Tigerbeetle and Ghostty for architectural guidance, contrasting them with less organized projects like Bun.
This conversation between Matthew Keil and Cayman Nava centers on lessons from Lighthouse's architecture and how to design Lodestar's Zig and JavaScript bindings for both current interoperability and future pure-Zig clients.
Looking to Lighthouse for Inspiration
- Both agree Lighthouse's structure—with well-organized classes/structs and clear module boundaries—offers relevant architectural patterns, even if Teku and others differ.
- However, Lighthouse operates without the cross-language binding layer they need for Lodestar/JS, so not all patterns translate directly.
Separation of Concerns: Native vs. JS Layer
- Matthew emphasizes that how things are structured at the native/Zig layer must be settled before considering the JS interface.
- Cayman agrees and suggests that lower layers (state transition, caches, etc.) should only care about internal logic and not about how the top-level orchestration happens, whether in JS-land or in a future Zig-only client.
Composability and Abstraction
- Cayman likens function organization to "Lego blocks," advocating for maximum composability and minimal coupling: the way you assemble and glue blocks together at the top should not dictate details at the bottom.
- He notes that the bindings layer (to JS) is a layer on top of reusable Zig systems. It's vital to avoid locking in design meant only for JS interop into lower-level state transition logic.
Forward Compatibility and Avoiding Premature Constraints
- Matthew is thinking ahead to a "full Zig client" (native, beacon chain, caches), wondering if current design will serve them there.
- Cayman responds that for pure Zig, the design space is much broader. The current need is to get the boundaries and bindings layer right for JS while ensuring the Zig code for things like state transition, Merkle trees, and caches does not become polluted with binding-layer concerns.
Reference Counting and Opaque Handles
- They discuss reference counting and life-cycle management:
- Cayman: Ref counting is only needed in some low-level, resource-managed Zig structures (like a Merkle node pool); most caches, beacon blocks, etc. don’t require it.
- Once a JS binding is involved, opaque tokens/handles are used: the JS layer holds a handle (pointer) to a native structure, and the VM/bindings manage reference lifetime—not the Zig code itself.
- Matthew argues that as long as the cache is properly locked (mutex) on updates and read access is synchronized, the same Zig cache code can be wrapped with JS handles or used natively in a pure Zig client.
Technical Consensus
- The recommended approach is to write core logic in Zig without binding-specific (JS) constraints or ref-counting unless necessary for native resource management.
- Expose these as opaque pointers/tokens with accessor methods to JS; let the JS VM manage lifetime/handles.
- With this separation, the same core code will be reusable for a future all-Zig beacon chain client.
Strong modularity and composability (Lego block analogy) at the underlying logic level. Binding-specific logic (handles, reference management) must be kept shallow, at the interop boundary, not in core state or cache logic. Future-proofing: Do not pollute Zig code with constraints required only by current JS interoperability.
Three Zig Object Access Patterns Identified (Matthew)
- Singletons: Native objects that exist only once (e.g., large caches), usually referenced globally.
- Non-singleton, Non-refcounted Objects: Regular native objects that, while not globally unique, do not need reference counting.
- Refcounted/Pool Objects: Special cases, e.g., the hash pool, where explicit tracking and cleanup (ref counting, circular buffers) are necessary to manage resources safely.
Separation of Zig Module vs. Binding Layer (Cayman, supported by Matthew)
- Core Zig modules should be written for "Zig consumption"—concerned with correctness, clarity, and idiomatic practices.
- The binding layer is a separate concern: it adapts those modules for JS or C, possibly translating or transforming data as needed.
- This distinction is crucial for maintainability and for applying best practices from the Zig ecosystem.
Data Sharing Restrictions and API Layer
- Nazar points out that, except for a handful of types the ABI doesn't support (e.g., arrays of null-terminated strings), the internal shape of Zig structs doesn’t constrain what is shared with C/JS.
- The binding layer must translate unsupported types into acceptable forms (e.g., array list in Zig becomes a single comma-separated string for C/JS).
Best Practices: Zig Data Structures and Dev Experience
- Zig’s native data structures (like
ArrayList) are great for internal use, and efficient. - When sharing with C/JS, choose the simplest format that meets functional and performance needs.
- Focus first on writing unit-tested, idiomatic Zig code.
Binding Layer Design Can Proceed In Parallel
- Core logic (e.g., state transition, getter/setter logic) can be developed or ported to Zig directly from the TypeScript reference.
- Meanwhile, the binding layer (function signatures, wrappers, what gets exposed, and how handles are managed) can be designed and iterated independently.
- This parallelization allows for rapid prototyping and refinement.
Stick to Spec, Avoid Premature Optimization or Divergence
- Bing and others caution that major architectural changes should be aligned with the official spec (or, if inspired by Lighthouse or others, done with care).
- Naive rewrites may already be sufficient, and performance/architecture tweaks should be considered incrementally.
- Cayman starts by clarifying that while the binding architecture shouldn’t differ dramatically between NAPI (Node.js n-api) and bun-ffi (Bun’s FFI), for consistency and coordination, the team should choose one as the primary target for their initial binding implementation.
- Nazar describes his recent success with a pattern: first build the Zig module using standard library idioms, then create a Bun FFI binding as an intermediate C-ABI-friendly layer, and finally layer the Node.js binding over that.
- This approach simplifies the interface, as C ABI types are easier to target and adapt.
- Nazar strongly recommends starting with bun-ffi, targeting a C-ABI, instead of beginning with NAPI and then later trying to backport to bun-ffi.
- The general agreement is:
- Build the Zig library in a modular, idiomatic way.
- Add a C ABI FFI layer for Bun.
- Use that as a basis for any needed Node.js bindings via NAPI later.
- Cayman notes that bun-ffi actually supports passing n-api value types (like n-api values) but points out that if you want to do things like async, deal with promises, callbacks, or non-FFI-supported types, this is possible, but it’s “not thread safe” and more complex than standard FFI.
- Decision: Settle on bun ffi as the initial target, with room to adapt to nappy later if needed.
- Cayman calls out the risk: they can’t actually run the full Lodestar codebase on Bun yet, so some test coverage will have to happen in isolation (e.g., running ETH spec tests on state transitions in Zig via bun ffi, but not the whole node).
Recording/Transcript: https://drive.google.com/drive/folders/1tXIxf_iOMGNLaIv1DRnUoWhLj3PERhDq?usp=sharing
- Good fit for “vanilla” Zig libs (e.g. SSZ).
- Poor fit for target-specific code (hash-tree-root, BLST, Blast, Snappy bindings)
- Generates fallback build.zig/.zon, so easy to abandon if it becomes a problem
- Adopt zbuild for new or simple libs; avoid for native/asm or multi-arch code.
- Requires further issue discussion: https://github.com/ChainSafe/zbuild/issues/1
- Prioritise making Beacon
state-transitionnative; - Minimal JS ↔︎ Zig interface is a single process_block(state*, block_bytes) call
- Requires native Merkle-tree BeaconState + all epoch caches (proposer, committee, shuffling, etc.).
- Other consumers (gossip-validation, fork-choice, regen) must read these caches via bindings.
- Continue building native state-transition and cache structures; expose only required getters/setters first.
- Tuyen will finish cache implementation & add Zig spec-tests — est. 1 month for unit tests + 1 month for full spec-tests (optimistic timeline).
- Bun-FFI easier & familiar; risk of tying client to Bun runtime.
- N-API heavier but Node-stable; unknown performance until benchmarked.
- Large surface (≥5 k SSZ types) makes hand-written bindings infeasible → code-gen likely.
- Defer decision but must choose within ~2 months (by mid-Sep) or November-integration slips.
- Explore tiny PoC library in N-API to gauge DX & perf (no one assigned yet).
- Parallel task: Try to make Lodestar run under Bun today to unblock Bun-FFI testing.
- Whack-a-mole vs Architectural Redesign
- Hot-spot driven (“moles”) is practical, but may miss systemic wins (data-flow, thread layout).
- After native state-transition lands we must re-profile; new bottlenecks unknown.
- Use perf metrics to guide; re-evaluate design once native caches are in.
- Add fine-grained metrics inside Zig state-transition for future profiling (no one assigned yet).
- Next unlocks: Fork-choice, Gossip validation, Block production.
- Rough vision diagram shows JS network & API workers around fully-native chain core (see Vision document):
- Thought experiment, not committed to yet. Focus resources on initial integration.
- Bun integration running in Lodestar by November on-site meetup.
- Bi-weekly roadmap syncs will continue.