Technical discussion: Selecting a Key-Value Store Database Engine for btczee

[AbdelStark]

Issue: Selecting a Key-Value Store Database Engine for btczee

Background

btczee is a Bitcoin protocol implementation in Zig. We need to choose a key-value store database for storing critical data like block data, UTXOs, etc. This decision will have significant implications for performance, maintainability, and development complexity.

Objectives

• Select a high-performance key-value store that can handle Bitcoin node data efficiently
• Ensure the chosen solution is well-maintained and has long-term support
• Find a database that integrates easily with Zig, either as a pure Zig library or via simple bindings
• Keep the implementation lean and avoid premature abstraction

Criteria

1. Performance: Database access speed is critical and can be a serious bottleneck
2. Maintenance: We want a well-maintained solution with active development and community support
3. Ease of integration: Should be simple to integrate with Zig
4. Scalability: Must handle large datasets efficiently
5. Consistency: Ensure data integrity, especially important for a Bitcoin node
6. Resource usage: Efficient use of memory and disk space
7. Licensing: Compatible with btczee's licensing requirements

Candidates

1. RocksDB

   • Pros:
     • High performance
     • Well-maintained by Facebook
     • Existing Zig bindings (zigrocks)
   • Cons:
     • Complex codebase
     • Potential overhead from unused features

2. LMDB

   • Pros:
     • Extremely fast read performance
     • Simple codebase
     • Existing Zig bindings
   • Cons:
     • Write performance may be slower than alternatives
     • Less active development compared to RocksDB

3. LevelDB

   • Pros:
     • Developed by Google, basis for RocksDB
     • Simpler than RocksDB
     • Good performance
   • Cons:
     • Less actively developed than RocksDB
     • May lack some advanced features

Comparison Table

Feature	RocksDB	LMDB	LevelDB
Read Performance	High	Very High	High
Write Performance	Very High	Good	High
Maintenance	Active	Stable	Less Active
Zig Integration	Good	Good	Unknown
Scalability	Excellent	Good	Good
Consistency	Strong	Strong	Strong
Memory Usage	Moderate	Low	Moderate
Disk Usage	Efficient	Efficient	Efficient
License	Apache/GPL	OpenLDAPv2.8	BSD 3-Clause

Resources

• RocksDB: Syndica RocksDB Zig repo
• LMDB: Zig bindings

Questions to Consider

1. What are our specific performance requirements? Do we prioritize read or write operations?
2. How important is database size? Will we need to support pruning or compaction?
3. Are there any specific Bitcoin Core optimizations we should consider that might favor one database over another?
4. How much effort are we willing to put into database integration and potential optimization?
5. Are there any other key-value stores we should consider?

Next Steps

1. Conduct benchmarks specific to Bitcoin node operations using each database
2. Investigate the effort required to integrate each option with Zig
3. Consider reaching out to other Bitcoin implementation projects for their experiences
4. Evaluate the long-term maintenance prospects for each option

Call for Input

We welcome input from the community on:

• Experiences with these databases in similar projects
• Additional criteria we should consider
• Other database options we may have overlooked
• Potential challenges or benefits specific to Zig integration

Please provide any insights, data, or opinions to help inform this important decision for btczee.

[GrapeBaBa]

What about https://github.com/erthink/libmdbx which used by erigon and reth, the performance seems better. And it was developed by C, which should be used with Zig easily.

[AbdelStark]

That's indeed a good suggestion. Would you like to check the ease of integration in the project ?

[GrapeBaBa]

That's indeed a good suggestion. Would you like to check the ease of integration in the project ?

Cool, will try to investigate and feedback.