feat: add Era File Reading and Writing

[guha-rahul]

Motivation

• tracks ERA file support #7048
• era specs.

Description

• Adds functionality to read/write to e2s files

import {open} from "node:fs/promises";
import {e2s} from "@lodestar/era";

const fh = await open("mainnet-xxxxxx-xxxxxxxx.era");
const entry = await e2s.readEntry(fh, 0);
entry.type == e2s.EntryType.Version

• Adds functionality to read/write era files

import {era} from "@lodestar/era";
import {config} from "@lodestar/config/default";
// open reader
const reader = await era.EraReader.open(config, "mainnet-xxxxx-xxxxxxxx.era");

// check number of groups
reader.groups.length === 1;

// read blocks
const slot = reader.groups[0].startSlot;

 // return snappy-frame compressed, ssz-serialized block at slot or null if a skip slot
 // throws if out of range
await reader.readCompressedBlock(slot);
// same, but for ssz-serialized block
await reader.readSerializedBlock(slot);
// same but for deserialized block
await reader.readBlock(slot);

// read state(s), one per group
// similar api to blocks, but with an _optional_ eraNumber param for specifying which group's state to read
await reader.readCompressedState(); 
await reader.readSerializedState();
await reader.readState();

// write era files
const writer = await era.EraWriter.create(config, "path/to/era");

// similar api to reader, can write compressed, serialized, or deserialized items
// first write all blocks for the era
await writer.writeBlock(block);
// ...
// then write the state
await writer.writeState(state);
// if applicable, continue writing eras of blocks and state (an era file can contain multiple eras, or "groups" as the spec states)
// when finished, must call `finish`, which will close the file handler and _rename_ the file to the spec-compliant name
await writer.finish();

• e2e test reads an era file, does all validation, writes an era fila, does validation on that freshly created file
  • requires the era file fixture to be downloaded (cd packages/era/test && ./download_era_file.sh)
  • e2e test is skipped (test:e2e is not defined for the era package)

[CLAassistant]

All committers have signed the CLA.

[nflaig]

why do we wanna add types separately? A more complete implementation makes more sense to me before reviewing

[guha-rahul]

@nflaig I had a chat with @wemeetagain where we agreed to proceed like that https://discord.com/channels/593655374469660673/1387128551962050751/1388555043921461300

[nflaig]

@nflaig I had a chat with @wemeetagain where we agreed to proceed like that https://discord.com/channels/593655374469660673/1387128551962050751/1388555043921461300

I agree with the part that we don't need to integrate it with the beacon node in the first iteration but just having the types without any actual code imo makes not much sense. I'd prefer a more complete implementation but leaving this up to @wemeetagain

[guha-rahul]

@nflaig while implementing, i found some small optimisations that deviated from the spec. So I may have to change the types later on. I think it would be better if I add the reading and writing of era files in this pr too so that we can consolidate on the major types we need.
what do you think @wemeetagain ?

[wemeetagain]

Yeah go for it. It will be easier to review the types if we can see how they're being used.

[guha-rahul]

I modified the pr to focus on era file reading and testing it with a script on .era file from https://mainnet.era.nimbus.team/

[codecov]

Codecov Report

❌ Patch coverage is 29.71014% with 97 lines in your changes missing coverage. Please review.
✅ Project coverage is 51.91%. Comparing base (6d7c41d) to head (6ac0887).
⚠️ Report is 1 commits behind head on unstable.

Additional details and impacted files

@@             Coverage Diff              @@
##           unstable    #8035      +/-   ##
============================================
- Coverage     51.95%   51.91%   -0.05%     
============================================
  Files           848      851       +3     
  Lines         65941    66096     +155     
  Branches       4814     4816       +2     
============================================
+ Hits          34258    34312      +54     
- Misses        31615    31715     +100     
- Partials         68       69       +1     

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[guha-rahul]

not quite sure why The CI/CD is showing test timed out in places where I have not touched , but I got this test time timed out issues on previous runs too. What is the root cause for those errors?

[nflaig]

@guha-rahul did you go through the gemini bot comments? feel free to resolve them if it's not a good suggestions / not applicable, you can also add a comment to it why you think it's wrong what the bot suggests

[guha-rahul]

hrough the gemini bot comments? feel free to resolve them if it's not a good suggestions / not applicable, you can also add a comment to it why you think it's wrong what the bot suggests

@nflaig Yes, all of them have been worked out(Some by cayman and some by me) except the one with console logs since for testing they are quite hellpful and I plan to remove them when its ready to be merged. Also should i remove the snappy code from utils package to packages/reqresp/utils?

[nflaig]

hrough the gemini bot comments? feel free to resolve them if it's not a good suggestions / not applicable, you can also add a comment to it why you think it's wrong what the bot suggests

@nflaig Yes, all of them have been worked out(Some by cayman and some by me) except the one with console logs since for testing they are quite hellpful and I plan to remove them when its ready to be merged. Also should i remove the snappy code from utils package to packages/reqresp/utils?

feel free to resolve them in that case if it's already addressed so we have a better overview of what's missing so we can merge this

[guha-rahul]

hrough the gemini bot comments? feel free to resolve them if it's not a good suggestions / not applicable, you can also add a comment to it why you think it's wrong what the bot suggests

@nflaig Yes, all of them have been worked out(Some by cayman and some by me) except the one with console logs since for testing they are quite hellpful and I plan to remove them when its ready to be merged. Also should i remove the snappy code from utils package to packages/reqresp/utils?

feel free to resolve them in that case if it's already addressed so we have a better overview of what's missing so we can merge this

done!

[nflaig]

docs checks are failing, see job run

@lodestar/era: ℹ Found 3 TypeScript snippets
@lodestar/era: ✖ Error compiling example code block 2 in file README.md:
@lodestar/era:   ⨯ Unable to compile TypeScript:
Error: @lodestar/era:       README.md → Code Block 2(11,31): error TS2339: Property 'startSlot' does not exist on type 'EraIndices'.
@lodestar/era: ✖ Error compiling example code block 3 in file README.md:
@lodestar/era: ✖ Compilation failed, see above errors

[nflaig]

LGTM