Skip to content

v1.15.8
Compare
Choose a tag to compare
@sduchesneau sduchesneau released this 13 Jun 17:29
v1.15.8
dca2459

CLI

  • substreams run now prints "Total Egress Bytes" as well as "Total Processed Bytes"

Server

Rework the execout File read/write:

  • This reduces the RAM usage necessary to read and stream data to the user on tier1,
    as well as to read the existing execouts on tier2 jobs (in multi-stage scenario)

  • The cached execouts need to be rewritten to take advantage of this, since their data is currently not ordered:
    the system will automatically load and rewrite existing execout when they are used.

  • Code changes include:

    • new FileReader / FileWriter that "read as you go" or "write as you go"
    • No more 'KV' map attached to the File
    • Split the IndexWriter away from its dependencies on execoutMappers.
    • Clock distributor now also reads "as you go", using a small "one-block-cache"

  • Removed env var and behaviors:

    • removed SUBSTREAMS_DISABLE_PRELOAD_EXEC_FILES (no more preloading, it was mostly useful because reading full file+unmarshal was necessary when streaming...)
    • removed SUBSTREAMS_OUTPUT_SIZE_LIMIT_PER_SEGMENT (this is not a RAM issue anymore)

  • Add uncompressed_egress_bytes field to substreams request stats log message. Only tier1 will produce a non-zero value there.

Assets 7
Loading