v1.16.1

CLI

• Fixed substreams build command when there is WASM file present yet.

v1.16.0

CLI

• Improved Improved substreams build, substreams protogen and substreams pack command outputs to be streamlined and condensed.

• Added support for reading manifest from stdin across all manifest-accepting commands using "-" as the manifest path. Affected commands: build, run, gui, info, graph, pack, protogen. This enables dynamic manifest generation and preprocessing workflows, including integration with tools like envsubst for environment variable substitution and CI/CD pipeline automation.

v1.15.10

• Re-release of v1.15.9 with missing dependency update.

v1.15.9

Server

• BREAKING CHANGE substreams-tier2 servers must be upgraded before tier1 servers, tier2 servers will stream outputs for the 'first segment', to speed up time to first block.
• Return processed_blocks counter to client at the end of the request.
• Progress notifications will only be sent every 500ms for the first minute, then reduce rate up to every 5 seconds (can be overridden per request).
• Added dev_output_modules to protobuf request (if present, in dev mode, only send the output of the modules listed).
• Added progress_messages_interval_ms to protobuf request (if present, overrides the rate of progress messages to that many milliseconds).

CLI

• Updated to latest networks registry version.
• Added --proto-path flag to substreams run and substreams gui commands: Allows loading protobuf definitions from a directory containing .proto files on top of the substreams package protobuf definitions
• Added --proto-descriptor-set flag to substreams run and substreams gui commands: Allows loading protobuf definitions from a single protobuf descriptor set file on top of the substreams package protobuf definitions
• Both flags work with both manifest files (.yaml) and pre-compiled packages (.spkg), enabling additional protobuf types to be available during execution
• Added substreams unpack command to extract the contents of a .spkg file to a tweakable YAML manifest.
• Added validation of protobuf outputs when doing 'pack' and 'publish' (they must have protobuf definitions attached to the manifest)
• Set dev_output_modules to only show the output_module when using substreams run, and all non-imported modules when using substreams gui
• Print the processed blocks counter to client at the end of the request

v1.15.8

CLI

• substreams run now prints "Total Egress Bytes" as well as "Total Processed Bytes"

Server

Rework the execout File read/write:

• This reduces the RAM usage necessary to read and stream data to the user on tier1,
  as well as to read the existing execouts on tier2 jobs (in multi-stage scenario)

• The cached execouts need to be rewritten to take advantage of this, since their data is currently not ordered:
  the system will automatically load and rewrite existing execout when they are used.

• Code changes include:

  • new FileReader / FileWriter that "read as you go" or "write as you go"
  • No more 'KV' map attached to the File
  • Split the IndexWriter away from its dependencies on execoutMappers.
  • Clock distributor now also reads "as you go", using a small "one-block-cache"

• Removed env var and behaviors:

  • removed SUBSTREAMS_DISABLE_PRELOAD_EXEC_FILES (no more preloading, it was mostly useful because reading full file+unmarshal was necessary when streaming...)
  • removed SUBSTREAMS_OUTPUT_SIZE_LIMIT_PER_SEGMENT (this is not a RAM issue anymore)

• Add uncompressed_egress_bytes field to substreams request stats log message. Only tier1 will produce a non-zero value there.

v1.15.7

Server

• Tier2 jobs now write mapper outputs "as they progress", preventing memory usage spikes when saving them to disk.
  This should considerably reduce the memory footprint of tier2 instances.
• Tier2 jobs now limit writing and loading mapper output files to a maximum size of 8GiB by default.
• AddedSUBSTREAMS_OUTPUT_SIZE_LIMIT_PER_SEGMENT environment variable to control this new limit.
• Gate the DebugAPI feature on tier2 with the SUBSTREAMS_DEBUG_API_ADDR environment variable (set it to localhost:8081 to keep behavior from v1.15.5)

CLI

• Removed the 'codegen subgraph' command from the CLI as SpS are being deprecated.
• Added --skip-package-validation and --extension-configs flags to tools tier2call dev command

v1.15.6

CLI

• The substreams run will now better render bytes depending on the network.

• The substreams run/gui JSON rendered is now able to render known anypb.Any type correctly.

• Integrated the Network Registry to better track supported networks.

Server

• Add SUBSTREAMS_STORE_SIZE_LIMIT env var to allow overwriting the default 1GiB value

v1.15.5

Server

• Add env var SUBSTREAMS_PRINT_STACK to enable printing full stack traces when caught panic occurs
• Prevent a deterministic failure on a module definition (mode, valueType, updatePolicy) from persisting when the issue is fixed in the substreams.yaml #621
• Metering events on tier2 now bundled at the end of the job (prevents sending metering events for failing jobs)
• Added metering for: "processed_blocks" (block * number of stages where execution happened) and "egress_bytes"

CLI

• Add unichain to the list of supported chains.

v1.15.4

• dedupe modules with same hash when computing graph. (#619)
• prevent memory usage burst when writing mapper by streaming protobuf items to writer
• ignore "service currently overloaded" worker errors in the "maxRetries" count. Tier1 requests should not error out because tier2 servers are ramping up, only when they fail multiple times.
• Default SUBSTREAMS_WORKER_MAX_RETRIES now set to 5.

v1.15.3

Server

• Catch "store errors" as deterministic (ex: invalid operation, store too big...), writing them to the module cache as well as errors that happen directly in the WASM code.
• Ensure the 'error cache' is effective even when the "stop block" is unset (0)
• Fix 'SUBSTREAMS_WORKERS_RAMPUP_TIME' environment variable that was not being honored

CLI

• substreams init: fix project creation when using the --force-download-cwd flag.