v0.1.0-beta4

RELEASE DATE: 2020-06-23

We’re constantly working on improving the user experience with each release, so here’s what changed since our last update:

Public API Changes

On the Public API side of things, which is meant for developers querying the data on chain through the dfuse hosted services or through a local version of the dfuseeos product, we introduced 1 BREAKING change for 2 fields;

Changed

• BREAKING: For more clarity when using these fields, the eosws transaction lifecycle fields creation_irreversible & cancelation_irreversible were renamed respectively to dtrx_creation_irreversible and dtrx_cancelation_irreversible.

System Administration Changes

On the side of System Administration, which is meant for developers running a local version of the dfuseeos product, we introduced many changes, some of them BREAKING;

Added

• App: mindreader-stdin — This app relays blocks and produces one-block-files (or merged-blocks-files) based on stdin, without trying to manage nodeos. This is an alternative way of seeding your dfuse system if you have existing tooling for managing nodeos operations. It uses only a subset of the mindreader flags and does not stop on TERM signal until it receives an EOF signal from stdin.
• App: booter — We’re especially excited about this new app! This app allows a boot sequence to be applied to a new chain (see the README for more information). It also brings new flags with it:
  • --booter-bootseq — File path to the desired boot sequence (default: "./bootseq.yaml").
  • --booter-data-dir — Booter's working directory (default: "{dfuse-data-dir}/booter").
  • --booter-nodeos-api — Target API address to communicate with underlying nodeos (default: “http://localhost:8888/”).
  • --booter-private-key — Genesis private key having control over an eosio account to be able to create all related system accounts & contracts.
  • --booter-vault-file — Wallet file that contains encrypted key material.
• Command: dmesh — This command was added to the dfuseeos tools with 2 new flags: --dsn & --service-version. This command allows you to inspect dmesh search peers. It currently only supports etcd (the server in which all the information is stored, see this graphic).
• Command: kv — This command was added to the dfuseeos tools with 8 new subcommands: get, scan, prefix, account, blk, blkirr, trx, trxtrace. This command allows you to retrieve data from trxdb at a lower lever (e.g. only rows of data).
• Command: db — This command was added to the dfuseeos tools with 2 new subcommands: blk, trx. This command allows you to retrieve data from trxdb at a higher level (e.g. assembling multiple rows into a data structure).
• Command: check trxdb-blocks — This command was added to the dfuseeos tools and ensures linearity of irreversible blocks in storage, which is useful to know if you've missed some block ranges when doing parallel insertions into your trxdb storage. This relates to trxdb-loader's job being done right.
• Flag: --mindreader-fail-on-non-contiguous-block — This flag lets you enable the mindreader ContinuityChecker (default:false). See mindreader under the CHANGED section below.
• Flag: --log-level-switcher-listen-addr — This flag lets you change the port to which you send your HTTP request to switch the log levels of a given component in real-time (default: ":1065") (see LOGGING.md).
• Flag: --common-ratelimiter-plugin — This flag lets you enable a rate limiter plugin (default: "null://").
• Flag: --pprof-listen-addr — This flag enables you to change the port listening to pprof (default: ":6060").
• Flag: --search-common-dfuse-events-unrestricted — This flag lets you lift all restrictions on Indexing Limits for searching dfuse Events (max field count, max key length, max value length).
• Flag: --mindreader-discard-after-stop-num — This flag lets you discard all blocks after stop-num if it is set to true (default:false). See the mindreader bullet below.
• mindreader now writes remaining one-block files after stop-block if --mindreader-merge-and-store-directly is set, unless --mindreader-discard-after-stop-num is set to true. This improves the experience of a user following PARTIAL_SYNC.md steps, producing merged files up to a certain block, then switching to one-block files with a separate merge instance. This improves the parallel reprocessing experience.
• The auto-backups feature will now only trigger if os.Hostname() returns the non-empty value of one of these new flags:
  • --node-manager-auto-backup-hostname-match
  • --node-manager-auto-snapshot-hostname-match
  • --mindreader-auto-backup-hostname-match
  • --mindreader-auto-snapshot-hostname-match
• The trxdb “deduper” now reduces storage by removing identical action data and calls the "reduper" to add this data back. EOS Mainnet saves 20%(!) off of storage because of this.

Changed

• BREAKING: eosdb was renamed to trxdb (the storage location where blocks & transactions are saved). Verify and adjust your command arguments and 'dfuse.yaml' file for references to eosdb. Specifically, eosdb’s healthz endpoint field is now trxdb’s healthz endpoint field, so you might need to adjust your monitoring.
• BREAKING: The search flag --search-common-dfuse-hooks-action-name was renamed to --search-common-dfuse-events-action-name.
• BREAKING: The abicodec app default value for its --abicodec-cache-base-url and --abicodec-export-cache-url flags was changed to {dfuse-data-dir}/storage/abicache (fixing a typo in abicahe). To remain compatible, simply do a manual rename on disk before starting the updated version such as mv {dfuse-data-dir}/storage/abicahe {dfuse-data-dir}/storage/abicache.
• BREAKING: The fluxdb flag --fluxdb-enable-dev-mode was removed. Use --fluxdb-enable-live-pipeline=false to get the same behavior as before.
• mindreader ContinuityChecker is not enabled by default anymore.
• The value of node-manager and mindreader’s respective --shutdown-delay flags is now also applied to commands like snapshot or backup, so they become "not-ready" on /healthz endpoint, allowing a load-balancer to take it out of the pool before they actually stop working.
• The dfuseeos tools check blocks command was renamed to dfuseeos tools check merged-blocks.
• The search roarCache is now based on a normalized version of the query string (e.g. a:foo b:bar is now equivalent to b:bar a:foo, etc.). Note that this will make previously-cached entries useless.
• The --node-manager-number-of-snapshots-to-keep and --mindreader-number-of-snapshots-to-keep flags now default to 0, which means an unlimited number of snapshots will be kept, instead of their previous default of 5 which only kept the latest 5 backups.
• Various startup speed improvements for the blockmeta, bstream, and search-indexer apps.

Removed

• The --search-indexer-num-blocks-before-start flag was removed from search-indexer. Because of that, search-indexer now automatically resolves its start block.

Fixed

• trxdb now correctly implements "BatchGet" on most operations, giving a great(!) performance increase over previous versions.
• Global flags and specifically the --search-common-dmesh-dsn flag are now correctly parsed from the config file.
• search-indexer no longer overflows on negative startblocks on new chains, it fails fast instead.
• search-archive relative-start-block truncation now works.
• search-forkresolver no longer throws a nil pointer (app was previously broken).
• mindreader now has a more resilient shutdown handling (expects EOF on nodeos stdout).
• The mindreader and node-manager logs from nodeos that go through zap now have their level parsed correctly instead of all being seen as DEBUG.