Merge pull request #76 from SurfingNerd/release_9_2

Release 0.9.2 pull request.

CHANGELOG.md

@@ -1,6 +1,10 @@
+## Diamond Node Software 3.3.5-hbbft-0.9.2
+
+- [FIXED: pruning as root cause for stage 3 errors] https://github.com/DMDcoin/diamond-node/issues/68
+
 ## Diamond Node Software 3.3.5-hbbft-0.9.1
 
-- [pruning protection for hbbft engine] https://github.com/DMDcoin/diamond-node/issues/6
+- [pruning protection for hbbft engine] https://github.com/DMDcoin/diamond-node/issues/62
 - [reported fault: UnknownSender] https://github.com/DMDcoin/diamond-node/issues/69
 
 ## Diamond Node Software 3.3.5-hbbft-0.9.0

Cargo.toml

@@ -2,7 +2,7 @@
 description = "Diamond Node"
 name = "diamond-node"
 # NOTE Make sure to update util/version/Cargo.toml as well
-version = "3.3.5-hbbft-0.9.1"
+version = "3.3.5-hbbft-0.9.2"
 license = "GPL-3.0"
 authors = [
 	"bit.diamonds developers",

README.md

@@ -1,19 +1,14 @@
-# OpenEthereum
+# Diamond Node
 
-Fast and feature-rich multi-network Ethereum client.
-
-[» Download the latest release «](https://github.com/openethereum/openethereum/releases/latest)
+Node client for the protocol version 4 of the bit.diamonds network.
 
 [![GPL licensed][license-badge]][license-url]
 [![Build Status][ci-badge]][ci-url]
 [![Discord chat][chat-badge]][chat-url]
 
 [license-badge]: https://img.shields.io/badge/license-GPL_v3-green.svg
 [license-url]: LICENSE
-[ci-badge]: https://github.com/openethereum/openethereum/workflows/Build%20and%20Test%20Suite/badge.svg
-[ci-url]: https://github.com/openethereum/openethereum/actions
-[chat-badge]: https://img.shields.io/discord/669192218728202270.svg?logo=discord
-[chat-url]: https://discord.io/openethereum
+[chat-url]: (https://dmdcoin.slack.com/)
 
 ## Table of Contents
 
@@ -32,29 +27,25 @@ Fast and feature-rich multi-network Ethereum client.
 
 ## 1. Description <a id="chapter-001"></a>
 
-**Built for mission-critical use**: Miners, service providers, and exchanges need fast synchronisation and maximum uptime. OpenEthereum provides the core infrastructure essential for speedy and reliable services.
-
-- Clean, modular codebase for easy customisation
-- Advanced CLI-based client
-- Minimal memory and storage footprint
-- Synchronise in hours, not days with Warp Sync
-- Modular for light integration into your service or product
+diamond-node is the node software for the upcomming V4 of the diamond network.
+The Node Software is on a alpha level and still under active development.
 
 ## 2. Technical Overview <a id="chapter-002"></a>
 
-OpenEthereum's goal is to be the fastest, lightest, and most secure Ethereum client. We are developing OpenEthereum using the **Rust programming language**. OpenEthereum is licensed under the GPLv3 and can be used for all your Ethereum needs.
+diamond-node builds on OpenEthereum, and shares a lot of base features, 
+covered in the [OpenEthereum Documentation](https://openethereum.github.io/).
 
-By default, OpenEthereum runs a JSON-RPC HTTP server on port `:8545` and a Web-Sockets server on port `:8546`. This is fully configurable and supports a number of APIs.
+By default, diamond-node runs a JSON-RPC HTTP server on port `:8545` and a Web-Sockets server on port `:8546`. This is fully configurable and supports a number of APIs.
 
-If you run into problems while using OpenEthereum, check out the [old wiki for documentation](https://openethereum.github.io/), feel free to [file an issue in this repository](https://github.com/openethereum/openethereum/issues/new), or hop on our [Discord](https://discord.io/openethereum) chat room to ask a question. We are glad to help!
+If you run into problems while using diamond-node, feel free to [file an issue in this repository](https://github.com/dmdcoind/diamond-node/issues/new), or hop on our [Slack](https://dmdcoin.slack.com/), [Telegram](https://t.me/DMDcoin) or [Discord](https://discord.gg/TStv6gm) chat room to ask a question. We are glad to help!
 
-You can download OpenEthereum's latest release at [the releases page](https://github.com/openethereum/openethereum/releases) or follow the instructions below to build from source. Read the [CHANGELOG.md](CHANGELOG.md) for a list of all changes between different versions.
+We do not provide binaries and suggest to build from source.
 
 ## 3. Building <a id="chapter-003"></a>
 
 ### 3.1 Build Dependencies <a id="chapter-0031"></a>
 
-OpenEthereum requires **latest stable Rust version** to build.
+diamond-node requires **latest stable Rust version** to build.
 
 We recommend installing Rust through [rustup](https://www.rustup.rs/). If you don't already have `rustup`, you can install it like this:
 
@@ -63,7 +54,7 @@ We recommend installing Rust through [rustup](https://www.rustup.rs/). If you do
   $ curl https://sh.rustup.rs -sSf | sh
   ```
 
-  OpenEthereum also requires `clang` (>= 9.0), `clang++`, `pkg-config`, `file`, `make`, and `cmake` packages to be installed.
+  diamond-node also requires `clang` (>= 9.0), `clang++`, `pkg-config`, `file`, `make`, and `cmake` packages to be installed.
 
 - OSX:
   ```bash
@@ -83,14 +74,14 @@ Once you have `rustup` installed, then you need to install:
 * [Perl](https://www.perl.org)
 * [Yasm](https://yasm.tortall.net)
 
-Make sure that these binaries are in your `PATH`. After that, you should be able to build OpenEthereum from source.
+Make sure that these binaries are in your `PATH`. After that, you should be able to build diamond-node from source.
 
 ### 3.2 Build from Source Code <a id="chapter-0032"></a>
 
 ```bash
 # download OpenEthereum code
-$ git clone https://github.com/openethereum/openethereum
-$ cd openethereum
+$ git clone https://github.com/DMDcoin/diamond-node
+$ cd diamond-node
 
 # build in release mode
 $ cargo build --release --features final
@@ -116,26 +107,26 @@ This always compiles the latest nightly builds. If you want to build stable, do
 $ git checkout stable
 ```
 
-### 3.3 Starting OpenEthereum <a id="chapter-0034"></a>
+### 3.3 Starting diamond-node <a id="chapter-0034"></a>
 
 #### Manually
 
-To start OpenEthereum manually, just run
+To start diamond-node manually, just run
 
 ```bash
 $ ./target/release/openethereum
 ```
 
-so OpenEthereum begins syncing the Ethereum blockchain.
+so diamond-node begins syncing the Ethereum blockchain.
 
 #### Using `systemd` service file
 
-To start OpenEthereum as a regular user using `systemd` init:
+To start diamond-node as a regular user using `systemd` init:
 
 1. Copy `./scripts/openethereum.service` to your
 `systemd` user directory (usually `~/.config/systemd/user`).
 2. Copy release to bin folder, write `sudo install ./target/release/openethereum /usr/bin/openethereum`
-3. To configure OpenEthereum, see [our wiki](https://openethereum.github.io/Configuring-OpenEthereum) for details.
+3. To configure diamond-node, see [our wiki](https://openethereum.github.io/Configuring-OpenEthereum) for details.
 
 ## 4. Testing <a id="chapter-004"></a>
 

bin/oe/cli/usage_header.txt

@@ -1,5 +1,5 @@
-OpenEthereum Client.
-    By Wood/Paronyan/Kotewicz/Drwięga/Volf/Greeff
+diamond-node bit.diamonds Node Software
+    By Haller/Forstenlechner/Wood/Paronyan/Kotewicz/Drwięga/Volf/Greeff
        Habermeier/Czaban/Gotchac/Redman/Nikolsky
        Schoedon/Tang/Adolfsson/Silva/Palm/Hirsz et al.
     Copyright 2015-2020 Parity Technologies (UK) Ltd.

bin/oe/metrics.rs

@@ -59,6 +59,8 @@ fn handle_request(
                 elapsed.as_millis() as i64,
             );
 
+            reg.register_version();
+
             let mut buffer = vec![];
             let encoder = prometheus::TextEncoder::new();
             let metric_families = reg.registry().gather();

crates/ethcore/src/client/client.rs

@@ -1313,7 +1313,7 @@ impl Client {
                 break;
             }
             match state_db.journal_db().earliest_era() {
-                Some(mut earliest_era) if earliest_era + self.history <= latest_era => {
+                Some(earliest_era) if earliest_era + self.history <= latest_era => {
                     let freeze_at = self.snapshotting_at.load(AtomicOrdering::SeqCst);
                     if freeze_at > 0 && freeze_at == earliest_era {
                         // Note: journal_db().mem_used() can be used for a more accurate memory
@@ -1326,9 +1326,10 @@ impl Client {
 
                     // if the engine still needs that block, we are not going to prune it.
                     if let Some(protected_block) = self.engine.pruning_protection_block_number() {
-                        if earliest_era < protected_block {
+                        if earliest_era > protected_block {
                             info!(target: "pruning", "Detected attempt from pruning ancient block that is still required by the engine. protected block: {protected_block}, earliest_era: {earliest_era}");
-                            earliest_era = protected_block - 1;
+                            //earliest_era = protected_block - 1;
+                            break;
                         }
                     }
                     trace!(target: "client", "Pruning state for ancient era {}", earliest_era);

crates/ethcore/src/engines/hbbft/hbbft_engine.rs

@@ -357,7 +357,6 @@ impl IoHandler<()> for TransitionHandler {
                 trace!(target: "consensus", "All Operation that had to be done after syncing have been done now.");
             }
         } else if timer == ENGINE_VALIDATOR_CANDIDATE_ACTIONS {
-            warn!(target: "consensus", "do_validator_engine_actions");
             if let Err(err) = self.engine.do_validator_engine_actions() {
                 error!(target: "consensus", "do_validator_engine_actions failed: {:?}", err);
             }
@@ -1106,20 +1105,6 @@ impl HoneyBadgerBFT {
         }
     }
 
-    /// hbbft protects the start of the current posdao epoch start from being pruned.
-    pub fn pruning_protection_block_number(&self) -> Option<u64> {
-        // we try to get a read lock for 500 ms.
-        // that is a very long duration, but the information is important.
-        if let Some(hbbft_state_lock) = self.hbbft_state.try_read_for(Duration::from_millis(500)) {
-            return Some(hbbft_state_lock.get_current_posdao_epoch_start_block());
-        } else {
-            // better a potential stage 3 verification error instead of a deadlock ?!
-            // https://github.com/DMDcoin/diamond-node/issues/68
-            warn!(target: "engine", "could not aquire read lock for retrieving the pruning_protection_block_number. Stage 3 verification error might follow up.");
-            return None;
-        }
-    }
-
     /** returns if the signer of hbbft is tracked as available in the hbbft contracts. */
     pub fn is_available(&self) -> Result<bool, Error> {
         match self.signer.read().as_ref() {
@@ -1602,6 +1587,25 @@ impl Engine<EthereumMachine> for HoneyBadgerBFT {
     fn prometheus_metrics(&self, registry: &mut stats::PrometheusRegistry) {
         self.hbbft_message_dispatcher.prometheus_metrics(registry);
     }
+
+    /// hbbft protects the start of the current posdao epoch start from being pruned.
+    fn pruning_protection_block_number(&self) -> Option<u64> {
+        // we try to get a read lock for 500 ms.
+        // that is a very long duration, but the information is important.
+        if let Some(hbbft_state_lock) = self.hbbft_state.try_read_for(Duration::from_millis(500)) {
+            if let Some(last_epoch_start_block) =
+                hbbft_state_lock.get_last_posdao_epoch_start_block()
+            {
+                return Some(last_epoch_start_block);
+            }
+            return Some(hbbft_state_lock.get_current_posdao_epoch_start_block());
+        } else {
+            // better a potential stage 3 verification error instead of a deadlock ?!
+            // https://github.com/DMDcoin/diamond-node/issues/68
+            warn!(target: "engine", "could not aquire read lock for retrieving the pruning_protection_block_number. Stage 3 verification error might follow up.");
+            return None;
+        }
+    }
 }
 
 #[cfg(test)]

crates/ethcore/src/engines/hbbft/hbbft_state.rs

@@ -40,6 +40,7 @@ pub(crate) struct HbbftState {
     public_master_key: Option<PublicKey>,
     current_posdao_epoch: u64,
     current_posdao_epoch_start_block: u64,
+    last_posdao_epoch_start_block: Option<u64>,
     future_messages_cache: BTreeMap<u64, Vec<(NodeId, HbMessage)>>,
 }
 
@@ -51,6 +52,7 @@ impl HbbftState {
             public_master_key: None,
             current_posdao_epoch: 0,
             current_posdao_epoch_start_block: 0,
+            last_posdao_epoch_start_block: None,
             future_messages_cache: BTreeMap::new(),
         }
     }
@@ -91,13 +93,19 @@ impl HbbftState {
         }
 
         let posdao_epoch_start = get_posdao_epoch_start(&*client, block_id).ok()?;
-        let synckeygen = initialize_synckeygen(
+        let synckeygen = match initialize_synckeygen(
             &*client,
             signer,
             BlockId::Number(posdao_epoch_start.low_u64()),
             ValidatorType::Current,
-        )
-        .ok()?;
+        ) {
+            Ok(synckey) => synckey,
+            Err(e) => {
+                error!(target: "engine", "error initializing synckeygen for block: {:?}: {:?}", block_id, e);
+                return None;
+            }
+        };
+
         assert!(synckeygen.is_ready());
 
         let (pks, sks) = synckeygen.generate().ok()?;
@@ -109,6 +117,7 @@ impl HbbftState {
         self.honey_badger = None;
         // Set the current POSDAO epoch #
         self.current_posdao_epoch = target_posdao_epoch;
+        self.last_posdao_epoch_start_block = Some(self.current_posdao_epoch_start_block);
         self.current_posdao_epoch_start_block = posdao_epoch_start.as_u64();
 
         trace!(target: "engine", "Switched hbbft state to epoch {}.", self.current_posdao_epoch);
@@ -481,7 +490,7 @@ impl HbbftState {
                 Ok(synckeygen) => synckeygen,
                 Err(e) => {
                     let diff = parent_block_nr - posdao_epoch_start.low_u64();
-                    error!(target: "consensus", "Synckeygen failed. parent block: {} epoch_start: {}  diff {} with error: {:?}  ", parent_block_nr, posdao_epoch_start, diff, e);
+                    error!(target: "consensus", "Error: Synckeygen failed. parent block: {} epoch_start: {}  diff {} with error: {:?}. current posdao: {:?} target epoch  {:?}", parent_block_nr, posdao_epoch_start, diff, e, self.current_posdao_epoch, target_posdao_epoch);
                     return false;
                 }
             };
@@ -544,4 +553,8 @@ impl HbbftState {
     pub fn get_current_posdao_epoch_start_block(&self) -> u64 {
         self.current_posdao_epoch_start_block
     }
+
+    pub fn get_last_posdao_epoch_start_block(&self) -> Option<u64> {
+        self.last_posdao_epoch_start_block
+    }
 }

crates/ethcore/src/engines/mod.rs

@@ -575,6 +575,12 @@ pub trait Engine<M: Machine>: Sync + Send {
         true
     }
 
+    /// allows engines to define a block that should not get pruned in the DB.
+    /// This is useful for engines that need to keep a certain block in the DB.
+    fn pruning_protection_block_number(&self) -> Option<u64> {
+        None
+    }
+
     /// Optional entry point for adding engine specific metrics.
     fn prometheus_metrics(&self, _registry: &mut stats::PrometheusRegistry) {}
 }
@@ -709,12 +715,6 @@ pub trait EthEngine: Engine<::machine::EthereumMachine> {
     fn minimum_gas_price(&self) -> Option<U256> {
         None
     }
-
-    /// allows engines to define a block that should not get pruned in the DB.
-    /// This is useful for engines that need to keep a certain block in the DB.
-    fn pruning_protection_block_number(&self) -> Option<u64> {
-        None
-    }
 }
 
 // convenience wrappers for existing functions.

crates/util/stats/Cargo.toml

@@ -6,3 +6,4 @@ authors = ["Parity Technologies <[email protected]>"]
 [dependencies]
 log = "0.4"
 prometheus = "0.13.0"
+vergen = "0.1"

crates/util/stats/src/lib.rs

@@ -108,6 +108,27 @@ impl PrometheusRegistry {
         );
         t
     }
+
+    pub fn register_version(&mut self) {
+        let sha3 = vergen::SHORT_SHA;
+        let version = sha3.bits();
+        self.register_gauge("version_sha3_bits", "version_sha3", version as i64);
+        self.register_gauge(
+            "version_semver_bits",
+            "Sementic Versioning bits",
+            vergen::SEMVER.bits() as i64,
+        );
+        self.register_gauge(
+            "version_commit_date",
+            "commit date",
+            vergen::COMMIT_DATE.bits() as i64,
+        );
+        self.register_gauge(
+            "version_vergen_target",
+            "vergen targets",
+            vergen::TARGET.bits() as i64,
+        );
+    }
 }
 
 /// Implements a prometheus metrics collector