Skip to content

Releases: ethereum/tests

Tests update 17.2: Reorganization

04 Jun 11:34
@winsvega winsvega
v17.2
c67e485
Compare
Choose a tag to compare
Tests update 17.2: Reorganization Latest
Latest

The recent tests changes:

Added tx sender field in all Legacy Cancun tests and Invalid block tests for the tools that does not support signature recovery.
GeneralStateTests vetors are fully managed by https://github.com/ethereum/execution-spec-tests
There are still a few Blockchain test vectors in this repo and unit tests (Transaction/RLP) that are not migrated to .py framework yet.

From now on there would be no new test vectors in this repository and existing ones will slowly be converted into https://github.com/ethereum/execution-spec-tests.
NOTE: Pyspecs doesn't cover all the vectors from this repo, but has many new vectors that are not covered in ethereum/tests
Eventually there will be only ethereum/legacytests repo with all .json tests from Frontier to Cancun.
And ethereum/execution-spec-tests will handle all the official ethereum testing vectors for all the froks.

Announcement

New docker instructions for all t8n supported clients: https://github.com/ethereum/retesteth/tree/develop#docker-instructions
The test process diagram: https://tinyurl.com/y525ezs7
Reminder of a brand new web interface for test debugging: http://retesteth.ethdevops.io/web/
Tests stats on latest builds: http://retesteth.ethdevops.io/
(implement test RPC support or t8ntool protocol to apply for the page. https://ethereum-tests.readthedocs.io/en/latest/t8ntool-ref.html)
Previous release link: https://github.com/ethereum/tests/releases/tag/v17.1

Test format

Tests Fillers

GeneralStateTests/BlockchainTests Filled

Test Regeneration

Update all:

  • Refill sender tx field in invalid blockchain tests and in legacy tests
    PR #1493

  • Remove GeneralStateTests (now are in execution-spec-tests releases)
    PR #1491
    PR #1492

  • Documentation

TESTS

Cosmetic

Upcoming changes:

Support ALL existing .json tests with pyspecs consume
Convert .json .yml yo .py

Assets 2
Loading

Tests update 17.1: Reorganization

13 May 14:50
@winsvega winsvega
v17.1
08b5fe3
Compare
Choose a tag to compare
Tests update 17.1: Reorganization

The recent tests changes:

Archive Transition tests to legacy tests repo
Unify _info section with pyspecs format
Test name inside files in pyspecs format
Insert Prague precompiles into blockchain version of the tests

From now on there would be no new test vectors in this repository and existing ones will slowly be converted into https://github.com/ethereum/execution-spec-tests.
NOTE: Pyspecs doesn't cover all the vectors from this repo, but has many new vectors that are not covered in ethereum/tests
Eventually there will be only ethereum/legacytests repo with all .json tests from Frontier to Cancun.
And ethereum/execution-spec-tests will handle all the official ethereum testing vectors for all the froks.

Announcement

New docker instructions for all t8n supported clients: https://github.com/ethereum/retesteth/tree/develop#docker-instructions
The test process diagram: https://tinyurl.com/y525ezs7
Reminder of a brand new web interface for test debugging: http://retesteth.ethdevops.io/web/
Tests stats on latest builds: http://retesteth.ethdevops.io/
(implement test RPC support or t8ntool protocol to apply for the page. https://ethereum-tests.readthedocs.io/en/latest/t8ntool-ref.html)
Previous release link: https://github.com/ethereum/tests/releases/tag/v17.0

Test format

Tests Fillers

BlockchainTests/TransitionTests folders are deleted

GeneralStateTests/BlockchainTests Filled

BlockchainTests/TransitionTests folders are deleted

Field change in _info section:
generatedTestHash renamed to hash
new field repo == ethereum/tests

Test Regeneration

Update all:

PR #1489

  • Documentation

TESTS

  • Archive transition tests to legacy tests
    PR #1473

Converted to python Tests

  • Remove converted blake2 tests
    PR #1479

  • Remove EOF Stack tests
    PR #1478

  • Remove converted EOF JUMPF stack overflow tests
    PR #1481

  • delete converted EOF stack underflow tests
    PR #1476

  • **Remove non constant stack EOF tests **
    PR #1483

  • delete converted EOF RJUMP to self tests*
    PR #1485

  • Remove stack range maximally broad
    PR #1484

  • Remove converted EOF CALLF stack overflow tests
    PR #1474

  • remove converted calldataload and calldatasize tests
    PR #1477

  • remove converted instrinsic gas tests*
    PR #1488

  • remove test scenarios
    PR #1490

Cosmetic

Upcoming changes:

Support ALL existing .json tests with pyspecs consume
Convert .json .yml yo .py

Loading

Tests update 17.0: Reorganization

13 Mar 09:47
@winsvega winsvega
v17.0
81862e4
Compare
Choose a tag to compare
Tests update 17.0: Reorganization

The recent tests changes:

Remove Pyspec versions of tests to avoid confusion
Remove EIPTests folder
Fill in fixture-format in tests _info section
Pack all fixtures into fixtures tarball

From now on there would be no new test vectors in this repository and existing ones will slowly be converted into https://github.com/ethereum/execution-spec-tests.
NOTE: Pyspecs doesn't cover all the vectors from this repo, but has many new vectors that are not covered in ethereum/tests
Eventually there will be only ethereum/legacytests repo with all .json tests from Frontier to Cancun.
And ethereum/execution-spec-tests will handle all the official ethereum testing vectors for all the froks.

Announcement

New docker instructions for all t8n supported clients: https://github.com/ethereum/retesteth/tree/develop#docker-instructions
The test process diagram: https://tinyurl.com/y525ezs7
Reminder of a brand new web interface for test debugging: http://retesteth.ethdevops.io/web/
Tests stats on latest builds: http://retesteth.ethdevops.io/
(implement test RPC support or t8ntool protocol to apply for the page. https://ethereum-tests.readthedocs.io/en/latest/t8ntool-ref.html)
Previous release link: https://github.com/ethereum/tests/releases/tag/v15.0

Test format

Tests Fillers

EIPTests folders are deleted
Pyspecs folders are deleted

GeneralStateTests/BlockchainTests Filled

EIPTests folders are deleted
Pyspecs folders are deleted
_info section has fixture-format field of values state_test, blockchain_test
pack all the fixtures separately into tarball

Test Regeneration

Update all:

  • Remove EIP tests support
    PR #1466

  • Remove Pyspec tests support
    PR #1467

  • Documentation

TESTS

  • Pack tests to fixtures tarball
    PR #1469
    PR #1471

  • Move bigint test vectors to a separate folder
    To keep track of it later
    PR #1470

Converted to python Tests

  • Delete converted EOF JUMPF stack overflow tests
    PR #1463

  • delete converted EOF unreachable instructions tests
    PR #1461

  • delete converted EOF RJUMPV tests
    PR #1458

Cosmetic

  • Remove test with empty account from the legacy tests
    PR #1464

  • Fill in fixture-format field in _info section
    PR #1468

Upcoming changes:

Convert .json .yml yo .py

Loading

Prague support v16.0

26 Feb 11:03
@winsvega winsvega
v16.0
afed83b
Compare
Choose a tag to compare
Prague support v16.0

The recent tests changes:

Generate all ethereum/tests on Prague devnet 6 using evmone t8n https://github.com/ethereum/evmone/releases/tag/v0.14.0

From now on there would be no new test vectors in this repository and existing ones will slowly be converted into https://github.com/ethereum/execution-spec-tests.
NOTE: Pyspecs doesn't cover all the vectors from this repo, but has many new vectors that are not covered in ethereum/tests
Eventually there will be only ethereum/legacytests repo with all .json tests from Frontier to Cancun.
And ethereum/execution-spec-tests will handle all the official ethereum testing vectors for all the froks.

Announcement

New docker instructions for all t8n supported clients: https://github.com/ethereum/retesteth/tree/develop#docker-instructions
The test process diagram: https://tinyurl.com/y525ezs7
Reminder of a brand new web interface for test debugging: http://retesteth.ethdevops.io/web/
Tests stats on latest builds: http://retesteth.ethdevops.io/
(implement test RPC support or t8ntool protocol to apply for the page. https://ethereum-tests.readthedocs.io/en/latest/t8ntool-ref.html)
Previous release link: https://github.com/ethereum/tests/releases/tag/v15.0

Test format

Tests Fillers

GeneralStateTests/BlockchainTests Filled

new field in filled state tests/blockchain test:

        "config" : {
            "blobSchedule" : {
                "Cancun" : {
                    "baseFeeUpdateFraction" : "0x32f0ed",
                    "max" : "0x06",
                    "target" : "0x03"
                },
                "Prague" : {
                    "baseFeeUpdateFraction" : "0x4c6964",
                    "max" : "0x09",
                    "target" : "0x06"
                }
            },
            "chainid" : "0x01"
        },

contains Cancun or Prague or both or none schedules depending if test is filled with this forks.
in blockchain test this config field contains also network field

new field in filled state tests:

        "post" : {
            "Cancun" : [
                {
                    "hash" :  ...,
                    "indexes" : ...,
                    "logs" : ...,
                    "state" : {
                             `post state after transaction execution`
                    }
                   ...
               }
      }

basically exports the post state in each test. following the new test format.

On blockchain tests a system block history address inserted into the pre state according to Prague rules.

Test Regeneration

Update all:

  • Refill using Prague:
    PR #1416

  • Documentation

TESTS

Converted to python Tests

  • delete converted EOF prefix tests
    PR #1450

  • delete converted EOF "no type section" tests
    PR #1445

  • delete covered tests for DUPN, SWAPN, EXCHANGE
    PR #1453

  • delete converted EOF "multiple type sections" tests
    PR #1452

  • delete converted EOF "no code section" tests
    PR #1451

  • delete converted EOF RJUMP tests
    PR #1455

  • delete converted EOFCREATE validation tests
    PR #1454

  • delete converted EOF RJUMPI tests
    PR #1457

  • Remove migrated ID Precompile tests
    PR #1444

Cosmetic

Upcoming changes:

Convert .json .yml yo .py
Pyspecs filling support

Loading

Tests update 15.0 Convert EOF/tests to python

07 Feb 09:34
@winsvega winsvega
v15.0
e2d83cf
Compare
Choose a tag to compare
Tests update 15.0 Convert EOF/tests to python

The recent tests changes:

From now on there would be no new test vectors in this repository and existing ones will slowly be converted into https://github.com/ethereum/execution-spec-tests.
NOTE: Pyspecs doesn't cover all the vectors from this repo, but has many new vectors that are not covered in ethereum/tests
Eventually there will be only ethereum/legacytests repo with all .json tests from Frontier to Cancun.
And ethereum/execution-spec-tests will handle all the official ethereum testing vectors for all the froks.

Announcement

New docker instructions for all t8n supported clients: https://github.com/ethereum/retesteth/tree/develop#docker-instructions
The test process diagram: https://tinyurl.com/y525ezs7
Reminder of a brand new web interface for test debugging: http://retesteth.ethdevops.io/web/
Tests stats on latest builds: http://retesteth.ethdevops.io/
(implement test RPC support or t8ntool protocol to apply for the page. https://ethereum-tests.readthedocs.io/en/latest/t8ntool-ref.html)
Previous release link: https://github.com/ethereum/tests/releases/tag/v14.1

Test format

Tests Fillers

GeneralStateTests/BlockchainTests Filled

Test Regeneration

Update all:

  • Documentation

TESTS

  • Fix broken test in CALLCODE case in costRevertFiller
    PR #1429

  • Update pyspec tests that are untill Cancun
    PR #1413
    PR #1440

  • rename Prague to Osaka in EOF tests
    PR #1409

Converted to python Tests

  • Remove migrated EOF execution tests
    PR #1410
    PR #1412

  • remove EEST covered tests: unreachable EOF code sections
    PR #1404

  • remove migrated EOF RJUMP validation tests
    PR #1421

  • remove migrated PUSH test
    PR #1427

  • remove migrated DUP test
    PR #1428

  • remove calldatacopy tests
    PR #1425

  • **delete converted tests for EOF functions **
    PR #1432

  • delete minimal EOF validation tests
    PR #1435
    PR #1447

  • delete converted CALLF to non-returning test case
    PR #1433

  • delete converted tests for DATALOADN
    PR #1434
    PR #1439

  • delete EOF validation tests for "too many sections"
    PR #1437

  • Remove migrated precompiled blake2B Tests
    PR #1431

  • delete converted tests for non-returning sections
    PR #1436

  • delete converted JUMPF tests
    PR #1438

  • delete already covered EOF validation tests
    PR #1441

Cosmetic

  • Fix typos and readme
    PR #1446
    PR #1443
    PR #1449

  • correct geth t9n exception on transaction
    PR #1448

  • correct input data length in calldatacopy. the test is not affected
    PR #1430

  • change test address from 100 to 1000 as the precompile numbers approach 100
    PR #1401

  • add comments to legacy tests folders
    PR #1424

Upcoming changes:

Convert .json .yml yo .py
Pyspecs filling support

Loading

Tests update 14.1 Legacy Tests

31 Aug 10:59
@winsvega winsvega
v14.1
9201075
Compare
Choose a tag to compare
Tests update 14.1 Legacy Tests

The recent tests changes:

From now on there would be no new test vectors in this repository and existing ones will slowly be converted into https://github.com/ethereum/execution-spec-tests. NOTE: Pyspecs doesn't cover all the vectors from this repo, but has many new vectors that are not covered in ethereum/tests
Eventually there will be only ethereum/legacytests repo with all .json tests from Frontier to Cancun.
And ethereum/execution-spec-tests will handle all the official ethereum testing vectors for all the froks.

Announcement

New docker instructions for all t8n supported clients: https://github.com/ethereum/retesteth/tree/develop#docker-instructions
The test process diagram: https://tinyurl.com/y525ezs7
Reminder of a brand new web interface for test debugging: http://retesteth.ethdevops.io/web/
Tests stats on latest builds: http://retesteth.ethdevops.io/
(implement test RPC support or t8ntool protocol to apply for the page. https://ethereum-tests.readthedocs.io/en/latest/t8ntool-ref.html)
Previous release link: https://github.com/ethereum/tests/releases/tag/v14.0

Test format

Tests Fillers

GeneralStateTests/BlockchainTests Filled

Test Regeneration

Update all:

  • Documentation

TESTS

  • Remove TXCREATE from EOF tests
    PR #1378
  • Update EOF Tests top level container rules
    PR #1381

Converted to python Tests

Cosmetic

  • Remove retesteth expect section tests from legacy folder
    PR #1392
  • Correct filenames
    PR #1393

Upcoming changes:

Convert .json .yml yo .py
Pyspecs filling support

Loading

Tests update 14 Legacy Tests

01 Jul 09:12
@winsvega winsvega
v14.0
e46e1db
Compare
Choose a tag to compare
Tests update 14 Legacy Tests

The recent tests changes:

Make all existing tests snapshot into LegacyTests folder Cancun
Make all recent test vectors >=Cancun only
Add PYSPEC style exceptions to develop tests
Begin convertion of json fillers to .py fillers (migrate to pyspecs repo)

Announcement

New docker instructions for all t8n supported clients: https://github.com/ethereum/retesteth/tree/develop#docker-instructions
The test process diagram: https://tinyurl.com/y525ezs7
Reminder of a brand new web interface for test debugging: http://retesteth.ethdevops.io/web/
Tests stats on latest builds: http://retesteth.ethdevops.io/
(implement test RPC support or t8ntool protocol to apply for the page. https://ethereum-tests.readthedocs.io/en/latest/t8ntool-ref.html)
Previous release link: https://github.com/ethereum/tests/releases/tag/v13.3

Test format

Tests Fillers

New folder LegacyTests/Cancun

GeneralStateTests/BlockchainTests Filled

New folder LegacyTests/src/LegacyTests/Cancun

Test Regeneration

Update all:
PR #1377 (make all tests snapshot)
PR #1380 (refill only cancun vectors)
PR #1382 (update pyspec tests)

  • Documentation
    PR #1376 (list KEVM in readme)

TESTS

  • Rename exceptions to PYSPEC style
    PR #1380

  • Replace addresses in tests 0x00..0100 to 0x00..01000
    PR #1372

  • Enable ecparing tests for cancun and beyond
    PR #1375

EIPTests/EOFTests

  • Reintroduce BLS precompile tests
    PR #1366

  • Remove TXCREATE from EOF tests
    PR #1378

Converted to python Tests
The following tests will now be managed by pyspecs test generator
PR #1383 (Cancun/stEIP1153)

Cosmetic

  • Fix invalid json fields from pyspecs
    PR #1385

  • Mark bigInt in test ValueOverflow when generated from state test
    PR #1387

  • Fix pyspec test filled with bug
    PR #1388

Upcoming changes:

Convert .json .yml yo .py
Pyspecs filling support

Loading

Tests update 13.3 Cancun tests

06 May 08:42
@winsvega winsvega
v13.3
faf33b4
Compare
Choose a tag to compare
Tests update 13.3 Cancun tests

The recent tests changes:

Fix the issue in blockchain tests with beacon root precompile, cosmetic updates
EIP7610 legacy tests fix
EIP tests for BLS

Announcement

New docker instructions for all t8n supported clients: https://github.com/ethereum/retesteth/tree/develop#docker-instructions
The test process diagram: https://tinyurl.com/y525ezs7
Reminder of a brand new web interface for test debugging: http://retesteth.ethdevops.io/web/
Tests stats on latest builds: http://retesteth.ethdevops.io/
(implement test RPC support or t8ntool protocol to apply for the page. https://ethereum-tests.readthedocs.io/en/latest/t8ntool-ref.html)
Previous release link: https://github.com/ethereum/tests/releases/tag/v13.2

Test format

Tests Fillers

GeneralStateTests/BlockchainTests Filled

Test Regeneration

Update all:

  • Documentation
  • PR #1356 (fix typos)

TESTS

  • Fix legacy tests to revert on create collision with non empty storage dest (EIP7610)
    PR #1367

  • **Fix the blockchain tests logic with beacon root in genesis **
    PR #1357

  • Add more ecpairing test vectors
    PR #1358

  • Update bad opcode tests for Cancun
    PR #1362

  • Update pyspec tests, rename Merge to Paris
    PR #1363

EIPTests/EOFTests

  • Reintroduce BLS precompile tests
    PR #1366

  • Fix and update EOF tests
    PR #1349

  • More EOF validation tests from evmone
    PR #1364

Upcoming changes:

Convert .json .yml yo .py
Pyspecs filling support

Loading

Tests update 13.2 Cancun tests

13 Mar 14:04
@winsvega winsvega
v13.2
066a587
Compare
Choose a tag to compare
Tests update 13.2 Cancun tests

The recent tests changes:

Cosmetic update
Insert beacon precompile into bc tests's pre >= Cancun
Fix genesis excessBlobGasUsed for first block excessBlobGasUsed to be = 0 or x calculation
Remove empty storage records from pre state

Announcement

New docker instructions for all t8n supported clients: https://github.com/ethereum/retesteth/tree/develop#docker-instructions
The test process diagram: https://tinyurl.com/y525ezs7
Reminder of a brand new web interface for test debugging: http://retesteth.ethdevops.io/web/
Tests stats on latest builds: http://retesteth.ethdevops.io/
(implement test RPC support or t8ntool protocol to apply for the page. https://ethereum-tests.readthedocs.io/en/latest/t8ntool-ref.html)
Previous release link: https://github.com/ethereum/tests/releases/tag/v13.1

Test format

Tests Fillers

GeneralStateTests/BlockchainTests Filled

Insert beacon precompile into tests pre if not exist for BlockchainTests >= Cancun

"0x000f3df6d732807ef1319fb7b8bb8522d0beac02" : {
                "balance" : "0x00",
                "code" : "0x3373fffffffffffffffffffffffffffffffffffffffe14604d57602036146024575f5ffd5b5f35801560495762001fff810690815414603c575f5ffd5b62001fff01545f5260205ff35b5f5ffd5b62001fff42064281555f359062001fff015500",
                "nonce" : "0x01",
                "storage" : {
                }
            },

Test Regeneration

Update all:

  • Pyspec update: remove beacon contract from tests, remove empty storage records
    PR #1345

  • Refill all blockchain tests, add beacon root on cancun
    PR #1351

  • Documentation

TESTS

  • --
    PR --

EIPTests/EOFTests

  • update for relaxed stack validation
    PR #1334

  • **Move EOF creation tests into EOF test format **
    PR #1337

Cosmetic

  • Move random blockhash calling state tests to bc format
    PR #1354

  • Change exception name in PYSPEC
    PR #1342

  • Remove empty storage keys (0x00) in pre state
    PR #1347

Upcoming changes:

Convert .json .yml yo .py
Pyspecs filling support

Loading

Tests update 13.1 Cancun tests

14 Feb 14:07
@winsvega winsvega
v13.1
853b1e0
Compare
Choose a tag to compare
Tests update 13.1 Cancun tests

The recent tests changes:

Rename Merge to Paris (note: Pyspecs tests still have Merge keyword)
Remove empty accounts from post Paris
Update cancun rules, state test format, update pyspecs vectors
Update filled state test format

Announcement

New docker instructions for all t8n supported clients: https://github.com/ethereum/retesteth/tree/develop#docker-instructions
The test process diagram: https://tinyurl.com/y525ezs7
Reminder of a brand new web interface for test debugging: http://retesteth.ethdevops.io/web/
Tests stats on latest builds: http://retesteth.ethdevops.io/
(implement test RPC support or t8ntool protocol to apply for the page. https://ethereum-tests.readthedocs.io/en/latest/t8ntool-ref.html)
Previous release link: https://github.com/ethereum/tests/releases/tag/v13

Test format

Tests Fillers

Pyspec tests folder moved from /src/BlockchainTestsFiller/Pyspecs to /src/GeneralStateTestsFiller/Pyspecs
and the generated tests moved accordingly. as .py tests now could have state and blockchain tests.
--filltests flag will generate all state tests out of .py files
--filltests --fillchain flag will generate all blockchain tests out of .py files.

GeneralStateTests/BlockchainTests Filled

rename forkname in all of the tests: "Merge" => "Paris"
empty accounts (zero balance, no code, zero nonce, with or without storage) in pre state are not allowed post Paris.

In state tests remove blockchain related fields:

 "currentBeaconRoot" : "0x0000000000000000000000000000000000000000000000000000000000000000",
 "currentWithdrawalsRoot" : "0x56e81f171bcc55a6ff8345e692c0f86e5b48e01b996cadc001622fb5e363b421",`
 "previousHash" : "0x5e20a0453cecd065ea59c37ac63e079ee08998b6045136a8ce6635c7912ec0b6"

In blockchain tests if invalid block rlp is unreadable, or supposed to fail during rlp parsing, rlp_decoded field no longer provided

Test Regeneration

Update all:

  • All tests regeneration with Cancun rules
    PR #1332
    PR #1333 (PYSPEC UPDATE)

  • Adapt tests and remove empty accounts from state/bc tests post Paris/Merge
    PR #1335
    PR #1336

  • Documentation
    PR #1323

TESTS

  • Update the TestEachOpcode template for Cancun
    PR #1320

  • Fix genesis blobgasused params
    PR #1328

EIPTests/EOFTests

  • Update EOF tests to not contain unreachable code sections
    PR #1319

Cosmetic

  • ---
    PR ---

Upcoming changes:

Convert .json .yml yo .py
Pyspecs filling support

Loading