Run pebble database tests in Nightly CI Builds

[KolbyML]

Replaces #3915 this PR uses a branch from origin instead of my remote branch. I will use the origin branch going forward.

Resolves NIT-3501

This PR

• makes "in-memory" the default database for system_tests
• you can set which database engine is used with the --test-database-engine flag
• adds a nightly ci test runner which runs all system_tests using the pebble database engine

[KolbyML]

@pmikolajczyk41 here is the nightly ci run https://github.com/OffchainLabs/nitro/actions/runs/19438293760 lets see what happens

[github-actions]

❌ 3 Tests Failed:

Tests completed	Failed	Passed	Skipped
2174	3	2171	0

View the top 3 failed tests by shortest run time

TestBroadcastClientConfirmedMessage

Stack Traces | 5.010s run time

=== RUN   TestBroadcastClientConfirmedMessage
=== PAUSE TestBroadcastClientConfirmedMessage
=== CONT  TestBroadcastClientConfirmedMessage
    broadcastclient_test.go:327: broadcasting seq 0 message
INFO [11-28|17:37:29.625] connecting to arbitrum inbox message broadcaster url=ws://127.0.0.1:35507/
INFO [11-28|17:37:29.625] connecting to arbitrum inbox message broadcaster url=ws://127.0.0.1:35507/
INFO [11-28|17:37:29.625] arbitrum websocket broadcast server is listening address=[::]:39581
INFO [11-28|17:37:29.628] connecting to arbitrum inbox message broadcaster url=ws://127.0.0.1:39581/
INFO [11-28|17:37:29.628] connecting to arbitrum inbox message broadcaster url=ws://127.0.0.1:44695/
INFO [11-28|17:37:29.630] Feed connected                           feedServerVersion=2 chainId=8744 requestedSeqNum=0
INFO [11-28|17:37:29.630] Feed connected                           feedServerVersion=2 chainId=8744 requestedSeqNum=0
INFO [11-28|17:37:29.630] Feed connected                           feedServerVersion=2 chainId=8742 requestedSeqNum=0
INFO [11-28|17:37:29.630] Feed connected                           feedServerVersion=2 chainId=8742 requestedSeqNum=0
    broadcastclient_test.go:338: Received Message, Sequence Message: {0 {0xc000432570 0} <nil> [37 51 190 58 198 7 229 240 56 134 13 67 235 225 242 197 228 174 219 186 176 227 85 27 177 253 210 43 197 160 24 146 0 142 80 53 110 168 22 247 244 66 121 93 112 173 244 210 20 144 10 46 219 11 214 196 254 121 172 131 4 176 36 10 1] [] 0}
WARN [11-28|17:37:29.632] confirmed sequence number is past the end of stored messages "confirmed sequence number"=42 "last stored sequence number"=0
    broadcastclient_test.go:359: Client did not receive confirm message
--- FAIL: TestBroadcastClientConfirmedMessage (5.01s)

TestAuctioneerFailover_RapidRecovery

Stack Traces | 15.060s run time

... [CONTENT TRUNCATED: Keeping last 20 lines]
INFO [11-28|17:39:37.686] Imported new potential chain segment     number=15 hash=4e9b59..447373 blocks=1 txs=1 mgas=0.979 elapsed=1.062ms     mgasps=921.361  triediffs=37.61KiB triedirty=0.00B
INFO [11-28|17:39:37.686] Chain head was updated                   number=15 hash=4e9b59..447373 root=26c6a0..57c3e4 elapsed="90.029µs"
    auctioneer_failover_test.go:45: 
        	Error Trace:	/home/runner/work/nitro/nitro/timeboost/auctioneer_failover_test.go:45
        	            				/home/runner/work/nitro/nitro/timeboost/auctioneer_failover_test.go:312
        	Error:      	Received unexpected error:
        	            	invalid length, need 256 bits
        	            	opening wallet
        	            	github.com/offchainlabs/nitro/timeboost.NewAuctioneerServer
        	            		/home/runner/work/nitro/nitro/timeboost/auctioneer.go:207
        	            	github.com/offchainlabs/nitro/timeboost.createAndStartAuctioneer
        	            		/home/runner/work/nitro/nitro/timeboost/auctioneer_failover_test.go:44
        	            	github.com/offchainlabs/nitro/timeboost.TestAuctioneerFailover_RapidRecovery
        	            		/home/runner/work/nitro/nitro/timeboost/auctioneer_failover_test.go:312
        	            	testing.tRunner
        	            		/opt/hostedtoolcache/go/1.25.4/x64/src/testing/testing.go:1934
        	            	runtime.goexit
        	            		/opt/hostedtoolcache/go/1.25.4/x64/src/runtime/asm_amd64.s:1693
        	Test:       	TestAuctioneerFailover_RapidRecovery
--- FAIL: TestAuctioneerFailover_RapidRecovery (15.06s)

TestOutOfGasInStorageCacheFlush

Stack Traces | 27.240s run time

... [CONTENT TRUNCATED: Keeping last 20 lines]
        	/home/runner/work/nitro/nitro/system_tests/program_test.go:2832 +0x1006
        testing.tRunner(0xc089fdac40, 0x3d277e0)
        	/opt/hostedtoolcache/go/1.25.4/x64/src/testing/testing.go:1934 +0xea
        created by testing.(*T).Run in goroutine 1
        	/opt/hostedtoolcache/go/1.25.4/x64/src/testing/testing.go:1997 +0x465
        
    program_test.go:2832: �[31;1m [] failed calculating position for validation: batch not found on L1 yet �[0;0m
�[90mposted new batch 46�[0;0m
�[38;5;48;1myay!! we validated block 31 in 61.82ms�[0;0m
INFO [11-28|17:47:30.271] HTTP server stopped                      endpoint=127.0.0.1:40311
TRACE[11-28|17:47:30.271] P2P networking is spinning down
--- FAIL: TestOutOfGasInStorageCacheFlush (27.24s)
�[38;5;48;1myay!! we validated block 32 in 71.92ms�[0;0m
�[38;5;48;1myay!! we validated block 33 in 172.15ms�[0;0m
�[38;5;48;1myay!! we validated block 34 in 59.11ms�[0;0m
�[38;5;48;1myay!! we validated block 35 in 62.27ms�[0;0m
�[90mposted new batch 54�[0;0m
�[38;5;48;1myay!! we validated block 36 in 75.97ms�[0;0m
INFO [11-28|17:47:30.743] HTTP server stopped                      endpoint=127.0.0.1:33415
TRACE[11-28|17:47:30.743] P2P networking is spinning down

📣 Thoughts on this report? Let Codecov know! | Powered by Codecov

[KolbyML]

Ok I am doing another run of nightly and will open this PR if tests pass https://github.com/OffchainLabs/nitro/actions/runs/19448297103

[codecov]

Codecov Report

❌ Patch coverage is 85.71429% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 31.64%. Comparing base (5d545a1) to head (601d818).

Additional details and impacted files

@@             Coverage Diff             @@
##           master    #4026       +/-   ##
===========================================
+ Coverage   18.02%   31.64%   +13.62%     
===========================================
  Files         388      446       +58     
  Lines       48612    54681     +6069     
===========================================
+ Hits         8760    17303     +8543     
+ Misses      38159    34223     -3936     
- Partials     1693     3155     +1462     

[KolbyML]

@pmikolajczyk41 https://github.com/OffchainLabs/nitro/actions/runs/19448297103 looks like the nightly tests pass now

[pmikolajczyk41]

do I understand correctly, that the tests where we add manually WithDatabase() just won't work with memory db for some reason?

apart from that clarification question, lgtm

[KolbyML]

do I understand correctly, that the tests where we add manually WithDatabase() just won't work with memory db for some reason?

you can, it is just there probably isn't a need to unless there is some extremely niche cache

[pmikolajczyk41]

you can, it is just there probably isn't a need to unless there is some extremely niche cache

hmm, so why do we add this line? we will run these tests with pebble in nightly anyway and I thought the ticket assumed running everything with memory db by default

[KolbyML]

you can, it is just there probably isn't a need to unless there is some extremely niche cache

hmm, so why do we add this line? we will run these tests with pebble in nightly anyway and I thought the ticket assumed running everything with memory db by default

You should be able to choose what database you need for a test, if does --database-engine=pebble, but you only want the test to be ran with memorydb

MemoryDB is one of the options

[eljobe]

@KolbyML, great work. Sorry it took me so long to review it. Please resolve the merge conflict, and add it to the merge queue.