Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix_: no test reporting to codecov #5968

Merged
merged 1 commit into from
Oct 21, 2024

Conversation

igor-sirotin
Copy link
Collaborator

@igor-sirotin igor-sirotin commented Oct 19, 2024

Part of #5963

Description

Remove uploading test results to Codecov. Can be brought back when we don't re-run tests on failure.

Re-running tests results in having both failure and success results in the same report, which Codecov treats as a failure and doesn't report coverage to Github. More details here: #5963

Result

Sems to work ok now, even when a test is failing once:

image image image
Flaky test example
package flaky_test
import (
	"os"
	"testing"
)
const stateFile = "test_state.txt"
func TestFlaky(t *testing.T) {
	if !checkState() {
		// First run: fail the test
		t.Errorf("Test failed on first run!")
	} else {
		// Second run: pass the test
		t.Log("Test passed on second run!")
		resetState()
	}
}
func checkState() bool {
	// Check if the state file exists
	if _, err := os.Stat(stateFile); !os.IsNotExist(err) {
		// If file exists, return true (pass on second run)
		return true
	}
	// If file doesn't exist, create it and return false (fail on first run)
	f, _ := os.Create(stateFile)
	defer f.Close()
	return false
}
func resetState() {
	// Remove the state file to reset the test state
	os.Remove(stateFile)
}

@igor-sirotin igor-sirotin self-assigned this Oct 19, 2024
@status-im-auto
Copy link
Member

status-im-auto commented Oct 19, 2024

Jenkins Builds

Click to see older builds (10)
Commit #️⃣ Finished (UTC) Duration Platform Result
✖️ 5b4c57d #1 2024-10-19 09:40:31 ~3 min tests 📄log
✔️ 5b4c57d #1 2024-10-19 09:41:54 ~4 min ios 📦zip
✔️ 5b4c57d #1 2024-10-19 09:42:18 ~4 min linux 📦zip
✔️ 5b4c57d #1 2024-10-19 09:42:47 ~5 min android 📦aar
✔️ 5b4c57d #1 2024-10-19 09:43:01 ~5 min tests-rpc 📄log
✔️ 8113237 #2 2024-10-19 10:04:39 ~1 min android 📦aar
✔️ 8113237 #2 2024-10-19 10:05:32 ~2 min linux 📦zip
✔️ 8113237 #2 2024-10-19 10:05:48 ~2 min ios 📦zip
✔️ 8113237 #2 2024-10-19 10:08:08 ~5 min tests-rpc 📄log
✔️ 8113237 #2 2024-10-19 10:34:46 ~31 min tests 📄log
Commit #️⃣ Finished (UTC) Duration Platform Result
✔️ f8d353c #3 2024-10-19 19:11:56 ~1 min android 📦aar
✔️ f8d353c #3 2024-10-19 19:12:42 ~2 min linux 📦zip
✔️ f8d353c #3 2024-10-19 19:13:11 ~2 min ios 📦zip
✔️ f8d353c #3 2024-10-19 19:15:44 ~5 min tests-rpc 📄log
✔️ f8d353c #3 2024-10-19 19:41:59 ~31 min tests 📄log
✔️ 4180006 #4 2024-10-19 19:16:58 ~1 min android 📦aar
✔️ 4180006 #4 2024-10-19 19:17:54 ~2 min linux 📦zip
✔️ 4180006 #4 2024-10-19 19:18:08 ~2 min ios 📦zip
✔️ 4180006 #4 2024-10-19 19:20:57 ~5 min tests-rpc 📄log
✔️ 4180006 #4 2024-10-19 20:15:27 ~33 min tests 📄log

Copy link

codecov bot commented Oct 19, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 47.63%. Comparing base (9f56401) to head (4180006).
Report is 1 commits behind head on develop.

Additional details and impacted files
@@             Coverage Diff             @@
##           develop    #5968      +/-   ##
===========================================
- Coverage    47.91%   47.63%   -0.29%     
===========================================
  Files          843      842       -1     
  Lines       138322   138318       -4     
===========================================
- Hits         66282    65884     -398     
- Misses       64400    64645     +245     
- Partials      7640     7789     +149     
Flag Coverage Δ
functional 10.59% <ø> (-0.05%) ⬇️
unit 46.94% <ø> (-0.29%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

see 74 files with indirect coverage changes

@igor-sirotin igor-sirotin force-pushed the fix/no-test-report-to-codecov branch 2 times, most recently from 8113237 to f8d353c Compare October 19, 2024 19:10
@igor-sirotin igor-sirotin marked this pull request as ready for review October 21, 2024 05:07
@igor-sirotin igor-sirotin requested review from osmaczko and a team October 21, 2024 05:08
@igor-sirotin igor-sirotin merged commit 4202259 into develop Oct 21, 2024
16 of 18 checks passed
@igor-sirotin igor-sirotin deleted the fix/no-test-report-to-codecov branch October 21, 2024 11:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Status: Done
Development

Successfully merging this pull request may close these issues.

4 participants