Finalize Codecov integration

[igor-sirotin]

1. Ensure all PRs have codecov status checks
   Looks like when 1 test failed, codecov doesn't report the status checks to the PR. Despite that we rerun the test.
   It's not always like that though. Maybe because of the flaky test detection Codecov feature.

   One of the ways to fix it could be remove test results report to Codecov, keep only the coverage.

2. Configure Codecov to satisfy our needs.

   • Set threshold for project coverage
   • Set minimum for the patch coverage

3. Remove codeclimate integration.

[igor-sirotin]

I did some investigation of the opened PRs. Need to filter by date, might be some old PRs when status checks were not set up yet.

PR	Created at	Last commit	Date OK	Commit checks	Comment
#5964	18.10.2024	18.10.2024	✅	✅	✅
#5962	17.10.2024	18.10.2024	✅	❌	✅
#5961	17.10.2024	17.10.2024	✅	✅	✅
#5960	17.10.2024	17.10.2024	✅	✅	⚠️ (4) (flaky: false)
#5959	17.10.2024	07.10.2024	✅	✅	✅
#5958	17.10.2024	11.10.2024	✅	✅	✅
#5957	17.10.2024	17.10.2024	✅	❌	⚠️ (1) (flaky: true)
#5956	16.10.2024	16.10.2024	✅	✅	⚠️ (1) (flaky: true)
#5955	16.10.2024	16.10.2024	✅	❌	❌
#5953	16.10.2024	17.10.2024	✅	❌	✅
#5952	16.10.2024	16.10.2024	✅	❌	✅
#5950	16.10.2024	16.10.2024	✅	✅	✅
#5948	15.10.2024	15.10.2024	✅	✅	✅
#5947	15.10.2024	15.10.2024	✅	❌	⚠️ (1) (flaky: true)
#5946	15.10.2024	17.10.2024	✅	❌	❌
#5945	15.10.2024	15.10.2024	✅	❌	⚠️ (1) (flaky: false)
#5943	14.10.2024	17.10.2024	✅	✅	✅
#5941	14.10.2024	14.10.2024	✅	❌	✅
#5939	11.10.2024	11.10.2024	✅	✅	✅
#5930	09.10.2024	09.09.2024	✅	✅	❌
#5929	08.10.2024	08.10.2024	✅	✅	✅
#5925	07.10.2024	07.10.2024	✅	✅	✅
#5907	03.10.2024	09.10.2024	✅	❌	❌
#5899	01.10.2024	19.09.2024	❌	✅	❌
#5885	27.09.2024	27.09.2024	❌	❌	❌
#5877	25.09.2024	03.10.2024	✅	❌	❌
#5864	23.09.2024	17.10.2024	✅	✅	✅
#5859	20.09.2024	19.09.2024	❌	❌	❌
#5857	19.09.2024	19.09.2024	❌	❌	✅
#5840	16.09.2024	18.10.2024	✅	❌	⚠️ (1) (flaky: false)

Code

package main

import (
	"context"
	"fmt"
	"log"
	"github.com/google/go-github/v48/github"
	"golang.org/x/oauth2"
	"strings"
	"time"
	"os"
)

func main() {
	ctx := context.Background()

	ghToken, ok := os.LookupEnv("GITHUB_TOKEN")
	if !ok {
		log.Fatalf("GITHUB_TOKEN not set")
	}

	ts := oauth2.StaticTokenSource(
		&oauth2.Token{AccessToken: ghToken},
	)

	tc := oauth2.NewClient(ctx, ts)

	client := github.NewClient(tc)

	owner := "status-im" // Replace with your repository owner
	repo := "status-go"  // Replace with your repository name
	thresholdDate, _ := time.Parse("02.01.2006", "03.10.2024")

	// Get open pull requests
	prs, _, err := client.PullRequests.List(ctx, owner, repo, &github.PullRequestListOptions{State: "open"})
	if err != nil {
		log.Fatalf("Error fetching pull requests: %v", err)
	}

	fmt.Printf("| PR | Created at | Last commit | Date OK | Commit checks | Comment |\n")
	fmt.Printf("|-|-|-|-|-|-|\n")

	//skip := true

	for _, pr := range prs {
		//if skip {
		//	if *pr.Number == 5943 {
		//		skip = false
		//	}
		//	continue
		//}

		prDate := pr.GetCreatedAt()

		// Check the date of the latest commit in the PR
		latestCommit, _, err := client.Repositories.GetCommit(ctx, owner, repo, pr.Head.GetSHA(), nil)
		if err != nil {
			log.Fatalf("Error fetching the latest commit: %v", err)
		}

		commitDate := latestCommit.Commit.Author.GetDate()
		tooOld := commitDate.Before(thresholdDate) && prDate.Before(thresholdDate)

		prComments, _, err := client.Issues.ListComments(ctx, owner, repo, *pr.Number, nil)
		if err != nil {
			log.Fatalf("Error fetching PR comments: %v", err)
		}

		var codecovComment *github.IssueComment
		var codecovChecks []string

		// Check for codecov comments
		for _, comment := range prComments {
			if comment.User != nil && comment.User.GetLogin() == "codecov[bot]" {
				codecovComment = comment
				break
			}
		}

		checkRuns, _, err := client.Checks.ListCheckRunsForRef(ctx, owner, repo, *pr.Head.SHA, nil)
		if err != nil {
			log.Fatalf("Error fetching combined check runs: %v", err)
		}

		for _, run := range checkRuns.CheckRuns {
			if run.Name == nil {
				continue
			}
			if !strings.HasPrefix(*run.Name, "codecov") {
				continue
			}
			codecovChecks = append(codecovChecks, strings.Replace(*run.Name, "codecov/", "", 1))
		}

		//if codecovStatusChecks < 6 {
		//	fmt.Printf("PR #%d: %s has %d codecov checks\n", *pr.Number, *pr.Title, codecovStatusChecks)
		//}

		fmt.Printf("| #%d | %s | %s | %v | %s | %s |\n",
			*pr.Number,
			prDate.Format("02.01.2006"),
			commitDate.Format("02.01.2006"),
			boolEmoji(!tooOld),
			boolEmoji(len(codecovChecks) > 0),
			firstLine(codecovComment),
		)
	}
}

func boolEmoji(b bool) string {
	if b {
		return "✅"
	}
	return "❌"
}

func checksString(checks []string) string {
	if len(checks) == 0 {
		return "❌"
	}

	return "✅"
}

func commentURL(comment *github.IssueComment) string {
	if comment == nil {
		return "---"
	}
	if comment.URL == nil {
		return "???"
	}
	return fmt.Sprintf("[url](%s)", *comment.URL)
}

func firstLine(comment *github.IssueComment) string {
	if comment == nil {
		return "❌"
	}
	if comment.Body == nil {
		return "🙈"
	}
	body := *comment.Body

	if strings.HasPrefix(body, "## [Codecov](https://app.codecov.io/gh/status-im/status-go/pull") {
		return "✅"
	}

	if strings.HasPrefix(body, "### :x:") {
		numberOfTestsFailed := strings.Split(body, " ")[2]
		flaky := strings.Contains(body, "flaky")
		//return fmt.Sprintf("%s failed", numberOfTestsFailed)
		return fmt.Sprintf("⚠️ (%s) (flaky: %v)", numberOfTestsFailed, flaky)
	}

	return strings.Split(*comment.Body, "\n")[0]
}

[igor-sirotin]

Analyzing issues from above:

PR	Codecov Checks	Codecov Tests	Comment	EL PROBLEMA
#5962	❌	✅	CI job failed, comment old
#5957	❌	⚠️ (1) ❄️	- different number of coverage report uploads - has changed files	👈
#5955	❌	❌	- different number of coverage report uploads - has changed files
#5953	❌	✅	- CI job failed, comment old (already fixed)
#5952	❌	✅	- CI job failed, comment old
#5947	❌	⚠️ (1) ❄️	- No Files covered by tests were changed - Commits have different number of coverage report uploads	👈
#5946	❌	❌	- CI job failed
#5945	❌	⚠️ (1)		👈
#5941	❌	✅	- different number of coverage report uploads - has changed files
#5930	✅	❌	Base branch is not develop
#5907	❌	❌		👈
#5877	❌	❌	- Commits have different number of coverage report uploads

[igor-sirotin]

Checking #5955.

tests log:

[2024-10-16T14:54:27.579Z] Uploading coverage report to Codecov
[2024-10-16T14:54:27.579Z] info - 2024-10-16 14:54:27,117 -- ci service found: jenkins
[2024-10-16T14:54:27.579Z] info - 2024-10-16 14:54:27,305 -- Found 2 test_results files to report
[2024-10-16T14:54:27.579Z] info - 2024-10-16 14:54:27,306 -- > /home/jenkins/workspace/status-go_prs_tests_PR-5955/report_0.xml
[2024-10-16T14:54:27.579Z] info - 2024-10-16 14:54:27,306 -- > /home/jenkins/workspace/status-go_prs_tests_PR-5955/report_1.xml
[2024-10-16T14:54:28.844Z] info - 2024-10-16 14:54:28,608 -- Process Upload complete
[2024-10-16T14:54:29.627Z] info - 2024-10-16 14:54:29,241 -- ci service found: jenkins
[2024-10-16T14:54:29.627Z] info - 2024-10-16 14:54:29,570 -- Process Commit creating complete
[2024-10-16T14:54:29.995Z] info - 2024-10-16 14:54:29,907 -- Process Report creating complete
[2024-10-16T14:54:29.995Z] info - 2024-10-16 14:54:29,908 -- Finished creating report successfully --- {"response": "{\"status\":\"queued\"}\n"}
[2024-10-16T14:54:29.995Z] warning - 2024-10-16 14:54:29,908 -- xcrun is not installed or can't be found.
[2024-10-16T14:54:30.363Z] warning - 2024-10-16 14:54:30,057 -- No gcov data found.
[2024-10-16T14:54:30.363Z] warning - 2024-10-16 14:54:30,057 -- coverage.py is not installed or can't be found.
[2024-10-16T14:54:30.363Z] info - 2024-10-16 14:54:30,214 -- Found 4 coverage files to report
[2024-10-16T14:54:30.363Z] info - 2024-10-16 14:54:30,214 -- > /home/jenkins/workspace/status-go_prs_tests_PR-5955/c.out
[2024-10-16T14:54:30.363Z] info - 2024-10-16 14:54:30,214 -- > /home/jenkins/workspace/status-go_prs_tests_PR-5955/test_1.coverage.out
[2024-10-16T14:54:30.363Z] info - 2024-10-16 14:54:30,214 -- > /home/jenkins/workspace/status-go_prs_tests_PR-5955/test_0.coverage.out
[2024-10-16T14:54:30.363Z] info - 2024-10-16 14:54:30,215 -- > /home/jenkins/workspace/status-go_prs_tests_PR-5955/coverage_merged.out
[2024-10-16T14:54:39.692Z] info - 2024-10-16 14:54:38,890 -- Your upload is now processing. When finished, results will be available at: https://app.codecov.io/github/status-im/status-go/commit/4b1d93b98e5476e14993b95f9cfb5391049c8941
[2024-10-16T14:54:40.958Z] info - 2024-10-16 14:54:40,694 -- Process Upload complete

tests-rpc log:

[2024-10-16T14:26:26.278Z] �[0;32mUploading coverage report to Codecov�[0m
[2024-10-16T14:26:26.645Z] info - 2024-10-16 14:26:26,458 -- ci service found: jenkins
[2024-10-16T14:26:26.645Z] warning - 2024-10-16 14:26:26,487 -- Some files being explicitly added are found in the list of excluded files for upload. We are still going to search for the explicitly added files. --- {"files": ["/home/jenkins/workspace/status-go_prs_tests-rpc_PR-5955/tests-functional/reports/report.xml"]}
[2024-10-16T14:26:27.025Z] info - 2024-10-16 14:26:26,633 -- Found 1 test_results files to report
[2024-10-16T14:26:27.026Z] info - 2024-10-16 14:26:26,634 -- > /home/jenkins/workspace/status-go_prs_tests-rpc_PR-5955/tests-functional/reports/report.xml
[2024-10-16T14:26:27.394Z] info - 2024-10-16 14:26:27,187 -- Process Upload complete
[2024-10-16T14:26:28.179Z] info - 2024-10-16 14:26:27,806 -- ci service found: jenkins
[2024-10-16T14:26:28.575Z] info - 2024-10-16 14:26:28,344 -- Process Commit creating complete
[2024-10-16T14:26:28.943Z] info - 2024-10-16 14:26:28,758 -- Process Report creating complete
[2024-10-16T14:26:28.943Z] info - 2024-10-16 14:26:28,758 -- Finished creating report successfully --- {"response": "{\"status\":\"queued\"}\n"}
[2024-10-16T14:26:28.943Z] warning - 2024-10-16 14:26:28,759 -- xcrun is not installed or can't be found.
[2024-10-16T14:26:28.943Z] warning - 2024-10-16 14:26:28,901 -- No gcov data found.
[2024-10-16T14:26:28.943Z] warning - 2024-10-16 14:26:28,902 -- coverage.py is not installed or can't be found.
[2024-10-16T14:26:29.311Z] info - 2024-10-16 14:26:29,060 -- Found 2 coverage files to report
[2024-10-16T14:26:29.311Z] info - 2024-10-16 14:26:29,060 -- > /home/jenkins/workspace/status-go_prs_tests-rpc_PR-5955/tests-functional/reports/report.xml
[2024-10-16T14:26:29.311Z] info - 2024-10-16 14:26:29,060 -- > /home/jenkins/workspace/status-go_prs_tests-rpc_PR-5955/tests-functional/coverage/coverage.out
[2024-10-16T14:26:38.650Z] info - 2024-10-16 14:26:37,861 -- Your upload is now processing. When finished, results will be available at: https://app.codecov.io/github/status-im/status-go/commit/4b1d93b98e5476e14993b95f9cfb5391049c8941
[2024-10-16T14:26:39.915Z] info - 2024-10-16 14:26:39,532 -- Process Upload complete

Both reports are detected by the Codecov:

Maybe it's because No files covered by tests were changed 🤔
But it doesn't make any sense. Especially if I'd have the commit check as required.

[igor-sirotin]

These are the PRs to consider as problems:

PR	Codecov Checks	Codecov Tests	Comment	EL PROBLEMA
#5957	❌	⚠️ (1) ❄️	- different number of coverage report uploads (has changed files)	👈
#5955	❌	❌	- different number of coverage report uploads (has changed files)
#5947	❌	⚠️ (1) ❄️	- different number of coverage report uploads (no files covered by tests were changed)	👈
#5945	❌	⚠️ (1)		👈
#5941	❌	✅	- different number of coverage report uploads (has changed files)
#5907	❌	❌		👈
#5877	❌	❌	- Commits have different number of coverage report uploads

Problems description

1. has changed files / No Files covered by tests were changed
   Doesn't seem to affect status checks.

2. different number of coverage report uploads
   Will be fixed with Run tests on each commit in develop branch #5809 (comment)
   It's now implemented, but required PRs to be rebased. Will watch PRs starting based on e611b1e and later.

3. When a test is failing, no status check is reported.

   Not sure how to tackle this. In general, it makes sense: when one test failed, no point of measuring the coverage.
   But we automatically re-run failed tests to fight flakiness. So in the same report there's definitely a test failing and same test passing. But Codecov seems to have precedence for the fails.

   I guess we have 2 options:

   • Don't report test results to Codecov
     Though it's fancy to have, I don't think someone is using it anyway, we have test results in Jenkins.
   • Try to modify the test report before uploading to Codecov and remove any failing tests if there's one passing
     Not sure how difficult it is. Need to investigate. Will quickly push for the first option first and then let's see if it works.

[igor-sirotin]

Another thing that bothers me:

For some reason the last commit for develop branch context is ba37c32.
Though it was not on develop. And it was quite long time ago 🤔

https://app.codecov.io/github/status-im/status-go/tree/develop

UPD:

Just noted that a new commit appeared for the develop branch context: 807397f
And it's again a commit from @jakubgs 👀 What kind of super-power is that?! 🤔

Interesting:

https://app.codecov.io/github/status-im/status-go/commits/develop

[igor-sirotin]

Another issue I found in #5985:

UPD: probably not an issue. Codecov is actually showing better results, because it counts the functional-tests 🎉

[igor-sirotin]

So it seems that only directly pushed commits were visible in develop branch context.

This could be because of the wrong default branch:

I changed it to develop, will monitor further.

UPD:

Doesn't seem to help 🙁

UPD 2:

ok maybe it is actually working now