Fix: Rebuild Go stdlib after installing gotip to prevent version mismatch. #7667
Conversation
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #7667 +/- ##
==========================================
+ Coverage 96.54% 96.56% +0.02%
==========================================
Files 384 384
Lines 19501 19501
==========================================
+ Hits 18827 18831 +4
+ Misses 489 486 -3
+ Partials 185 184 -1
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
Metrics Comparison SummaryTotal changes across all snapshots: 53 Detailed changes per snapshotsummary_metrics_snapshot_cassandra📊 Metrics Diff SummaryTotal Changes: 53
🆕 Added Metrics
View diff sample+http_server_request_body_size_bytes{http_request_method="GET",http_response_status_code="503",le="+Inf",network_protocol_name="http",network_protocol_version="1.1",otel_scope_name="go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp",otel_scope_schema_url="",otel_scope_version="0.63.0",server_address="localhost",server_port="13133",url_scheme="http"}
+http_server_request_body_size_bytes{http_request_method="GET",http_response_status_code="503",le="0",network_protocol_name="http",network_protocol_version="1.1",otel_scope_name="go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp",otel_scope_schema_url="",otel_scope_version="0.63.0",server_address="localhost",server_port="13133",url_scheme="http"}
+http_server_request_body_size_bytes{http_request_method="GET",http_response_status_code="503",le="10",network_protocol_name="http",network_protocol_version="1.1",otel_scope_name="go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp",otel_scope_schema_url="",otel_scope_version="0.63.0",server_address="localhost",server_port="13133",url_scheme="http"}
+http_server_request_body_size_bytes{http_request_method="GET",http_response_status_code="503",le="100",network_protocol_name="http",network_protocol_version="1.1",otel_scope_name="go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp",otel_scope_schema_url="",otel_scope_version="0.63.0",server_address="localhost",server_port="13133",url_scheme="http"}
+http_server_request_body_size_bytes{http_request_method="GET",http_response_status_code="503",le="1000",network_protocol_name="http",network_protocol_version="1.1",otel_scope_name="go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp",otel_scope_schema_url="",otel_scope_version="0.63.0",server_address="localhost",server_port="13133",url_scheme="http"}
+http_server_request_body_size_bytes{http_request_method="GET",http_response_status_code="503",le="10000",network_protocol_name="http",network_protocol_version="1.1",otel_scope_name="go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp",otel_scope_schema_url="",otel_scope_version="0.63.0",server_address="localhost",server_port="13133",url_scheme="http"}
+http_server_request_body_size_bytes{http_request_method="GET",http_response_status_code="503",le="25",network_protocol_name="http",network_protocol_version="1.1",otel_scope_name="go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp",otel_scope_schema_url="",otel_scope_version="0.63.0",server_address="localhost",server_port="13133",url_scheme="http"}
...View diff sample+http_server_request_duration_seconds{http_request_method="GET",http_response_status_code="503",le="+Inf",network_protocol_name="http",network_protocol_version="1.1",otel_scope_name="go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp",otel_scope_schema_url="",otel_scope_version="0.63.0",server_address="localhost",server_port="13133",url_scheme="http"}
+http_server_request_duration_seconds{http_request_method="GET",http_response_status_code="503",le="0.005",network_protocol_name="http",network_protocol_version="1.1",otel_scope_name="go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp",otel_scope_schema_url="",otel_scope_version="0.63.0",server_address="localhost",server_port="13133",url_scheme="http"}
+http_server_request_duration_seconds{http_request_method="GET",http_response_status_code="503",le="0.01",network_protocol_name="http",network_protocol_version="1.1",otel_scope_name="go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp",otel_scope_schema_url="",otel_scope_version="0.63.0",server_address="localhost",server_port="13133",url_scheme="http"}
+http_server_request_duration_seconds{http_request_method="GET",http_response_status_code="503",le="0.025",network_protocol_name="http",network_protocol_version="1.1",otel_scope_name="go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp",otel_scope_schema_url="",otel_scope_version="0.63.0",server_address="localhost",server_port="13133",url_scheme="http"}
+http_server_request_duration_seconds{http_request_method="GET",http_response_status_code="503",le="0.05",network_protocol_name="http",network_protocol_version="1.1",otel_scope_name="go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp",otel_scope_schema_url="",otel_scope_version="0.63.0",server_address="localhost",server_port="13133",url_scheme="http"}
+http_server_request_duration_seconds{http_request_method="GET",http_response_status_code="503",le="0.075",network_protocol_name="http",network_protocol_version="1.1",otel_scope_name="go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp",otel_scope_schema_url="",otel_scope_version="0.63.0",server_address="localhost",server_port="13133",url_scheme="http"}
+http_server_request_duration_seconds{http_request_method="GET",http_response_status_code="503",le="0.1",network_protocol_name="http",network_protocol_version="1.1",otel_scope_name="go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp",otel_scope_schema_url="",otel_scope_version="0.63.0",server_address="localhost",server_port="13133",url_scheme="http"}
...View diff sample+http_server_response_body_size_bytes{http_request_method="GET",http_response_status_code="503",le="+Inf",network_protocol_name="http",network_protocol_version="1.1",otel_scope_name="go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp",otel_scope_schema_url="",otel_scope_version="0.63.0",server_address="localhost",server_port="13133",url_scheme="http"}
+http_server_response_body_size_bytes{http_request_method="GET",http_response_status_code="503",le="0",network_protocol_name="http",network_protocol_version="1.1",otel_scope_name="go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp",otel_scope_schema_url="",otel_scope_version="0.63.0",server_address="localhost",server_port="13133",url_scheme="http"}
+http_server_response_body_size_bytes{http_request_method="GET",http_response_status_code="503",le="10",network_protocol_name="http",network_protocol_version="1.1",otel_scope_name="go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp",otel_scope_schema_url="",otel_scope_version="0.63.0",server_address="localhost",server_port="13133",url_scheme="http"}
+http_server_response_body_size_bytes{http_request_method="GET",http_response_status_code="503",le="100",network_protocol_name="http",network_protocol_version="1.1",otel_scope_name="go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp",otel_scope_schema_url="",otel_scope_version="0.63.0",server_address="localhost",server_port="13133",url_scheme="http"}
+http_server_response_body_size_bytes{http_request_method="GET",http_response_status_code="503",le="1000",network_protocol_name="http",network_protocol_version="1.1",otel_scope_name="go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp",otel_scope_schema_url="",otel_scope_version="0.63.0",server_address="localhost",server_port="13133",url_scheme="http"}
+http_server_response_body_size_bytes{http_request_method="GET",http_response_status_code="503",le="10000",network_protocol_name="http",network_protocol_version="1.1",otel_scope_name="go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp",otel_scope_schema_url="",otel_scope_version="0.63.0",server_address="localhost",server_port="13133",url_scheme="http"}
+http_server_response_body_size_bytes{http_request_method="GET",http_response_status_code="503",le="25",network_protocol_name="http",network_protocol_version="1.1",otel_scope_name="go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp",otel_scope_schema_url="",otel_scope_version="0.63.0",server_address="localhost",server_port="13133",url_scheme="http"}
... |
|
How did you test this? |
|
@yurishkuro I analyzed the CI error logs showing the version mismatches and developed this fix based on that analysis. |
yurishkuro
added
the
run-all-workflows
|
We specifically defined the logic in this workflow to run even on PRs when |
| go clean -cache | ||
| go clean -modcache | ||
| go install std |
There was a problem hiding this comment.
this doesn't make sense to me. The fact that you have to do that indicates that we still have some cross-pollution happening between two Go versions. If go clean does not help we need to find a more reliable approach - once a gotip is installed no leftovers of the previous toolchain should be accessible. Maybe we need to be more aggressive about env variables, to ensure none of them point to the previous toolchain.
There was a problem hiding this comment.
@yurishkuro That's a totally valid point and i am working on the issue and looking for the root cause of this cross-pollution that has occurred after the 2nd PR was merged. Just give me some more time to solve this issue.
- Disable Go module caching with cache: false - Rebuild standard library after installing gotip Signed-off-by: Kavish-12345 <[email protected]>
- Unset GOTOOLCHAIN env var in setup-go-tip action - Remove stale go1.25.4 binaries to prevent shadowing - Add GOTOOLCHAIN: auto to workflow steps Signed-off-by: Kavish-12345 <[email protected]>
Kavish-12345
force-pushed
the
issue-7664-revisited
branch
from
4141930 to
e3030bd
Compare
| echo "GOTOOLCHAIN=" >> $GITHUB_ENV | ||
| # Remove any stale go binaries from previous toolchain installations | ||
| # This prevents shadowing of gotip's go binary | ||
| for go_path in /opt/hostedtoolcache/go/*/x64/bin/go /home/runner/go/bin/go; do |
There was a problem hiding this comment.
/opt/hostedtoolcache/go/*/x64/bin/go
can we not do this via some env var reference? Hardcoding the path will make this action unstable in the future
There was a problem hiding this comment.
Instead of hardcoding /opt/hostedtoolcache/go/*/x64/bin/go, I'll use command -v go to dynamically find and remove the old go binary. This is future-proof and doesn't depend on GitHub's directory structure.
| uses: actions/setup-go@4dc6199c7b1a012772edbd06daecab0f50c9053c # v6.1.0 | ||
| with: | ||
| go-version: 1.25.x | ||
| cache: false |
There was a problem hiding this comment.
according to me the cache: false setting prevents the go1.25.4 module cache from persisting and interfering with gotip's isolated environment.
| go clean -cache | ||
| go clean -modcache | ||
| env: | ||
| GOTOOLCHAIN: auto |
There was a problem hiding this comment.
GOTOOLCHAIN is not something that must be managed by use. According to AI, there are 5 env vars we need to control (actual paths are examples):
To use the 1.25.x toolchain:
- export GOROOT=$HOME/sdk/go1.25.x
- export GOPATH=$HOME/go/1.25
- export PATH=$GOROOT/bin:$PATH
- export GOCACHE=$GOPATH/cache
- export GOMODCACHE=$GOPATH/pkg/mod
To use the gotip toolchain:
- export GOROOT=$HOME/sdk/go-tip
- export GOPATH=$HOME/go/tip
- export PATH=$GOROOT/bin:$PATH
- export GOCACHE=$GOPATH/cache
- export GOMODCACHE=$GOPATH/pkg/mod
There was a problem hiding this comment.
i am addressing this . Also after checking the failure point this time it is not a version mismatch like earlier but due to the TestSpanCollectorHTTP test failing due to a pre-existing IPv6 port parsing issue that is unrelated to the gotip environment changes.
PATH) and using dynamic path detection instead of hardcoded values. Signed-off-by: Kavish-12345 <[email protected]>
Signed-off-by: Kavish-12345 <[email protected]>
Signed-off-by: Kavish-12345 <[email protected]>
![graphite-app[bot]](https://avatars.githubusercontent.com/in/158384?s=60&v=4){kind=small}
| env: | ||
| GOTOOLCHAIN: auto | ||
|
|
There was a problem hiding this comment.
Setting GOTOOLCHAIN: auto conflicts with the GOTOOLCHAIN= (unset) in the setup-go-tip action (line 69). When GOTOOLCHAIN is set to "auto", Go will automatically select and download a toolchain version, potentially bypassing gotip. This contradicts the stated goal of ensuring gotip is used. Either remove this env block or change to GOTOOLCHAIN: local to force using the installed gotip.
env:
GOTOOLCHAIN: local| env: | |
| GOTOOLCHAIN: auto | |
| env: | |
| GOTOOLCHAIN: local | |
Spotted by Graphite Agent
Is this helpful? React 👍 or 👎 to let us know.
|
@yurishkuro I have implemented the dynamic approach for the binary files , also tried to control the 5 ENV variables and along with it tried the GOTOOLCHAIN:LOCAL as well as AUTO , but the version error mismatch is back and during the last time when i made the changes it had worked correctly ....... the tests failed due to the other preexisting issues .... need your guidance on it for the implementation right now i am pushing the changes in which the version mismatch errors were not encountered. |
2 participants
Fixes #7664
Problem
After the initial fix separated stable Go for tools and gotip for tests, cached standard library packages compiled with Go 1.25.x were still being used by gotip, causing compilation errors.
Root Cause
When gotip runs, it tries to link against pre-compiled standard library packages (like
internal/goarch,sync/atomic, etc.) from Go 1.25.x, causing version mismatches.Changes
cache: falseto disable automatic caching.go install stdto rebuild standard library with gotip.