feat: improve error handling and logging in the compute package #4850
Conversation
WalkthroughThis pull request refines the error handling across the compute package. In the bidder module, error checks now distinguish specific error types, and additional logging is implemented. In the executor watcher, errors during bidding, execution, and cancellation are handled with improved logging and corresponding metric increments. Additionally, three new error metrics are introduced in the metrics file to track execution errors more closely. Changes
Sequence Diagram(s)sequenceDiagram
participant B as Bidder
participant S as Store
participant L as Logger
B->>S: UpdateExecutionState()
alt Error is ErrInvalidExecutionState
B->>L: Log debug: "Skipping update owing to invalid state"
else Other error encountered
B->>B: Call handleError()
Note right of B: Update execution state to failed
B->>L: Log detailed error if failErr exists
end
sequenceDiagram
participant E as ExecutionUpsertHandler
participant O as Operation
participant M as Metrics
participant L as Logger
E->>O: Perform bidding/execution/cancellation
alt Error occurs during operation
E->>M: Increment corresponding error metric
E->>L: Log error with execution ID & state details
else No error
E->>E: Continue normal flow
end
Possibly related PRs
Poem
Warning There were issues while running some tools. Please review the errors and either fix the tool’s configuration or disable the tool if it’s a critical failure. 🔧 golangci-lint (1.62.2){"Issues":[{"FromLinter":"typecheck","Text":": # github.com/bacalhau-project/bacalhau/pkg/compute/watchers\npkg/compute/watchers/executor_watcher.go:57:7: duplicate case models.ExecutionStateCancelled (constant 11 of type "github.com/bacalhau-project/bacalhau/pkg/models".ExecutionStateType) in expression switch\n\tpkg/compute/watchers/executor_watcher.go:56:13: previous case\npkg/compute/watchers/executor_watcher.go:60:6: logger.Error().Err(err).Msg("failed to cancel execution") (no value) used as value","Severity":"","SourceLines":["package watchers"],"Replacement":null,"Pos":{"Filename":"pkg/compute/watchers/bprotocol_dispatcher.go","Offset":0,"Line":1,"Column":0},"ExpectNoLint":false,"ExpectedNoLintLinter":""},{"FromLinter":"typecheck","Text":"could not import github.com/bacalhau-project/bacalhau/pkg/compute/watchers (-: # github.com/bacalhau-project/bacalhau/pkg/compute/watchers\npkg/compute/watchers/executor_watcher.go:57:7: duplicate case models.ExecutionStateCancelled (constant 11 of type "github.com/bacalhau-project/bacalhau/pkg/models".ExecutionStateType) in expression switch\n\tpkg/compute/watchers/executor_watcher.go:56:13: previous case\npkg/compute/watchers/executor_watcher.go:60:6: logger.Error().Err(err).Msg("failed to cancel execution") (no value) used as value)","Severity":"","SourceLines":["\t"github.com/bacalhau-project/bacalhau/pkg/compute/watchers""],"Replacement":null,"Pos":{"Filename":"pkg/transport/bprotocol/compute/transport.go","Offset":0,"Line":14,"Column":2},"ExpectNoLint":false,"ExpectedNoLintLinter":""},{"FromLinter":"typecheck","Text":"could not import github.com/bacalhau-project/bacalhau/pkg/compute/watchers (-: # github.com/bacalhau-project/bacalhau/pkg/compute/watchers\npkg/compute/watchers/executor_watcher.go:57:7: duplicate case models.ExecutionStateCancelled (constant 11 of type "github.com/bacalhau-project/bacalhau/pkg/models".ExecutionStateType) in expression switch\n\tpkg/compute/watchers/executor_watcher.go:56:13: previous case\npkg/compute/watchers/executor_watcher.go:60:6: logger.Error().Err(err).Msg("failed to cancel execution") (no value) used as value)","Severity":"","SourceLines":["\t"github.com/bacalhau-project/bacalhau/pkg/compute/watchers""],"Replacement":null,"Pos":{"Filename":"pkg/node/compute.go","Offset":0,"Line":22,"Column":2},"ExpectNoLint":false,"ExpectedNoLintLinter":""},{"FromLinter":"typecheck","Text":"pattern build/: no matching files found","Severity":"","SourceLines":["//go:embed build/"],"Replacement":null,"Pos":{"Filename":"webui/webui.go","Offset":0,"Line":23,"Column":12},"ExpectNoLint":false,"ExpectedNoLintLinter":""}],"Report":{"Linters":[{"Name":"asasalint"},{"Name":"asciicheck"},{"Name":"bidichk"},{"Name":"bodyclose","Enabled":true},{"Name":"canonicalheader"},{"Name":"containedctx"},{"Name":"contextcheck"},{"Name":"copyloopvar","Enabled":true},{"Name":"cyclop"},{"Name":"decorder"},{"Name":"deadcode"},{"Name":"depguard","Enabled":true},{"Name":"dogsled","Enabled":true},{"Name":"dupl"},{"Name":"dupword"},{"Name":"durationcheck"},{"Name":"errcheck","Enabled":true,"EnabledByDefault":true},{"Name":"errchkjson"},{"Name":"errname"},{"Name":"errorlint"},{"Name":"execinquery"},{"Name":"exhaustive"},{"Name":"exhaustivestruct"},{"Name":"exhaustruct"},{"Name":"exportloopref"},{"Name":"forbidigo","Enabled":true},{"Name":"forcetypeassert"},{"Name":"fatcontext"},{"Name":"funlen","Enabled":true},{"Name":"gci"},{"Name":"ginkgolinter"},{"Name":"gocheckcompilerdirectives"},{"Name":"gochecknoglobals"},{"Name":"gochecknoinits","Enabled":true},{"Name":"gochecksumtype"},{"Name":"gocognit"},{"Name":"goconst","Enabled":true},{"Name":"gocritic"},{"Name":"gocyclo","Enabled":true},{"Name":"godot"},{"Name":"godox"},{"Name":"err113"},{"Name":"gofmt","Enabled":true},{"Name":"gofumpt"},{"Name":"goheader"},{"Name":"goimports","Enabled":true},{"Name":"golint"},{"Name":"mnd","Enabled":true},{"Name":"gomnd"},{"Name":"gomoddirectives"},{"Name":"gomodguard"},{"Name":"goprintffuncname","Enabled":true},{"Name":"gosec","Enabled":true},{"Name":"gosimple","Enabled":true,"EnabledByDefault":true},{"Name":"gosmopolitan"},{"Name":"govet","Enabled":true,"EnabledByDefault":true},{"Name":"grouper"},{"Name":"ifshort"},{"Name":"iface"},{"Name":"importas"},{"Name":"inamedparam"},{"Name":"ineffassign","Enabled":true,"EnabledByDefault":true},{"Name":"interfacebloat"},{"Name":"interfacer"},{"Name":"intrange"},{"Name":"ireturn"},{"Name":"lll","Enabled":true},{"Name":"loggercheck"},{"Name":"maintidx"},{"Name":"makezero"},{"Name":"maligned"},{"Name":"mirror"},{"Name":"misspell"},{"Name":"musttag"},{"Name":"nakedret","Enabled":true},{"Name":"nestif"},{"Name":"nilerr"},{"Name":"nilnil"},{"Name":"nlreturn"},{"Name":"noctx","Enabled":true},{"Name":"nonamedreturns"},{"Name":"nosnakecase"},{"Name":"nosprintfhostport"},{"Name":"paralleltest"},{"Name":"perfsprint"},{"Name":"prealloc"},{"Name":"predeclared"},{"Name":"promlinter"},{"Name":"protogetter"},{"Name":"reassign"},{"Name":"recvcheck"},{"Name":"revive"},{"Name":"rowserrcheck"},{"Name":"sloglint"},{"Name":"scopelint"},{"Name":"sqlclosecheck"},{"Name":"spancheck"},{"Name":"staticcheck","Enabled":true,"EnabledByDefault":true},{"Name":"structcheck"},{"Name":"stylecheck","Enabled":true},{"Name":"tagalign"},{"Name":"tagliatelle"},{"Name":"tenv"},{"Name":"testableexamples"},{"Name":"testifylint"},{"Name":"testpackage"},{"Name":"thelper"},{"Name":"tparallel"},{"Name":"typecheck","Enabled":true,"EnabledByDefault":true},{"Name":"unconvert","Enabled":true},{"Name":"unparam"},{"Name":"unused","Enabled":true,"EnabledByDefault":true},{"Name":"usestdlibvars"},{"Name":"varcheck"},{"Name":"varnamelen"},{"Name":"wastedassign"},{"Name":"whitespace","Enabled":true},{"Name":"wrapcheck"},{"Name":"wsl"},{"Name":"zerologlint"},{"Name":"nolintlint","Enabled":true}]}} ✨ Finishing Touches
Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media? 🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (Invoked using PR comments)
Other keywords and placeholders
CodeRabbit Configuration File (
|
|
@wdbaruni Can you please take a look at this |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 3
📜 Review details
Configuration used: CodeRabbit UI
Review profile: ASSERTIVE
Plan: Pro
📒 Files selected for processing (3)
pkg/compute/bidder.go(2 hunks)pkg/compute/metrics.go(1 hunks)pkg/compute/watchers/executor_watcher.go(2 hunks)
⏰ Context from checks skipped due to timeout of 90000ms (1)
- GitHub Check: testcontainers-suite / tests
🔇 Additional comments (2)
pkg/compute/watchers/executor_watcher.go (1)
33-36: LGTM! Well-structured logging setup.The logger is properly configured with relevant context fields that will help with debugging and tracing issues.
pkg/compute/bidder.go (1)
223-232: LGTM! Robust error type handling.Good use of error type assertion to handle invalid state errors gracefully. The debug logging provides excellent context for troubleshooting.
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
There was a problem hiding this comment.
Thanks @virajbhartiya for your contribution. Looks great, but with minor comments
| logger := log.Ctx(ctx).With(). | ||
| Str("executionID", execution.ID). | ||
| Str("state", execution.ComputeState.StateType.String()). | ||
| Logger() |
There was a problem hiding this comment.
Actionable comments posted: 2
♻️ Duplicate comments (1)
pkg/compute/watchers/executor_watcher.go (1)
48-55: 🛠️ Refactor suggestionMaintain consistency in error wrapping for run state.
The error is logged but not wrapped with bacerrors, unlike other states.
err = h.executor.Run(ctx, execution) if err != nil { compute.ExecutionRunErrors.Add(ctx, 1) logger.Error(). Err(err). Msg("failed to run execution") - return err + return bacerrors.Wrap(err, "failed to handle execution state %s", execution.ComputeState.StateType) }
📜 Review details
Configuration used: CodeRabbit UI
Review profile: ASSERTIVE
Plan: Pro
📒 Files selected for processing (3)
pkg/compute/bidder.go(2 hunks)pkg/compute/metrics.go(1 hunks)pkg/compute/watchers/executor_watcher.go(2 hunks)
⏰ Context from checks skipped due to timeout of 90000ms (1)
- GitHub Check: build / Build Binary (linux, arm64)
🔇 Additional comments (1)
pkg/compute/bidder.go (1)
223-232: LGTM! Well-structured error handling.The error handling for invalid execution state is well-implemented:
- Appropriate use of errors.As for type checking
- Debug-level logging for expected errors
- Clear message indicating why the update is skipped
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
…larity and propagation
There was a problem hiding this comment.
Actionable comments posted: 2
📜 Review details
Configuration used: CodeRabbit UI
Review profile: ASSERTIVE
Plan: Pro
📒 Files selected for processing (3)
pkg/compute/bidder.go(3 hunks)pkg/compute/metrics.go(1 hunks)pkg/compute/watchers/executor_watcher.go(2 hunks)
⏰ Context from checks skipped due to timeout of 90000ms (2)
- GitHub Check: build / Build Binary (darwin, amd64)
- GitHub Check: build / Build Binary (linux, arm64)
🔇 Additional comments (5)
pkg/compute/watchers/executor_watcher.go (2)
34-37: Use structured logging effectively.
AttachingexecutionIDandstatedirectly to the logger provides more context for debugging, making logs significantly more traceable. Good approach!
66-68: Proper error wrapping.
Usingbacerrors.Wrapat the end ensures that errors carry clear context, making them easier to troubleshoot downstream.pkg/compute/bidder.go (2)
235-238: Clear error propagation.
Wrapping errors withbacerrors.Wrappreserves crucial context. Returningnilotherwise keeps behavior consistent when skipping state updates for invalid transitions.
190-190:❓ Verification inconclusive
Method signature now returns an error.
AllowinghandleBidResultto surface errors to its callers promotes proper error handling. Verify that all call sites handle the new return type to prevent unhandled errors.
🏁 Script executed:
#!/bin/bash # Searching for calls to handleBidResult in the codebase ast-grep --pattern $'handleBidResult($_, $_, $_)'Length of output: 50
Verify error propagation for handleBidResult call sites
The method’s signature change now returns an error, which is a good step toward improving error handling. However, our initial search did not produce any output for calls to
handleBidResultusing the patternhandleBidResult($_, $_, $_). This absence suggests we need to manually verify that all invocations ofhandleBidResult(which may not match that specific pattern) are appropriately updated to handle the new error return. Please double-check by either running a broader search (for example, withrg "handleBidResult\s*\(" -n .) or manually inspect the call sites to ensure proper error propagation.pkg/compute/metrics.go (1)
34-50: New error counters introduced.
These counters for bidding, run, and cancellation errors are well-defined, following the established naming pattern. Great addition to improve observability.
|
|
||
| if err != nil { | ||
| log.Ctx(ctx).Error().Err(err).Msg("failed to update execution state") | ||
| return | ||
| var invalidStateErr store.ErrInvalidExecutionState | ||
| if errors.As(err, &invalidStateErr) { | ||
| log.Ctx(ctx).Debug(). | ||
| Err(err). | ||
| Str("executionID", execution.ID). | ||
| Str("expectedState", models.ExecutionStateNew.String()). | ||
| Str("actualState", invalidStateErr.Actual.String()). | ||
| Msg("skipping execution state update - execution no longer in expected state") | ||
| return nil | ||
| } |
There was a problem hiding this comment.
🧹 Nitpick (assertive)
Skips update if ErrInvalidExecutionState.
Returning nil here is appropriate if you intentionally ignore an outdated or conflicting state. To enhance clarity, add a comment explaining the decision and how it aligns with the desired concurrency or state-management logic.
if errors.As(err, &invalidStateErr) {
+ // The execution has likely advanced to a newer state. Skipping update is intentional.
log.Ctx(ctx).Debug().📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| if err != nil { | |
| log.Ctx(ctx).Error().Err(err).Msg("failed to update execution state") | |
| return | |
| var invalidStateErr store.ErrInvalidExecutionState | |
| if errors.As(err, &invalidStateErr) { | |
| log.Ctx(ctx).Debug(). | |
| Err(err). | |
| Str("executionID", execution.ID). | |
| Str("expectedState", models.ExecutionStateNew.String()). | |
| Str("actualState", invalidStateErr.Actual.String()). | |
| Msg("skipping execution state update - execution no longer in expected state") | |
| return nil | |
| } | |
| if err != nil { | |
| var invalidStateErr store.ErrInvalidExecutionState | |
| if errors.As(err, &invalidStateErr) { | |
| // The execution has likely advanced to a newer state. Skipping update is intentional. | |
| log.Ctx(ctx).Debug(). | |
| Err(err). | |
| Str("executionID", execution.ID). | |
| Str("expectedState", models.ExecutionStateNew.String()). | |
| Str("actualState", invalidStateErr.Actual.String()). | |
| Msg("skipping execution state update - execution no longer in expected state") | |
| return nil | |
| } |
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
There was a problem hiding this comment.
Actionable comments posted: 1
📜 Review details
Configuration used: CodeRabbit UI
Review profile: ASSERTIVE
Plan: Pro
📒 Files selected for processing (1)
pkg/compute/watchers/executor_watcher.go(2 hunks)
🧰 Additional context used
🪛 GitHub Check: lint / go-lint (ubuntu-latest)
pkg/compute/watchers/executor_watcher.go
[failure] 57-57:
duplicate case models.ExecutionStateCancelled (constant 11 of type "github.com/bacalhau-project/bacalhau/pkg/models".ExecutionStateType) in expression switch
[failure] 56-56:
previous case
[failure] 60-60:
logger.Error().Err(err).Msg("failed to cancel execution") (no value) used as value (typecheck)
[failure] 57-57:
duplicate case models.ExecutionStateCancelled (constant 11 of type "github.com/bacalhau-project/bacalhau/pkg/models".ExecutionStateType) in expression switch
[failure] 56-56:
previous case
[failure] 60-60:
logger.Error().Err(err).Msg("failed to cancel execution") (no value) used as value) (typecheck)
[failure] 57-57:
duplicate case models.ExecutionStateCancelled (constant 11 of type "github.com/bacalhau-project/bacalhau/pkg/models".ExecutionStateType) in expression switch
[failure] 56-56:
previous case
⏰ Context from checks skipped due to timeout of 90000ms (8)
- GitHub Check: build / Build Binary (windows, amd64)
- GitHub Check: build / Build Binary (darwin, arm64)
- GitHub Check: build / Build Binary (darwin, amd64)
- GitHub Check: build / Build Binary (linux, armv6)
- GitHub Check: build / Build Binary (linux, armv7)
- GitHub Check: build / Build Binary (linux, arm64)
- GitHub Check: lint / lint
- GitHub Check: build / Build Binary (linux, amd64)
🔇 Additional comments (4)
pkg/compute/watchers/executor_watcher.go (4)
34-37: LGTM! Well-structured logging setup.The logger setup with execution ID and state provides excellent context for debugging and tracing issues.
42-47: LGTM! Consistent error handling in bidding state.The error handling follows the recommended pattern with metric tracking and error logging.
49-55: LGTM! Consistent error handling in running state.The error handling matches the pattern used in the bidding state, maintaining consistency.
64-70: LGTM! Clean error handling with bacerrors.The final error handling correctly uses
bacerrors.Wrapto provide context about the execution state.
| case models.ExecutionStateCancelled: | ||
| return h.executor.Cancel(ctx, execution) | ||
| case models.ExecutionStateCancelled: | ||
| err = h.executor.Cancel(ctx, execution) | ||
| if err != nil { | ||
| + logger.Error().Err(err).Msg("failed to cancel execution") | ||
| compute.ExecutionCancelErrors.Add(ctx, 1) | ||
| } |
There was a problem hiding this comment.
Fix duplicate case statement and logger syntax.
There are two critical issues in the cancelled state handling:
- Duplicate case statement for
ExecutionStateCancelled - Syntax error in the logger statement
Apply this diff to fix the issues:
- case models.ExecutionStateCancelled:
-case models.ExecutionStateCancelled:
+ case models.ExecutionStateCancelled:
err = h.executor.Cancel(ctx, execution)
if err != nil {
- logger.Error().Err(err).Msg("failed to cancel execution")
+ logger.Error().
+ Err(err).
+ Msg("failed to cancel execution")
compute.ExecutionCancelErrors.Add(ctx, 1)
}📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| case models.ExecutionStateCancelled: | |
| return h.executor.Cancel(ctx, execution) | |
| case models.ExecutionStateCancelled: | |
| err = h.executor.Cancel(ctx, execution) | |
| if err != nil { | |
| + logger.Error().Err(err).Msg("failed to cancel execution") | |
| compute.ExecutionCancelErrors.Add(ctx, 1) | |
| } | |
| case models.ExecutionStateCancelled: | |
| err = h.executor.Cancel(ctx, execution) | |
| if err != nil { | |
| logger.Error(). | |
| Err(err). | |
| Msg("failed to cancel execution") | |
| compute.ExecutionCancelErrors.Add(ctx, 1) | |
| } |
🧰 Tools
🪛 GitHub Check: lint / go-lint (ubuntu-latest)
[failure] 57-57:
duplicate case models.ExecutionStateCancelled (constant 11 of type "github.com/bacalhau-project/bacalhau/pkg/models".ExecutionStateType) in expression switch
[failure] 56-56:
previous case
[failure] 60-60:
logger.Error().Err(err).Msg("failed to cancel execution") (no value) used as value (typecheck)
[failure] 57-57:
duplicate case models.ExecutionStateCancelled (constant 11 of type "github.com/bacalhau-project/bacalhau/pkg/models".ExecutionStateType) in expression switch
[failure] 56-56:
previous case
[failure] 60-60:
logger.Error().Err(err).Msg("failed to cancel execution") (no value) used as value) (typecheck)
[failure] 57-57:
duplicate case models.ExecutionStateCancelled (constant 11 of type "github.com/bacalhau-project/bacalhau/pkg/models".ExecutionStateType) in expression switch
[failure] 56-56:
previous case
Closes #4684 #4849
This pull request includes several changes to improve error handling and logging in the compute package. The most important changes include adding error metrics, handling specific error cases, and enhancing logging to provide more detailed information.
Error Handling Improvements:
pkg/compute/bidder.go: Improved error handling in thehandleBidResultfunction by checking for specific error types and logging detailed messages.Logging Enhancements:
pkg/compute/watchers/executor_watcher.go: Enhanced logging in theHandleEventfunction to include execution ID and state, and added error logging for bidding, running, and cancellation errors.Metrics Addition:
pkg/compute/metrics.go: Added new metrics to track errors encountered during execution bidding, running, and cancellation.Miscellaneous Changes:
cmd/cli/config/list.go: Updated the table column configuration to include a maximum width and a soft wrap for the "Value" column.pkg/compute/bidder.go: Added theerrorspackage import to handle error type assertions.pkg/compute/watchers/executor_watcher.go: Added thezerolog/logpackage import for enhanced logging.Summary by CodeRabbit
Bug Fixes
New Features
Refactor