Skip to content

Conversation

Copy link

Copilot AI commented Oct 2, 2025

Problem

When RPC endpoints become unavailable (due to rate limits, connection issues, etc.), the error logs don't explicitly indicate which endpoint is affected. This makes troubleshooting difficult, especially in multi-chain deployments where operators need to quickly identify which specific endpoint is having issues.

Example of the current error log:

Error: HTTP error: 429 Too Many Requests
Location:
    /app/src/chains.rs:435:33

While the chain_id is logged, the actual RPC endpoint URL is missing, requiring operators to cross-reference configuration files to determine which endpoint needs attention.

Solution

This PR adds endpoint URL logging to all RPC error scenarios by:

  1. Storing the endpoint URL in the Chain struct - Added an endpoint field and getter method so the URL is accessible throughout the codebase

  2. Enhanced transport layer tracing - Modified TraceLayer and TraceTransport to include rpc.endpoint in OpenTelemetry span attributes, ensuring all RPC calls now include endpoint information in their traces

  3. Explicit health check logging - Updated the health check error logging to include the endpoint URL alongside chain_id

Impact

After these changes, when an RPC endpoint becomes unavailable, logs will clearly show:

Failed to obtain block number for health check
  err: HTTP error: 429 Too Many Requests
  chain_id: 8453
  endpoint: https://mainnet-preconf.base.org/

This enables operators to:

  • Immediately identify which specific endpoint is having issues
  • Quickly correlate errors with rate limits, regional outages, or provider-specific problems
  • Respond faster to infrastructure issues without manually checking configurations

Testing

  • ✅ Code compiles and passes clippy linting
  • ✅ All existing tests continue to pass
  • ✅ No breaking changes to existing functionality

Fixes #issue_number

Warning

Firewall rules blocked me from connecting to one or more addresses (expand for details)

I tried to connect to the following addresses, but was blocked by firewall rules:

  • arb1.arbitrum.io
    • Triggering command: /home/REDACTED/work/relay/relay/target/debug/deps/relay-0116f8e29f20dbc0 (dns block)
  • eu-central-mainnet.rpc.ithaca.xyz
    • Triggering command: /home/REDACTED/work/relay/relay/target/debug/deps/relay-0116f8e29f20dbc0 (dns block)
  • mainnet.optimism.io

If you need me to access, download, or install something from one of these locations, you can either:

Original prompt

This section details on the original issue you should resolve

<issue_title>Additional logging in cases one or multiple rpc endpoints become unavailable</issue_title>
<issue_description>## Description

Additional logging in cases one or multiple rpc endpoints become unavailable.

Problem Statement

At the moment relay does not explicitly log which chain/rpc endpoint is affected. Example error log due to rate limits being hit:

2025-10-01 14:29:54.865	
    /app/src/chains.rs:435:33
2025-10-01 14:29:54.865	
Location:
2025-10-01 14:29:54.865	

2025-10-01 14:29:54.865	
    HTTP error: 429 Too Many Requests
2025-10-01 14:29:54.865	
Caused by:
2025-10-01 14:29:54.865	

2025-10-01 14:29:54.865	
Error: HTTP error: 429 Too Many Requests
2025-10-01 14:29:54.857	
2025-10-01T06:29:54.857867Z  INFO alloy_pubsub::service: Pubsub service request channel closed. Shutting down.
2025-10-01 14:29:54.853	
2025-10-01T06:29:54.853721Z DEBUG alloy_rpc_client::call: sending request method=eth_getBlockByNumber id=5
2025-10-01 14:29:54.850	
2025-10-01T06:29:54.850349Z DEBUG poller{method=eth_getBlockByNumber}:ReqwestTransport{url=https://mainnet-preconf.base.org/}: hyper_util::client::legacy::connect::http: connected to 104.18.40.153:443
2025-10-01 14:29:54.849	
2025-10-01T06:29:54.849754Z DEBUG poller{method=eth_getBlockByNumber}:ReqwestTransport{url=https://mainnet-preconf.base.org/}: hyper_util::client::legacy::connect::http: connecting to 104.18.40.153:443
2025-10-01 14:29:54.844	
2025-10-01T06:29:54.844756Z DEBUG poller{method=eth_getBlockByNumber}:ReqwestTransport{url=https://mainnet-preconf.base.org/}: reqwest::connect: starting new connection: https://mainnet-preconf.base.org/
2025-10-01 14:29:54.844	
2025-10-01T06:29:54.844695Z DEBUG alloy_rpc_client::call: sending request method=eth_getBlockByNumber id=4
2025-10-01 14:29:54.844	
2025-10-01T06:29:54.844683Z DEBUG alloy_rpc_client::call: sending request method=eth_chainId id=3
2025-10-01 14:29:54.844	

Implementation Hints

User story

  • Easier troubleshooting
    </issue_description>

Comments on the Issue (you are @copilot in this section)

Fixes #1448

💬 Share your feedback on Copilot coding agent for the chance to win a $200 gift card! Click here to start the survey.

Copilot AI changed the title [WIP] Additional logging in cases one or multiple rpc endpoints become unavailable feat: add endpoint URL logging for RPC errors Oct 2, 2025
Copilot AI requested a review from yongkangc October 2, 2025 09:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Additional logging in cases one or multiple rpc endpoints become unavailable

2 participants