Unhealthy e2b API

[ayshsens]

After deploying the stack
e2b-orch-api-ig is unhealthy

[ayshsens]

no service for api port 50001 is running on gcp host

[linear]

E2B-2493 Unhealthy e2b API

[jakubno]

Hi @ayshsens, can you share your logs from nomad? Are you on latest main, there was a fixed regression in #794.

[ayshsens]

Thanks for prompt help on this.

looks like db ipv6 address in not accepted as published in recent docs.
but as per docs we are using supabase db.

{"level":"fatal","timestamp":"2025-06-17T21:22:44.49369519Z","message":"failed to check migration version","service":"orchestration-api","internal":true,"pid":1,"error":"failed to get database version: dial tcp [2600:1f18:2e13:9d13:c9af:7320:a3a3:1c59]:5432: connect: network is unreachable; dial tcp [2600:1f18:2e13:9d13:c9af:7320:a3a3:1c59]:5432: connect: network is unreachable","stacktrace":"main.run\n\t/build/api/main.go:255\nmain.main\n\t/build/api/main.go:424\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:272"}
{"level":"fatal","timestamp":"2025-06-17T21:23:00.097453489Z","message":"failed to check migration version","service":"orchestration-api","internal":true,"pid":1,"error":"failed to get database version: dial tcp [2600:1f18:2e13:9d13:c9af:7320:a3a3:1c59]:5432: connect: network is unreachable; dial tcp [2600:1f18:2e13:9d13:c9af:7320:a3a3:1c59]:5432: connect: network is unreachable","stacktrace":"main.run\n\t/build/api/main.go:255\nmain.main\n\t/build/api/main.go:424\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:272"}
{"level":"fatal","timestamp":"2025-06-17T21:23:15.62036184Z","message":"failed to check migration version","service":"orchestration-api","internal":true,"pid":1,"error":"failed to get database version: dial tcp [2600:1f18:2e13:9d13:c9af:7320:a3a3:1c59]:5432: connect: network is unreachable; dial tcp [2600:1f18:2e13:9d13:c9af:7320:a3a3:1c59]:5432: connect: network is unreachable","stacktrace":"main.run\n\t/build/api/main.go:255\nmain.main\n\t/build/api/main.go:424\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:272"}

[ayshsens]

supabase db is only providing ipv6 address

host db.evdqqguxvnqcqmxvwdta.supabase.co

db.evdqqguxvnqcqmxvwdta.supabase.co has IPv6 address 2600:1f18:2e13:9d13:c9af:7320:a3a3:1c59

[jakubno]

Your Supabase database needs to have enabled IPv4 access. You can do that in Connect screen

[ayshsens]

noted, moved to pro plan on supabase for ipv4 support.

have redeployed and have API keeps in pending state only with errors on log collector

2025-06-17T21:59:20.679033Z  WARN vector::config::loading: Transform "internal_routing.internal" has no consumers
2025-06-17T21:59:20.744767Z  WARN vector::internal_events::http_client: HTTP error. error=error trying to connect: dns error: failed to lookup address information: Name does not resolve error_type="request_failed" stage="processing" internal_log_rate_limit=true
2025-06-17T21:59:20.744819Z ERROR vector::topology::builder: msg="Healthcheck failed." error=Failed to make HTTP(S) request: error trying to connect: dns error: failed to lookup address information: Name does not resolve component_kind="sink" component_type="loki" component_id=local_loki_logs
2025-06-17T21:59:20.744828Z ERROR vector::topology::running: Sinks unhealthy.
2025-06-17T21:59:39.372964Z  WARN vector::config::loading: Transform "internal_routing.internal" has no consumers
2025-06-17T21:59:39.425942Z  WARN vector::internal_events::http_client: HTTP error. error=error trying to connect: dns error: failed to lookup address information: Name does not resolve error_type="request_failed" stage="processing" internal_log_rate_limit=true
2025-06-17T21:59:39.425990Z ERROR vector::topology::builder: msg="Healthcheck failed." error=Failed to make HTTP(S) request: error trying to connect: dns error: failed to lookup address information: Name does not resolve component_kind="sink" component_type="loki" component_id=local_loki_logs
2025-06-17T21:59:39.425999Z ERROR vector::topology::running: Sinks unhealthy.
2025-06-17T21:59:55.670971Z  WARN vector::config::loading: Transform "internal_routing.internal" has no consumers
2025-06-17T21:59:55.723334Z  WARN vector::internal_events::http_client: HTTP error. error=error trying to connect: dns error: failed to lookup address information: Name does not resolve error_type="request_failed" stage="processing" internal_log_rate_limit=true
2025-06-17T21:59:55.723386Z ERROR vector::topology::builder: msg="Healthcheck failed." error=Failed to make HTTP(S) request: error trying to connect: dns error: failed to lookup address information: Name does not resolve component_kind="sink" component_type="loki" component_id=local_loki_logs
2025-06-17T21:59:55.723398Z ERROR vector::topology::running: Sinks unhealthy.

[ayshsens]

placement failure on api and other jobs.

[jakubno]

Can you also share your screenshot from Clients in nomad? If you miss something from following list, check in GCP if they are running. And if they are running replace them (in instance group), that should make them connect to the server

[ayshsens]

we have 1 client only which is failing at logs collector and clickhouse backup
our make plan and apply does not deploy stack further.

are you referring to deleting all instance groups and reapplying again ?

[jakubno]

Can you rebuild the cluster image and redeploy? There was a bug fixed by #794, which could address the problems you describe.

[ayshsens]

thanks, we redeployed stack and api is up but failing with connection to template manager.
and there is no allocation for template manager

ithub.com/e2b-dev/infra/packages/api/internal/template-manager.(*GRPCClient).healthCheckSync
	/build/api/internal/template-manager/client.go:87
2025-06-18T06:57:48.443Z  INFO  API internal status  {"service": "orchestration-api", "internal": true, "pid": 1, "sandboxes_count": 0, "nodes_count": 0, "nodes": []}
2025-06-18T06:57:53.443Z  ERROR  failed to get health status of template manager  {"service": "orchestration-api", "internal": true, "pid": 1, "error": "rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp: lookup template-manager.service.consul on 127.0.0.53:53: no such host\""}
github.com/e2b-dev/infra/packages/api/internal/template-manager.(*GRPCClient).healthCheckSync
	/build/api/internal/template-manager/client.go:87
2025-06-18T06:57:58.443Z  ERROR  failed to get health status of template manager  {"service": "orchestration-api", "internal": true, "pid": 1, "error": "rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp: lookup template-manager.service.consul on 127.0.0.53:53: no such host\""}
github.com/e2b-dev/infra/packages/api/internal/template-manager.(*GRPCClient).healthCheckSync

[jakubno]

This issue might be due to a broken connection between the server and worker nodes. If you have a client and build node in GCP, you need to replace them so they connect to the server properly.

I recommend setting up 3 server nodes. This way, any updates to the Nomad/Consul server—though infrequent—won’t cause sync issues between the worker nodes and the server node. With 3 nodes, you can perform rolling updates without any problems.