Thumper Load Testing Fails with "DeadlineExceeded" Against SpiceDB via Kubernetes Port-Forwarding

[saytoaditya]

Title:
Thumper Load Testing Fails with "DeadlineExceeded" Against SpiceDB via Kubernetes Port-Forwarding
Description:
I'm trying to perform load testing on SpiceDB using Thumper but keep getting DeadlineExceeded errors. Here's my complete setup:

1. Environment
   • SpiceDB running in Kubernetes (spicedb-operator-1 namespace)
   • Thumper CLI (version 0bc1b8762bac) running locally
   • Using port-forwarding to connect to SpiceDB:
   • kubectl port-forward deployment/dev-spicedb 50051:50051 -n spicedb-operator-1
2. Thumper Configuration (thumper_test.yaml):

• name: "check_permission"
weight: 1
steps:
- op: "CheckPermission"
resource: "module:BusinessTransaction"
permission: "approver"
subject: "user:aksdah"

3.Command Used: thumper run thumper_test.yaml --token "xxxx"

4. Error Received:

• {"severity":"warn","script":"check_permission","step":0,"worker":0,"op":"CheckPermission",
  "error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded",...}

5.What I've Tried:

• Verified port-forwarding is active (Handling connection for 50051 appears)
• Confirmed the preshared key matches SpiceDB's configuration
• Increased timeout in YAML configuration to "10s"
• Tested basic connectivity with grpcurl (works intermittently)
• Checked SpiceDB pod logs - no obvious errors
• Tried with both localhost and 127.0.0.1

6. Additional Context:

   • A simple Python client works sometimes but fails under load
   • SpiceDB resources appear adequate (CPU/Memory not maxed out)
   • Network connectivity seems fine (can telnet to port 50051)

Question:

• What could be causing these consistent deadline exceeded errors during Thumper load testing, and how can I properly diagnose and resolve them? Are there specific:
• Kubernetes/network configurations I should check?
• Thumper settings I might be missing?
• SpiceDB server-side configurations that could help?
• Alternative approaches to connecting Thumper to a Kubernetes-hosted SpiceDB?

[tstirrat15]

I wouldn't expect kubectl port-forwarding to be a good representative load test, because kubectl becomes a part of the networking stack and therefore is a part of your load test. I don't think it's optimized for networking performance.

The behavior you're seeing is still strange; it sounds like thumper isn't calling the endpoint that you think it is.

I'd recommend either running the thumper instance in your cluster as its own workload or setting up proper external ingress and then running the load test through that ingress.

[saytoaditya]

thanks @tstirrat15 for the update. any document which can help me to run through external ingress or how to run via its own workload?