Bug Report: Simple Cache Incorrectly Converted to Semantic Cache Causing Platform Outage

[leofmarciano]

What Happened?

Bug Report: Simple Cache Incorrectly Converted to Semantic Cache Causing Platform Outage

Severity: Critical
Impact: Platform-wide outage affecting all customers
Environment: Production

Description

A critical issue occurred where a simple cache implementation was inadvertently converted to semantic caching, resulting in a cache hit rate approaching 100%. This caused a complete platform failure while all customers were actively using the system.

Expected Behavior

• Simple cache should maintain exact key matching
• Cache hit rates should reflect normal usage patterns
• Platform should remain operational under standard load

Actual Behavior

• Cache began matching semantically similar requests instead of exact keys
• Cache hit rate increased to nearly 100%
• Platform became unresponsive and stopped functioning
• All active customer operations were disrupted

Impact

• Complete platform outage
• All customers unable to access services
• Business operations halted

Steps to Reproduce

1. System was operating normally with simple cache configuration
2. Cache behavior changed to semantic matching (root cause to be investigated)
3. Cache hit rate rapidly increased to ~100%
4. Platform performance degraded until complete failure

Priority

This requires immediate attention as it resulted in a production outage affecting all customers.

We are on portkey cloud, started with these PR: #1236

What Should Have Happened?

No response

Relevant Code Snippet

No response

Your Twitter/LinkedIn

No response

[leofmarciano]

#1236

[narengogi]

⚠️ Bug Report: Cache Collision Issue (Now Resolved)

While working to fix some caching issues yesterday, we inadvertently introduced a bug that caused erroneous cache collisions. This issue may have affected your requests that had cache enabled. If you were using different metadata or cache namespaces, your requests were properly isolated and unaffected.

Important notes:
The collisions did not happen between cache partitions.
The collisions did not happen across accounts
The issue has been fully reverted as of Jul 23, 2025 3:23 UTC
Our team is actively monitoring the system to ensure stability
If you experienced unexpected behavior with cached responses between Jul 22, 2025 19:41 UTC and Jul 23, 2025 3:23 UTC, please let us know in this thread.

We apologize for any inconvenience this may have caused.