High load on ORY Keto database with increased RPS on permission checking (self-hosted)

[DenisPnko]

Preflight checklist

• I could not find a solution in the existing issues, docs, nor discussions.
• I agree to follow this project's Code of Conduct.
• I have read and am following this repository's Contribution Guidelines.
• I have joined the Ory Community Slack.
• I am signed up to the Ory Security Patch Newsletter.

Ory Network Project

No response

Describe the bug

(help wanted) Hi all, I have run into an issue with using ORY Oathkeeper as a reverse proxy and ORY Keto for authorisation and permission checking. With increased RPS (> 1000 req/sec) I have noticed that the database used for checking ORY Keto permissions gets overloaded because for every request from ORY Oathkeeper (authorizer: remote_json) ORY Keto makes a query to the database to check the namespace relationship.

Is this a normal behaviour? I wonder if there is a an option either on the Oathkeeper or Keto side to implement caching or a temporary in memory storage for permission checking and not having a separate call to the database on each permission checking request. I have looked through the configuration for both and haven't found a solution, would greatly appreciate your help or advice.

Reproducing the bug

ORY Oathkeeper (v0.40.6) Reverse proxy -> remote_json authorizer (ORY keto read endpoint) ->

ORY Keto (v0.14.0) (simple namespace)

~1000 requests/sec for permissions checking

Relevant log output

A separate query to the ORY Keto database with each permissions check.

Relevant configuration

Version

Keto v0.14.0, Oathkeeper v0.40.6

On which operating system are you observing this issue?

Linux

In which environment are you deploying?

Kubernetes with Helm

Additional Context

No response