v13: JWT and JWK key ids must match

[pebosi]

Environment

• PostgreSQL version: 13.3
• PostgREST version: latest / devel
• Operating system: Docker

Description of issue

Currently using latest images with this JWT Secret Config:

{
  "alg":"RS256",
  "e":"AQAB",
  "key_ops":["verify"],
  "kty":"RSA",
  "n":"ryPOQAv29sSO9jbDWkte3exY...."
}

is working with latest version, but fails with this error on devel:

No suitable key or wrong key type

[pebosi]

Using Config from Docker Swarm Secrect via this ENV's:

PGRST_JWT_SECRET: "@/run/secrets/jwt_sig"
PGRST_JWT_CACHE_MAX_LIFETIME: 900

[wolfgangwalther]

For devel/v13 we switched from https://hackage.haskell.org/package/jose to https://hackage.haskell.org/package/jose-jwt.

The error is thrown here:

postgrest/src/PostgREST/Auth.hs

Line 77 in f531476

jwtDecodeError (JWT.KeyError _) = JwtDecodeError "No suitable key or wrong key type"

It's unfortunate that we're not rethrowing the Text argument that JWT.KeyError has. As a first step we should expose that, so that we can see the underlying error here.

[taimoorzaeem]

Agree, the underlying error can be exposed as details in the error response.

[pebosi]

Tested with the artifacts from the pull request, the message is now:

{
"code":"PGRST301",
"details":"No suitable key was found to decode the JWT",
"hint":null,
"message":"No suitable key or wrong key type"
}

[pebosi]

Tried setting from file, inline in config, as base64, everything worked on latest, no version worked on devel.

[taimoorzaeem]

While I can't understand why this error is coming up, I did a little bit of tracing and this error comes from here:

canDecodeJws :: JwsHeader -> Jwk -> Bool
canDecodeJws hdr jwk = jwkUse jwk /= Just Enc &&
    keyIdCompatible (jwsKid hdr) jwk &&
    algCompatible (Signed (jwsAlg hdr)) jwk &&
    case (jwsAlg hdr, jwk) of
        (EdDSA, Ed25519PublicJwk {}) -> True
        (EdDSA, Ed25519PrivateJwk {}) -> True
        (EdDSA, Ed448PublicJwk {}) -> True
        (EdDSA, Ed448PrivateJwk {}) -> True
        (RS256, RsaPublicJwk {}) -> True
        (RS384, RsaPublicJwk {}) -> True
        (RS512, RsaPublicJwk {}) -> True
        (RS256, RsaPrivateJwk {}) -> True
        (RS384, RsaPrivateJwk {}) -> True
        (RS512, RsaPrivateJwk {}) -> True
        (HS256, SymmetricJwk {}) -> True
        .
        .

This function is not returning True on any of your keys.

[wolfgangwalther]

is working with latest version, but fails with this error on devel:

Can you show the (header of) the JWT used for the request, too?

[pebosi]

Header is this:

{
"alg":"RS256",
"typ" : "JWT",
"kid" : "....iaIux4MFz2LnGCVKWJAqVCzZKVlW....."
}

[wolfgangwalther]

Aha, so the header has a kid. I think the condition fails on keyIdCompatible (jwsKid hdr) jwk then:

https://github.com/tekul/jose-jwt/blob/1d59c9fdeb6159431ea12289784d487da86914fd/Jose/Jwk.hs#L183-L185

I read this as: If the token provides a kid, this must be set in the JWK, too. Which does not appear to be the case.

[pebosi]

No change, when adding kid to jwt-secret config.

[taimoorzaeem]

Hmm, I think it would be hard to diagnose the issue here unless we reproduce this error locally with our spec tests.

[pebosi]

The tokens are generated from an OpenId Connect Keycloak client.

[yobottehg]

Just wanted to tell that we have the same with JWT generated by Ping identity. We wanted to use V13.0 because of #3813 but the jet fails to validate afterwards with the above error message.

[steve-chavez]

@yobottehg Can you share a sample JWT so we can reproduce?

[wolfgangwalther]

I think we need a sample key + jwt, the JWT alone won't help much. Of course with a fake key, not a real one.

[laurenceisla]

I think I managed to reproduce the issue in a local Keycloak environment. Didn't check where the issue is, but in the meantime here's the relevant data:

JWK

{
  "e": "AQAB",
  "kty": "RSA",
  "n": "qyTSXmEPF6ZqXZyiH1mnVZwK_TkWkzvqaUeqN5nBkKFFHyJlgqKRpeuwaOgius67CvjQry21HKreRrq755kxP7ecOwQW-QM0qSVv8EGg5tpNc8yDu1N3DfeGaw_XCXOr-ETgIvjHNCn_6f2OaGO8qTIFH4nHZ8L0Prlxj4fU0HgzMFaeS45Y80TkxQEgAjQZE8MGtF2yZ50wu1jCjeFwVuv5qS_puuN_7dyBq5WEx-OBto_ykLyNCHFWhbbGxJC6gIIqgga7heqxBdPNqFqfGQG2gjiF171cJQCDhf6XnCkQASmlgeLZoKmg19AC0ihepZDsNjOjT57WjqFvHunD_w"
}

JWT (can be decoded in https://jwt.io/)

eyJhbGciOiJSUzI1NiIsInR5cCIgOiAiSldUIiwia2lkIiA6ICJDSUJiRUFsZ1lYbXRiXzlucy11aVlRcDE2V2RaUXZlYWJFVzlmT3dZUk9VIn0.eyJleHAiOjE3NDc0NjM3MTgsImlhdCI6MTc0NzQ2MzQxOCwianRpIjoidHJydGNjOjVjMjA2ZDE5LTcwYTUtNGY2MS1iODYzLTY0ZDA5NTQyYjhjMyIsImlzcyI6Imh0dHA6Ly9sb2NhbGhvc3Q6ODE4MS9yZWFsbXMvcG9zdGdyZXN0IiwiYXVkIjoiYWNjb3VudCIsInN1YiI6ImQ2NDZkOTFhLWViOTMtNDA5ZS04MzVmLTg5NDgwZjJlZmMwOSIsInR5cCI6IkJlYXJlciIsImF6cCI6ImNvbmZpZGVudGlhbC1jbGllbnQiLCJhY3IiOiIxIiwiYWxsb3dlZC1vcmlnaW5zIjpbIi8qIl0sInJlYWxtX2FjY2VzcyI6eyJyb2xlcyI6WyJvZmZsaW5lX2FjY2VzcyIsInVtYV9hdXRob3JpemF0aW9uIiwiZGVmYXVsdC1yb2xlcy1wb3N0Z3Jlc3QiXX0sInJlc291cmNlX2FjY2VzcyI6eyJhY2NvdW50Ijp7InJvbGVzIjpbIm1hbmFnZS1hY2NvdW50IiwibWFuYWdlLWFjY291bnQtbGlua3MiLCJ2aWV3LXByb2ZpbGUiXX19LCJzY29wZSI6Im9wZW5pZCBlbWFpbCBwcm9maWxlIiwiZW1haWxfdmVyaWZpZWQiOmZhbHNlLCJjbGllbnRIb3N0IjoiMTkyLjE2OC4yMTUuMSIsInByZWZlcnJlZF91c2VybmFtZSI6InNlcnZpY2UtYWNjb3VudC1jb25maWRlbnRpYWwtY2xpZW50IiwiY2xpZW50QWRkcmVzcyI6IjE5Mi4xNjguMjE1LjEiLCJjbGllbnRfaWQiOiJjb25maWRlbnRpYWwtY2xpZW50In0.OWpd8_vyX5nEd_Q1fcvSQkawEQnIQWFh1N7K7Rr89JGP4aoLIg3XLAUbXzJMvAJCfj0kjM4MJp8P8sHvwXuvD8jqIafX63Ypy9o68d-cnhYZyd-0n14OpWaX0Fs5YRq5gFaRd1J3zePnZsbKGPMarMbG51TejoowrQcgyp-U3G8ayl7O3tmHt8UH2dRYktMHD7MbcfVAJT_1fTjZlwa-4FIZHxpo2Bay1xTRNp9ahR8Jc24PQa2rnN476vtMNC33mK9jLvWQDT8BA7GEb4sHfibvpgA9X4GzBs2JlwF-sG9PycjzRPM4uLf7cTCIO6oEBY8cmjj9nwGlBVAQiCMPVA

Similar example working in v12 (should return a "JWT expired" message by now), and failing with OP's error in v13:

curl 'localhost:3000/projects' -H 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsInR5cCIgOiAiSldUIiwia2lkIiA6ICJDSUJiRUFsZ1lYbXRiXzlucy11aVlRcDE2V2RaUXZlYWJFVzlmT3dZUk9VIn0.eyJleHAiOjE3NDc0NjM3MTgsImlhdCI6MTc0NzQ2MzQxOCwianRpIjoidHJydGNjOjVjMjA2ZDE5LTcwYTUtNGY2MS1iODYzLTY0ZDA5NTQyYjhjMyIsImlzcyI6Imh0dHA6Ly9sb2NhbGhvc3Q6ODE4MS9yZWFsbXMvcG9zdGdyZXN0IiwiYXVkIjoiYWNjb3VudCIsInN1YiI6ImQ2NDZkOTFhLWViOTMtNDA5ZS04MzVmLTg5NDgwZjJlZmMwOSIsInR5cCI6IkJlYXJlciIsImF6cCI6ImNvbmZpZGVudGlhbC1jbGllbnQiLCJhY3IiOiIxIiwiYWxsb3dlZC1vcmlnaW5zIjpbIi8qIl0sInJlYWxtX2FjY2VzcyI6eyJyb2xlcyI6WyJvZmZsaW5lX2FjY2VzcyIsInVtYV9hdXRob3JpemF0aW9uIiwiZGVmYXVsdC1yb2xlcy1wb3N0Z3Jlc3QiXX0sInJlc291cmNlX2FjY2VzcyI6eyJhY2NvdW50Ijp7InJvbGVzIjpbIm1hbmFnZS1hY2NvdW50IiwibWFuYWdlLWFjY291bnQtbGlua3MiLCJ2aWV3LXByb2ZpbGUiXX19LCJzY29wZSI6Im9wZW5pZCBlbWFpbCBwcm9maWxlIiwiZW1haWxfdmVyaWZpZWQiOmZhbHNlLCJjbGllbnRIb3N0IjoiMTkyLjE2OC4yMTUuMSIsInByZWZlcnJlZF91c2VybmFtZSI6InNlcnZpY2UtYWNjb3VudC1jb25maWRlbnRpYWwtY2xpZW50IiwiY2xpZW50QWRkcmVzcyI6IjE5Mi4xNjguMjE1LjEiLCJjbGllbnRfaWQiOiJjb25maWRlbnRpYWwtY2xpZW50In0.OWpd8_vyX5nEd_Q1fcvSQkawEQnIQWFh1N7K7Rr89JGP4aoLIg3XLAUbXzJMvAJCfj0kjM4MJp8P8sHvwXuvD8jqIafX63Ypy9o68d-cnhYZyd-0n14OpWaX0Fs5YRq5gFaRd1J3zePnZsbKGPMarMbG51TejoowrQcgyp-U3G8ayl7O3tmHt8UH2dRYktMHD7MbcfVAJT_1fTjZlwa-4FIZHxpo2Bay1xTRNp9ahR8Jc24PQa2rnN476vtMNC33mK9jLvWQDT8BA7GEb4sHfibvpgA9X4GzBs2JlwF-sG9PycjzRPM4uLf7cTCIO6oEBY8cmjj9nwGlBVAQiCMPVA'

[Yasirunet]

I encountered a similar issue with Keycloak and PostgREST (v13). Adding the kid (Key ID) to the JWK resolved the problem.

Here's an example JWK configuration for PostgREST

{
  "kty": "RSA",
  "alg": "RS256",
  "kid": "your-key-id",
  "n": "xxxxxxxxxxxxxxxxxx",
  "e": "AQAB"
}

[laurenceisla]

Adding the kid (Key ID) to the JWK resolved the problem.

Yup can confirm! In my example above, adding the kid works:

{
  "e": "AQAB",
  "kty": "RSA",
  "n": "qyTSXmEPF6ZqXZyiH1mnVZwK_TkWkzvqaUeqN5nBkKFFHyJlgqKRpeuwaOgius67CvjQry21HKreRrq755kxP7ecOwQW-QM0qSVv8EGg5tpNc8yDu1N3DfeGaw_XCXOr-ETgIvjHNCn_6f2OaGO8qTIFH4nHZ8L0Prlxj4fU0HgzMFaeS45Y80TkxQEgAjQZE8MGtF2yZ50wu1jCjeFwVuv5qS_puuN_7dyBq5WEx-OBto_ykLyNCHFWhbbGxJC6gIIqgga7heqxBdPNqFqfGQG2gjiF171cJQCDhf6XnCkQASmlgeLZoKmg19AC0ihepZDsNjOjT57WjqFvHunD_w",
  "kid": "CIBbEAlgYXmtb_9ns-uiYQp16WdZQveabEW9fOwYROU"
}

No change, when adding kid to jwt-secret config.

@pebosi, can you retry to verify this?

[antonymott]

Attn: as workaround for MacOS users who use jwt-secret is to set key "kid" = undefined. This may help others who upgraded PostgREST to 13.0.0 with homebrew, and who ran into this issue.

Summary

Workaround we eventually found for us, until the bug is fixed:
sample postgrest.conf

db-uri = "..."
db-schemas = "public"
jwt-secret = "someSecretAtLeast31CharsLong"
server-port = 3001

Our setup:

Postgresql: 17
PostgREST: 13.0.0
macOS Sequoia 15.4.1

Unlike other unix users who can easily downgrade from 13.0.0 to 12.2.17, we found on the gitHub repo no easy way to downgrade to PostgREST 12.2.17 using brew. For example brew install [email protected] does not work, and brew list postgrest lists only "PostgREST" with no explicit version. However, this is expected behavior for many formulae.

Also, in many uses, the "hint" key is not sent, only the message:

{
  "code": "PGRST301",
  "details": null,
  "hint": null,
  "message": "No suitable key or wrong key type"
}

Just using the message from 13.0.0, AI/google first suggest the Schema needs reloading (message does not actually mention jwt). It took us a while to narrow down the cause of the issue to jwt.

Hope this helps other MacOS users.

Great thanks to the maintainers of this extraordinarily useful rest api.

[wolfgangwalther]

Workaround we eventually found for us, until the bug is fixed, was to set "kid" key to undefined

Can you clarify - did you set kid to undefined in the JWT (token) or in the JWK provided to jwt-secret ?

[antonymott]

@wolfgangwalther: we set "kid" = undefined only in the JWT (token), in our postgrest.conf we have jwt-secret but no "kid" key. I updated my answer and put our sample postgrest.conf in case that helps

[pebosi]

The kid is already contained in my JWT Header.

[wolfgangwalther]

The kid is already contained in my JWT Header.

If the kid is in the JWT, then it must be in the JWK (the key in jwt-secret) as well. And it must match.

[pebosi]

I thought i already checked this. Re-checked again and when kid is in secret and in the jwt header it works!

[wolfgangwalther]

Cool!

So at this stage, I think this becomes a documentation issue. The new behavior is correct, I think, but we should document it.

[laurenceisla]

The new behavior is correct, I think, but we should document it.

Yeah, just to add some info that backs using the kid to validate the JWT:

The JWS RFC mentions: "When used with a JWK, the "kid" value is used to match a JWK "kid" parameter value." Our library interprets this as if it should be enforced (this is internal and cannot be changed); there are others that suggest the same too, e.g. Microsoft Entra also mentions this: "Use the kid claim to validate the token".

---

So at this stage, I think this becomes a documentation issue

I think we need to add some tests too. We only check without kid and for a JWK set with a single element.

postgrest/test/spec/SpecHelper.hs

Line 226 in 1258ea6

let secret = encodeUtf8 [str|{"keys": [{"alg":"RS256","e":"AQAB","key_ops":["verify"],"kty":"RSA","n":"0etQ2Tg187jb04MWfpuogYGV75IFrQQBxQaGH75eq_FpbkyoLcEpRUEWSbECP2eeFya2yZ9vIO5ScD-lPmovePk4Aa4SzZ8jdjhmAbNykleRPCxMg0481kz6PQhnHRUv3nF5WP479CnObJKqTVdEagVL66oxnX9VhZG9IZA7k0Th5PfKQwrKGyUeTGczpOjaPqbxlunP73j9AfnAt4XCS8epa-n3WGz1j-wfpr_ys57Aq-zBCfqP67UYzNpeI1AoXsJhD9xSDOzvJgFRvc3vm2wjAW4LEMwi48rCplamOpZToIHEPIaPzpveYQwDnB1HFTR1ove9bpKJsHmi-e2uzQ","use":"sig"}]}|]

It seems that the previous library didn't care about the kid at all. For example for this JWK set in jwt-secret:

{
  "keys": [
    {"kid": "not the correct kid in JWT", ...},
    {"kid": "the correct kid in JWT", ...}
  ]
}

In v12 it takes the first JWK and doesn't authenticate (uses the anon role by default). In v13 it gets the correct JWK (the second one) and authenticates. So what we have right now is an improvement. I'm surprised nobody reported this, unless there's another method used to select the correct JWK or maybe I'm missing something?

[laurenceisla]

It seems that the previous library didn't care about the kid at all [...] In v12 it takes the first JWK and doesn't authenticate (uses the anon role by default).

OK, I made some mistake while testing here. So, I tried to reproduce this again for v12, but I couldn't... the validation works correctly for v12. It still doesn't look for or validate the kid, but it will go through all the JWKs in the set until it finds the correct one and succeed (or fail with "JWSInvalidSignature" if none is found).

I'm surprised nobody reported this, unless there's another method used to select the correct JWK or maybe I'm missing something?

So yeah, not surprising at all since this was working.

[SidPatel-TH]

Does this apply to the case of symmetric keys and setting the jwt-token to a simple string? A couple days ago we upgraded to v13, and after facing PGRST301 error, had to downgrade back to v12. Our jwt never had kid and it works fine after downgrading.

[wolfgangwalther]

If neither your key, nor your token, had kid, then this must be a different problem. In that case, please open a new issue with more details and ideally a reproducer.