feat(backend,nextjs): Introduce machine authentication

[wobsoriano]

Description

This PR adds machine authentication support (atm only in the backend SDK) by introducing support for 4 token types: api_key, oauth_token, machine_token, and session_token. To maintain backwards compatibility, session_token remains the default authentication method when no specific token type is specified. This ensures existing apps continue to work without modification while allowing new applications to opt-in to machine authentication methods through the acceptsToken option.

Key changes:

• Deprecated SignedInState and SignedOutState in favor of AuthenticatedState and UnauthenticatedState to better represent both session and machine authentication states. They still return the same properties, with an added tokenType and isAuthenticated properties (deprecating isSignedIn).
• The toAuth() method now returns a different value if the tokenType is not a session_token. For now, we landed on the id, name, subject, claims and scopes property for machine auth tokens.
• Added two new internal functions in authenticateRequest: authenticateAnyRequestWithTokenInHeader and authenticateMachineRequestWithTokenInHeader to handle machine authentication.
• The internal signedIn and signedOut functions have been updated to accommodate machine auth.
• Added new error types and codes specific to machine token verification (MachineTokenVerificationErrorCode)
• Added new APIs (APIKeysApi, IdPOAuthAccessTokenApi, and MachineTokensApi) used inside a new verifyMachineAuthToken function to validate tokens against their respective endpoints
• Added test for various scenarios for token validation, handling different token types, token mismatch, and proper error responses when verification fails

Here's an example usage pattern with API key:

Say C1 wants to protect their endpoints in a Hono app:

import { serve } from '@hono/node-server'
import { createMiddleware } from 'hono/factory'
import { Hono } from 'hono'
import { clerkClient } from './client'
import { HTTPException } from 'hono/http-exception'

const app = new Hono()

const clerkMiddleware = createMiddleware(async (c, next) => {
  const authReq = await clerkClient.authenticateRequest(c.req.raw, {
    acceptsToken: 'api_key'
  })

  if (!authReq.isAuthenticated) {
    throw new HTTPException(401, { message: 'Unauthorized' })
  }

  await next()
})

app.post('/api/protected', clerkMiddleware, async (c, next) => {
  return c.text('Hello from /api/protected')
})

Then C2 can access it by passing the api_key:

const resp = await fetch('http://localhost:3000/api/protected', {
  method: 'POST',
  headers: {
    'Content-Type': 'application/json',
    'Authorization': `Bearer ${process.env.API_KEY}`
  },
})

const data = await resp.text()

P.S. I attempted to break this down into smaller PRs but the changes are tightly coupled 😞. So sorry and thank you in advance reviewer! I believe 30-40% of the total changes are from the test files.

Resolves ROBO-36

Checklist

• pnpm test runs as expected.
• pnpm build runs as expected.
• (If applicable) JSDoc comments have been added or updated for any package exports
• (If applicable) Documentation has been updated

Type of change

• 🐛 Bug fix
• 🌟 New feature
• 🔨 Breaking change
• 📖 Refactoring / dependency upgrade / documentation
• other:

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
clerk-js-sandbox	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	May 7, 2025 9:33pm

[changeset-bot]

🦋 Changeset detected

Latest commit: 178e823

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 11 packages

Name	Type
@clerk/backend	Major
@clerk/tanstack-react-start	Minor
@clerk/agent-toolkit	Minor
@clerk/react-router	Minor
@clerk/express	Minor
@clerk/fastify	Minor
@clerk/astro	Minor
@clerk/remix	Minor
@clerk/nuxt	Minor
@clerk/nextjs	Minor
@clerk/testing	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[jescalan]

Looking great so far!

[jescalan]

Just wanna note that we do plan to allow custom prefixes in the future - likely these end up being prepended to the token type prefix so i think it should be a fairly straightforward change

[wobsoriano]

Added a note to cover this in the future.

[jescalan]

I'm mildly confused by the typecasting here

[wobsoriano]

Yeah the as const is needed here so TS knows these are literal types that match the keys in our mock objects, otherwise it would just see it as string[]

[jescalan]

Something seems weird about this logic...

[brkalow]

I believe in practice this shouldn't be hit, as we check the existence of the header token before calling this method.

[wobsoriano]

Yeah either that or we remove and do non-null assertions

[LekoArts]

So do we want/need to add a TODO to change this behavior in the future?

[wobsoriano]

I don't think there are plans to change the behavior in the future yet. But I will update as soon as I get more info!

[LekoArts]

I personally prefer APIs where there is no mix between string and array of strings.

So default would be ['session_token']. Then I don't need to wrap it with an array when I quickly want to switch to two token types.

[wobsoriano]

We need a minor update for these packages since we changed the auth type from AuthObject to SignedInAuthObject | SignedOutAuthObject for backwards compat

[wobsoriano]

Even though this is a major update, the only public API that is breaking is AuthObject. Previously, it's a union of SignedInAuthObject | SignedOutAuthObject but now it's

export type AuthObject =
  | SignedInAuthObject
  | SignedOutAuthObject
  | AuthenticatedMachineObject
  | UnauthenticatedMachineObject;