Performance trade offs between initialization and validation speed

[ottomated]

I was looking into migrating from Zod and was drawn by the validation benchmarks on both the homepage and blog post. My intuition, however, was that the expensive schema initialization step would make arktype a mistake for serverless environments such as Cloudflare workers where schemas are reinitialized on each request.

I verified it with a quick benchmark that shows a significant performance hit that unfortunately more than outweighs the gain on validation speed:

import { run, bench, boxplot, summary } from 'mitata';
import { z } from 'zod';
import { type } from 'arktype';

boxplot(() => {
  summary(() => {
    bench('zod', () => z.object({
      string: z.string(),
      stringRange: z.string().min(0).max(10),
      optional: z.array(z.number()).optional(),
    }));
    bench('arktype', () => type({
      string: 'string',
      stringRange: '0 < string <= 10',
      'optional?': 'number[]',
    }));
  });
});

await run();

I understand that the library is not focused on initialization performance, but it feels a bit misleading to advertise a 100x speedup when a significant portion of your users will actually experience the opposite. Would it be possible to add a note mentioning this to the relevant docs pages?

[gregg-cbs]

I agree on this. One of the reasons we moved from zod was the performance benefits. Although this does not stray us from arktype, it solves some problems zod could not - like good union types. In mongodb this is essential.

Good post OP.

[ssalbdivad]

Yeah, it would be valuable to give a more nuanced breakdown of the tradeoffs here. I will try to do that in an upcoming comparisons section I have planned for the docs.

That said, I wouldn't say the existing framing is going out of its way to be misleading. It just references the only community-maintained validation benchmarks that exist. 100% agree it would be valuable for those to cover other scenarios like initialization or complex unions.

However, for environments sensitive to initialization speed/bundle size, there will be an opt-in build-step to avoid these costs in an upcoming release:

#810

For now, you may be interested in jitless mode. There will still be some overhead to intializing the fully-reduced types, but it would help mitigate it at the expense of validation speed.

[johnpyp]

I think an important consideration here is that in environments where initialization performance is important, the relative performance of any validation library like Zod or ArkType will be completely outweighed by the runtime startup cost. It's multiple orders of magnitude. Any difference in validation performance will be likely literally impossible to measure in those scenarios just because of jitter noise in the time it takes to start up the runtime.

Any benchmark that considers initialization performance is almost guaranteed to not actually make an impactful difference in any real-world application.

Additionally, I'm curious to see if someone makes a combined benchmark which measures validation against a large input document, if the costs in startup time go away. In other words, if you're in a situation where you do in fact care about performance and startup time (parsing a large object in a serverless runtime) , do the benefits in startup time of Zod even matter?

[johnpyp]

Another thing to consider - Cloudflare Workers actually isn't a great analogy here, because they reuse global variables across multiple requests (the isolate that's started up persists to serve additional requests over some period of minutes). So as long as you aren't freshly initializing the schema variables in the request loop, you should benefit from it under load.