JSON formatter/linter performance could be improved?

[dalisoft]

I am creating all-in-one linter (yes, another one) and found Biome isn't fastest JSON linter/formatter yet.
Here benchmark repo: https://github.com/dalisoft/json-lint-benchmark

Here project which does JSON linting/formatting very fast: https://github.com/jsona/jsona
I am migrated project into JS ecosystem by using download binary method, see at here if you want check

package-lock.json

Command	Mean [ms]	Min [ms]	Max [ms]	Relative
./node_modules/@biomejs/biome/bin/biome format data/package-lock-test.json	51.3 ± 1.7	50.3	53.3	7.94 ± 1.08
./node_modules/biome-rs-npm/biome format data/package-lock-test.json	6.9 ± 1.1	6.2	8.2	1.07 ± 0.22
prettier -c data/package-lock-test.json	176.2 ± 32.9	155.6	214.1	27.25 ± 6.23
./node_modules/dprint/bin.js check data/package-lock-test.json	53.1 ± 1.6	51.2	54.0	8.21 ± 1.11
./node_modules/dprint-rs-npm/dprint check data/package-lock-test.json	12.3 ± 0.7	11.6	13.0	1.91 ± 0.27
deno fmt data/package-lock-test.json	7.6 ± 0.2	7.4	7.8	1.18 ± 0.16
jsona fmt --option trailing_newline=true --check data/package-lock-test.json	6.5 ± 0.9	5.7	7.4	1.00
spectral lint --ignore-unknown-format data/package-lock-test.json	385.7 ± 1.9	384.6	387.8	59.65 ± 7.87

package.json

Command	Mean [ms]	Min [ms]	Max [ms]	Relative
./node_modules/@biomejs/biome/bin/biome format data/package-template.json	71.9 ± 49.9	42.9	129.5	13.86 ± 15.18
./node_modules/biome-rs-npm/biome format data/package-template.json	8.3 ± 8.9	2.8	18.6	1.60 ± 2.19
prettier -c data/package-template.json	112.1 ± 4.6	107.8	117.0	21.62 ± 18.33
./node_modules/dprint/bin.js check data/package-template.json	75.7 ± 31.9	49.0	111.1	14.60 ± 13.81
./node_modules/dprint-rs-npm/dprint check data/package-template.json	12.1 ± 7.5	7.6	20.8	2.34 ± 2.46
deno fmt data/package-template.json	20.9 ± 23.0	7.1	47.4	4.04 ± 5.59
jsona fmt --option trailing_newline=true --check data/package-template.json	5.2 ± 4.4	2.5	10.3	1.00
spectral lint --ignore-unknown-format data/package-template.json	423.1 ± 75.9	377.4	510.7	81.58 ± 70.62

[arendjr]

Please keep in mind that Biome uses a daemon architecture for communication with both the CLI and the VS Code extension. This does cause some overhead which is especially noticeable if you run it against a single small file. I have no illusions for Biome to be the fastest (even though it is fast), but running it against a single small file is kind of its worst-case scenario :) When you use more/larger files at least some of that overhead should be amortized.

[ematipico]

I'm not really sure how the benchmark is working. I tried locally and here's the result

hyperfine --runs 3 "npx @biomejs/biome format ./data/package-lock-test.json" --show-output
Benchmark 1: npx @biomejs/biome format ./data/package-lock-test.json
Checked 1 file in 6ms. No fixes needed.
Checked 1 file in 1955µs. No fixes needed.
Checked 1 file in 1938µs. No fixes needed.
  Time (mean ± σ):     284.4 ms ±  81.0 ms    [User: 238.8 ms, System: 54.3 ms]
  Range (min … max):   237.4 ms … 377.9 ms    3 runs

  Warning: The first benchmarking run for this command was significantly slower than the rest (377.9 ms). This could be caused by (filesystem) caches that were not filled until after the first run. You should consider using the '--warmup' option to fill those caches before the actual benchmark. Alternatively, use the '--prepare' option to clear the caches before each timing run.

I don't know why hyperfine marks 81.0 ms as range.

[ematipico]

There's one thing to note, running using npx isn't ideal because we incur into npm delays. I suggest downloading Biome using homebrew or downloading the binary. That's where you can test the real speed without any external factors.

[dalisoft]

@ematipico Or binary linking locally helps with performance for Node.js projects. See updated benchmark result

[Conaclos]

If I remember correctly, Deno uses dprint. Why is there a such difference between deno fmt and dprint?

[ematipico]

Caching: dprint/dprint-plugin-json#35

[dsherret]

When deleting deno's cache before each run to cause it to not read from a cache, the performance is about the same in this benchmark because the actual JSON file formatting time is < 2ms.

The main reason was actually dprint was loading system certificates on startup for fetching the Wasm plugins instead of doing it lazily and that was very slow on Mac/Linux. I did a new release of dprint fixing the issue and now they're about the same speed dprint/dprint-plugin-json#35 (comment) (see second set of benchmarks in that post... one thing is it seems deno fmt is slower at outputting a diff because they use a different set of code to do that)

[arendjr]

Doesn't this highlight that your benchmark is really just about measuring startup performance rather than JSON formatting or linting? I'm a little confused at what value we should derive from these numbers...

[dsherret]

That's what I said in the issue (dprint/dprint-plugin-json#35 (comment)). The numbers in the original benchmark of this post aren't even running JSON formatting in dprint because it was using the cache. This isn't my benchmark.

[dalisoft]

@arendjr When using CLI, on lint-staged or CI these numbers really could matter. I am benchmarked every JSON lint/formatter i've found and know as startup time -> runtime -> finish time. Seems there maybe could be improved.
And binary linking locally helped with performance a lot.