Benchmarks

[danini-the-panini]

Has anyone made any benchmarks for testing KDL implementation performance? I'm thinking of writing some to compare between my pure-ruby implementation and a ckdl ruby binding I just wrote.

Any ideas for some particularly gnarly edge cases to consider, e.g. 10,000 nested children.

I'm also considering looking for some XML benchmarks and porting them over with XiK.

[zkat]

I don’t even want to think about how slow kdl-rs would be on big docs

i am personally opposed to benchmarking because it would be personally embarrassing and harmful to my reputation 🫣🫣🫣

[zkat]

(I think this is a good idea and would be very useful and tbh we can probably just start with benchmarking against the examples/, which should be fairly representative of real-world docs)

[danini-the-panini]

I guess just running against the existing examples in the examples folder would be a good start. My dart implementation would take several seconds parsing the scheme one, until I realised I didn't have to try decode every character as a UTF code point 🤦🏻‍♀️

[tabatkins]

The HTML spec is still pretty useful as a browser stress test. I could write a quick converter to turn it into KDL...

[zkat]

@danini-the-panini haha it’s fine. I’m genuinely surprised kdl-rs is that fast in the first place. It’s incredibly inefficient about backtracking everywhere and allocates all over the place. I can probably do better but tbh this seems fast enough that I can focus on other high-pri work :)

[zkat]

As a side note, if you’re evaluating ckdl vs kdl-rs bindings, I would recommend checking if ckdl can do error recovery (and multi-error collection)

[bgotink]

I'm at 1.19 seconds for my js parser and 1.6 seconds to parse and format.
Of course comparisons between any of the numbers posted are inherently invalid unless everyone's using the exact same device on the exact same load… and as Kat mentioned, some implementations made wildly different choices.

[zkat]

I do also want to point out: I think for the foreseeable futurefuture, KDL is going to primarily be used for hand-authored configurations. It’s very unlikely for those configs to be 20mb, or for an equivalent number of smaller configs to need rapid processing. I think if and when we start seeing people using KDL as a network exchange format to the extent JSON is, perf will start mattering a lot more and at least for kdl-rs, that’s when I plan on revisiting that

[tabatkins]

Right, I'm not overly concerned with my runtime right now (for any reasonable-sized documents it's plenty fast; runtimes are dominated by Python startup), I'm just wondering if there's an accidental quadratic somewhere or if I'm just paying for way too many allocations or something.

[eugenesvk]

Oh, that'd be very useful even for regular use as some parsers can be noticeably slow even for regular config use cases!

A benchmark idea (based off a slightly non-trivial app configuration):

• input: a dozen of low complexity files (simulating one config importing other configs): ~a few hundred lines max each, a few nodes with a dozen children, each child having a few args/props of different value types with/without tags
• parsing:
  • preparing a list of values for properties with given parent node / node / property name (simulating finding a specific config value)
  • preparing a list of all arguments with values having some predefined tag