General performance issue

[woutslakhorst]

While looking into some performance challenges I noticed some slow operations or at least operations that can cause peak load.

• JSON-LD processing is really slow. The json-gold library creates the context and validates VCs every time. The validating step does a lot of regexes.
• we do a lot of json (un)marshalling. For VC/JSON-LD processing, DID resolving and internals where we marshal/unmarshal to/from interface{}
• the memory allocations for json processing also causes GC cycles.

There are some quick fixes:

• cache DID resolve after unmarshal
• fork json-gold and disable some validation checks
• check functions/methods on interface{} arguments and remove some of the marshal/unmarshal operations where possible.

[reinkrul]

I don't think we process DID documents as JSON-LD? And we should cache did:web documents in the HTTP client, meaning if there's improvements to be made there, it's in the JSON (unmarshalling).?

[gerardsn]

encoding/json (un)marshaling is slow, which is why there are a bunch of different implementations. There is an ongoing discussion to replace the current implementation.
There is an experimental implementation for encoding/json/v2 from the go team? that also compares some alternatives with benchmarks. We could also consider using one of the alternatives to reduce JSON processing times if that makes sense. Can you post some of the results so we can make a better estimate of where to improve performance?

Most alternative implementations don't have deterministic map ordering which is probably going to break a bunch of tests.
update: I get ~50 failing tests after replacing the package with the experimental v2 package. Most errors seem te be related to how empty values are handled. This used to return nil (unmarshal) or was omitted from marshalling but now seems to add the empty value for the type. Maybe the tags changed?

[woutslakhorst]

I don't think we process DID documents as JSON-LD? And we should cache did:web documents in the HTTP client, meaning if there's improvements to be made there, it's in the JSON (unmarshalling).?

yes did resolving performance is mainly about json unmarshal. Indeed nothing to do with JSON-LD

[woutslakhorst]

For test I requested 100 access tokens in parallel every second for 60 seconds.

First is baseline.
Second has json-ld validating disabled
third has less marshalling
Click for svg

[reinkrul]

encoding/json (un)marshaling is slow, which is why there are a bunch of different implementations. There is an ongoing discussion to replace the current implementation. There is an experimental implementation for encoding/json/v2 from the go team? that also compares some alternatives with benchmarks. We could also consider using one of the alternatives to reduce JSON processing times if that makes sense. Can you post some of the results so we can make a better estimate of where to improve performance?

Most alternative implementations don't have deterministic map ordering which is probably going to break a bunch of tests. update: I get ~50 failing tests after replacing the package with the experimental v2 package. Most errors seem te be related to how empty values are handled. This used to return nil (unmarshal) or was omitted from marshalling but now seems to add the empty value for the type. Maybe the tags changed?

It was pretty easy to switch to sonnet; #3734

[reinkrul]

Plan of attack, in order of importance:

• Cache resolved DID documents client-side. Currently, it's cached by the (custom) HTTP client, but this doesn't cache the unmarshaled result (only the HTTP response bytes)
• Rewrite go-did's json.Unmarshal to not normalize the document with (un)marshalling, but unmarshal into a map and populate the VerifiableCredential struct by hand (NormalizeDocument()) is expensive.
• Change go-did's CredentialSubject property to []map[string]any
  • Use this if we fetch a known property using casting instead of (re)marshalling.
• Cache loaded DID documents server-side, this also mitigates a potential DoS vector.
• Remove "is valid field" check in the JSON-LD normalization library, which performs 2 regexes for each line

[woutslakhorst]

add a benchmark to get baseline and improvements

[reinkrul]

add a benchmark to get baseline and improvements

Can you document how you performed the test, so we can repeat it?

[reinkrul]

We need more metrics to analyze this issue. We'll be adding:

• BBolt: total time spend writing (prometheus)
• BBolt/stoabs: read lock acquire time (prometheus)
• Crypto: time spent, bucket per operation (prometheus)
• HTTP: latency per request
• SQL: query duration (prometheus), optional

See #metrics branch