Can we make validation faster for common scenarios -- especially on mobile devices?

[swcurran]

From @stormer78

There are kind of two scenario's I see:

1. Regular resolution of the DID using the latest published version - likely to be the highest use case by high frequency services like messaging systems etc.
2. Historical lookups for issued data from the past (critical to support, likely lowers in frequency over the years).

Right now webvh has the inverse problem. The oldest LogEntries (which are likely to be lowest frequency over time) are the cheapest to resolve. The latest version which is likely to be highest frequency has the highest cost to resolve (cost being CPU/time). Especially on mobile devices or wearables.

Ideally a shortcut for the latest version that provides the option to do a quick resolution with least amount of checking would be best.

In some ways the witnesses can help if we apply the logic that for a witness to provide proof, they are doing a full end-to-end validation and is that enough that if the latest LogEntry is correctly witnessed, check the integrity of the newest Entry and Proofs - and it is ok to use.

If you want to do a full validation, feel free, but it is not always required. The more the heavy lifting can be shifted back to the controller, witnesses and watchers - the better. Client side should be as cheap as we can make it (if possible).

TTL will help with caching this, but that initial resolution is high.

[swcurran]

Your observations are accurate, but I think there are some things in the specification that allow for some shortcuts without the loss of verifiability.

If you have no cache and no other source for data about a given DID, then there is no choice for a resolver but to process the full file to ensure verifiability. That is a given. I did mention in my deep dive at the W3C, making the equivalent of a did:web retrieval of a did:webvh is dead simple -- grab the file, find the last entry, and grab the value of the state entity, and you are done. No verification, but if that is OK with you -- you are set. We won't be using that approach :-).

If a resolver has a cache, they can keep some state around to reduce the verification effort on subsequent retrievals. Notably, if you keep the currently active parameters, the latest versionId, perhaps the latest DIDDoc (in case it hasn't changed) and the length of the Log when last accessed, you are able to:

• Use an HTTP header and the cached file length to only retrieve the part of the file added from the last retrieval. That said, this may not be worth it vs. just getting the full file and jumping into the file at the point already processed. Either way, this is one reason why JSONL is used -- it supports jumping straight to the "new stuff" in a re-retrieved Log.
• The remaining entries (if any) in the file can be properly processed from that point based on the held state. As you note in your other comments, the witness.json file may also need to be downloaded.

For a mobile agent -- using a server-based resolver that is trusted can greatly reduce the effort of the mobile/wearable device. Instead of getting the did:webvh data from the source and then processing it, call out to a "trusted" resolver, and get back only what you need -- the relevant DIDDoc or resource. Leave it to the resolver to determine how best to manage the DIDs and limit the verification processing. What makes for a 'trusted" resolver is out of scope of the spec -- that governance topic again.

I'm not sure what else can be done to reduce the processing time while retaining verifiability but definitely open to ideas.

[stormer78]

It is a tough one to crack.

Addressing it in a "performance" or "optimisation" section in the spec doc or webvh implementors guide around some best practices would help immensely.

DID Resolvers cashing the last verified version ID is a good suggestion; the best result is likely to be the only need to validate forward from that.

Agreed on jsonl - it is a lot easier to work with and efficient to stream a validation process from than regular old json.

Is there an option to skip LogEntry version-iD's? In some cases if the LogEntry isn't changing the state (DID Document) then do clients care that it is a parameters only change? (i.e. adding/removing witness or watcher, or rolling the updateKeys every 7 days - this could be a in-place replacement of the version-ID and potentially cut down on number of entries to process.

History is more important for DID Doc changes, not sure if any one outside of the controller (and witness/watchers) would care about parameter set changes - could be another possible area of optimisation.

Will run some scaled tests and come back with some real numbers on what this looks like across hundreds or thousands of log entries

[swcurran]

Funny you mention non-DIDDoc updates. Our first versions used JSON-DIFF for the DIDDoc on that premise -- that changes to DIDDocs would be insubstantive and so using a DIFF would be useful. However, we found the diffs were often larger than the actual DIDDoc, and it was an added dependency, so we dropped it. But if what you are proposing is a common model -- that many Log Entries don't alter the DIDDoc, it is one we could put back.

If you do get a load test setup, that might be a fun variation to try. Does the use of JSON-DIFF on the DIDDoc help in scenarios? If so, we could consider bring it back in a later version.