support JSON Merge Patch (RFC 7396) diff creation

[satelliteprogrammer]

The JSON merge patch document format describes the set of modifications to a resource's content, that more closely mimics the syntax of the resource being modified.

However, and in contrast to JSON Patch (RFC 6902), a JSON Merge Patch cannot express certain modifications, e.g., changing an array element at a specific index, or setting a specific object value to null. The null value in a JSON Merge Patch is used to remove the key from the object.

The diff algorithm is not part of the RFC 7396, but it was tested against all examples provided, plus additional cases on how null values are handled.

JSON Merge Patch PR: #876
PR discussing the diff: #2018

If the content is approved, please let me know if/what documentation needs to be updated.

[Describe your pull request here. Please read the text below the line and make sure you follow the checklist.]

• The changes are described in detail, both the what and why.
• If applicable, an existing issue is referenced.
• The Code coverage remained at 100%. A test case for every new line of code.
• If applicable, the documentation is updated.
• The source code is amalgamated by running make amalgamate.

Read the Contribution Guidelines for detailed information.

[github-actions]

🔴 Amalgamation check failed! 🔴

The source code has not been amalgamated. @satelliteprogrammer
Please read and follow the Contribution Guidelines.

[coveralls]

coverage: 99.194% (+0.003%) from 99.191%
when pulling d32edb3 on satelliteprogrammer:develop
into 29913ca on nlohmann:develop.

[nlohmann]

I am not sure if this feature would be widely used, so I'm opening a discussion.

[satelliteprogrammer]

We had a (somewhat niche) need for this at work, that's why I've contributed here.

It's essentially a combination of 2 things:

1. we log all our data structures in JSON, this makes it easier to parse the logs and pretty-format large data structures;
2. we have a container that only notifies listeners on data changes.

For very large data structures, reading through a log line to find the one variable that did change is a pain. Not only that, but on some interfaces only a small amount of member variables on the entire structure are actually changing.
Given that we are already serializing in JSON, on those containers that track differences, we thought why not log only those differences using one of the available JSON patch methods. So here we are.

[github-actions]

This pull request has been marked as stale because it has had no activity for 30 days. While we won’t close it automatically, we encourage you to update or comment if it is still relevant. Keeping pull requests active and up-to-date helps us review and merge changes more efficiently. Thank you for your contributions!

[cschreib-ibex]

[..] we thought why not log only those differences using one of the available JSON patch methods. So here we are.

We had this exact need as well: logging changes in simple JSON structures without all the noise of the formal JSON Patch syntax. Would have been convenient to have this capability built in.

[cschreib-ibex]

This may be overly constraining. An alternative would be to interpret "field set to null" as "field removed".

The RFC isn't very clear on this. While it says:

This design means that merge patch documents are suitable for describing modifications to JSON documents that primarily use objects for their structure and do not make use of explicit null values.

It also lists examples in appendix where some fields are set to null in the source JSON. So they may have meant "do not make use of explicit null values" as "do not attribute specific meaning to explicit null values".

[satelliteprogrammer]

I've changed the if to explicitly check if we have a transition from !null -> null. That one is impossible to generate, as it would collide with the actual meaning of null in the patch file.

Null values in the merge patch are given special meaning to indicate the removal of existing values in the target.

However, the end result is the same. We would reach here if the source != target AND target == null. There's no other way for the diff to be null.

[cschreib-ibex]

Inefficient; merge_diff was already called on the same inputs above and diff could be reused here.

[satelliteprogrammer]

Thanks! I had figured this one out already, but since the PR wasn't going anywhere I didn't fix it here.

[cschreib-ibex]

Inefficient: this will traverse the whole depth of the JSON structure to check for equality of all sub fields, and we'll do that again in merge_diff if going inside the if. This check could be removed, calling merge_diff unconditionally, then checking the output isn't an empty object.

[cschreib-ibex]

Though, this will need checking for value types, and only call merge_diff if both source and target are objects.

if (it.value().is_object() && itf.value().is_object()) 
{
    auto diff = merge_diff(it.value(), itf.value());
    if (!diff.empty()) 
    {
        result[it.key()] = std::move(diff);
    }
} 
else if (it.value() != itf.value()) 
{
    result[it.key()] = itf.value();
}

[satelliteprogrammer]

this will traverse the whole depth of the JSON structure to check for equality of all sub fields

I realised it, but since it would break on the first inequality, I conceded. What nudged me in this direction was that we were already checking if the target is not an object at the start, so it didn't feel right to check again before calling the recursive function.

I've now realised where I was wrong.
For two non-object identical values, it they are present at the top-level of the JSON, then the patch must contain them, as otherwise it will not apply them.
However, if the identical non-objects are nested within an object, then, even though it would still work, they aren't necessary in the patch file, as the target object will retain its previous values.

I think this difference in behaviour is what requires the extra work within the for-loop. And then you're correct, we need to check the type to make sure we're applying the most efficient comparison.

[benpeck-eepics]

This is a feature that would be very useful to me. Our usecase is we would like to create a hierarchical data storage system that allows users to override values set in a base layer of json, storing user "overrides" as a merge patch. So being able to easily apply and generate merge patch documents would be essential to such a use case.

I find the merge patch formatting to be more human-readable than the json patch "action list" format. They are more intuitive to interpret for my usecase, where you are essentially looking at a sparse overlay document. You have the context of the changed element's path in the document providing self-documentation as to the intent of the change. A flat list of change actions lacks some of this context when it isn't structured like a typical document.

Thank you for your time, have a nice day.

[satelliteprogrammer]

@cschreib-ibex just in case it's useful for you. You can represent null values iff the source and target represent the same container. In that scenario you don't need null to represent removing existing values.