Any plan to support the JSON in kvrocks?

[simplechen]

As the title

[torwig]

Hi @simplechen.
Do you mean to support operations like in MongoDB?

[git-hulk]

Should be like Redis JSON: https://redis.io/docs/stack/json/

[simplechen]

yes,like rejson,

[simplechen]

Is json data type support in our plan, thank you

[simplechen]

Hi @simplechen. Do you mean to support operations like in MongoDB?

no,mogodb is so complex,like redis json https://redis.io/docs/stack/json/

[simplechen]

Thank you, I see that there is a standardized JSON in the 3.0 plan, and I am looking forward to it

[git-hulk]

Yes, feedback is welcome if you have any thoughts.

[xiaobiaozhao]

I think if you need to store json strings, consider using lua scripts instead, as cjson codec support is available in lua vm

[simplechen]

thanks

[git-hulk]

ReJSON is expected to support in 2023 roadmap: Kvrocks 2025 Roadmap

[mapleFU]

RedisJSON

Type System

Complex Types:

• Array
• Object

Basic Types:

• Int: a 64-bit signed integer number
• Double: float64 number
• Bool
• String

Bottom Type:

• Null

Format

Storage

Just stores "Raw IJSON" values for json. It's a cbor-like typed json binary format.

When stored in rdb, the "Raw IJSON" format would be cast to "JSON Text".

Memory

• serde_json: User input value will be wrapped in SerdeJson
• ijson: ijson is an in-place replacement for serde_json, it provides light-weight in-memory format. The storage value will be deserialized with ijson format.

Commands

Commands are Listed below:

• https://github.com/RedisJSON/RedisJSON/blob/master/redis_json/src/lib.rs#L190-L214

Path

• JsonPath
  • Require Path and Filter Expression
• Legacy Path Syntax
  • Use a.b a["b"]

Kvrocks Design

Method 1: Integrate or Reuse RedisJSON

RedisJson can not be well-optimized if "string" representation is used. Instead, we should pay some efforts to re-implement the old Legact Path Syntax.

If we just reuse RedisJSON, we can:

1. Make RedisJSON supports redis modules
2. Maintain a fork of RedisJSON, and replace redismodules interface with kvrocks' callback

Pros:

• Can always be compatible with RedisJson
• Easy to integrate Redisearch
• Not need to maintain "Legacy Path Syntax" in kvrocks' code

Cons:

• It's hard to encapsulate kvrocks's content into Redis context.
• If fork RedisJson, we need to maintain a fork for RedisJSON, and currently we just need to maintain get/set /event/redis-protocol-version and rdb. But if it requires more features, maintaining a fork might be hard.
• The underlying storage format might change, and RedisJSON doesn't have KeyMeta in storage, if RedisJSON or ijson change the underlying implemention

Method 2: Re-implement JSON in kvrocks

Re-implement JSON in kvrocks is not hard, we need to consider the case below:

• Json Path implementions
• Storage format
• In-memory format
• Thirdparty libraries

Thirdparty libraries

• Jsoncons: https://github.com/danielaparker/jsoncons
  • Support JSON Path, and has cbor, bson extensions
  • Not support querying objects directly on binary / text
• SIMDJson:
  • Query json directly on raw text

Storage Format

Raw Text Data

Pros:

1. Easy to use
2. Can be well compacted
3. Human readable

Cons:

• May consume lots of memory

Cbor

Pros:

1. Easy to use
2. Usally smaller than json text
3. Strong typed

Cons:

• May complex during parsing

Jsonb

Pros:

1. It can be fast when reading, because it supports fast seek
2. Strong typed

Cons:

• May complex during parsing
• Might be larger than json text

In-memory format

• Memory format can be the same as storage format or jsoncons::json. The latter might be memory-consuming
• When updating or querying:
  • When updating, we might need to support casting it to jsoncons::json, and update it, then casting it back to storage format
  • When querying, we will use storage format if it's possible

Corner cases

Huge JSON Value

If the json value is small, using all kinds of data is OK, if json value is huge, we need to consider that:

• Would it be heavy when parsing or updating?
• Should it be separate in kvrocks/rocksdb?

Pros:

• Might have better performance if we can use jsonb without parsing the object
• If the json is huge, we're able to separate it into different parts

Cons:

• Need lots of compatibility works
• Might not have good performance the first time, and the underlying storage format might change

References

1. PostgreSQL JSON / JSONB
2. RedisJson
3. Redis Modules
4. Jsoncons

[mapleFU]

@PragmaTwice @git-hulk
I've listed some possible ways to implement RedisJSON in kvrocks, would you mind take a look?

[mapleFU]

I can also implement the logic same as IValue and jsonpath in kvrocks, but I guess it's a bit time-consuming.
It could be fast to just using jsoncons in memory, and parse it when every time get/set is called, but I guess the performance would be pool

[git-hulk]

@mapleFU Thanks for your great proposals.

For the implementation method, I prefer to re-implement in Kvrocks which is more coherent than use the redis module. As the formatting, jsonb is a good choice if it wouldn’t bring too much work while comparing to the raw text. And for the performance, I don’t worry much about this since we can try to improve if users did suffer from the performance issues.

[mapleFU]

Okay, I'll try to draft one without legacy path, using jsoncons::json in memory. The backend implemention can switch from cbor and json text.