Msgspec

[shyuep]

https://jcristharif.com/msgspec/

Seems like a good package that can replace most of monty's current serialization with speed?

@DanielYang59 @mkhorton

[shyuep]

I did a quick test.

%timeit orjson.dumps(s.as_dict())
%timeit msgspec.json.encode(s.as_dict())

The results are indistinguishable

717 μs ± 8.45 μs per loop (mean ± std. dev. of 7 runs, 1,000 loops each)
722 μs ± 7.48 μs per loop (mean ± std. dev. of 7 runs, 1,000 loops each)

However, msgspec allows one to specify an enc_hook. That means we can now use custom encoders:

from monty.json import MontyEncoder
enc_hook = MontyEncoder().encode
%timeit msgspec.json.encode(s, enc_hook=enc_hook)

This gives 803 μs ± 10.6 μs per loop (mean ± std. dev. of 7 runs, 1,000 loops each). Somewhat slower (not sure why) but acceptable.

Native json is strange.

import json
%timeit json.dumps(s.as_dict())
%timeit json.dumps(s, cls=MontyEncoder)

always gives speed around the same as the one using encode hooks.
788 μs ± 8.9 μs per loop (mean ± std. dev. of 7 runs, 1,000 loops each)
784 μs ± 3.91 μs per loop (mean ± std. dev. of 7 runs, 1,000 loops each)

[shyuep]

@DanielYang59 I actually found that you can use orjson with MontyEncoder.

orjson.dumps(s, default=MontyEncoder().encode)

actually works.

[DanielYang59]

Great thanks for letting me know! I didn't know the encode method could also be passed as default to orjson... I thought we could "probably" pass default=MontyEncoder().default but it's not completely a static method and might rely on self state for certain types.

Both methods do work for certain types but encode gives an escaped stringified JSON, so probably we should pass default=MontyEncoder().default? But again I'm not quite confident about this usage as the MontyEncoder().default is not static (therefore I didn't replace in #4411).

import json

import orjson
import numpy as np

from monty.json import MontyEncoder


obj = np.array([1, 2, 3])

print(json.dumps(obj, cls=MontyEncoder))
print(orjson.dumps(obj, option=orjson.OPT_SERIALIZE_NUMPY))
print(orjson.dumps(obj, default=MontyEncoder().encode))
print(orjson.dumps(obj, default=MontyEncoder().default))

{"@module": "numpy", "@class": "array", "dtype": "int64", "data": [1, 2, 3]}
b'[1,2,3]'
b'"{\\"@module\\": \\"numpy\\", \\"@class\\": \\"array\\", \\"dtype\\": \\"int64\\", \\"data\\": [1, 2, 3]}"'
b'{"@module":"numpy","@class":"array","dtype":"int64","data":[1,2,3]}'