Use orjson for json_dumps

[EugeneChung]

What type of PR is this?

• Refactor
• Feature
• Bug Fix
• New Query Runner (Data Source)
• New Alert Destination
• Other

Description

Following the discussion in #7339 (comment), I updated utils.json_dumps to use orjson for improved serialization performance.

Key implementation details:

• orjson 3.10.15 is used, as it's the last version compatible with Python 3.8.
  • orjson 3.10.16 is currently out but it stops Python 3.8 support.
• Pre-processing: Before calling orjson.dumps, data is pre-processed recursively using the existing custom JSONEncoder to maintain compatibility with Redash's current serialization specifications.
  • For instance, datetime serialization differs:
    • With the custom JSONEncoder: {"time": "2024-03-01T15:30:45.123"}
    • With orjson: {"time": "2024-03-01T15:30:45.123456"}
  • Note: Unlike the standard json module, orjson does not allow overriding serialization behavior for supported native types. The provided default function isn't called for these built-in supported types.
• Option Mapping:
  • Default options are set to orjson.OPT_NON_STR_KEYS | orjson.OPT_UTC_Z, aligning with ensure_ascii=False behavior.
  • The sort_keys parameter maps directly to OPT_SORT_KEYS.
• Testing: Added pytest cases to validate behavior aligned with the existing JSONEncoder specifications.

How is this tested?

• Unit tests (pytest, jest)
• E2E Tests (Cypress)
• Manually
• N/A

For Athena and Trino, the result of select 1.0, cast('NaN' as double), cast('Infinity' as double), cast('-Infinity' as double) is 1.0, null, null, null as expected.

Related Tickets & Documents

• Issue: Redash queries crash if numerical column contain NaN  #6992
• Previous PRs:
  • Sanitize NaN, Infinite, -Infinite causing error when saving as PostgreSQL JSON #7339
  • Sanitize NaN, Infinite, -Infinite causing error when saving as PostgreSQL JSON #7339 (2nd try) #7348

[eradman]

Basic test works

SELECT 'NaN'::float AS not_a_number, 'Inf'::float AS inf, now() AS date

[EugeneChung]

(venv) ~/project/private/redash git:[master]
ruff check --fix tests/test_utils.py
warning: The top-level linter settings are deprecated in favour of their counterparts in the `lint` section. Please update the following options in `pyproject.toml`:
  - 'ignore' -> 'lint.ignore'
  - 'select' -> 'lint.select'
  - 'mccabe' -> 'lint.mccabe'
  - 'per-file-ignores' -> 'lint.per-file-ignores'
All checks passed!
(venv) ~/project/private/redash git:[master]
ruff --version
ruff 0.11.2

[eradman]

Is there a difference in the way key-value values are formatted?

- Options: {"dbname":"testdb1","host":"example.com"}
+ Options: {"dbname": "testdb1", "host": "example.com"}

https://github.com/getredash/redash/actions/runs/14329251848/job/40418005004?pr=7391

[EugeneChung]

@eradman Yes. As I commented, orjson always uses compact separators. I'm going to fix the test.

[EugeneChung]

Passed all failed tests.