Assertion error when converting to and from BJdata

[jkrshnmenon]

Description

An assertion error was discovered when fuzzing using the parse_bjdata_fuzzer harness which is used by OSS Fuzz.

The error occurs due to the assertion failure here

Reproduction steps

Instructions for reproducing the crash

1. Build the parse_bjdata_fuzzer with the following command:

python </path/to/oss-fuzz>/infra/helper.py build_fuzzers --engine libfuzzer --sanitizer address json

2. Run the reproduce feature with the following command:

python </path/to/oss-fuzz>/infra/helper.py reproduce json parse_bjdata_fuzzer </path/to/crash>

Expected vs. actual results

The input file should not crash with an assertion failure.

However, the following error is reported:

INFO:__main__:Running: docker run --privileged --shm-size=2g --platform linux/amd64 --rm -i -e HELPER=True -e ARCHITECTURE=x86_64 -v /home/jay/git_stuff/oss-fuzz/build/out/json:/out -v /tmp/crash:/testcase -t gcr.io/oss-fuzz-base/base-runner reproduce parse_bjdata_fuzzer -runs=100.
+ FUZZER=parse_bjdata_fuzzer
+ shift
+ '[' '!' -v TESTCASE ']'
+ TESTCASE=/testcase
+ '[' '!' -f /testcase ']'
+ export RUN_FUZZER_MODE=interactive
+ RUN_FUZZER_MODE=interactive
+ export FUZZING_ENGINE=libfuzzer
+ FUZZING_ENGINE=libfuzzer
+ export SKIP_SEED_CORPUS=1
+ SKIP_SEED_CORPUS=1
+ run_fuzzer parse_bjdata_fuzzer -runs=100 /testcase
vm.mmap_rnd_bits = 28
/out/parse_bjdata_fuzzer -rss_limit_mb=2560 -timeout=25 -runs=100 /testcase -max_len=456 -timeout=10 < /dev/null
INFO: Running with entropic power schedule (0xFF, 100).
INFO: Seed: 3140016452
INFO: Loaded 1 modules   (6020 inline 8-bit counters): 6020 [0x55aa4942fa48, 0x55aa494311cc), 
INFO: Loaded 1 PC tables (6020 PCs): 6020 [0x55aa494311d0,0x55aa49448a10), 
/out/parse_bjdata_fuzzer: Running 1 inputs 100 time(s) each.
Running: /testcase
parse_bjdata_fuzzer: src/fuzzer-parse_bjdata.cpp:60: int LLVMFuzzerTestOneInput(const uint8_t *, size_t): Assertion `json::to_bjdata(j2, false, false) == vec2' failed.
AddressSanitizer:DEADLYSIGNAL
=================================================================
==18==ERROR: AddressSanitizer: ABRT on unknown address 0x000000000012 (pc 0x7febd8b4f00b bp 0x7febd8cc4588 sp 0x7ffcd4405bd0 T0)
SCARINESS: 10 (signal)
    #0 0x7febd8b4f00b in raise (/lib/x86_64-linux-gnu/libc.so.6+0x4300b) (BuildId: 5792732f783158c66fb4f3756458ca24e46e827d)
    #1 0x7febd8b2e858 in abort (/lib/x86_64-linux-gnu/libc.so.6+0x22858) (BuildId: 5792732f783158c66fb4f3756458ca24e46e827d)
    #2 0x7febd8b2e728  (/lib/x86_64-linux-gnu/libc.so.6+0x22728) (BuildId: 5792732f783158c66fb4f3756458ca24e46e827d)
    #3 0x7febd8b3ffd5 in __assert_fail (/lib/x86_64-linux-gnu/libc.so.6+0x33fd5) (BuildId: 5792732f783158c66fb4f3756458ca24e46e827d)
    #4 0x55aa49321f02 in LLVMFuzzerTestOneInput /src/json/tests/src/fuzzer-parse_bjdata.cpp:60:13
    #5 0x55aa491d6350 in fuzzer::Fuzzer::ExecuteCallback(unsigned char const*, unsigned long) /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerLoop.cpp:614:13
    #6 0x55aa491c15c5 in fuzzer::RunOneTest(fuzzer::Fuzzer*, char const*, unsigned long) /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerDriver.cpp:327:6
    #7 0x55aa491c705f in fuzzer::FuzzerDriver(int*, char***, int (*)(unsigned char const*, unsigned long)) /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerDriver.cpp:862:9
    #8 0x55aa491f2302 in main /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerMain.cpp:20:10
    #9 0x7febd8b30082 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x24082) (BuildId: 5792732f783158c66fb4f3756458ca24e46e827d)
    #10 0x55aa491b97ad in _start (/out/parse_bjdata_fuzzer+0x6e7ad)

DEDUP_TOKEN: raise--abort--
AddressSanitizer can not provide additional info.
SUMMARY: AddressSanitizer: ABRT (/lib/x86_64-linux-gnu/libc.so.6+0x4300b) (BuildId: 5792732f783158c66fb4f3756458ca24e46e827d) in raise
==18==ABORTING
MS: 0 ; base unit: 0000000000000000000000000000000000000000
0x5b,0x5b,0x24,0x42,0x23,0x75,0x0,0x0,0x5b,0x24,0x42,0x23,0x75,0x19,0x0,0x64,0x0,0x5b,0x5b,0x5b,0x5b,0x24,0x42,0x23,0x75,0x0,0x0,0x64,0x0,0x64,0x0,0x5b,0x4d,0x4d,0x5a,0x5a,0x44,0x1a,0x6d,0x5a,0x5d,
[[$B#u\000\000[$B#u\031\000d\000[[[[$B#u\000\000d\000d\000[MMZZD\032mZ]

Minimal code example

Error messages

Compiler and operating system

This bug can be reproduced in Docker container used by OSS-Fuzz

Library version

The latest commit on the github repository

Validation

• The bug also occurs if the latest version from the develop branch is used.
• I can successfully compile and run the unit tests.

[jkrshnmenon]

The input that triggers this error can be generated using the following command:

echo "W1skQiN1AABbJEIjdRkAZABbW1tbJEIjdQAAZABkAFtNTVpaRBptWl0=" | base64 -d

[nlohmann]

#include <nlohmann/json.hpp>
#include <iostream>

int main() {
  std::vector<uint8_t> input = {{0x5b, 0x5b, 0x24, 0x42, 0x23, 0x75, 0x00, 0x00, 0x5b, 0x24, 0x42, 0x23, 0x75, 0x19, 0x00, 0x64, 0x00, 0x5b, 0x5b, 0x5b, 0x5b, 0x24, 0x42, 0x23, 0x75, 0x00, 0x00, 0x64, 0x00, 0x64, 0x00, 0x5b, 0x4d, 0x4d, 0x5a, 0x5a, 0x44, 0x1a, 0x6d, 0x5a, 0x5d}};
  auto j = nlohmann::json::from_bjdata(input);
  std::cout << j << std::endl;
  auto input2 = nlohmann::json::to_bjdata(j, false, false);
  for(int x : input2) {
    std::cout << std::hex << x << " ";
  }
  std::cout << std::endl;
  std::cout << nlohmann::json::from_bjdata(input2) << std::endl;
}

yields:

[{"bytes":[],"subtype":null},{"bytes":[100,0,91,91,91,91,36,66,35,117,0,0,100,0,100,0,91,77,77,90,90,68,26,109,90],"subtype":null}]

5b 5b 5d 5b 55 64 55 0 55 5b 55 5b 55 5b 55 5b 55 24 55 42 55 23 55 75 55 0 55 0 55 64 55 0 55 64 55 0 55 5b 55 4d 55 4d 55 5a 55 5a 55 44 55 1a 55 6d 55 5a 5d 5d 

[[],[100,0,91,91,91,91,36,66,35,117,0,0,100,0,100,0,91,77,77,90,90,68,26,109,90]]

Maybe @nebkat has an idea.

[nebkat]

The input is using optimized binary arrays which are a feature in BJData draft 3. This has to be explicitly enabled in to_bjdata.