Is there a fast way to search JSON keys?

[SheepChef]

I am working on a project involving huge amount of searching in JSON keys.

{
    "basic":{
         "a":[...],
         "b":[...],
         "c":[...],
         "d":[...],
         "e":[...],
          ...
    }
}

Basically I need to search from A to Z until there is a match, and the search need to be performed for many times (around 10 Million times).

For now I just use iterations mentioned in the docs:

        for (auto& el : Map_Obj["basic"].items())
        {   
            if(el.key() == letter){
                 //Do something
            }
        }

When the iteration needs to be perform for over 10M times, the speed is dramatically slow, completing in 10~20 minutes or so.

However, when I rewrite the code in Node.js and run it on V8, the speed boosts to less than 30 seconds, while the JSON is the same and I am using the same iteration method.

    for (let key in Map_Obj["basic"]) {
      if (Map_Obj["basic"].hasOwnProperty(key)) {
        if (key == letter) {
           //Do something
        }
      }
    }

I've checked my code for multiple times and found nothing, so I am here to enquiry if there is anyway to improve the performance on C++.

[nlohmann]

Do you compile in Release mode?

[SheepChef]

No -g when compiling

[nlohmann]

Please compile with optimizations enabled -O3 and -DNDEBUG to switch off assertions. Also, don't use -g. This should improve the runtime speed.

[SheepChef]

-O3 was already implemented before.

Tried with -DNDEBUG but there is no obvious improvement.

[gregmarr]

You are doing a linear search on a map, which is about the most inefficient thing you can do.

void doSomething(nlohmann::json const &j) {}

int main()
{
    std::string letter("a");
    nlohmann::json Map_Obj;
    auto basic = Map_Obj["basic"];
    auto it = basic.find(letter);

    if (it != basic.end())
    {   
        doSomething(it.value());
    }
}

[SheepChef]

thanks, I will try!

[SheepChef]

Well, the mothod above is actually slower for 3x(according to my test)

[SheepChef]

I have to say it's weird.

I used the same iteration logic in JS and C++, but the speed is so different.

[SheepChef]

Technically C++ is faster, but V8 is faster in the situation.

[nlohmann]

Well V8 is written in C++, so there is not so much about that. But it is weird that your code is so slow. Can you share a runnable example?

[SheepChef]

Well V8 is written in C++, so there is not so much about that. But it is weird that your code is so slow. Can you share a runnable example?

Here is the function:
https://github.com/SheepChef/Abracadabra/blob/main/src/abracadabra.cpp#L521

Here is the repo:
https://github.com/SheepChef/Abracadabra

Executable code is in release.

In contrast, here is a demo using Node.js:
https://sheepchef.github.io/Abracadabra/

As I am using Chinese to write the README, if you want to test the speed, run:

 ./abracadabra -f <FILE-PATH> -o <OUTPUT-PATH>

Thank you for reviewing the issue.

[SheepChef]

Alright, I mapped the JSON mannually in C++, using

map<string,vector<string>> = {{"a",{"x","y","z"}}, ...}

it turns out that even if I use such a hard-coded map, the performance is totally not improved.

So, obviously, the problem has nothing to do with JSON lib, and I am closing the quesion.

Such a weird problem, I literally have no idea what caused it.