Are Qlever indices portable?

[C-Loftus]

Thank you for your great work on Qlever.

I have a very large nq file that is over 200gb and, to my understanding, takes around 20gb of RAM to index. I would like to generate the index on my own VM that has more resources but also be able share it more easily with others.

I was wondering if it is possible to generate the index once and then share it with others even if they are on a different platform?

Or is it advised not to do that and to just share the nq directly and compress it, even though it will take some time to index?

Context

I am using ORAS to upload and share data artifacts via OCI registries; I am trying to get something sort of like a data container that allows for easy open data sharing and my thought was that by bundling the index directly it would more easily allow others to spin up and query my data on their own machine.

[ktk]

This works fine as long as you stay in the same CPU architecture. In fact we do that all the time as it makes a lot of sense in a cloud-env to pre-calculate the indexes on fast machines & then deploy them on a much smaller instance.

[C-Loftus]

This works fine as long as you stay in the same CPU architecture. In fact we do that all the time as it makes a lot of sense in a cloud-env to pre-calculate the indexes on fast machines & then deploy them on a much smaller instance.

Thank you very much for your response. Just to clarify then, is it correct that it isn't possible to share it across machines with different architecture?

For instance if someone generates the index on a ARM based Mac and then distributes it to be ran on a amd64 Linux machine?

[joka921]

The indices use low-level binary serialization, so roughly

• the endianess of the systems has to be the same
• the sizeof() for builtin types like size_t , int etc. has to be the same.

So the ARM->AMD portability is something we have never tested and currently do not guarantee, but you can tell us your experiences if you try it out.

[C-Loftus]

Makes sense thank you very much.

[ktk]

Please follow up if you try to build on Mac and use on Intel, I have not tried that so far :)