FlatBuffers 64 for C++

[dbaileychess]

This introduces 64-bit FlatBuffers.

This allows buffers to be larger than 2 GiB limit due to the addressable range of the uoffset_t (aka uint32_t) used. This add a uoffset64_t (aka uint64_t) as a possible offset backing type, allowing the addressable range to be much larger.

Overview

The buffer is now conceptually two regions of contiguous memory:

[           binary           ]
[32-bit region][64-bit region]

Where the 32-bit region was historically the whole of the FlatBuffer. All 32-bit offsets (Offset) are relative to the end of the 32-bit region, and thus can only address objects within that region. The new 64-bit offset (Offset64) is relative to end of the 64-bit region (or conceptually the tail of the whole buffer). So Offset64 can address any object within the buffer.

This leads to an important concept for using 64-bit FlatBuffers:

All 64-bit offsets MUST be serialized to the binary first, before adding any 32-bit offsets

Attempting otherwise will lead to an assertion.

Schema

Two new attributes are added:

• offset64

  This will generate methods to produce Offset64 return types. This can be used on strings and vectors.

• vector64

  This implies offset64 but also expands the type of the length field of a vector from 32-bits to 64-bits. This allows you to store a single large vector that is > 2 GiB. This also works with nested_flatbuffer attributes.

Code Generation

Only C++ code generation is supported at the moment.

For the most part, this code is semantically similar to the 32-bit version, it just switches out some types. It also requires use of the new FlatBufferBuilder64 to handle the larger buffer.

Builder

There is a new FlatBufferBuilder64 that is used to build these large buffers. They have various methods to create the supported 64-bit enabled types: vectors and strings (sorry no tables yet).

Semantically the builder is the same as the 32-bit one (its just a templated version of it), so the API and flow of building will be identical. The only difference is the inclusion of more template parameters to dictate the use Offset64 or Offset.

For example there are now:

FlatBufferBuilder64 builder;
Offset<T> offset = builder.CreateVector(T t); // create a normal 32-bit vector with 32-bit length field.
Offset64<T> offset = builder.CreateVector<Offset64>(T t); // create a 64-bit vector with 32-bit length field,
Offset64<T> offset = builder.CreateVector<Offset64, Vector64>(T t); // create a 64-bit vector with 64-bit length field.

// Same with strings
Offset<String> offset = builder.CreateString("hi");
Offset64<String> offset = builder.CreateString<Offset64>("hi");

Accessing

Accessing a 64-bit FlatBuffer is almost identical to the current ways. Only the returned types differ a bit for vectors, where now there is a Vector64<T> (which is just a Vector<T, uoffset64_t>). So it will have identical API, just operates on different length types.

Compatibility

Adding either offset64 or vector64 to an existing field is an evolution error (it would fail the compatibility check), so backwards compatibility is preserved, as any 64-bit field would have to be a new field.

Implementation Notes

Here is what an annotated binary looks for a example schema that uses various 64-bit fields.

Primarily the difference is the support for UOffset64 in the table definition. The associated vtable doesn't need special treatment since it natively supported various offsets.

root_table (RootTable):
  +0x1C | 14 00 00 00             | SOffset32  | 0x00000014 (20) Loc: 0x08          | offset to vtable
  +0x20 | D0 00 00 00 00 00 00 00 | UOffset64  | 0x00000000000000D0 (208) Loc: 0xF0 | offset to field `far_vector` (vector)
  +0x28 | 00 00 00 00             | uint8_t[4] | ....                               | padding
  +0x2C | D2 04 00 00             | uint32_t   | 0x000004D2 (1234)                  | table field `a` (Int)
  +0x30 | 8C 00 00 00 00 00 00 00 | UOffset64  | 0x000000000000008C (140) Loc: 0xBC | offset to field `far_string` (string)
  +0x38 | 00 00 00 00             | uint8_t[4] | ....                               | padding
  +0x3C | 40 00 00 00             | UOffset32  | 0x00000040 (64) Loc: 0x7C          | offset to field `near_string` (string)
  +0x40 | 70 00 00 00 00 00 00 00 | UOffset64  | 0x0000000000000070 (112) Loc: 0xB0 | offset to field `big_vector` (vector64)
  +0x48 | 08 00 00 00 00 00 00 00 | UOffset64  | 0x0000000000000008 (8) Loc: 0x50   | offset to field `big_struct_vector` (vector64)

The other interesting case is the vector64 that now supports a uint64_t length field:

vector64 (RootTable.big_vector):
  +0xB0 | 04 00 00 00 00 00 00 00 | uint64_t   | 0x0000000000000004 (4)             | length of vector (# items)
  +0xB8 | 05                      | uint8_t    | 0x05 (5)                           | value[0]
  <2 regions omitted>
  +0xBB | 08                      | uint8_t    | 0x08 (8)                           | value[3]

Fixes: #7537

[aardappel]

This is awesome! Appreciate the great care you've taken to not disrupt the 32-bit eco system much :) Overall complexity surprisingly low too. Very neat!

[battre]

FYI, the tests are now really slow...

time ./flatbuffers_unittests
ALL TESTS PASSED

real    1m24.133s
user    1m23.377s
sys     0m0.752s

[aardappel]

@battre @dbaileychess odd, they used to be really fast, worth seeing which test that is..

[battre]

I attached a debugger because I thought that the tests were in an infinite loop. When I triggered a break, I was in the resize here:

    std::vector<uint8_t> big_data;
    big_data.resize(big_vector_size);

[dbaileychess]

Yeah, I actually made a giant buffer to test and it takes a while to make. Let me fix it so the average case doesn't have to do it.

[dbaileychess]

Fixed in 66e9d98

Before:

[I] derekbailey@lysine ~/P/d/flatbuffers (master)> time ./flattests
ALL TESTS PASSED

________________________________________________________
Executed in    4.44 secs    fish           external
   usr time    1.31 secs    1.35 millis    1.31 secs
   sys time    3.11 secs    0.40 millis    3.11 secs

After:

[I] derekbailey@lysine ~/P/d/flatbuffers (master)> time ./flattests
ALL TESTS PASSED

________________________________________________________
Executed in  121.91 millis    fish           external
   usr time   93.57 millis    1.43 millis   92.14 millis
   sys time   28.39 millis    0.35 millis   28.04 millis

I guess my machine was beefy enough that the 100 ms to 4 s wasn't too bad that I noticed.

[battre]

Hm... My machine that needed 1m24.133s has 128 virtual cores and 512 GB of RAM :-) - But it's a virtual machine and I compiled it within a Chrome checkout. I wonder whether Chrome has special parameters for the memory allocator...

[battre]

That helped here as well!

time ./flatbuffers_unittests
ALL TESTS PASSED

real    0m0.236s
user    0m0.228s
sys     0m0.008s

Thank you.