Storage Size Limitation Exception

[Bios-Marcel]

Environment Details

• EclipseStore Version: 2.1.3
• JDK version: 24
• OS: Windows

Describe the bug

I checked the recent PRs / Issues and it doesn't seem this was addressed in a newer version, hence I assume updating won't change anything.

Just happens sometimes when storing a bunch of data at once in a big transaction:

Caused by: org.eclipse.store.storage.exceptions.StorageExceptionCommitSizeExceeded: Channel 0 store size 2157149122 bytes exceeds technical limit of 2^31 bytes
	at org.eclipse.store.storage.types.StorageDataChunkValidator$MaxFileSize.validateDataChunk(StorageDataChunkValidator.java:358) ~[storage-2.1.3.jar:?]
	at org.eclipse.store.storage.types.StorageRequestAcceptor$Default.storeData(StorageRequestAcceptor.java:175) ~[storage-2.1.3.jar:?]
	at org.eclipse.store.storage.embedded.types.EmbeddedStorageBinaryTarget$Default.write(EmbeddedStorageBinaryTarget.java:80) ~[storage-embedded-2.1.3.jar:?]
	... 25 more

It doesn't break anything as far as I know, BUT, I also don't know why this would be an exception.

Expected behavior

I am not entirely sure, since I don't quite understand why this is an issue. I am assuming if the file is too big for some reason, we can create another one? Or is this an actual limit of the channel size as a whole?

Or would the fix be my job, by increasing the amount of channels? I feel like it's a bit hard to say how many channels you'll need. Is there an downside to have a massive amount of channels, like tons of synchronisation overhead or so?

Maybe the documentation should provide more explanation and guidances regarding pros, cons and limits.

[hg-ms]

The maximum size of a single commit is limited to 2^31 bytes to ensure that a commit never creates files that are larger than GBs, because the current loading mechanism can’t handle file larger than those 2GB. This limit applies to a single commit (store) only.
The storage size is not limited by those 2GB. Using more channels will not bypass the limitation.
If you have more than 2GB data to persist you need to split it up in several stores.

[Bios-Marcel]

I see? Is this a fixable limitation though? If so, maybe I could look into it?

[hg-ms]

Many thanks for your offer to have a look at that issue.

I had some initial thoughts on the issue:

• Exception was introduced with PR 371 to fix issues Storage position for entity 1000000000000029026 exceeds the technical int value limit of 2147483647 #319 and Storage file size exceeds Java technical IO limitations #321.
  PR371 ensures that files never exceed the max loadable size of 2GB.
• File loading is done in org.eclipse.store.storage.types.StorageEntityInitializer.
  I’d assume that increasing the max file size is only the first and simplest step. As this will affect other parts of the storage as well: The storage relies on ByteBuffers quite a lot. The limited size of those buffers causes restrictions at other storage components too that must be considered if the file size is increased beyond the max buffer size. E.g.: a single entities binary data must fit in single byte buffer because the StorageEnityCache uses ByteBuffers too. That limitation is currently assured by the file size limit.

Please don’t hesitate if you have any questions.

[Bios-Marcel]

OK, so basically I could see if I can replace whatever stdlib classes are used with custom implementations that allow bigger sizes. I guess I could just write a test that writes more than 2gb, restarts the storage and verifies that the data is still correct.

As long as we don't have to split the store dynamically it shouldn't be too complicated I guess?

I'll check with my manager to see if I can allocate some time here 😁