Large amount of StorageEntity$Default instances

[mainzelM]

I'm writing a small application (based on MicroStream 07.01.00-MS-GA) that uses a Map<Long, Foo> as root (wrapped in a simple DataRoot class). After storing about 700 Foo instances (which of course reference other custom instances and JDK instances), I took a memory snapshot and analyzed it with YourKit profiler.

Much to my surprise, the snapshot contained about 18 million (strongly reachable) instances of StorageEntity$Default (corresponding to 97% of the total number of instances in the JVM) and a shallow size of about 1.2GB (corresponding to 86% of the total heap).

The Foo class has a List field that can grow quite large and which I thus made Lazy:

private Lazy<List<Record>> records;

Right after storing a Foo instance I call foo.records.clear() and I consequently don't see any instances of Record in the heap dump:

public long addFoo(Foo foo) {
        dataRoot.getContent().put(foo.getId(), foo);
        long id = storageManager.store(dataRoot.getContent());
        foo.getRecords().clear();
        return id;
}

If I however remove the records field, the problem goes away. So I suspect the StorageEntity$Default instances have something to do with this field.

Here is the shortest path of one random instance to a GC root:

one.microstream.storage.types.StorageEntity$Default
[10] of one.microstream.storage.types.StorageEntity$Default[64]
oidHashTable of one.microstream.storage.types.StorageEntityCache$Default
entityCache of one.microstream.storage.types.StorageChannel$Default [Stack Local]
target of java.lang.Thread [Thread, Stack Local] "MicroStream-StorageChannel-0" tid=85 [TIMED_WAITING]

What is also strange: if I programatically clear and save the data root Map and take another snapshot, the Foo instances are gone as expected, but the StorageEntity$Default instances are still there, even after restarting the application.

As I'm not very experienced with MicroStream, I suppose I did something wrong. Could you please enlighten me?

[hg-ms]

Hello,
The StorageEntity instances are part of the internal cache, there is an entry for each persisted object.
Those StorageEnities are not cleared by intend when LazyRefences are unloaded.
If you remove the record field you reduce the number of persisted objects and therefore the required StorageEnities.
Why those entities survive the map.clear(), store and restart is hard to say without a code example. But a common cause is that the wrong object is stored to update an already persisted object. In your case the code should be:

map.clear();
storageManager.store(map);

please see lazy / eager storing for details.

[mainzelM]

Hello @hg-ms,
Thanks for your reply! I've created a small sample project at https://github.com/mainzelM/microstream-sandbox

1. Initializing the store via

./mvnw clean package exec:java -Dexec.args="init"

creates 1'400'000 StorageEntity$Default instances occupying about 850MB (probably related to the 700'000 Recordand 700'000 String instances, taking together about 270MB). Clearing the LazyReferences (line 29) gets rid of the Records/Strings but not of the StorageEntity$Default instances as you explained.
2. Quite to my surprise, the StorageEntity$Default instances are created again after a JVM restart:

./mvnw clean package exec:java -Dexec.args="size"

3. Next, I clear the map and store it again as you suggested:

./mvnw clean package exec:java -Dexec.args="clear"

The StorageEntity$Default instances are still there
4. A JVM restart shows that the Map is indeed empty now, but the StorageEntity$Default instances are still there:

./mvnw clean package exec:java -Dexec.args="size"

So I really wonder, what I'm doing wrong when clearing the Map. In addition, I wonder how to store an even larger amount of objects without hitting a memory limit. In my real application, I've got enough memory to hold all Foo instances, but not for the Record instances (not to speak of the StorageEntity$Default instances).

[hg-ms]

Many thanks for your sample.

With that sample I can explain why you still see that StorageEntity$Default instances after clearing your data set and reloading. They still exist because the related objects are not yet removed from the storage. Your code almost immediately exits so the storage has no time to clean up those delete objects.
To can manually trigger the storages garbage collector manually by calling if the automatics are not sufficient:

storageManager.issueFullGarbageCollection();
storageManager.issueFullFileCheck();

For some more details please see Housekeeping

If low on memory the only option is to avoid persisting that much object instances. This may be quite tricky. One thing might be to use primitive arrays instead of object-based collections or to implement specialized handlers.

[mainzelM]

Ok, thanks, I can confirm that triggering the storage garbage collector indeed gets rid of the StorageEntity$Default instances after clearing the Map.

But I still don't see why the Lazy references have such a huge memory impact behind the back and why there is no way to clear this cache. Even in my real application where Record contains 11 fields, the StorageEntity$Default instances use 100 times more memory than the Record instances (which I could get rid of using Lazy#clear()). Using primitive arrays (I guess you mean one array per field of the Record class?) seems to contradict MicroStream`s promise "Design your model freely.".

[hg-ms]

The StorageEntity$Default instances are also used by the storage garbage collector (don’t mix up with the Java-GC) to ensure the referential integrity of the persisted graph. They hold the information whether an object is still part of the persisted data. E.g.: even if an object has been deleted from the applications data in memory it can still be part of the persisted graph and therefore must not be deleted from that. Without that management data in memory the StorageGC would be very slow.
With primitive arrays I mean arrays of primitive value types like int[]. Those arrays require only one StorageEntity$Default independent of the size as they store values not references. Arrays (or lists) of non-primitive types require 1 + size StorageEntity$Default instances (one for the array itself and one for each referenced object).

[mainzelM]

I'm now serializing the List<Record> instances myself and store Lazy references to the resulting byte[]s in the data root (mapped by Foo's id). The corresponding field in Foo is now transient and populated only on request. This greatly reduces the memory needed for the StorageEntity$Default instances. However, in my real application, MicroStream instances still seem to occupy about 10x more memory than the actually stored instances. I don't consider this a good ratio but from what you describe, I have to live with that. For my current use case, it's now no longer a problem but I'll certainly have a close look at that topic next time I start to build an application based on MicroStream. Thank's a lot for your explanations!