Memory Leak When Processing Text Batches with miniLML6

[zhibisora]

Memory usage increases continuously as more text is added, and it is not automatically released after the vector calculations are complete.

When I performed computations on 3,000 text chunks (512 characters each) using miniLML6, the memory usage grew from 200MB to 4.4GB and was not reclaimed after the process finished.

[ozziek]

FWIW:

I’m not using this library directly yet, but I ported over the embedding code. At first, I thought there was a memory leak, but it turned out to be caused by the MLX cache. Once I set a cache limit, the (seemingly) unbounded memory growth stopped:

MLX.GPU.set(cacheLimit: 100 * 1024 * 1024)

The MLX docs has a some good notes on this: https://swiftpackageindex.com/ml-explore/mlx-swift/0.25.6/documentation/mlx/running-on-ios#Controlling-Memory-Use