Gpu memory increase

[assistechsol]

i am using tensorflow model of DocTr and during prediction GPU memory increase incrementally

[felixdittrich92]

Hi @assistechsol,
Could you please share a bit more information about your environment and maybe a code snippet you have used ? :)
cc @charlesmindee

[assistechsol]

I replace tensor flow code to pytorch

[frgfm]

Again, the snippet you just shared is only instantiating a model. While the process on your GPU points towards FastAPI. Are you using our API template?

We need to check the code used for inference if we are to help you 😅

[assistechsol]

Here we load model

And this portion is used for prediction

[felixdittrich92]

Hi @assistechsol
That's only for testing it will decrease the performance but can you please set torch.cuda.empty_cache() after response = predict([image]) and track the mem usage again and share with us ? :)

[frgfm]

I'd suggest something extra:

• if you don't necessarily need GPU, running an API on GPU might be tricky
• I don't know how you ran the uvicorn process for fastAPI but it looks like you have 5. If you launched several workers, it's as if you had N models running (where N is the number of workers). That could help
• for a single worker, it looks like it can go up to 2149Mb of GPU RAM. The first picture you shared was when the model were initiated, for orchestration, you need to give models on GPU a few warm-up runs. If after those warmups, the memory is still increasing, then there might be a memory leak 🤔