OpenVino-Genai Model Loading and Inference Using GPU, Automatic Model "Release" After Idle State, and Garbled Response During Subsequent Inference

[nnbw-liu]

ov-genai version: 25.1-25.4.2
Model: phi-4-mini
PC: PTL new platform device

Reproduction Steps:
1.Launch the application and perform initial inference.
2.Open Task Manager → Performance → GPU, and monitor the GPU memory usage.
3.Remain idle for 2 to 20 minutes while observing GPU memory. Perform inference again after the GPU memory usage drops significantly.
4.The issue occurs probabilistically under the above conditions. It appears occasionally in the chat sample, but consistently reproduces when using WinUI3.
This issue can only be reproduced on PTL devices, not on LNL devices.

Observation:
It appears that LNL devices do not automatically release GPU memory, whereas PTL devices exhibit automatic GPU memory release behavior.

[as-suvorov]

It's not very clear what do you mean by issue. Could you please provide expected and actual behavior. Do you have same issue on CPU device?

[nnbw-liu]

It's not very clear what do you mean by issue. Could you please provide expected and actual behavior. Do you have same issue on CPU device?

We have not tested with the CPU, and the device we configured is set to AUTO.

[nnbw-liu]

@as-suvorov
First, initialize the ov::genai::LLMPipeline and check the GPU memory after the loading is complete.

Then perform inference with ov::genai::LLMPipeline.

After the inference is finished, leave the system idle for 3 to 20 minutes; you may try the 20-minute idle scenario for testing.

After 3 minutes of idling, the GPU memory usage drops to a range of 0 to 1.5 GB. When inference is initiated again after this drop, the GPU memory usage ramps back up to the normal level, and the following content may appear at this point with a certain probability.
The longer the idle time, the more likely the issue is to reproduce.

weakening weakening weakening weakening weakening weakening weakening weakening weakening weakening weakening weakening weakening weakening weakening weakening weakening weakening weakening weakening weakening weakening weakening weakening weakening weakening weakening weakening weakening weakening weakening weakening weakening weakening weakening weakening weakening weakening weakening weakening weakening weakening weakening weakening weakening weakening weakening weakening weakening weakening weakening weakening weakening weakening weakening weakening weakening weakening weakening weakening weakening weakening weakening weakening weakening weakening weakening weakening weakening weakening weakening weakening weakening weakening weakening weakening weakening weakening weakening weakening weakening weakening weakening weakening weakening weakening weakening weakening weakening weakening weakening weakening weakening weakening weakening weakening weakening weakening weakening weakening weakening weakening weakening weakening 

[nnbw-liu]

@as-suvorov
This issue only occurs on PCs with Intel's new platform (PTL). Is there any way to prevent the memory occupied by the GPU from decreasing? I tried the code below, but it doesn't seem to take effect—the PC still cleans up the GPU memory.

ov::AnyMap pipeline_config = {
    { "PERFORMANCE_HINT", "LATENCY"},
    { "IDLE_SEQUENCE_CLEANUP", "false"},
    //{ "SEQUENCE_CLEANER_POLL_WAIT_MINUTES", "true"},
};

[as-suvorov]

@nnbw-liu Please try CPU device as well. The workaround I see to prevent memory offloading is to run model with dummy inputs with some interval.
Assinging to GPU team

[nnbw-liu]

@as-suvorov @geunhwan
The dummy input method does not seem feasible, as it would excessively consume system resources and increase power consumption for users. Are there any alternative solutions?

[nnbw-liu]

@nnbw-liu Please try CPU device as well. The workaround I see to prevent memory offloading is to run model with dummy inputs with some interval. Assinging to GPU team

@as-suvorov It seems that the CPU has not experienced such a situation

[Wovchena]

Transferred to the main repo so GPU team could take a look