PaddleOCR-VL runtime error: image_features is None in FastDeploy 2.4.0

[byeongjaeSon]

Bug Description

Hello, I am trying to serve PaddleOCR-VL using the latest version of FastDeploy. However, I encounter the following error during runtime. I would like to understand what might be causing this issue and how it can be resolved.

Environment

fastdeploy-gpu 2.4.0
paddlepaddle-gpu 3.3.0

Error Message

Traceback (most recent call last):
  File "/path/to/env/lib/python3.12/site-packages/fastdeploy/worker/worker_process.py", line 1089, in <module>
    run_worker_proc()
  File "/path/to/env/lib/python3.12/site-packages/fastdeploy/worker/worker_process.py", line 1085, in run_worker_proc
    worker_proc.event_loop_normal()
  File "/path/to/env/lib/python3.12/site-packages/fastdeploy/worker/worker_process.py", line 522, in event_loop_normal
    self.worker.execute_model(req_dicts, max_occupied_batch_index)
  File "/path/to/env/lib/python3.12/site-packages/fastdeploy/worker/gpu_worker.py", line 193, in execute_model
    output = self.model_runner.execute_model(model_forward_batch, num_running_request)
  File "/path/to/env/lib/python3.12/site-packages/fastdeploy/worker/gpu_model_runner.py", line 2238, in execute_model
    model_output = self.model(
  File "/path/to/env/lib/python3.12/site-packages/paddle/nn/layer/layers.py", line 1833, in __call__
    return self.forward(*inputs, **kwargs)
  File "/path/to/env/lib/python3.12/site-packages/fastdeploy/model_executor/models/paddleocr_vl/paddleocr_vl.py", line 240, in forward
    input_embeddings = self.get_input_embeddings(
  File "/path/to/env/lib/python3.12/site-packages/fastdeploy/model_executor/models/paddleocr_vl/paddleocr_vl.py", line 231, in get_input_embeddings
    input_embeddings[image_mask] = image_features.cast(self._dtype)
AttributeError: 'NoneType' object has no attribute 'cast'

[Wanglongzhi2001]

Could you please provide a minimal code example for reproduction?

[byeongjaeSon]

@Wanglongzhi2001
Hi,
When running PaddleOCR-VL with FastDeploy under very high concurrent requests (PDF parsing workload), the serving process sometimes crashes due to a runtime error.

In most cases it works well, but the process occasionally dies unexpectedly, requiring a manual restart.

Is there any guidance, recommended configuration, or known limitations when running FastDeploy in production or high-concurrency environments?

Reproduction

1. Launch FastDeploy server

python -m fastdeploy.entrypoints.openai.api_server \
  --model PaddlePaddle/PaddleOCR-VL \
  --workers 4 \
	--port 8000 \
	--metrics-port 3000 \
	--host 0.0.0.0 \
  --gpu-memory-utilization 0.45 \
  --max-model-len 16384 \
  --max-num-batched-tokens 131072 \
  --max-num-seqs 256 \
  --enable-mm \
  --max-concurrency 8192 \
  --graph-optimization-config '{"graph_opt_level":0, "use_cudagraph":true}'

2. Send requests using PaddleOCR-VL pipeline (sample)

from paddleocr import PaddleOCRVL
pipeline = PaddleOCRVL()
output = pipeline.predict("https://paddle-model-ecology.bj.bcebos.com/paddlex/imgs/demo_image/paddleocr_vl_demo.png")
for res in output:
    res.print()

[Wanglongzhi2001]

Hello,
This issue appears to be caused by excessive GPU memory usage.

1. You can increase the gpu-memory-utilization to 0.9 to allocate more KV cache.
2. You can also reduce the max-num-batched-tokens to 4096 to use a smaller number of chunks in the chunked prefill.
3. We also recommend setting the graph optimization configuration to: --graph-optimization-config '{"use_cudagraph":true, "use_unique_memory_pool":true, "cudagraph_capture_sizes":[1, 4, 16, 64, 128, 256]}. This will reduce the number of graphs in cudagraph and enable use_unique_memory_pool to reduce cudagraph's memory usage.

[byeongjaeSon]

@Wanglongzhi2001

Hi, thank you for response.

I tried the suggested options, but I encountered another error at the same line of code.

When running inference with paddleocr_vl using FastDeploy, the worker process crashes due to a shape mismatch during input embedding construction.
The error occurs in get_input_embeddings() when assigning image_features to input_embeddings[image_mask].

How can I handle this issue?

Reproduction

 python -m fastdeploy.entrypoints.openai.api_server \
  --model PaddlePaddle/PaddleOCR-VL \
  --workers 4 \
	--port 8002 \
	--metrics-port 3002 \
	--host 0.0.0.0 \
  --gpu-memory-utilization 0.9 \
	--max-model-len 16384 \
  --max-num-batched-tokens 4096 \
  --max-num-seqs 256 \
  --enable-mm \
  --max-concurrency 8192 \
  --graph-optimization-config '{"use_cudagraph":true, "use_unique_memory_pool":true, "cudagraph_capture_sizes":[1, 4, 16, 64, 128, 256]}'

Error log

Traceback (most recent call last):
  File ".../site-packages/fastdeploy/worker/worker_process.py", line 1089, in <module>
    run_worker_proc()
  File ".../site-packages/fastdeploy/worker/worker_process.py", line 1085, in run_worker_proc
    worker_proc.event_loop_normal()
  File ".../site-packages/fastdeploy/worker/worker_process.py", line 522, in event_loop_normal
    self.worker.execute_model(req_dicts, max_occupied_batch_index)
  File ".../site-packages/fastdeploy/worker/gpu_worker.py", line 193, in execute_model
    output = self.model_runner.execute_model(model_forward_batch, num_running_request)
  File ".../site-packages/fastdeploy/worker/gpu_model_runner.py", line 2238, in execute_model
    model_output = self.model(
  File ".../site-packages/paddle/nn/layer/layers.py", line 1833, in __call__
    return self.forward(*inputs, **kwargs)
  File ".../fastdeploy/model_executor/models/paddleocr_vl/paddleocr_vl.py", line 240, in forward
    input_embeddings = self.get_input_embeddings(
  File ".../fastdeploy/model_executor/models/paddleocr_vl/paddleocr_vl.py", line 231, in get_input_embeddings
    input_embeddings[image_mask] = image_features.cast(self._dtype)
ValueError: (InvalidArgument) shape mismatch: value tensor of shape [944, 1024]
cannot be broadcast to indexing result of shape [945, 1024].

Environment

fastdeploy-gpu : 2.4.0
paddlepaddle-gpu : 3.3.0
GPU / CUDA : A100 80G, CUDA Version: 12.2, Driver Version: 535.230.02
Python : 3.12

[ming1753]

We recommend using Docker to install FastDeploy according to install document, and then start the service following parctice document.
We have routine testing and monitoring for this usage scenario.

[Wanglongzhi2001]

We recommend using Docker to install FastDeploy according to install document, and then start the service following parctice document. We have routine testing and monitoring for this usage scenario.

@byeongjaeSon Hi, has this resolved your issue?

[byeongjaeSon]

@Wanglongzhi2001 @ming1753

Unfortunately, the same issue occurs even when applying the suggested options or using the official Docker image.

Additionally, during long-running operation, another issue was observed where the following error occurs and incoming requests remain stuck in the waiting queue without transitioning to the running state.

ERROR:legacy.api_server:request[chatcmpl-50a6b82b-1777-4eec-93aa-977bd62a6698]full generator error: Engine is not healthy: Worker Service Not Healthy, Traceback (most recent call last):
  File "/usr/lib/python3.10/asyncio/queues.py", line 159, in get
    await getter
asyncio.exceptions.CancelledError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.10/asyncio/tasks.py", line 456, in wait_for
    return fut.result()
asyncio.exceptions.CancelledError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/fastdeploy/entrypoints/openai/serving_chat.py", line 556, in chat_completion_full_generator
    response = await asyncio.wait_for(response_queue.get(), timeout=10)
  File "/usr/lib/python3.10/asyncio/tasks.py", line 458, in wait_for
    raise exceptions.TimeoutError() from exc
asyncio.exceptions.TimeoutError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/fastdeploy/entrypoints/openai/serving_chat.py", line 172, in create_chat_completion
    return await self.chat_completion_full_generator(
  File "/usr/local/lib/python3.10/dist-packages/fastdeploy/entrypoints/openai/serving_chat.py", line 563, in chat_completion_full_generator
    raise ValueError(f"Engine is not healthy: {msg}")
ValueError: Engine is not healthy: Worker Service Not Healthy