PaddleOCR-VL Serving with Data Parallel?

[wavy-jung]

First, thank you for your great contribution to open-source environments.
Is serving PaddleOCR-VL model with data parallel not available yet?

[Jiang-Jia-Jun]

@wavy-jung Hello
FastDeploy supports DP (Data Parallel) deployment of PaddleOCR-VL, but currently only supports tensor-parallel-size=1.

Moreover, it relies on the latest version of FastDeploy (you can manually install the nightly build version, but we will soon release version v2.4 to support it, and at the same time, we will provide a Router implemented in Go to achieve request scheduling among DP instances).

Refer to this document for details of DP: https://github.com/PaddlePaddle/FastDeploy/blob/develop/docs/features/data_parallel_service.md

[byeongjaeSon]

@Jiang-Jia-Jun
Hi, I'm trying to serve PaddleOCR-VL model with data parallel. But i faced several errors when i use nightly version of fastdeploy. Could you confirm whether this is a known issue, and if there is a recommended Paddle version or commit to use with FastDeploy 2.4.0.dev?

Environment

Case 1: nightly + nightly
paddlepaddle-gpu 3.4.0.dev20260112
fastdeploy-gpu 2.4.0.dev20260112
Python 3.12

Case 2: stable Paddle + nightly FastDeploy
paddlepaddle-gpu 3.2.1
fastdeploy-gpu 2.4.0.dev20260112
Python 3.12

Reproduction

export FD_ENABLE_MULTI_API_SERVER=1 
export CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7

python -m fastdeploy.entrypoints.openai.multi_api_server \
  --num-servers 8 \
  --ports "8000,8001,8002,8003,8004,8005,8006,8007" \
  --metrics-ports "3000,3001,3002,3003,3004,3005,3006,3007" \
  --args --model PaddlePaddle/PaddleOCR-VL \
  --engine-worker-queue-port "25611,25621,25631,25641,25651,25661,25671,25681" \
  --data-parallel-size 8 \
  --max-model-len 8192 \
  --max-num-seqs 256 \
  --max-concurrency 8192 \
  --gpu-memory-utilization 0.46 \
	--max-num-batched-tokens 100000 \
	--workers 4 \
	--graph-optimization-config '{"graph_opt_level":0, "use_cudagraph":true}'

Errors :

Case 1: nightly Paddle
WARNING Paddle version mismatch:
Build-time commit: 4ff6cb2d01c2eb5d7ecf38e7cabdc00a885b3fde
Runtime commit: 2594854f8ccf925232637e0a42bfd8f2d877e492

TypeError: 'NoneType' object is not callable
at prefix_cache_manager.py:254

Case 2: stable Paddle
AttributeError: module 'paddle.compat' has no attribute 'enable_torch_proxy'

[liyonghua0910]

@byeongjaeSon Hi, both issues are known.

For the first issue, it is recommended to run FastDeploy with Python 3.10, which is continuously monitored for stability. Support for newer Python versions is actively in progress.

For the second issue, FastDeploy 2.4.0.dev should be used together with a Paddle 3.4.0 nightly build; otherwise, calls to paddle.compat.enable_torch_proxy may fail.