[Bug] Regression: "opea/vllm-gaudi:latest" container in crash loop

[eero-t]

Priority

Undecided

OS type

Ubuntu

Hardware type

Gaudi2

Installation method

• Pull docker images from hub.docker.com
• Build docker images from source

Deploy method

• Docker compose
• Docker
• Kubernetes
• Helm

Running nodes

Single Node

What's the version?

https://hub.docker.com/layers/opea/vllm-gaudi/latest/images/sha256-d2c0b0aa88cd26ae2084990663d8d789728f658bacacd8a49cc5b81a6a022c58

Description

vllm-gaudi:latest container does not find devices, and is in crash loop.

But if I change latest tag to 1.1, it works fine, i.e. this is regression.

Reproduce steps

Apply: opea-project/GenAIInfra#610

Then run ChatQnA from GenAIInfra:
$ helm install chatqna chatqna/ --skip-tests --values chatqna/gaudi-vllm-values.yaml ...

Raw log

$ kubectl logs chatqna-vllm-75dfb59d66-wp4vs
...
  File "/usr/local/lib/python3.10/dist-packages/habana_frameworks/torch/hpu/__init__.py", line 132, in current_device
    init()
  File "/usr/local/lib/python3.10/dist-packages/habana_frameworks/torch/hpu/__init__.py", line 71, in init
    _hpu_C.init()
RuntimeError: synStatus=8 [Device not found] Device acquire failed.

[eero-t]

Only change in OPEA Git since v1.1 is dropping of the eager option, but v1.1 image works fine with that change.
https://github.com/opea-project/GenAIComps/commits/main/comps/llms/text-generation/vllm/langchain/dependency/

However, comparing the latest image layers to ones in earlier v1.1 one:
https://hub.docker.com/layers/opea/vllm-gaudi/1.1/images/sha256-c75d22e05ff23e4c0745e9c0a56ec74763f85c7fecf23b7f62e0da74175ddae7

Shows quite a few differences, also in sizes of the installed layers.

=> I think the problem is in Habana repo side.

Recent Gaudi vLLM dependency changes is one possibility: https://github.com/HabanaAI/vllm-fork/commits/habana_main/requirements-hpu.txt

Maybe new HPU deps do not handle correctly pod's Gaudi plugin device request allowing vLLM (write) access only to one of the 8 devices in the node?

[xiguiw]

Not sure how the docker file are buid.
I find out one build command, but did not find Dockerfile.hpu

Here both v1.1 and latest are built from the same commit id git checkout 3c39626.
because Dockerfile.hpu is missed. And from the behavior, it should be Gaudi vllm service issue. not OPEA level.

@ashahba
Would you let us know the build command for docker image on docker hub?

[eero-t]

Not sure how the docker file are buid. I find out one build command, but did not find Dockerfile.hpu

@xiguiw As you can see from the OPEA script, it git clones Habana repo [1], cds to repo's vllm-fork dir, and builds Dockerfile.hpu from there.

[1] It would be faster to clone specific commit instead of first cloning whole repo and only then checking out that specific commit.

[xiguiw]

Not sure how the docker file are buid. I find out one build command, but did not find Dockerfile.hpu

@xiguiw As you can see from the OPEA script, it git clones Habana repo [1], cds to repo's vllm-fork dir, and builds Dockerfile.hpu from there.

@eero-t
I did not find the exact build command.

I mean if the vllm-gaudi image build from the same commit in v1.1 and latest, it does not make sense that v1.1 works but latest failed.

If v1.1 and latest docker images are built with different commit ID, that possible.
We can try to build vllm-gaudi docker image independently (without OPEA), then to verify if vllm-gaudi service works.