add dynamic batching embedding/reranking

[Spycsh]

Description

To solve the issue that multiple "small" GenAI services cannot be launched on one Gaudi card, here we draft one implementation of replacing TEI logics with local inference using optimum habana + dynamic batching. This requires the following functionalities:

• request buffering
• HPU batch inference on embedding/reranking

Issues

n/a

Type of change

List the type of change like below. Please delete options that are not relevant.

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds new functionality)
• Breaking change (fix or feature that would break existing design and interface)
• Others (enhancement, documentation, validation, etc.)

Dependencies

Under implementation

Tests

Under implementation

[codecov]

Codecov Report

Attention: Patch coverage is 68.29268% with 13 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
comps/cores/mega/micro_service.py	68.29%	13 Missing ⚠️

Files with missing lines	Coverage Δ
comps/cores/mega/micro_service.py	82.90% <68.29%> (-8.24%)	⬇️

[Spycsh]

Benchmark repo: https://github.com/Spycsh/dynamic-batching-benchmark