phi 4 multimodal training version 1 ( with limitations ) #1555
Conversation
src/oumi/datasets/chat_templates/phi4-multimodal-instruct.jinja
Outdated
Show resolved
Hide resolved
| setup: | | ||
| set -e | ||
|
|
||
| pip install uv && uv pip install oumi[gpu] hf_transfer |
There was a problem hiding this comment.
is Phi4 compatible with the current transformers version ?
There was a problem hiding this comment.
yes. please see here https://huggingface.co/microsoft/Phi-4-multimodal-instruct
flash_attn==2.7.4.post1
torch==2.6.0
transformers==4.48.2
accelerate==1.3.0
soundfile==0.13.1
pillow==11.1.0
scipy==1.15.2
torchvision==0.21.0
backoff==2.2.1
peft==0.13.2
the one place I see a potential issue is peft (they use a lower version than our lowest allowed) -- I plan to investigate this more when we publish a more LoRA oriented version of it.
optas
changed the title
| @override | ||
| def ignore_features(self) -> list[str]: | ||
| """Returns a list of keys of features to ignore from feeding the model.""" | ||
| return self._ignore_features if self._ignore_features else [] |
There was a problem hiding this comment.
Consider returning a shallow copy. Just to make sure that external users can't mutate our member variables.
return copy.copy(self._ignore_features) if self._ignore_features else []
| @@ -75,6 +79,7 @@ def __init__( | |||
| self._worker_processor.image_processor | |||
| ) | |||
| self._label_ignore_index: Optional[int] = label_ignore_index | |||
| self._ignore_features: Optional[list[str]] = ignore_features | |||
There was a problem hiding this comment.
Consider making our own private copy. Just to make sure that external users can't mutate our member variable.
self._ignore_features: Optional[list[str]] = copy.copy(ignore_features) if ignore_features else []
| Configs for Phi-4-multimodal-instruct 14B model. See https://huggingface.co/microsoft/Phi-4-multimodal-instruct | ||
|
|
||
| This is a multimodal model that combines text, visual, and audio inputs. | ||
| It uses a "Mixture of LoRAs" approach, allowing you to plug in adapters for each |
There was a problem hiding this comment.
if there is relevant paper for "Mixture of LoRAs" and/or the model itlself , please quote it here.
Description
Adds the necessary changes for SFT training (vision/text-only) with phi-4 in Oumi.
Given that we are using here the
vision_language_with_paddingwe are limited to size batch 1, and we cannot accommodate multi-image training.Related issues
Towards OPE-1102
Before submitting
Reviewers
At least one review from a member of
oumi-ai/oumi-staffis required.