feat: support cpu and xpu devices in llama4 #326
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
CPU support tried on Intel Xeon. XPU support tried on Intel Data Center GPU Max series. Note that a fix in fairscale for facebookresearch/fairscale#1195 is required to make XPU working (CPU not affected). I believe that this fix might also be needed for CUDA as well since issue seems to be device agnostic and I see it with the simplified reproducer on NVidia A10. Verified on the platforms named above with LLama4 sample completion and chat completion scripts. Requires: facebookresearch/fairscale#1196 Signed-off-by: Dmitry Rogozhkin <[email protected]>
dvrogozh
requested review from
ashwinb,
yanxi0830,
hardikjshah,
dltn,
raghotham and
ehhuang
as code owners
facebook-github-bot
added
the
CLA Signed
|
@ashwinb : can you, please, help review this PR and maybe the one to fairscale (facebookresearch/fairscale#1196)? |
This comment was marked as abuse.
This comment was marked as abuse.
Sorry, something went wrong.
|
Thank you @dvrogozh especially for fixing the fairscale issue. I had observed that myself and it caused me much pain. I am not sure about the release cadence for fairscale packages, so will figure that out. Do we need that fix to land (in pypi) before this PR can work? |
API wise - no. That's fairscale internal issue and we did not update fairscale API. Functionally wise:
|
|
@ashwinb : do we have a way forward making a release in fairscale and merging in this PR? |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Chang is similar to the one previously done for Llama3.
CPU support tried on Intel Xeon.
XPU support tried on Intel Data Center GPU Max series. Note that a fix in fairscale for facebookresearch/fairscale#1195 is required to make XPU working (CPU not affected). I believe that this fix might also be needed for CUDA as well since issue seems to be device agnostic and I see it with the simplified reproducer on NVidia A10.
Verified on the platforms named above with LLama4 sample completion and chat completion scripts.
Requires: facebookresearch/fairscale#1196
CC: @ashwinb, @raghotham