Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add supports for Mac M1 #227

Open
wants to merge 3 commits into
base: main
Choose a base branch
from
Open

Add supports for Mac M1 #227

wants to merge 3 commits into from

Conversation

tomy128
Copy link

@tomy128 tomy128 commented Apr 1, 2024

No description provided.

@tomy128
Copy link
Author

tomy128 commented Apr 1, 2024

Tested on Mac m1 silicon ship. Below is the output of test command:

python3 inference.py \
  --face examples/face/1.mp4 \
  --audio examples/audio/1.wav \
  --outfile results/1_1.mp4
show output logs

(venv) ops@arm64:~/www/video-retalking $ PYTORCH_ENABLE_MPS_FALLBACK=1 python3 inference.py
--face examples/face/1.mp4
--audio examples/audio/1.wav
--outfile results/1_1.mp4
[Info] Using mps for inference.
[Step 0] Number of frames available for inference: 135
[Step 1] Using saved landmarks.
[Step 2] 3DMM Extraction In Video:: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████| 135/135 [00:02<00:00, 47.97it/s]
using expression center
Load checkpoint from: checkpoints/DNet.pt
Load checkpoint from: checkpoints/LNet.pth
Load checkpoint from: checkpoints/ENet.pth
[Step 3] Using saved stabilized video.
[Step 4] Load audio; Length of mel chunks: 109
[Step 5] Reference Enhancement: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████| 109/109 [02:07<00:00, 1.17s/it]
landmark Det:: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 109/109 [00:47<00:00, 2.29it/s]
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 109/109 [00:00<00:00, 39592.89it/s]
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 109/109 [00:00<00:00, 997.79it/s]
FaceDet:: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 28/28 [00:20<00:00, 1.35it/s]
[Step 6] Lip Synthesis:: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████| 7/7 [16:40<00:00, 142.87s/it]
outfile: results/1_1.mp4

@tomy128
Copy link
Author

tomy128 commented Apr 1, 2024

The PYTORCH_ENABLE_MPS_FALLBACK=1 variable is solve Error of pytorch below:

    ....
    ffted = torch.fft.rfftn(x, dim=fft_dim, norm=self.fft_norm). 
NotImplementedError: The operator 'aten::_fft_r2c' is not currently implemented for the MPS device.   
 If you want this op to be added in priority during the prototype phase of this feature, please comment on https://github.com/pytorch/pytorch/issues/77764.   
As a temporary fix, you can set the environment variable `PYTORCH_ENABLE_MPS_FALLBACK=1` to use the CPU as a fallback for this op.   
WARNING: this will be slower than running natively on MPS.

@tomy128
Copy link
Author

tomy128 commented Apr 1, 2024

completion for #217 and #30

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant