Editing without any mask

[jh5442]

Hi there! Thank you for making your great work open sourced! I was wondering whether there is a way to achieve video editing without providing any mask and only using text prompt? It seems like the current edit.sh requires mask input. Thank you!

[yxbian23]

Hi there! Thank you for making your great work open sourced! I was wondering whether there is a way to achieve video editing without providing any mask and only using text prompt? It seems like the current edit.sh requires mask input. Thank you!

Thanks for your recognition of our work!

However, since masks and text prompts are the two required inputs of videopainter, it's impossible to inpaint without any masks. The reason is that masks describe the areas that need to be inpainted, and texts describe the desired inpaint results.

You can refer to SAM2 or other VLMs for generating desired masks~

[li-wenzhan]

Hi there! Thank you for making your great work open sourced! I was wondering whether there is a way to achieve video editing without providing any mask and only using text prompt? It seems like the current edit.sh requires mask input. Thank you!

Thanks for your recognition of our work!

However, since masks and text prompts are the two required inputs of videopainter, it's impossible to inpaint without any masks. The reason is that masks describe the areas that need to be inpainted, and texts describe the desired inpaint results.

You can refer to SAM2 or other VLMs for generating desired masks~

Is there any script that generates the video masks based on SAM2? I've installed SAM2, but I haven't seen where it is used in /infer yet.

[yxbian23]

Hi there! Thank you for making your great work open sourced! I was wondering whether there is a way to achieve video editing without providing any mask and only using text prompt? It seems like the current edit.sh requires mask input. Thank you!

Thanks for your recognition of our work!
However, since masks and text prompts are the two required inputs of videopainter, it's impossible to inpaint without any masks. The reason is that masks describe the areas that need to be inpainted, and texts describe the desired inpaint results.
You can refer to SAM2 or other VLMs for generating desired masks~

Is there any script that generates the video masks based on SAM2? I've installed SAM2, but I haven't seen where it is used in /infer yet.

Thanks for the follow-up!

In the current implementation, the inference scripts assume that the SAM2 masks have already been generated and saved. The pipeline simply loads these precomputed masks and performs inpainting based on them.

We do have code for using SAM2 to create masks, but it is currently implemented only in the Gradio demo interface. You can refer to app.py for the SAM2-based interactive mask annotation workflow.

Hope this helps! And feel free to reach out if you have further questions. 😊