Skip to content

Add video-to-video task #1772

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 29 commits into
base: main
Choose a base branch
from
Open

Add video-to-video task #1772

wants to merge 29 commits into from
+157 −1

Conversation

MrShahzebKhoso
Copy link
Contributor

  • Added about.md with a definition and summary of the task.
  • Added data.ts with task metadata:
    • Datasets (VIRESET, LongV-EVAL, SeedVR_VideoDemos)
    • Models (e.g., Lucy-Edit-Dev, SeedVR2, Shape-for-Motion, etc.)
    • Demo Spaces (lucy-edit-dev, SeedVR2-3B)
    • Task summary describing scope and applications.

@pcuenca
Copy link
Member

pcuenca commented Oct 1, 2025

cc @merveenoyan @sergiopaniego

merveenoyan
merveenoyan reviewed Oct 2, 2025
View reviewed changes
Copy link
Contributor

@merveenoyan merveenoyan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

very neat and thanks a ton for opening this PR!
these pages are targeted for devs to get started from scratch so let's explain a bit better and from scratch!

sergiopaniego
sergiopaniego reviewed Oct 2, 2025
View reviewed changes
Copy link
Member

@sergiopaniego sergiopaniego left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the contribution! I agree with Merve's suggestions and just leaving a few additional minor nits :)

MrShahzebKhoso and others added 11 commits October 5, 2025 03:23
@MrShahzebKhoso @merveenoyan
Update packages/tasks/src/tasks/video-to-video/about.md
21ed0a0 
Co-authored-by: Merve Noyan <[email protected]>
@MrShahzebKhoso @merveenoyan
Update packages/tasks/src/tasks/video-to-video/about.md
88182f2 
Co-authored-by: Merve Noyan <[email protected]>
@MrShahzebKhoso @merveenoyan
Update packages/tasks/src/tasks/video-to-video/about.md
26cc86c 
Co-authored-by: Merve Noyan <[email protected]>
@MrShahzebKhoso @merveenoyan
Update packages/tasks/src/tasks/video-to-video/about.md
ad9f2eb 
Co-authored-by: Merve Noyan <[email protected]>
@MrShahzebKhoso @merveenoyan
Update packages/tasks/src/tasks/video-to-video/about.md
f9fc4b9 
Co-authored-by: Merve Noyan <[email protected]>
@MrShahzebKhoso @merveenoyan
Update packages/tasks/src/tasks/video-to-video/about.md
418bd48 
Co-authored-by: Merve Noyan <[email protected]>
@MrShahzebKhoso @merveenoyan
Update packages/tasks/src/tasks/video-to-video/about.md
aeed0f4 
Co-authored-by: Merve Noyan <[email protected]>
@MrShahzebKhoso @merveenoyan
Update packages/tasks/src/tasks/video-to-video/data.ts
9ab6da4 
Co-authored-by: Merve Noyan <[email protected]>
@MrShahzebKhoso @sergiopaniego
Update packages/tasks/src/tasks/video-to-video/data.ts
ae71347 
Co-authored-by: Sergio Paniego Blanco <[email protected]>
@MrShahzebKhoso @sergiopaniego
Update packages/tasks/src/tasks/video-to-video/about.md
663b190 
Co-authored-by: Sergio Paniego Blanco <[email protected]>
@MrShahzebKhoso
Merge branch 'main' into add-video-to-video-task
aa18356
@MrShahzebKhoso
Copy link
Contributor Author

Thank you for considering this PR and taking time to review.

Updated both the about.md and data.ts files based on all the helpful feedback from @merveenoyan and @sergiopaniego — thank you both for the detailed and thoughtful reviews. Really appreciate the guidance.

Also added a Video Super-Resolution demo (low-res to high-res) via PR #10
on the datasets repo.

Everything is now cleaned up, clarified, and aligned with the review suggestions.

Vaibhavs10
Vaibhavs10 reviewed Oct 6, 2025
View reviewed changes
Copy link
Member

@Vaibhavs10 Vaibhavs10 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

make sure that the changes reflect here: as well packages/tasks/src/tasks/index.ts

cc: @merveenoyan

@MrShahzebKhoso
Copy link
Contributor Author

Thanks for catching that, @Vaibhavs10!
I’ve added the corresponding entry to packages/tasks/src/tasks/index.ts so the changes are now reflected there as well.
Appreciate the review and guidance!

merveenoyan
merveenoyan reviewed Oct 9, 2025
View reviewed changes
Copy link
Contributor

@merveenoyan merveenoyan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thank you for re-iterating!

@MrShahzebKhoso
Copy link
Contributor Author

Thank you, @merveenoyan for the review. I have made the changes including:

  • Linking the Lucy-Edit-Dev, and providing the context on what it's trained on.
  • Removing the second inference example and adding the sentence to direct to the model cards and try example codes provided.
  • Removing '##Repositories' and adding a 'read more' sentence.
  • Removing older model from model.ts.
    With all these changes, I believe the PR is now ready to be merged. If any further feedback or adjustments are needed, I’d be happy to iterate!

merveenoyan
merveenoyan reviewed Oct 13, 2025
View reviewed changes
Copy link
Contributor

@merveenoyan merveenoyan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thank you! we can merge after these ones are addressed

packages/tasks/src/tasks/video-to-video/about.md
- [VIRES](https://github.com/suimuc/VIRES) - Implementation for sketch- and text-guided video instance repainting.
- [ECCV2022-RIFE: Video Frame Interpolation](https://github.com/hzwer/ECCV2022-RIFE)- Real-time video frame interpolation via intermediate flow estimation.
- [StableVSR: Enhancing Perceptual Quality in Video](https://github.com/claudiom4sir/StableVSR)- Super-resolution method to enhance perceptual video quality.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we can remove these lines below

packages/tasks/src/tasks/video-to-video/data.ts
id: "ByteDance-Seed/SeedVR2-7B",
},
{
description: "Framework that uses 3D mesh proxies for precise, consistent video editing.",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we try to add most used ones, let's exclude this

packages/tasks/src/tasks/video-to-video/data.ts
},
{
description: "Model for relighting videos by manipulating illumination distributions.",
id: "TeslaYang123/TC-Light",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this one as well. would be nice to find more used ones

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@sergiopaniego sergiopaniego sergiopaniego left review comments

@Vaibhavs10 Vaibhavs10 Vaibhavs10 left review comments

@merveenoyan merveenoyan merveenoyan left review comments

@SBrandeis SBrandeis Awaiting requested review from SBrandeis SBrandeis is a code owner

@gary149 gary149 Awaiting requested review from gary149 gary149 is a code owner

@Wauplin Wauplin Awaiting requested review from Wauplin Wauplin is a code owner

@julien-c julien-c Awaiting requested review from julien-c julien-c is a code owner

@pcuenca pcuenca Awaiting requested review from pcuenca pcuenca is a code owner

@ngxson ngxson Awaiting requested review from ngxson ngxson is a code owner

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@MrShahzebKhoso @pcuenca @sergiopaniego @Vaibhavs10 @merveenoyan