[unbloating] move pipeline utilities to pipeline_utils

[sayakpaul]

What does this PR do?

Move methods like calculate_shift(), retrieve_latents(), and retrieve_timesteps() to pipeline_utils.py to reduce a lot of LOC.

Didn't make these changes to the files under examples/community.

Note

I didn't start any deprecation for this because the respective scripts from which these methods are removed will anyway contain their imports.

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[JoeGaffney]

Nice, i do know there is a do repeat yourself ethos and I'm not a massive fan of overly DRY code with confusing abstractions.

But think in allot of cases lifting more of the common things out instead of the # Copied from makes allot more sense in readability. Especially as more and pipelines are sharing ideas from one another.

[sayakpaul]

Thanks @JoeGaffney! Yes, those will essentially be considered similar to how transformers does it (something like unpacking inheritance to not distort readability).

[DN6]

Looks good to me. cc: @yiyixuxu here too.

[sayakpaul]

@yiyixuxu a gentle ping when you get time.

[yiyixuxu]

So on a high level, I have concerns about moving commonly used pipeline methods to one single pipeline_utils.py file like this.

How do you decide which methods go there in the future? If you think about it, we don't actually decide which methods are "common" - the # Copied from mechanism does. We didn't know the exact impact of each pipeline at the time of integration, but some of them (like SDXL/Flux) had methods that were naturally adopted by other pipelines (e.g., calculate_shift from Flux).

That's why # Copied from is brilliant - it allows pipelines to evolve independently without worrying about breaking changes in shared utilitiies. Methods become "common" completely organic through adoption, not something we decide upfront.

Instead of a single pipeline_utils.py, I think we can have pipeline-specific utils for each pipeline group. we continue using # Copied from between the pipelines. This way, you can probably abstract more common methods too - you'd only find methods here that are shared by all pipelines, but there will be many more within a single pipeline group.

[JoeGaffney]

Hey,

These are pure, self-contained functions that don’t depend on any specific pipeline logic. Since they’re already used across multiple pipelines and unlikely to change in behavior, it makes sense to centralize them to avoid drift and duplication. The # Copied from pattern is great for early-stage development, but it can introduce maintenance risk in mature pipelines because a function marked “copied from” is not guaranteed to be identical without checking each file.

Having a common.py per pipeline family would also make sense. For example, in Flux there are several functions copied from the main pipeline that could be lifted into pure helpers, keeping the code consistent while still allowing family-level flexibility. Maybe more care and thought is needed here though.

[yiyixuxu]

These are pure, self-contained functions that don’t depend on any specific pipeline logic

I don't agree, these are pipeline logic that are just "common"
we have DiffusionPipeline base class for things that are truly not depends on any specific pipeline logic

[JoeGaffney]

ah yes, you’re right retrieve_timesteps does mutate the scheduler so it’s more pipeline-adjacent than a pure utility. Still the implementation is identical across so many files that it seems worth centralizing the core logic in a shared helper

[sayakpaul]

Having a common.py per pipeline family would also make sense. For example, in Flux there are several functions copied from the main pipeline that could be lifted into pure helpers, keeping the code consistent while still allowing family-level flexibility. Maybe more care and thought is needed here though.

Which is why we're doing #12322.

[sayakpaul]

Instead of a single pipeline_utils.py, I think we can have pipeline-specific utils for each pipeline group. we continue using # Copied from between the pipelines. This way, you can probably abstract more common methods too - you'd only find methods here that are shared by all pipelines, but there will be many more within a single pipeline group.

@yiyixuxu I like the idea! So, on a high level, it could look something like so:

src/diffusers/pipelines/stable_diffusion/pipeline_utils.py
|-> retrieve_timesteps()
|-> retrieve_latents()

src/diffusers/pipelines/flux/pipeline_utils.py
|-> calculate_shift()

And then, src/diffusers/pipelines/stable_diffusion/pipeline_stable_diffusion.py will have an import like from .pipeline_utils import retrieve_timesteps, retrieve_latents.

Pipelines that use those methods could have similar pipeline_utils.py files under their modules and we append those with "# Copied from ...".

Does that reflect what you're envisioning?