New Pipeline: FluxFillControlNetInpaintPipeline for FLUX Fill-Based Inpainting with ControlNet

[pratim4dasude]

What does this PR do

This PR adds a new pipeline — FluxFillControlNetInpaintPipeline — located in pipeline_flux_fill_controlnet_inpaint.py.

This pipeline extends FLUX.1-Fill-dev with full ControlNet support for depth, canny, union, and other conditioning models. It enables fill-style inpainting + ControlNet conditioning in a single unified workflow.

We chose FLUX.1-Fill-dev instead of the main FLUX.1-dev model because the regular model does not handle inpainting or masked edits well, especially when combined with styling from Flux Redux.

This variant is specifically designed for mask-based inpainting and produces far more stable and coherent results in these workflows.

How I identified the gap

Existing FLUX pipelines were split:

FluxFillPipeline → fill inpainting only
FluxControlNetInpaintPipeline → ControlNet, with inpainting but no feature for the fill based model

There was no single pipeline combining all three.

How to Use the New Pipeline

Below is the updated example with the correct pipeline name and file import:

import torch
from diffusers import (
    FluxControlNetModel,
    FluxPriorReduxPipeline,
)
from diffusers.utils import load_image

# NEW PIPELINE (updated name)
from pipline_flux_fill_controlnet_Inpaint import  FluxFillControlNetInpaintPipeline

device = "cuda" if torch.cuda.is_available() else "cpu"
dtype = torch.bfloat16

# Models
base_model = "black-forest-labs/FLUX.1-Fill-dev"
controlnet_model = "Shakker-Labs/FLUX.1-dev-ControlNet-Union-Pro-2.0"
prior_model = "black-forest-labs/FLUX.1-Redux-dev"

# Load ControlNet
controlnet = FluxControlNetModel.from_pretrained(
    controlnet_model,
    torch_dtype=dtype,
)

# Load Fill + ControlNet Pipeline
fill_pipe = FluxFillControlNetInpaintPipeline.from_pretrained(
    base_model,
    controlnet=controlnet,
    torch_dtype=dtype,
).to(device)

# OPTIONAL FP8
# fill_pipe.transformer.enable_layerwise_casting(
#     storage_dtype=torch.float8_e4m3fn,
#     compute_dtype=torch.bfloat16
# )

#  OPTIONAL Prior Redux
#pipe_prior_redux = FluxPriorReduxPipeline.from_pretrained(
#    prior_model,
#    torch_dtype=dtype,
#).to(device)

# Inputs
cloth_image = load_image("images.png")
cloth_prompt = "A clean, minimal outfit"

combined_image = load_image("person_input.png")
combined_mask = load_image("mask.png")
control_image_depth = load_image("control_depth.png")

# 1. Prior conditioning
#prior_out = pipe_prior_redux(
#    image=cloth_image,
#    prompt=cloth_prompt,
#)

# 2. Fill Inpaint with ControlNet
result = fill_pipe(
    prompt="A woman wearing the outfit, futuristic and stylish.",
    image=combined_image,
    mask_image=combined_mask,

    control_image=control_image_depth,
    control_mode=[2],  # union mode
    control_guidance_start=0.0,
    control_guidance_end=0.5,
    controlnet_conditioning_scale=0.7,

    height=1024,
    width=768,

    strength=1.0,
    guidance_scale=50.0,
    num_inference_steps=60,
    max_sequence_length=512,

#    **prior_out,
)

result.images[0].save("flux_fill_controlnet_inpaint.png")

Who can review

Anyone in the community is free to review the PR once the tests have passed.
I'm new to contributing here, so please feel free to point out mistakes or roast the code if needed - it will help me improve.
@yiyixuxu and @asomoza

[asomoza]

hi @pratim4dasude can you post a couple of images so we can see the quality of using this pipeline?