ControlNet + Inpainting Projection

[carson-katri]

Usage

Improved UI still needs to be created. For now:

1. Place cameras around a single object
2. Select the object and go into edit mode
3. In the projection panel, enable "Use ControlNet" and choose a depth ControlNet model
4. Ensure you have downloaded the models runwayml/stable-diffusion-v1-5 and runwayml/stable-diffusion-inpainting as the model is currently hard-coded
5. Press "Project", Blender will hang while the generation occurs

Method Overview

For the first perspective

1. Generate the first angle with a depth ControlNet and text to image model.

dream-textures/operators/project.py

Lines 528 to 537 in 7884149

	res = gen.control_net(
	model='models--runwayml--stable-diffusion-v1-5',
	control=[depth],
	image=None,
	inpaint=False,
	inpaint_mask_src='alpha',
	**generated_args
	).result()
	generation_result.pixels[:] = res[-1].images[0].ravel()
	generation_result.update()

2. Bake the texture to the result.

dream-textures/operators/project.py

Line 538 in 7884149

color = bake(context, split_mesh, 512, 512, res[-1].images[0].ravel(), projection_uvs, uvs)

For subsequent perspectives

1. Create a mask for faces oriented toward this camera, in both projected and baked forms.

dream-textures/operators/project.py

Line 353 in 7884149

fragColor = dot(NormalMatrix * normal, CameraNormal) > Threshold ? vec4(1, 1, 1, 1) : vec4(0, 0, 0, 1);

2. Capture the object with the latest result texture from this new perspective, and use the mask as the alpha channel.

dream-textures/operators/project.py

Lines 510 to 511 in 7884149

	color, mask, baked_mask = self.masked_init_image(context, camera, np.array(texture.pixels, dtype=np.float32), mesh, split_mesh, uvs)
	color[:, :, 3] = 1 - mask[:, :, 0]

3. Inpaint using the masked color with a depth ControlNet and inpainting model.

dream-textures/operators/project.py

Lines 512 to 523 in 7884149

	res = gen.control_net(
	model='models--runwayml--stable-diffusion-inpainting',
	control=[depth],
	image=np.flipud(color * 255),
	inpaint=True,
	inpaint_mask_src='alpha',
	**generated_args
	).result()
	inpaint_result.pixels[:] = res[-1].images[0].ravel()
	inpaint_result.update()
	mask_result.pixels[:] = mask.ravel()
	mask_result.update()

7. Bake and merge with the latest result texture.

dream-textures/operators/project.py

Lines 524 to 526 in 7884149

	color = bake(context, split_mesh, 512, 512, res[-1].images[0].ravel(), projection_uvs, uvs).ravel()
	baked_mask = baked_mask.ravel()
	color = (np.array(texture.pixels, dtype=np.float32) * (1 - baked_mask)) + (color * baked_mask)

Results

I have not thoroughly tested this, but here is one result using only two camera angles.

Coherence between the two sides seems better than if two completely random generations were merged.