Graph Dependencies between Rendered Images

[Constantin1709]

I am rendering several images one by one with the Mesh Renderer (with HardPhongShader and MeshRasterizer) that I am concatenating and then passing to a Classifier in batches.
I have no problem with the 1st batch but when performing backward.loss() after the 2nd batch has been passed through the Classifier I am getting a dependency error.

_RuntimeError: Trying to backward through the graph a second time, but the saved intermediate results have already been freed. Specify retain_graph=True when calling backward the first time._

Every time I am rendering a new image I am defining again all the elements that make up the Renderer (cameras, lights, blend_params, raster_settings, MeshRenderer), mainly because I am changing some of the parameters in between generating the individual images.

`

 for obj in meshes:

      for i in range(0, 10):

          # Blender Settings
          cameras = FoVPerspectiveCameras(device=device)

          blend_params = BlendParams_bg(sigma=1e-5, gamma=1e-4, background_image=backgrounds[i])

          raster_settings = RasterizationSettings(
              image_size=256,
              blur_radius=np.log(1. / 1e-4 - 1.) * blend_params.sigma,
              faces_per_pixel=100, )

          # Lights
          lights_location = L[i]
          lights = PointLights(device=device, location=lights_location)

          # Renderer
          phong_renderer = MeshRenderer(
              rasterizer=MeshRasterizer(cameras=cameras, raster_settings=raster_settings),
              shader=HardPhongShader(device=device, cameras=cameras, lights=lights))

          image_ref = phong_renderer(meshes_world=obj, R=R[i]), T=T[i])

          Training_Images.append(image_ref)
          Labels.append(obj_idx)
...
# DataLoader
data = torch.utils.data.TensorDataset(Training_Images, Labels)
data_loader = torch.utils.data.DataLoader(data, batch_size=1, shuffle=True, num_workers=0, drop_last=True)
 
for out_img, ground_truth in dataloader:

        outputs = class_model(out_img) 

        loss = cec(out_img, ground_truth)  
        loss.backward() `

Is there a way to get rid of the dependency between the rendered images that is stored in the computational graph of the image tensors?

Any help or input would be highly appreciated.

[bottler]

The structure of the code shown doesn't quite reflect what you say (it looks like the second loop happens after the first loop). Are there any parameters or other things requiring grad involved in the rendering? I think seeing more of your code may make it easier to help.

[Constantin1709]

Thank you very much for your answer.

It is indeed true that some of the parameters used in the Rendering have a computational Graph attached as they are being predicted by a separate model (lets call it Render_Params_Model ) themselves (Lights Location, Translation, Rotation and Background).
I am trying to preserve these graph informations of the 4 parameters used when rendering an image (the image supposedly owns then the computational graph information of the parameters used to create it).

Furthermore, the generated tensors/images are then passed to classifier. The important bit is that I want to be able to compute the gradient of the final output and update the weights of the Render_Params_Model, hence I need the computational Graph of the Rendered Images that comes from the 4 Parameters.

Finally, yes there are 3 stages that are being executed sequentially: Parameter Prediction, Image Rendering and Image Classification.

`

# Predicting
T, Q, 
L,
 backgrounds = Render_Params_Model(inp_noise)

# Rendering
for obj in meshes:

   for i in range(0, 10):

        cameras = FoVPerspectiveCameras(device=device)
  
        blend_params = BlendParams_bg(sigma=1e-5, gamma=1e-4, background_image=backgrounds[i])
  
        raster_settings = RasterizationSettings(
            image_size=256,
            blur_radius=np.log(1. / 1e-4 - 1.) * blend_params.sigma,
            faces_per_pixel=100, )
  
        lights_location = L[i]
        lights = PointLights(device=device, location=lights_location)
  
        phong_renderer = MeshRenderer(
            rasterizer=MeshRasterizer(cameras=cameras, raster_settings=raster_settings),
            shader=HardPhongShader(device=device, cameras=cameras, lights=lights))
  
        image_ref = phong_renderer(meshes_world=obj, R=R[i]), T=T[i])
  
        Training_Images.append(image_ref)
        Labels.append(obj_idx)

# Training
Training_Images = torch.stack(Training_Images, dim=0)
Training_Images = Training_Images.permute(0, 3, 2, 1)
Labels = torch.tensor(Labels)

data = torch.utils.data.TensorDataset(Training_Images, Labels)
data_loader = torch.utils.data.DataLoader(data, batch_size=1, shuffle=True, num_workers=0, drop_last=True)

for out_img, ground_truth in dataloader:

   outputs = class_model(out_img) 

   loss = cec(out_img, ground_truth)  
   loss.backward() 

`

[bottler]

You want the outer actual parameters to accumulate grads from all the backwards. It could be enough to replace loss.backward() with loss.backward(retain_graph=True).

[Constantin1709]

Thank you Jeremy for your answer. I have been playing around lately with my model and your provided input.
Indeed when setting loss.backward(retain_graph=True) the error disappears, but I am not sure that this is the correct solution. Normally, every forward pass has its own Graph that is only being deleted when backward.loss() is performed on the output of this exact forward pass.

I have 2 open points I am trying to figure out about the Rendering Function:

I am unsure if the rendering step of the Mesh Renderer is keeping the graphs of the individual set of parameters generated in the forward pass (through the Render_Params_Model) as they were before the rendering. So basically: Is the graph information conserved?

Or what might also be a potential cause of the problem is that the Rendering Step is adding information to the Graph of the images and that this reference gets lost when performing the first backward.loss(). So basically: Is the graph being changed?

Maybe it might be important to clarify that the Parameter Prediction Step is actually a loop with several forward passes:

`

 # Predicting
 for k in range(0, len(inp_noise)):
      inp_noise[k] = inp_noise[k].to(device)
      parameters_raw.append(decep_model(inp_noise[k])) 
      # 1 Forward Pass per Parameter Set for 1 Image that is going to be rendered

 T, Q, 
L,
 backgrounds = output_edit(parameters_raw) # Reshaping Outputs to obtain separate parameter lists

`

Thank You!