VTK Volumes disappear after indefinite amount of time

[maflancer]

I am currently working on an app using Trame and VTK. Users are able to upload a few different file types and the app will display the 3D volume. I have now been working on setting up slicing of the volumes and have run into an issue that I've been stuck on for a bit.

When I first run the app, everything works correctly, and users can use the slice plane dropdown and slice index slider to view 2D slices of the 3D volume. Here is a screenshot of what this looks like:

After moving the slider around for some time, some of the slices will not display. It is very inconsistent, sometimes it will take 30 seconds before slices stop showing or sometimes it will take a few minutes before it stops working. Also, no errors show up in the console when this occurs.

I am not sure if this is a rendering issue or if there is a bug in my code that I am missing. One interesting thing I noticed is that when I toggle the image slice view to off and then back to on the slice that previously didn't display now shows up.

Here is the function that handles the slice plane and slice index changes:

@state.change("slicing_plane")
@state.change("slice_index")
def update_slice_renderer(slicing_plane, slice_index, **kwargs):
    global img_arr
    if state.show_slice_view and slices:
        if (
            (slicing_plane == "Z" and 0 <= slice_index < img_arr.shape[0])
            or (slicing_plane == "Y" and 0 <= slice_index < img_arr.shape[2])
            or (slicing_plane == "X" and 0 <= slice_index < img_arr.shape[3])
        ):
            create_slice_data(
                img_arr,
                state,
                ctrl,
                slices,
                slice_plane_actor,
                renderer,
                slice_renderer,
                slice_index=slice_index,
                slice_plane=slicing_plane,
            )

    ctrl.view_update()
    ctrl.view_reset_camera()
    ctrl.slice_view_update()
    ctrl.slice_view_reset_camera()

Here is the function that is triggered when the slice view is toggled:

@state.change("show_slice_view")
def toggle_slice_plane_visibility(show_slice_view, **kwargs):
    if show_slice_view:
        state.volume_visible = True
        for i in range(state.channels):
            volumes[i].SetVisibility(1)
            slices[i].SetVisibility(1)

    if slice_plane_actor:
        slice_plane_actor.SetVisibility(1 if show_slice_view else 0)

    ctrl.slice_view_update()
    ctrl.view_update()

Here is the create_slice_data function:

def create_slice_data(
    img_arr,
    state,
    ctrl,
    slices,
    slice_plane_actor,
    renderer,
    slice_renderer,
    slice_index=0,
    slice_plane="Z",
):
    """
    Creates 2D VTK volume objects from multi-channel image data to
    render image slices in the Trame application.

    This function handles the visualization of multi-channel image slices by
    creating and updating 2D VTK volume objects and adding them to the renderer.
    It normalizes and scales the raw image data from the input array,
    configures the VTK data structures, and sets up the slice rendering
    parameters.

    Args:
        img_arr (numpy.ndarray): The image array with shape (Z, C, Y, X), where
                                 Z is the number of slices,
                                 C is the number of channels,
                                and Y, X are the pixel dimensions of each slice.
        state (trame.State): The state object from the Trame application,
                             used to manage and update the UI state.
        ctrl (trame.Controller): The controller object from the Trame
                                 application, used to trigger UI updates.
        slices (list[vtk.vtkVolume]): A list that holds the slice objects
                                      that are currently being managed
                                      and displayed in the VTK renderer.
        slice_plane_actor (vtk.vtkActor): The VTK actor object associated with
                                          the slice plane.
        renderer (vtk.vtkRenderer): The VTK renderer object used for
                                    displaying the slice plane.
        slice_renderer (vtk.vtkRenderer): The VTK renderer object used for
                                          displaying the image slices.
        slice_index (int, optional): The index of the slice to be displayed.
                                     Defaults to 0.
        slice_plane (str, optional): The plane in which to slice the data.
                                     Must be one of 'X', 'Y', or 'Z'.
                                     Defaults to 'Z'.

    Returns:
        None: The function directly modifies the `slices` list and updates the
              renderer state.
    """
    # with open("/deploy/config.yml", "r") as file:
    with open("config.yml", "r") as file:
        config = yaml.safe_load(file)

    for slice in slices:
        slice_renderer.RemoveVolume(slice)
    slices.clear()

    for channel in range(state.channels):
        slice_data = None
        try:
            if slice_plane == "Z":
                slice_data = img_arr[slice_index, channel, :, :]
            elif slice_plane == "Y":
                slice_data = img_arr[:, channel, slice_index, :]
                slice_data = np.transpose(slice_data, (1, 0))
            elif slice_plane == "X":
                slice_data = img_arr[:, channel, :, slice_index]
                slice_data = np.transpose(slice_data, (1, 0))
            else:
                raise ValueError(
                    "Invalid slice_plane. Must be one of 'X', 'Y', or 'Z'."
                )
        except IndexError:
            return

        slice_data_scaled = (
            (slice_data - slice_data.min())
            / (slice_data.max() - slice_data.min())
            * 255
        ).astype(np.uint8)

        slice_vtk_data_array = numpy_support.numpy_to_vtk(
            num_array=slice_data_scaled.ravel(),
            deep=True,
            array_type=vtk.VTK_UNSIGNED_CHAR,
        )

        slice_vtk_image = vtk.vtkImageData()
        if slice_plane == "Z":
            slice_vtk_image.SetDimensions(
                slice_data_scaled.shape[1], slice_data_scaled.shape[0], 1
            )
        elif slice_plane == "Y" or slice_plane == "X":
            slice_vtk_image.SetDimensions(
                slice_data_scaled.shape[1], 1, slice_data_scaled.shape[0]
            )

        slice_vtk_image.SetSpacing(1, 1, 1)
        slice_vtk_image.GetPointData().SetScalars(slice_vtk_data_array)

        slice_mapper = vtk.vtkFixedPointVolumeRayCastMapper()
        slice_mapper.SetInputData(slice_vtk_image)

        # Create transfer mapping scalar value to opacity
        slice_opacity_transfer_function = vtk.vtkPiecewiseFunction()
        slice_opacity_transfer_function.AddPoint(0, 0.0)
        slice_opacity_transfer_function.AddPoint(255, config["slice_opacity"])

        # Create transfer mapping scalar value to color
        slice_color_transfer_function = vtk.vtkColorTransferFunction()
        slice_color_transfer_function.AddRGBPoint(
            0, *config["default_channel_colors"]["min"]
        )
        slice_color_transfer_function.AddRGBPoint(
            255, *config["default_channel_colors"]["max"]
        )

        slice_property = vtk.vtkVolumeProperty()
        slice_property.SetColor(slice_color_transfer_function)
        slice_property.SetScalarOpacity(slice_opacity_transfer_function)
        slice_property.ShadeOn()
        slice_property.SetAmbient(config["slice_ambient"])
        slice_property.SetDiffuse(config["slice_diffuse"])
        slice_property.SetSpecular(config["slice_specular"])
        slice_property.SetInterpolationTypeToLinear()

        slice = vtk.vtkVolume()
        slice.SetMapper(slice_mapper)
        slice.SetProperty(slice_property)
        slice_renderer.AddVolume(slice)
        slices.append(slice)

    if slice_plane_actor:
        renderer.RemoveActor(slice_plane_actor)
        slice_plane_source = vtk.vtkPlaneSource()
        spacing = [1, 1, 1]
        margin = config["slice_plane_margin"]

        if slice_plane == "Z":
            origin = [-margin, -margin, slice_index * spacing[2]]
            point1 = [
                img_arr.shape[3] + margin,
                -margin,
                slice_index * spacing[2],
            ]
            point2 = [
                -margin,
                img_arr.shape[2] + margin,
                slice_index * spacing[2],
            ]
        elif slice_plane == "Y":
            origin = [-margin, slice_index * spacing[1], -margin]
            point1 = [
                img_arr.shape[3] + margin,
                slice_index * spacing[1],
                -margin,
            ]
            point2 = [
                -margin,
                slice_index * spacing[1],
                img_arr.shape[0] + margin,
            ]
        elif slice_plane == "X":
            origin = [slice_index * spacing[0], -margin, -margin]
            point1 = [
                slice_index * spacing[0],
                img_arr.shape[2] + margin,
                -margin,
            ]
            point2 = [
                slice_index * spacing[0],
                -margin,
                img_arr.shape[0] + margin,
            ]

        slice_plane_source.SetOrigin(origin)
        slice_plane_source.SetPoint1(point1)
        slice_plane_source.SetPoint2(point2)

        slice_plane_mapper = vtk.vtkPolyDataMapper()
        slice_plane_mapper.SetInputConnection(
            slice_plane_source.GetOutputPort()
        )
        slice_plane_actor.SetMapper(slice_plane_mapper)
        slice_plane_actor.GetProperty().SetColor(*config["slice_plane_color"])
        slice_plane_actor.GetProperty().SetOpacity(
            config["slice_plane_opacity"]
        )
        renderer.AddActor(slice_plane_actor)

    renderer.ResetCamera()
    slice_renderer.ResetCamera()
    ctrl.slice_view_reset_camera()
    ctrl.view_reset_camera()

And here is my trame setup:

def content_layout(ctrl, state, render_window, slice_window):
    with vuetify.VContainer(fluid=True, classes="pa-0 fill-height"):
        with vuetify.VRow(classes="fill-height"):
            with vuetify.VCol(xs12=True, classes="pa-1 fill-height"):
                view_3d = vtkwidget.VtkLocalView(render_window)
                ctrl.view_update = view_3d.update
                ctrl.view_reset_camera = view_3d.reset_camera

            with vuetify.VCol(
                xs12=True, v_if="show_slice_view", classes="pa-1 fill-height"
            ):
                view_2d = vtkwidget.VtkLocalView(slice_window)
                ctrl.slice_view_update = view_2d.update
                ctrl.slice_view_reset_camera = view_2d.reset_camera

Please let me know if there is anything else I can provide that would help debug this issue! Thanks!

[jourdain]

Can you replace VtkLocalView by VtkRemoteView(..., interactive_ratio=1) or do you need it to be local rendered?

Long story short, you are running into a bug in the state synchronization protocol from the VtkLocalView. Ideally that will naturally be fixed by trame-vtklocal once available (although volume rendering is not supported in WASM currently).

[maflancer]

I went through those steps and made some changes and I can now run the Nvidia and egl commands in my docker container so it does seem like the container has access to my GPU.

However, I still am having issues where I am unable to rotate the 3D volume and 2D slices are not displayed correctly at all.

I tried changing my volume mappers from vtk.vtkFixedPointVolumeRayCastMapper() to vtk.vtkGPUVolumeRayCastMapper() but my slices are still not displaying correctly. I didn't think I'd have to modify my code for the VtkRemoteView to work since it all worked in VtkLocalView but is there anything else you think i could try to fix this?

I get that this could still just be an issue with my Docker/NVIDIA/EGL setup but I am curious if there's anything else I could try before diving back into that.

[jourdain]

I don't know why you can't rotate the 3D view. Did you put an invisible layer in html that grab the mouse interaction?
For the planes, if they don't render the way you want, you need to fix your pipeline that defines them, not the volume rendering.
Unless you have some custom volume orientation and when you say the planes don't render correctly, you mean the volume don't render in the right spot? But as you can imagine, it is fairly hard to guess what you've done, and what you expect with just word and not picture or context.

I don't think you have any issue with your docker EGL setup unless it is super slow.

[maflancer]

So this is what the app looks like when I run with vtkLocalView. I am able to rotate the volume smoothly and can see the slice:

And here is what it looks like when I run the same code but switch vtkLocalView to vtkRemoteView and vtk.vtkFixedPointVolumeRayCastMapper() to vtk.vtkGPUVolumeRayCastMapper(). I am also now unable to rotate the volume. I use the same renderwindowinteractor setup as I did with the local view:

Here is some of my code for reference:

View setup:

with vuetify.VContainer(fluid=True, classes="pa-0 fill-height"):
        with vuetify.VRow(classes="fill-height"):
            with vuetify.VCol(xs12=True, classes="pa-1 fill-height"):
                # view_3d = vtkwidget.VtkLocalView(render_window)
                view_3d = vtkwidget.VtkRemoteView(
                    render_window, interactive_ratio=1
                )
                ctrl.view_update = view_3d.update
                ctrl.view_reset_camera = view_3d.reset_camera

            with vuetify.VCol(
                xs12=True, v_if="show_slice_view", classes="pa-1 fill-height"
            ):
                # view_2d = vtkwidget.VtkLocalView(slice_window)
                view_2d = vtkwidget.VtkRemoteView(
                    slice_window, interactive_ratio=1
                )
                ctrl.slice_view_update = view_2d.update
                ctrl.slice_view_reset_camera = view_2d.reset_camera

# VTK renderer setup
renderer = vtk.vtkRenderer()
render_window = vtk.vtkRenderWindow()
render_window.AddRenderer(renderer)
render_window_interactor = vtk.vtkRenderWindowInteractor()
render_window_interactor.SetRenderWindow(render_window)

slice_renderer = vtk.vtkRenderer()
slice_window = vtk.vtkRenderWindow()
slice_window.AddRenderer(slice_renderer)
slice_window_interactor = vtk.vtkRenderWindowInteractor()
slice_window_interactor.SetRenderWindow(slice_window)

# Trame server setup
server = get_server(client_type="vue2")
state, ctrl = server.state, server.controller

Slice setup

for channel in range(state.channels):
        slice_data = None
        try:
            if slice_plane == "Z":
                slice_data = img_arr[slice_index, channel, :, :]
            elif slice_plane == "Y":
                slice_data = img_arr[:, channel, slice_index, :]
                slice_data = np.transpose(slice_data, (1, 0))
            elif slice_plane == "X":
                slice_data = img_arr[:, channel, :, slice_index]
                slice_data = np.transpose(slice_data, (1, 0))
            else:
                raise ValueError(
                    "Invalid slice_plane. Must be one of 'X', 'Y', or 'Z'."
                )
        except IndexError:
            return

        slice_data_scaled = (
            (slice_data - slice_data.min())
            / (slice_data.max() - slice_data.min())
            * 255
        ).astype(np.uint8)

        slice_vtk_data_array = numpy_support.numpy_to_vtk(
            num_array=slice_data_scaled.ravel(),
            deep=True,
            array_type=vtk.VTK_UNSIGNED_CHAR,
        )

        slice_vtk_image = vtk.vtkImageData()
        if slice_plane == "Z":
            slice_vtk_image.SetDimensions(
                slice_data_scaled.shape[1], slice_data_scaled.shape[0], 1
            )
        elif slice_plane == "Y" or slice_plane == "X":
            slice_vtk_image.SetDimensions(
                slice_data_scaled.shape[1], 1, slice_data_scaled.shape[0]
            )

        slice_vtk_image.SetSpacing(1, 1, 1)
        slice_vtk_image.GetPointData().SetScalars(slice_vtk_data_array)

        slice_mapper = vtk.vtkGPUVolumeRayCastMapper()
        slice_mapper.SetInputData(slice_vtk_image)

        # Create transfer mapping scalar value to opacity
        slice_opacity_transfer_function = vtk.vtkPiecewiseFunction()
        slice_opacity_transfer_function.AddPoint(0, 0.0)
        slice_opacity_transfer_function.AddPoint(255, config["slice_opacity"])

        # Create transfer mapping scalar value to color
        slice_color_transfer_function = vtk.vtkColorTransferFunction()
        slice_color_transfer_function.AddRGBPoint(
            0, *config["default_channel_colors"]["min"]
        )
        slice_color_transfer_function.AddRGBPoint(
            255, *config["default_channel_colors"]["max"]
        )

        slice_property = vtk.vtkVolumeProperty()
        slice_property.SetColor(slice_color_transfer_function)
        slice_property.SetScalarOpacity(slice_opacity_transfer_function)
        slice_property.ShadeOn()
        slice_property.SetAmbient(config["slice_ambient"])
        slice_property.SetDiffuse(config["slice_diffuse"])
        slice_property.SetSpecular(config["slice_specular"])
        slice_property.SetInterpolationTypeToLinear()

        slice = vtk.vtkVolume()
        slice.SetMapper(slice_mapper)
        slice.SetProperty(slice_property)
        slice_renderer.AddVolume(slice)
        slices.append(slice)

[jourdain]

The local view ignore the interactor setup and just force its own locally.
So like I said, if it is not working with remote view, that is because your VTK code is not correct.
Also why do you render your slice in volume rendering and why do you build the data by hand? Why don't you use VTK to generate those slice for you?

[jourdain]

It seems that your main issue is not with trame but VTK. Please interact with vtk discourse to solve your viz or look at the vtk examples for reference.

[maflancer]

I attempted to use the vtkImageReslice but was running into some issues. I am going to try that again though.

I'm going to spend some time looking over all of my VTK setup to try and fix it.

I'll mark this discussion as answered since I was able to figure out the cause of the issue and get my docker setup working with GPU and EGL.

Thanks again for all of your help!

[wyattnapier]

I'm having a similar issue, except I haven't gotten to the deployment stage yet. I'll make a new discussion post with my own code and maybe there is some similarity there that will help us both find a solution unless Sebastien's solution worked for Matthew.