Mouse tracking in 3d space of pyvista using local rendering

[HametAl]

Hi there,

I want to track the mouse position in pyvista's 3d space. I tried the following, but it didn't work. Can someone please assist me with this?

import pyvista as pv
from pyvista import examples

from trame.app import get_server
from trame.ui.html import DivLayout
from trame.widgets import html, client, vtk as vtk_widgets
from trame_vtklocal.widgets import vtklocal

WASM = True  # "USE_WASM" in os.environ

# -----------------------------------------------------------------------------


def setup_pyvista():
    p = pv.Plotter()
    mesh = examples.download_st_helens()
    warped = mesh.warp_by_scalar("Elevation")
    surf = warped.extract_surface().triangulate()
    surf = surf.decimate_pro(0.75)  # reduce the density of the mesh by 75%
    p.add_mesh(surf, cmap="gist_earth")
    p.reset_camera()
    p.show_axes()
    widgets = [r.axes_widget for r in p.renderers if hasattr(r, "axes_widget")]
    # p.ren_win.OffScreenRenderingOn()

    return p.ren_win, widgets


# -----------------------------------------------------------------------------


class TrameApp:
    def __init__(self, server=None):
        self.server = get_server(server, client_type="vue3")
        self.render_window, self.widgets = setup_pyvista()
        self.html_view = None
        self.ui = self._ui()

    def _ui(self):
        with DivLayout(self.server) as layout:
            client.Style("body { margin: 0; }")
            with html.Div(
                style="position: absolute; left: 0; top: 0; width: 100vw; height: 100vh;"
            ):
                if WASM:
                    self.html_view = vtklocal.LocalView(self.render_window)
                    for w in self.widgets:
                        self.html_view.register_widget(w)
                else:
                    self.html_view = vtk_widgets.VtkRemoteView(
                        self.render_window, interactive_ratio=1
                    )

        return layout


# -----------------------------------------------------------------------------

if __name__ == "__main__":
    app = TrameApp()
    app.server.start()

Regards
HA

[jourdain]

Can you explain what you mean by track the mouse position in pyvista's 3d space?

[HametAl]

I have a button to draw geometries, say for instance a line. Once the user clicks the button, track_mouse_position, which I think is in screen space, is activated. If the user moves the mouse to select the start point of the line, pick_mouse_position() will pick a point under the mouse in 3D (world) space. I will need to fix the z to 0 so the user always select a point on z=0 plane. After selecting the starting point of the line, user will move the mouse to select the end point of the line. So, a line will be shown between the selected starting point and the current location of the mouse, and I want to print the line's length on the screen so users can see the length while they are moving the mouse to select the end point of the line. It's exactly same as AutoCad feature.

[jourdain]

Ok so the vtkCamera has API to move from display coordinates to 3D space.

[HametAl]

Oh, sorry. Just realised I included the wrong code in my question. I actually meant:

def setup_pyvista():
    p = pv.Plotter()
    mesh = pv.Sphere()
    p.add_mesh(mesh)
    p.show_axes()
    widgets = [r.axes_widget for r in p.renderers if hasattr(r, "axes_widget")]
    p.ren_win.OffScreenRenderingOn()
    def on_mouse_move(obj, event):
         print(p.pick_mouse_position())
    p.track_mouse_position()
    p.iren.add_observer("MouseMoveEvent", on_mouse_move)
    return p.ren_win, widgets

[jourdain]

On VtkRemoteView your need to enable_picking=True to push mouse move if you are not dragging the mouse. That flag is not ideal for you but it should be enough to fix your issue without too much of side effect.

[HametAl]

Hi Jourdain,

Thanks for your reply. Even with enable_picking=True did not work. It looks like pyvista's bug to me. I share 3 different codes here to just see how MouseMoveEvent works:

1. Normal pyvista without trame:

import pyvista as pv
pl = pv.Plotter()
mesh = pv.Sphere()
pl.add_mesh(mesh)

def on_mouse_move(*args):
    print("mouse is moving")
pl.iren.add_observer("MouseMoveEvent", on_mouse_move)
pl.show()

This code works without any problems.

2. Pyvista with RemoteView in Trame:

import pyvista as pv
from trame.app import get_server
from trame.ui.vuetify3 import SinglePageLayout
from pyvista.trame import PyVistaLocalView
import multiprocessing

pl = pv.Plotter(off_screen=True)
mesh = pv.Sphere()
pl.add_mesh(mesh)

def on_mouse_move(*args):
    print("mouse is moving")
pl.iren.add_observer("MouseMoveEvent", on_mouse_move)

server = get_server()
state, ctrl = server.state, server.controller
with SinglePageLayout(server) as layout:
    with layout.content:
        view = PyVistaLocalView(pl)
        ctrl.view_update = view.update

if __name__ == "__main__":
    multiprocessing.freeze_support()
    server.start(exec_mode="desktop")

This code works but not in the way that it is expected. It prints "mouse is moving" whenever you right/left click and zoom in/out, but not when you move your mouse.

3. Pyvista with LocalView in trame:

import pyvista as pv
from trame.app import get_server
from trame.ui.vuetify3 import SinglePageLayout
from pyvista.trame import PyVistaLocalView
import multiprocessing

pl = pv.Plotter(off_screen=True)
mesh = pv.Sphere()
pl.add_mesh(mesh)

def on_mouse_move(*args):
    print("mouse is moving")
pl.iren.add_observer("MouseMoveEvent", on_mouse_move)

server = get_server()
state, ctrl = server.state, server.controller
with SinglePageLayout(server) as layout:
    with layout.content:
        view = PyVistaLocalView(pl, enable_picking=True)
        ctrl.view_update = view.update

if __name__ == "__main__":
    multiprocessing.freeze_support()
    server.start(exec_mode="desktop")

This code doesn't work. Not interaction with mouse at all.

[jourdain]

VtkRemoteView != *LocalView

Any local view is not supposed to send the mouse move.