Depth feedback / GPU Picking?

[unphased]

I'm a curious about whether it is possible to have hookable approaches to these techniques.

Here's an example for picking: https://threejsfundamentals.org/threejs/lessons/threejs-picking.html

A common use case for depth feedback is reading the rendered depth value at a particular pixel offset (The mouse position), this would constitute an alternative approach to raycasting, and indeed GPU Picking is somewhat of a generalization of raycasting done from the camera. (Indeed reading depth produces the 3d position, and Picking is a way to obtain some kind of object ID)

For example my present desire is to improve usability in my app by performing raycasting under the mouse to aid in navigating the 3D space (for a CAD application this isn't really an optional feature). Although it's readily doable to use CPU raycasting, it would be nice not to have to worry about complex geometry potentially impacting the performance of the routine. It also seems like there is an opportunity to put a foundation in place so that gpu based techniques can be used universally for mouse/ui raycasting needs, although it may be quite an undertaking considering the complexity of replicating a bubbling event model.

If we try to simplify and only climb a small way up the ladder of ambition, I wonder if perhaps there's something that can be hacked together with EffectComposer to just allow for reading out pixels from a depth render target or texture?

[unphased]

I'll be back after doing more required reading on the post processing tooling that's already available to three.js.

[gsimone]

Trying to reason through this:

1. add the id attribute to the geometry, so that you can eventually use it in a shader
2. add a pass to your postprocessing pipeline that ( I personally use https://github.com/vanruesc/postprocessing/ )

• replaces material with a shader material that uses id to color
• renders to a buffer

3. in your "MousePicking" component get the buffer and read from pixel with readRenderTargetPixels, obtaining a color that you'd convert to the object id.
4. dispatch the event to your components

[drcmda]

i've implemented mouse picking before - but it comes with greater caveats. to wire that into pointer events, well, that's gonna be complex but i think r3f exposes enough data to make this possible in user-land.

[unphased]

Thanks for the notes! Yes this is helpful. if you guys think there's no obvious roadblocks down that path then it's good enough for me to consider the path as a potential way forward.

I also forgot yesterday to mention that another method is of course utilizing a spatial acceleration structure to continue doing raycasts on the CPU!

[gsimone]

If that's an option, there's is a newish example in THREE that uses an octree https://threejs.org/examples/#games_fps

[unphased]

Wow I have not seen this example! It's a very nice example. I'm also impressed that the bulk of its code is actually physics computations being baked in (no doubt to show off/leverage the octree). Although perhaps not a sane way to architect an actual project, this really simplifies the example and tickles the minimalist in me a lot.

[unphased]

Hi all. I've actually gotten mouse picking working now after going through an interesting journey into postprocessing. It seems to work! However I'm stuck on a bit of a funny problem.

As you see I need to obtain the NDC coordinates of e.g. a mouse click in order to pass it in (to determine the precise location of the depth buffer to sample from the GPU back out to the CPU), I've found that attaching an event handler doesn't fire. I wonder if R3F explicitly captures all mouse events within. I wonder if there's a simple way I can get this back without hacking too much into R3F?

[drcmda]

i guess the simplest would be to overwrite Mesh.prototype.raycast and give it your own behaviour, that way you dont need to do anything and you would profit from r3f's event system.

[gsimone]

I think you should be able to get what you need in useFrame

useFrame(({ mouse }) => {
   // mouse holds NDC for the frame
})

[unphased]

wow if useFrame has mouse coords then that would do the trick!!! I'll take a look at this. Thanks.

With replacing Mesh.prototype.raycast would that work globally no matter where I click in the scene?

[unphased]

@gsimone useFrame works great, but only runs when a frame is rendered. So in the frame invalidation enabled scenario, this is a catch-22. I'm back at square one when it comes to binding a mouse event on the canvas element in order to be able to run invalidate().

[michaeldll]

Hello @unphased, I'm about to implement a GPU picking solution to my app and I'd be very interested in seeing how you went about it !

[drcmda]

have you guys tried https://github.com/pmndrs/drei#usebvh ?

[unphased]

Having a bvh implementation available is very awesome. But if the GPU read approach is a good fit for the application, it's a much more predictable constant runtime cost to use it. You can refer to this thread @michaeldll pmndrs/postprocessing#270

Note what I implemented is just to pick a 3D position out of a mouse click, it does not provide any object picking capability, as that would require another deferred render pass to be done and I did not need that for what I was working on.