Support individual viewport settings / LOD selection when using multiple views

[Lee-Bull]

Hi, I'm using Cesium Native for a project and wondered if you could clarify some issues I'm considering with the API.

My software requires multi viewport rendering, with all views sequentially, progressively read and rendered over a short period of time until they all converge at the required LODs. Usually what happens in engines of this type is that multiple queries of various types, including for rendering, are created and synchronously or asynchronously executed with data fetched in the backgrounds thread(s) with LOD control and fetch ordering considering all query requirements and prioritizations.

Cesium3DTilesSelection::Tileset::updateView() would appear to provide a feature for multiple view rendering, given that it receives a vector of ViewState frustums. However, given that it returns a single ViewUpdateResult with a single tilesToRenderThisFrame tile list, it doesn't seem obvious how the tiles appropriate to each frustum would be obtained, or sorted out of that list.

Alternatively, if it's feasible to only provide a single ViewState to multiple calls to updateView() for each view, that would be doable, but looking at the object lifetime for returned Tiles purging between calls (and that they're not on smart pointers) it would suggest that's not necessarily an intended approach. The internals of updateView() would need to be able to handle this kind of thrashing around between views too.

I guess it's possible to have an entire Tileset pipeline per view, but that seems wasteful and would need to rely on caching to share data between instances for some period.

Would you please be able to shed some light on this one?

Kind Regards, Lee.

[kring]

Hi @Lee-Bull,

The intention is that you call updateView with the multiple views. Then, render all of the tilesToRenderThisFrame for every view. You may be able to skip some of the tiles in some of the views by doing extra frustum or occlusion checks. In Unreal and Unity, for example, these visibility checks are automatically done by the engine.

It's conceptually possible to build up an optimal set of tiles for each view, but that's not currently implemented. It would inherently require loading more tiles (because tiles covering the same area at multiple LODs would be required, whereas the current approach only covers any given part of the globe with one LOD), though it might lighten the rendering load.

[Lee-Bull]

Hi Kevin,

Thanks very much for your swift reply. It would be preferable to have individual updateView queries called on each View at different times, asynchronously returning data currently available (or waiting to complete), with results that use smart pointers to required Tiles for the view so the result object lifetime remains as required for processing.

I can see some benefit of taking the union of all LOD requirements for the views. I can see issues with it too, for example the rendering performance drop as it's not really per view LOD control, also maybe some aliasing. Also popping/hysteresis if one view A of a region requires a high LOD, forcing view B to use the high LOD, but view A then moves away and the LOD drops suddenly in B.

You're right in saying dealing with each view individually would require more tile reads, at least for replacement hierarchies.

If we use the current approach, it means we need to cull working set leaf tiles to views in linear time. Presumably culling has already been done internally during the update traversal, so it's a shame we can't leverage that.

I'll have a think. We'll either think of a way of using Cesium Native's updateView approach or develop custom traversal code that's closer to our needs, though ideally we'd rather keep as closely in sync with the Native API as possible.

Thanks very much for your time, Lee.

[kring]

Yeah I don't disagree with anything you've written. It would be possible and useful to extend cesium-native to support proper per-viewport tile selection, but it will take some work.

[Lee-Bull]

Thanks, still thinking this one over, it's tough to shoehorn it into our current engine framework which requires separate, iterative queries performed at any time. It's still 50/50 whether I use Native's updateView() or write something custom. Contributing to the Cesium code base might be possible but require more knowledge of it than I have at the moment.

[kring]

I'm starting to look at what it will take to do this. Just random notes here so far.

The first major impediment to true "multi view" support in cesium-native is that the selection algorithm uses and modifies the _lastSelectionState field on the Tile. Which tiles are selected for a view is a function, in part, of which tiles were selected last frame. See Selecting Loaded Tiles in the selection algorithm documentation. To have multiple views, we need a way to store the last selection state "per view".

The simple and obvious way to do this is to remove the _lastSelectionState field from the Tile, and instead have a per-view map of Tile* to last selection state. This will work fine, but will have significant overhead compared to a field, and that overhead might matter. Last selection states are looked up a thousand times per frame in some cases.

If that does turn out to be a bottleneck, we can instead store Tile* and selection state pairs in a flat array in the order in which the tiles are traversed in that view. As the view changes (or as new tile subtrees are loaded), a tile subtree might be visited that wasn't previously. Or we might not visit a subtree that we did visit last frame. But in either case, we should be able to efficiently find the last selection state of any given tile as we traverse it by tracking our position in the array and updating the current position in the array during the traversal. Details are vague here because I haven't thought through it completely, but I'm pretty sure it's possible.

[kring]

The next challenge will be tile load prioritization. If we don't do anything to avoid it, whichever view goes first will use up all the maximumSimultaneousTileLoads slots, and other views will be starved.

Instead, we need to give every view a chance to select tiles, and then load the "most important" tiles across all views. If all views are passed simultaneously to updateView, as they are currently, then we can simply prioritize and load tiles after running selection against all views. A tile is loaded with the highest priority it has in any view. If each view is allowed to call updateView independently (as has been requested), we'll need some way for the user to mark the end of a "frame" of updates (it need not corresponding with actual render frames).

That's all relatively straightforward, but the next thing that users will surely want is the ability to prioritize some views over others. For example, the current view the user is interacting with should be prioritized over ancilliary views.

The simplest thing I can think of is to give each view a priority factor that is multiplied with the standard priority for that view. In theory, that should allow us to give a particular view 50% higher priority or similar. But there are problems with this. For one thing, there are priority groups. How can a priority boost factor make a Normal priority tile in one important view load before an Urgent priority tile in an unimportant view? Is that even desirable?

The other problem is that the priority value itself (ignoring the groups) is sort of fuzzy, and we can't assume it's linear. The current metric is useful for prioritizing tiles that are visible in a single view, but not necessarily across multiple views. Because it's a function of distance, zoomed-out views will always be lower priority than zoomed-in ones. And no reasonable priority scaling factor wil fix this.

It's not obvious to me how to fix this, but it probably requires a normalized priority metric of some sort that is meaningful across views and is roughly linear, i.e., we should be able to say that if tile A has a priority value double tile B's, then tile A is twice as important as tile B. Need to think through what this might look like. Screen-space error is a good place to start, but we'd also like to increase the priority of tiles near the center of the view and decrease the priority of tiles that are behind the camera (such tiles may still be loaded for various reasons, and SSE is oblivious to direction).