Flows incredibly slow to render on large traces. Optional rendering or depth limit?

[Holorite]

Problem

Hello, I have been having some issues with flow events. I have a large trace (1GB+) where all track events are connected via flows. Clicking on an event takes more than a minute to load and render since any connected flows are all automatically rendered.

Furthermore, since selecting an event automatically renders flows, any details normally present in the Current Selection tab are not available until after all flows have rendered which makes the UI essentially unusable.

The ability to show flows on a selection is a nice feature that helps prevent rendering explosions. However, it does not help in this case since the connectivity causes any selection to eventually render every following event's flow.

Example (Tiny trace)

Here is an example showing the dependency chain being rendered on a small trace. When dependency chains expand the load time explodes.
test2.json

Question

Have I missed any existing features able to reduce the render time? Each dependency is currently represented by s, f flow event pairs in the old json format with separate ids. I mention this as I am not sure if the new format allows some optimization which I can take advantage of in the trace.

Possible Solutions

1. Only render flows if the user selects Preceding Flows or Following Flows similar to the behavior of chrome://tracing.
   a. While this would not solve the render time problem it would allow Current Selection details to be viewable.
2. Provide a default depth limit parameter that truncates flow chains after n layers of preceding/following events.
3. Allow viewing full flow chains within a selected time frame. Similar to the current feature but only following a single events chain instead of all events within the selection.

[rsavitski]

I believe we're simply not expecting the connected set of flows to be nontrivial. The rendering is known to be expensive - that's why we don't render all flows by default.

A connected-depth limit sounds reasonable.

@stevegolton to keep me honest

[LalitMaganti]

Alongside what Ryan has said (which I totally agree with), I'd like to offer a workaround which you can today which can somewhat solve this problem without any Perfetto changes: instead of directly attaching the flow to the slice itself, create a child slice (potentially even an instant event) and attach it to that instead.

At that point, the the UI will only show directly connected flows instead of recursively traversing backwards recursively. At that point, see below

Allow viewing full flow chains within a selected time frame. Similar to the current feature but only following a single events chain instead of all events within the selection.

You can actually do this today! Do an area/box select of a region of the trace, then click on the "flow events" panel and then enable "show":

[LalitMaganti]

So I've fixed one obvious performance issue with the cost of getting the flows with #3223.

While looking at this though, I think there are actually several overlapping issues:

1. Even after an order of magnitude speedup, fetching all the flows from the database is still expensive as materialize ~400k rows into Javascript at once which is quite painful
2. Even if we were to fix the query performance, when you click on a flow while zoomed out, the causes catastrophic slowdowns in the rendering performance of the UI due to how many lines it's trying to draw. This is not a problem when zoomed in.

Maybe instead of caching the entire dependency chain, cache a range within some multiple of the view width and update on moves?

The problem is when zoomed out, unless we do something like limiting how many flows there are, this is still going to cause a full materialisation. And honestly thinking more, unless we have some toggle or similar, just not rendering flows is going to be confusing for people as there is a general expectation of all connected flows being rendered. So we cannot just do a naive depth limit.

[LalitMaganti]

To be clear, on out_large.zip [1] reduces the query part from >30s to ~100ms. So multiple orders of magnitude. But it's still not a great experience.

[1] #3223

[LalitMaganti]

You can try out that version of the UI at https://storage.googleapis.com/perfetto-ci-artifacts/gh-18357868856-ui/ui/index.html

[LalitMaganti]

I've filed #3224. We can continue discussion there as issues is better for long chains of discussions like this

[Holorite]

Thanks for the quick patch! I'll test my traces and update on the issue