Skip to content

Float32Array has insufficient precision in call tree timings - incorrect byte sums displayed in call tree #5310

Closed
@mstange

Description

@mstange

Profile: https://share.firefox.dev/40hmh0v

In this profile, the call tree displays "Total Size (bytes): 485,408,992" for the root node.
But if you run filteredThread.samples.weight.reduce((a, b) => a + b, 0) on the console, you get a sum of 485,409,290.

Those two values should be the same but they aren't. One ends in 8,992 (incorrect) and the other ends in 9,290 (correct).

This happens because the call tree timings store the total value per node in a Float32Array. But 32-bit floating point values can only exactly represent integers up to 16,777,216.

I must have chosen Float32Array when I wasn't thinking of byte values.

┆Issue is synchronized with this Jira Task

Metadata

Metadata

Assignees

Labels

call treeRelated to the call tree panel

Type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions