MarkerArray renders in the wrong position

[rg-gravis]

Found a strange bug in rendering visualization_msgs/msg/MarkerArray. Attached are two bags in a zip file, and a layout to make the bug obvious. If you open arrow_bug.mcap and jump to timestamp 1729779302, you'll see that the two marker arrows are not close to the custom grid square, despite the fact that they should be on top of each other - both are relative to the BASE frame and both have a position of roughly 7.39, 0.04, -1.05. The markers instead appear to have an x coordinate of approximately 9 relative to BASE.
However, the location of the arrows is correct in arrow_shorter_no_bug.mcap, which was produced from the exact same source bag, just trimmed in time with mcap filter. The exact commands to produce each were

mcap filter full_bag.mcap -o arrow_bug.mcap -y /tf -y /grading/vis_markers
mcap filter full_bag.mcap -o arrow_shorter_no_bug.mcap -y /tf -y /grading/vis_markers -s 1729779301 -e 1729779321

The behavior I would expect here is the one seen in the shorter bag, where the arrows appear in the same space as the custom grid. That is how this appears in rviz.

Attached
arrow_bug_layout.json
marker_array_bug_bags.zip

[defunctzombie]

Tracking issue: https://linear.app/foxglove/issue/FG-9260/markerarray-renders-in-the-wrong-position

[mkerpelman]

Hello, we took a look and spotted the problem: your header timestamps in your MarkerArray messages is zero, whereas Foxglove uses this timestamp to determine where to place your markers in the scene by referencing your transforms.

This explains why the shorter recording works: it starts when the BASE frame is in a fixed position for the rest of the recording. The longer recording has some early movement that is cut off in the shorter one. The timestamp being 0 for all MarkerArray messages, Foxglove matches up where to place the markers with the first available transform message.

To get the expected behavior, you'd need to publish your MarkerArrays with the appropriate timestamp, e.g., publish time. Let me know if that makes sense.

[rg-gravis]

Yes, that makes sense and fixed the issue, thanks! A bit confusing that it works differently than in RViz, but I think Foxglove's way of handling this is more correct.