Better testing UX for telemetry

[whatyouhide]

Testing with telemetry is not as good as it could be, says the guy who initially implemented telemetry_test. 🤡

The Problem

You do this:

telemetry_ref = :telemetry_test.attach_event_handlers(self(), events)

Great. Now your test receives all events in events, even if they're emitted from other tests. Basically, you get an n × m problem where n tests subscribe to m events and each test gets all of em. Yuck.

This can result in:

1. False positives. assert_receive on those telemetry events might be a false positive because you're testing that your system under test is emitting the telemtry event, but if you're not strict enough with the assertions on meta/measurements, then you might be catching an event emitted by code being tested by another test.
2. Kind of impossible to refute events. refute_received will mostly catch events from other async tests, same as the previous point.

Solution

Need help here 🙃 What we're doing at work is having an abstraction layer over telemetry (we had this anyway for other stuff), so that essentially every call to telemetry goes through our layer. Then, in our layer, we inject this in the meta of every event:

Map.put(meta, :__test_pid__, ProcessTree.get({:our_telemetry_layer, :test_pid})

All our tests that use telemetry have some sort of

setup do
  Process.put({:our_telemetry_layer, :test_pid}, self())
  :ok
end

This allows us to "scope" telemetry events for our tests. We have helpers to assert on events that essentially generate this code:

assert_receive ..., %{__test_pid__: ...}
refute_received ..., %{__test_pid__: ...}

This works, but maybe there's a better way for this in telemetry itself. Thoughts?