[SPARK-52409][SDP] Only use PipelineRunEventBuffer in tests

[sryza]

What changes were proposed in this pull request?

This PR inverts the relationship between event callbacks and PipelineRunEventBuffer.

Prior to this PR:

• PipelineUpdateContext had a PipelineRunEventBuffer, that all events from the run were written to.
• This PipelineRunEventBuffer contained an eventCallback val which the Connect backend used to forward events to the client
• The only code that reads from the buffer is test code

After this PR:

• PipelineUpdateContext has an eventCallback, which is invoked on all events.
• The test harness constructs a PipelineRunEventBuffer and passes its addEvent method to the eventCallback

Why are the changes needed?

With the prior code, there was a risk of memory pressure from the event buffer for long-running pipelines.

Does this PR introduce any user-facing change?

No

How was this patch tested?

Existing tests

Was this patch authored or co-authored using generative AI tooling?

No

[sryza]

@jonmio – I'm not sure how to add you as a reviewer, but heads up about this PR.