Calling stop() can cause projections to loose stream positions #233
Description
I have observed this issue over some time, but had a hard time reproducing it. In rare instances during deployments, projections could loose their stream positions and start from the beginning.
This can happen in instances where a pcntl stop handler invokes a projection's stop() method before the projection has loaded the stream positions.
The problematic section is in the run() method, before the actual main loop:
// Initial State is ProjectionStatus::IDLE()
// $this->streamPositions is an empty array
if (! $this->projectionExists()) {
$this->createProjection();
}
$this->acquireLock();
if (! $this->readModel->isInitialized()) {
$this->readModel->init();
}
$this->prepareStreamPositions();
$this->load(); // << Only here $this->streamPositions will be initialized
Calling stop() from the signal handler before load() loaded the stream positions, causes a call to persist((). The persist call saves the uninitialized stream positions (an empty array).
This issue becomes more frequent / likely the longer it takes the projection to load the stream positions. Putting a sleep() call after the acquireLock() call in the example above makes it rather easy to reproduce (we have replaced the locking mechanism with metadata locks on a 60 second timeout, hence more likely for us to run into this issue).
The current workaround for us seems to be to introduce a new flag $this->streamPositionsLoaded and have an early return in persist() if positions have not been loaded yet.
private function persist(): void
{
if (!$this->streamPositionsLoaded) {
return;
}
$this->readModel->persist();
// ...
We are running this in testing at the moment to see if this actually fixes the issue. Any other input / feedback / solutions are welcome though.