BaseByteArrayJournalDao lacks resiliency against stream failure

[Arkatufus]

BaseByteArrayJournalDao does not check to see if any of the streams it uses failed/completed/terminated, causing the journal to fail with no way of recovering, the only way to recover is to restart the node.

Chain of event:

[Arkatufus]

Possible scenario:

1. An exception/failure/timeout happened inside the SelectAsync() stage while a write was happening, stopping the stream (first figure).
2. The exception did not get propagated, causing an NRE to happen when it is being propagated up the stream (second figure).
3. The journal was not aware that the stream has stopped and failed to restart either itself or the stream, poisoning the persistence system (third figure).

[to11mtm]

This is confusing to me, primarily because AFAIK we have a fairly simple SelectAsync stage that should prevent stream stop normally:

               .SelectAsync(
                    JournalConfig.DaoConfig.Parallelism,
                    async promisesAndRows =>
                    {
                        try
                        {
                            await WriteJournalRows(promisesAndRows.Rows);
                            foreach (var taskCompletionSource in promisesAndRows.Tcs)
                                taskCompletionSource.TrySetResult(NotUsed.Instance);
                        }
                        catch (Exception e)
                        {
                            foreach (var taskCompletionSource in promisesAndRows.Tcs)
                                taskCompletionSource.TrySetException(e);
                        }

                        return NotUsed.Instance;
                    })

I'm honestly confused by this one, unless something with how shutdownToken (the only thing that could trigger a cancellation where the error makes sense in my head), and it's use in calls to ExecuteWithTransactionAsync are done is a culprit.

🤔 I take it half back. I guess the one other thing to look into, is whether linq2db has some of their own cancellationtoken stuff going on and however we are hitting it is wrong. I will note that if we give a good repo, it will get fixed (although providing our own PR may help that go faster >_<)