`Ptr Word8` instead of `MutableByteArray` in `IOOp`?

[ozkutuk]

First of all, thanks for this nice little library! I am experimenting with doing batch file writes, but I found the shape of IOOpWrite quite inconvenient for the usual things I might want to write (e.g. ByteString), the use of MutableByteArray being the main issue. Looking at the source code, the MutableByteArray is immediately fed to mutableByteArrayContents to access the Ptr Word8 underneath. Therefore, my question is: would it be possible for IOOp (specifically IOOpWrite, but it should probably be fine for IOOpRead too) to take Ptr Word8 directly? That way, it would be possible to, say, write ByteStrings by extracting the ForeignPtr Word8, and in turn the Ptr Word8 within, and creating an IOOp out of that.

I have actually applied this change locally and it seems to be working fine. One thing worth mentioning is that it is not possible to use isMutableByteArrayPinned to guard against unpinned buffers since we would have a plain Ptr Word8. I don't know if there is a way to infer "pinnedness" from a Ptr, but I would guess not.

[MagicRB]

One issue is lifetime, the bytestring must be alive for the duration of the iouring "call"

[ozkutuk]

I have used Foreign.withForeignPtr to extract the Ptr Word8, whose docs state:

The foreign object is kept alive at least during the whole action, even if it is not used directly inside.

...or do you mean something else?

[dcoutts]

The issue is lifetime, but it's even more subtle. Yes the buffer must be live for the duration of the I/O, but that's not quite the same as the duration of the call to submitIO: if you interrupt the thread (by throwing an async exception) then the submitIO will abort (internally it's just waiting on an MVar) but the I/O is not aborted. So in the current design it's really important the buffer be kept alive by the library itself, not just externally by the caller.

This works pretty easily when it's a byte array, but yes for other use cases Ptr would be useful. But it'd have to be Ptr + {thing to keep alive for the duration}.

And we should perhaps revisit the story of async exceptions for threads waiting in submitIO.

[jorisdral]

One could go with a slightly awkward design where, in addition to a Ptr, the user has to provide a "parent object to keep alive" as part of the IOOp. Then blockio-uring can make sure to keep the keepalive object alive for as long as it needs to, though the user could still abuse this API by passing in a Ptr and keepalive object that are unrelated

[ozkutuk]

Admittedly, I am not very well-versed in the intricacies of concurrent Haskell and async exceptions, so I do not have any alternate proposals at this time. Maybe an alternative question could be then: how is this library meant to be used in its current design? Am I trying to stretch it to do something that it was not designed for? If you have any suggestions on how to use this library to do bulk-writes of ByteString/Text, I am all ears.

[dcoutts]

Text is no problem since it uses ByteArrays under the hood. Yes, ForeignPtr for ByteString will need a bit more work.

You're not doing anything wrong or asking for anything unreasonable. It's simply that we developed this lib in a context where we were not using ByteString but heap allocated buffers instead.

[ozkutuk]

Text is no problem since it uses ByteArrays under the hood.

But the ByteArray used by Text is (in general) unpinned, isn't it? Is there a straightforward way to use it with blockio-uring without copying to a new pinned buffer first?

[MagicRB]

All bytearray beyond a certain size are pinned, whether you told GHC to make them pinned or not.

As for Ptr a couldn't we take a (Ptr a, IO ()) where the IO is the "freeing after async exception" action? But idk, best left to @dcoutts. I specifically care about whether I can write a Ptr () piece by piece, where the source of it is FUSE (hi Duncan, remember me from Zurihac? I'm the filesystem thesis dude :P )

[ozkutuk]

All bytearray beyond a certain size are pinned, whether you told GHC to make them pinned or not.

Well, yes, but that "certain size" is 80% of a 4KB block, i.e. 3277 bytes. I expect for a lot of use cases the size of a Text would be smaller, so there is still value in trying do it without copying if possible. Though my hope has diminished a bit since even Data.Text.IO.writeFile seems to be turning the Text into a ByteString (through encodeUtf8) and writing that in the happy path, hence copying in the process.

[jorisdral]

If I find the time, I'll try to prototype some of the suggested approaches soon