Application Level Acknowledgements

[sam-golioth]

This changes the Pouch GATT transport to use application-level sliding window acknowledgments on top of unreliable Bluetooth GATT transfers. Doing so eliminates the need to wait for round-trip acknowledgements on each packet, and results in significant speedups, as much as 10x in testing. The window size is configurable, and severely constrained devices can lower the window all the way back down to 1, in which case the transfer speed is essentially equivalent to the previous speed when using reliable GATT writes.

[trond-snekvik]

There are slightly too many corner cases in the sliding window mechanism for me to confidently say whether everything works as it should, but overall, this looks good to me.

[trond-snekvik]

The compiler is probably able to optimize this into a bitmask, but even so, this feels like it might be a little easier to understand, and still works across the wraparound:

Suggested change

	const uint32_t modulus = 1 << (CHAR_BIT * sizeof(uint8_t));
	return (modulus + atomic_get(&sender->last_sent) - atomic_get(&sender->last_acknowledged))
	% modulus;
	return (atomic_get(&sender->last_sent) - atomic_get(&sender->last_acknowledged)) & UINT8_MAX;

[trond-snekvik]

Should this set receiver->complete = true? And potentially call receiver->push with empty data to trigger functionality like downlink_finish()? Presumably, if we get FIN, we could still be in an open context

[sam-golioth]

Yeah I think that's right. The FIN handling in general is a little incomplete/inconsistent, so I'll clean that up. The reason I put this here originally was mostly for debugging, because at first I was handling the FIN message in the caller, so that the receiver could respond to a FIN received while it was idle. But I think it's better to handle FINs received while the receiver is active within the receiver itself, as you suggest. So the caller only handles received messages directly when the receiver is idle (NACKs in response to packets, nothing in response to FINs). We don't respond to FINs, because the sender will send a FIN if it receives an ACK while idle, so we need to be careful to not get into a loop of the sender and receiver acknowledging each other.

Anyway, this comment is only half responding to you and half me getting my thoughts out for myself 😅. But yes, I think when the receiver is active, we should handle the FIN here and I'll update the PR to reflect that. In thinking about this, I made a bunch of sequence diagrams to work the various scenarios, and those will make good figures to include in the design doc.