Networking: support capturing all packets to a pcap file

[tridge]

This allows capturing of all packets (ethernet or PPP) to a pcap file which can be read by wireshark

this is very useful for debugging tricky networking issues

tested on CubeOrange with BotbloxDroneNet and on Pixhawk6X

depends on: ArduPilot/lwip#4

this also includes a new set of buffered write operations for logging in AP_Filesystem, based on stdio-like API calls

[tpwrules]

Regarding the AP_Filesystem buffering, I did some testing and instrumentation with Lua and FATFS does have a sector buffer per file and it does work. I wrote a Lua script that writes one byte every 10ms and in the SD card interface layer in fatfs_diskio.c I see one sector written every five seconds and no reads (except on cluster boundaries). Certainly not multiple transactions per byte as we had previously thought. I don't think there is justification to be scared of misaligned writes, FATFS handles them fine. It's easy to misread the code at first and believe it does not though, I certainly did too!

Timing using the Lua script stats on Cube Orange suggests writing one byte takes about 10 microseconds which does not blow my socks off but is fine. Writing ten bytes also takes about 10 microseconds which matches my expectations of most of the time being locking and call overhead. Writing a sector takes about 4 milliseconds though, from the Lua thread at least, which is pretty bad. I merged this PR in to benchmark it and it seems to have shaved a couple microseconds off each call, but has had no effect on the sector write time, despite the fact that it's now writing four sectors at a time. Which is good as that means 4x the bandwidth.

The FATFS docs suggest a buffer of at least one cluster, which is usually 32K bytes IIRC. But this is a lot of RAM to sacrifice for each file particularly on F4. Your API could be used to amortize the sector write time but we might have to make the buffer size a processor-specific define. I think 8K is too often to sync, it should be once per cluster. I wanted to modify the logger code to sync after the cluster size and also after a fixed time, but haven't got around to it.

Ultimately I think it would be good to centralize the logic for streaming writing files (maybe with a new open code) but as it is doesn't quite feel right. Maybe I just need to sit with it more or see it as its own PR. I do have re-imaginings for AP_Filesystem in my head too which are doubtless conflicting... but we can always change the code again, there are not that many users.

[peterbarker]

Off by default; 1.2kB on CubeOrangePlus Plane (etc.)

[tridge]

@tpwrules @peterbarker I've broken out the filesystem changes into #30671 for separate discussion

[peterbarker]

Double-shuffle done?

[tridge]

Double-shuffle done?

yes. We use the 2.2 branch, here:
https://github.com/ArduPilot/lwip/commits/ArduPilot_LWIP_2_2/