Replace imx500 network weight loading mechanism #6904
Conversation
The imx500 driver currently provides for network weight loading with a custom V4L2 control that allows userspace to pass a file descriptor to the weights, which are then read out. This is a fairly uncommon method of allowing userspace to pass data to the kernel - the function kernel_read_file_from_fd() has no users in mainstream drivers. It additionally requires that the network weights be stored as a file, which is not necessarily the case and would currently mean userspace would need to write them out as a file before passing them to the driver. Replace the V4L2 control with a new function that handles custom ioctls (via the v4l2_subdev_core_ops.ioctl operation) and instead allows userspace to pass a buffer and size to the driver. The handling of the new ioctl differs slighly from the original V4L2 control handling; in the original control a value of S32_MAX was used to trigger the clearing of the network weights - in the new ioctl a size of 0 triggers the same behaviour. The ioctl additionally waits until it has read the buffer successfully before touching the driver's internal copy of the weights data, so a failure to read the buffer properly will leave the original network weights intact. Signed-off-by: Daniel Scally <[email protected]>
|
@roliver for comments. I'm not opposed to this change, although it would be nice to have the ctrl still active for a bit to maintain backward compatibility. From the commit message:
How is this expected or intended to work? The conversion tools all work on offline files, including the fpk generation and fpk to rpk cpio generation. I don't see how a rpk network can be generated through the toolchain without going to a file. You probably also want to update Picamera2 to use the new ioctl: (https://github.com/raspberrypi/picamera2/blob/c50033891450f5082740da8acba2f5f3fa94301d/picamera2/devices/imx500/imx500.py#L591). |
|
Excuse the intrusion, but can't the current file descriptor mechanism be used with a pipe? That would eliminate the need for an intemediate file. |
So both methods at the same time for a while?
What we're envisioning is an ecosystem in which they'll be generated remotely and deployed to a board by some service, which might choose to store them as a file or as blobs in a database or whatever.
Ah, thanks, I had missed that indeed.
No intrusion :)
I did briefly try a pipe which gave me an -EBADF error from the kernel. I didn't really spend any time trying to diagnose that problem though as I was already leaning towards this approach being better. I did also try memfd which seems to work ok though. |
This type of workflow might not be suitable for the picamere2 library as-is. Picamera2 is a bit more advanced compared to rpicam-apps, and we extract the neural network info from the cpio archive in the rpk file. Without this, some of the library functionality will not be available. Input tensor extraction comes to mind, but there may be other things. |
Hello
Working with the imx500 lately, the requirement for the network weights to be passed as a file descriptor gave me some pause, as I was instead storing them as database objects which meant I'd have had to write them out before passing them to the driver. I thought that switching the interface to expect a buffer / size would be a bit more flexible and possibly also a bit more upstream friendly.
I also opened a parallel pull request for rpicam-apps that switches it to the new ioctl: raspberrypi/rpicam-apps#825
Thanks
Dan