DNG encoding

[dgalland]

I think it's essential for any photo device to offer proper raw/dng encoding. In Picamera2, DNG encoding is done by a third party PiDNG library. This has the merit of existing, but it's far from optimal, as the encoding is extremely slow, probably due to the use of Python, despite good use of numpy.
I think the Raspberry Foundation should take on the task of making this encoding more efficient.
I'd also like to point out that when using DNG encoding, do not use CSI2P compressed formats, as decompression before encoding is catastrophically slow!
Regards

[davidplowman]

Do you have a specific example script that you could share that you feel is slow? I have in the past occasionally helped people get quite good performance saving DNG files, such as here: https://forums.raspberrypi.com/viewtopic.php?t=360334 (sorry, quite a long thread, but we were able to get some pretty blazing speeds using a thread pool and an NVMe drive).

My recollection was that you have to avoid using packed or compressed formats. Of course it's extremely sensitive to the speed of whatever disk you're writing too - that Samsung NVMe was just awesome and blew away my SATA SSDs. I don't recall that I thought the speed of PiDNG was an issue, but it was a while ago. We do know the author of PiDNG, and I'm sure he would be amenable to improvements if we could identify any.

[dgalland]

I'm coming back to this discussion because the status of the PiDNG version is not clear to me. While doing some tests of a compression of the raw (uncompressed) request buffer outside of picamera2 I noticed that the github version of PiDNG generates a bad DNG, I don't know why?
I also see that the rpicam-apps github has a C++ function that does the same processing, wouldn't it make sense to use it from picamera2?

[davidplowman]

I'm coming back to this discussion because the status of the PiDNG version is not clear to me.

It's a 3rd party library, like many others that we use, but we do know the author as he's a Pi user and is normally helpful to us in implementing features or fixing problems.

While doing some tests of a compression of the raw (uncompressed) request buffer outside of picamera2 I noticed that the github version of PiDNG generates a bad DNG, I don't know why?

I'm not aware of any problems myself, though we would recommend installing the PiDNG from our apt package as that's what we've tested. If there are bugs, I would encourage them to be reported, otherwise they might not get fixed. If you have no luck reporting them in the PiDNG repo, you could report them here and we could see if there's anything we can do to help.

I also see that the rpicam-apps github has a C++ function that does the same processing, wouldn't it make sense to use it from picamera2?

In C++ we use libtiff because it's the only library I know of that even comes close to writing DNG files. However, I'm not very keen on it. It has some strange behaviours that I don't understand requiring hacky workarounds to produce usable DNG files. Recently, also, there was an issue where some versions of libtiff "misplaced" some of the tags that are necessary for DNG files, breaking them completely. It seems to me that maintenance of DNG file support is not always great, and DNG files are a bit of a 2nd class citizen there. It also has no Python bindings (AFAIK) so there's be further work to do. I would rate our odds of getting things fixed and improved in PiDNG as a lot higher.

But in general - if there are any issues, please report them. We'll always do what we can to fix and improve things.

[dgalland]

Thanks for your answer, no problems actually with the installed version of PiDNG. It was just a remark about a potential problem with the Github version, there is one more SampleFormat tag and it's normal but the DNG file does not seem correct. I did not investigate further. Nothing urgent in any case.

[davidplowman]

Sure, no problem. But if anything does crop up that isn't working properly, please do let us know! :)