Temporal delay while setting/changing exposure

[cpixip]

I have a use case where I need to switch rapidly between different exposure values (HDR capture of movie frames).

Now, setting AnalogueGain = 1 and ExposureTime = 5000 initially, and, after letting the camera settle to these values (the actual exposure time is 4989), I switch to a different exposure time, namely ExposureTime = 50000.

The code used is mainly (omitting a few lines)

picam2.start()
controls = {}
controls['AnalogueGain'] = 1.0
controls['ExposureTime'] = 5000
picam2.set_controls(controls)
time.sleep(1)

for index in range(-1,20):
    
    if index==0:
        controls = {}
        controls['AnalogueGain'] = 1.0
        controls['ExposureTime'] = 50000
        picam2.set_controls(controls)    
    
    request = picam2.capture_request()
    metadata = request.get_metadata()
    request.release()

The exposure stays at 4989 for a total of 7-8 frames until it jumps to 49989, a value close to the requested one. This behavior is mostly independent of the actually used values.

Now, in an HDR context, you want to capture differently exposed images (which are combined into an HDR) in quick succession. This is especially true in the context of telecine applications where you scan tens of thousands frames - which takes anyway a long time.

But even with classic HDR work, it is important that the images are taken with different exposures at close intervals, otherwise motion in the image will spoil the fun.

So it would be great if that is possible. I tried renaming the AGC algorithm in the camera tuning file to x.rpi.agc in order to deactivate it, but that did not change much on this behaviour (it actually got worse: 9 frames until the request exposure was achieved).

Maybe there is another way to achieve a desired exposure time immediately or at least for the next picture or the one after that?

Tests were done on a RP3, with HQ camera. (PS: changing instead of exposure time the gain value in the test program shows the same delay of 7,8 or 9 frames.)

[davidplowman]

Hi, there is unfortunately no way to get gain/exposure requests to take place immediately. For a start the sensor takes a few frames anyway, and then they get backed up at the end of libcamera's request queue.

One easy approach is the one taken by the Mertens merge example which simply stops and restarts the camera, setting the desired exposure/gain before restarting it. This is guaranteed to work, but it is somewhat slow.

Another idea that might help, especially if you want to capture many different exposures, would be to start the camera and then keep requesting the exposures you haven't seen yet, and "ticking them off" when you do. You have to be a bit careful because you can't guarantee not to drop any images, so you do really have to keep tabs on what has or hasn't turned up. Here's an example:

from picamera2 import Picamera2
import time

picam2 = Picamera2()
picam2.configure(picam2.still_configuration())
picam2.start()
time.sleep(0.5)

metadata = picam2.capture_metadata()
picam2.stop()
total_exp = int(metadata["AnalogueGain"] * metadata["DigitalGain"] * metadata["ExposureTime"])
wanted_exps = [total_exp // 4, total_exp // 2, total_exp, total_exp * 2, total_exp * 4]
print("Want the following exposures:", wanted_exps)
frame_duration = max(wanted_exps[-1], 140000)
outputs = []

picam2.set_controls({"ExposureTime": wanted_exps[0], "AnalogueGain": 1,
                     "FrameDurationLimits": (frame_duration, frame_duration)})
picam2.start()

def exp_matches(wanted, got):
    return abs(wanted - got) < wanted * 0.01 + 100

while wanted_exps:
    wanted_exps = wanted_exps[1:] + wanted_exps[:1]
    picam2.set_controls({"ExposureTime": wanted_exps[0]})
    request = picam2.capture_request()
    got_exp = request.get_metadata()["ExposureTime"]
    matches = [exp for exp in wanted_exps if exp_matches(exp, got_exp)]
    wanted_exps = [exp for exp in wanted_exps if exp not in matches]
    print("Got exposure", got_exp)
    if not matches and len(outputs) == 1 and exp_matches(outputs[0][0], got_exp):
        matches = [outputs[0][0]]
        outputs = []
    if matches:
        outputs += [(got_exp, request.make_array("main"))]
    request.release()

This is a bit of a quick hack, but to summarise what it does:

• It starts the camera, waits a moment for it to settle, and then calculates the effective "total exposure".
• It constructs a list of all the exposures it wants to capture. (Should probably ensure none of those are beyond the range the sensor can deliver.)
• It starts the camera with the first of these exposures. Now, on every frame it requests, from among the exposures it hasn't seen yet, the one that it asked for least recently.
• When you run it, you'll notice that the very first exposure comes through multiple times (because of the queuing mechanism in libcamera). It actually updates it's copy of the image at that exposure on every repeat, so that you end up with a final collection of images that are pretty much "back to back".

[cpixip]

one additional note for others having a similar challenge:

tl;dr: include an additional check for the digital gain when checking for requested exposure time.

Now to the details. Both code snippets above aim to circumvent the work of the AGC to get a precise exposure time. They need an additional check included to function properly.

As described by @davidplowman, the actual requested exposure time is made up by the AGC with a discretely constrained exposure time (which is related to the sensor's mode) plus a tiny correction via the digital gain. The real effective exposure time is simply the product of these two values.

Now, rapidly requesting very different exposure times in a sequence sometimes pushes the AGC into some creative choices. Because an exposure time of, say, 18928 can be realized with different combinations of exposure time and digital gain:

(A) 18928 = (DigitalGain = 1,0) * (ExposureTime = 18928)
(B) 18928 = (DigitalGain = 2,0) * (ExposureTime =  9464)
(C) 18928 = (DigitalGain = 4,0) * (ExposureTime =  4732)

Well, and all of these combinations can be observed occationally. These combinations yield different images; for starters, the 0.1-quantile of (A)=12, (B) = 11 and (C) = 10.

This tiny variation of image intensities creates in my application (film scanner) a visible flicker.

So in an application requiring really fixed, constant exposures over several (thousands of) captures, you will need to do an additional check on digital gain, similar to this code:

if (metadata['ExposureTime']-requestedExposure)<200 and metaData['DigitalGain']<1.05:
      print('I got my exposure!')
else:
      print('... still waiting for the digital gain to settle ... ')

[davidplowman]

Yes, I do think there might be some slight problems here. If you're changing the overall total exposure on every frame then the digital gain is going to get in the way of things by changing too. Perhaps we need to reconsider slightly how some of that works.

[cpixip]

Well, at least in my use case, it would be perfect if digital gain would be setable. Maybe similar to the color balance gains: stay fixed as soon as they are set, adjust to optimal values if set to zero again. That would be a rather transparent user interface.

There are probably other use cases as well, like science applications, where you would prefer to work with known and setable gains. For example, some folks use Raspberry hardware in DIY-spectrometers. In order to get reproducable measurements, you'd prefer setable analog and digital gains (and fixed exposure times, of course).

As far as I know, the old picamera/Broadcom combination allowed this. Such a possibility would certainly also be helpful for some other legacy software trying to make the step to the new library. So maybe the picamera2/libcamera combination could support a similar approach, while keeping the usefulness of the AGC for the average user.

[meawoppl]

This all is a little extra annoying for people interested in doing metered -long- exposures, as the number of frames you have to move through is conserved, but the amount of time that takes goes up quite a bit.

In my application, we are using still frames between 2-10 seconds, and having to shoot extra frames for the gain/exposure to settle takes a lot of time!

[davidplowman]

I'm sure I've said this on some other threads, but perhaps not here! There is some work planned to address these problems, and I'll post back once I have more information.