How to capture multiple images with different exposure time for exposure stacking?

[Milchgraf]

Hello, I would like to migrate my project to picamera2. With the outdated picamera library, I was able to pre-set the exposure time for each individual image. For example image1 - 1ms, image2 - 5ms and image3 - 9ms. I am having a hard time figuring out how to solve this problem with picamera2. How can I change the exposure time before each picture is taken? Preferably so that the images can be taken one after the other as fast as possible?
Would be very happy about your suggestions.

[davidplowman]

Hi, thank you for the question. There is some work going on in this area which makes the answer a little complicated. For now, the easiest thing is to stop and restart the camera, changing the exposure every time, something like this:

from picamera2 import Picamera2
picam2 = Picamera2()
picam2.configure(picam2.create_still_configuration())

def do_capture(exp_time):
    picam2.set_controls({"ExposureTime": exp_time, "AnalogueGain": 1.0})
    picam2.start()
    array = picam2.capture_array()
    picam2.stop()
    return array

short = do_capture(1000)
medium = do_capture(5000)
long = do_capture(9000)

Unfortunately this process is a bit slow, and this is where some of the work currently happening will help.

For example, if you wanted raw images (not processed by the ISP) you could fire updated exposure times at the camera without stopping it and you'd get the exposures you want returned to you back-to-back (after a latency of a few frames). Unfortunately that doesn't quite work with the processed images (output by the ISP), but I'm hoping there will be some fixes for that merged into libcamera any day now.

Looking slightly further out we plan to have a mechanism for sending controls to the camera (as above), but where you will get notified exactly when those controls have been applied, without even having to check whether the image metadata "looks like what you wanted", and there will be a reduction in the number of frames of latency too. But no estimated time scale for that yet, I'm afraid.

[Milchgraf]

Hi David, thanks for your detailed reply. I like the solution of creating RAW images with different exposure times. The delay of a few frames is acceptable for me.
What would the code for this look like? Can I just keep changing the controls as in your example?

picam2.set_controls({"ExposureTime": exp_time, "AnalogueGain": 1.0})

[cstaton-ftlg]

Hello,
This is a few months old and I am wondering if these changes you spoke of have been released or do we still need to do this workaround you provided.

[davidplowman]

I've pasted some code below. The ugliest thing is the match_exp function which is trying to tell when the exposure values you requested have happened. As I suggested previously, there will come a point when it will be "easy" to know this, but that's not imminent.

from picamera2 import Picamera2

def capture_multiple_exposures(picam2, exp_list, callback):

    def match_exp(metadata, indexed_list):
        err_factor = 0.01
        err_exp_offset = 30
        exp = metadata["ExposureTime"]
        gain = metadata["AnalogueGain"]
        for want in indexed_list:
            want_exp, _ = want
            if abs(gain - 1.0) < err_factor and abs(exp - want_exp) < want_exp * err_factor + err_exp_offset:
                return want
        return None

    indexed_list = [(exp, i) for i, exp in enumerate(exp_list)]
    while indexed_list:
        request = picam2.capture_request()
        match = match_exp(request.get_metadata(), indexed_list)
        if match is not None:
            indexed_list.remove(match)
            exp, i = match
            callback(i, exp, request)
        if indexed_list:
            exp, _ = indexed_list[0]
            picam2.set_controls({"ExposureTime": exp, "AnalogueGain": 1.0})
            indexed_list.append(indexed_list.pop(0))
        request.release()

def callback_func(i, wanted_exp, request):
    print(i, "wanted", wanted_exp, "got", request.get_metadata()["ExposureTime"])
    images.append((i, request.make_array("main")))


with Picamera2() as picam2:
    config = picam2.create_preview_configuration(raw={})
    picam2.configure(config)
    picam2.start()

    images = []
    exposure_list = [1000, 5000, 9000]
    capture_multiple_exposures(picam2, exposure_list, callback_func)

    picam2.stop()

The other thing to be aware of is that you don't necessarily get your exposures in the order you list, but the callback knows which you have. The whole process is somewhat influenced by camera framerate, the number of buffers you allocate for the camera to use and so on. But maybe give it a try and see how you get on.

[Milchgraf]

Hi, thanks for taking the extra time to write me this detailed code. I am surprised how fast the exposure time can be changed this way. I have one last question about the code. At which point do I convert the output of images.append((i, request.make_array("main"))) into a .dng file?

[davidplowman]

Hi, you will need to do this:

picam2.helpers.save_dng(buffer, metadata, config['raw'], "file.dng")

In the callback you will have to store the buffer (request.make_buffer['raw'] this is the same data as you get from make_array just not formatted into 2D), and the metadata (request.get_metadata()).

config is the configuration object that you passed to picam2.configure(), and which updates it with the correct raw stream format parameters.

I would probably save them after capturing them all, because writing those DNGs is quite slow and will cause frame drops, but obviously you can experiment.

[Milchgraf]

Hi, thanks again for your reply. Here is my code for my project. It captures 3 frames in a row, then the motor winds to the next frame (from 8mm film) and then it takes 3 frames again. It takes about 1.2 seconds to capture the 3 frames. In my eyes this is quite fast. I used a ramdisk for testing, where I buffer the images. But I couldn't see any difference in the capturing speed compared to saving directly to a usb-stick. Where exactly does the time for capturing the image or saving it as a .dng file come from? Does it depend on the cpu power?

from picamera2 import Picamera2
from os import makedirs
import shutil
import glob
from time import sleep
from gpiozero import Button as btn
from gpiozero import LED

path =  '/mnt/ramdisk'
motor_btn = LED(26)
button = btn(2)
counter = 1
dest_dir = '/media/***/stick/raw_images'
save_to_stick_counter = 50
exposure_list = [1000, 4500, 9000]
    
makedirs(f'{path}/test', exist_ok=True)
makedirs(f'{path}/raw-images', exist_ok=True)
makedirs(f'/media/***/stick/raw_images', exist_ok=True)

print('created directories on stick.')

def capture_multiple_exposures(picam2, exp_list, callback, counter):

    def match_exp(metadata, indexed_list):
        err_factor = 0.01
        err_exp_offset = 30
        exp = metadata["ExposureTime"]
        gain = metadata["AnalogueGain"]
        for want in indexed_list:
            want_exp, _ = want
            if abs(gain - 1.0) < err_factor and abs(exp - want_exp) < want_exp * err_factor + err_exp_offset:
                return want
        return None

    indexed_list = [(exp, i) for i, exp in enumerate(exp_list)]
    while indexed_list:
        request = picam2.capture_request()
        match = match_exp(request.get_metadata(), indexed_list)
        if match is not None:
            indexed_list.remove(match)
            exp, i = match
            callback(i, exp, request)
        if indexed_list:
            exp, _ = indexed_list[0]
            picam2.set_controls({"ExposureTime": exp, "AnalogueGain": 1.0})
            indexed_list.append(indexed_list.pop(0))
        request.release()

def callback_func(i, wanted_exp, request):
    #buff = request.make_buffer['main']
    meta = request.get_metadata()
    #print(i, "wanted", wanted_exp, "got", meta["ExposureTime"])
    #images.append((i, request.make_buffer("main")))
    
    picam2.helpers.save_dng(request.make_buffer("raw"), meta, config['raw'], f"{path}/raw-images/image-{counter}-{i}.dng")
    print(f'captured image-{counter}-{i} with exposure time of {meta["ExposureTime"]}')

#def save_dng(image_array)

with Picamera2() as picam2:
    
    config = picam2.create_preview_configuration(raw={})
    picam2.configure(config)
    picam2.start()
    
    input('Mit dem Scannen beginnen? [ENTER]')
    
    while True:
        if button.is_pressed:
            
            sleep(.05)
            motor_btn.on()
            
            print('')
            capture_multiple_exposures(picam2, exposure_list, callback_func, counter)
            
            if counter == save_to_stick_counter:
                print('start transfering images to stick')
                
                for file in glob.glob(r'/mnt/ramdisk/raw-images/image-*.dng'):
                    print(file)
                    shutil.move(file, dest_dir)
               
                save_to_stick_counter += 50
                print('transfered images to stick')
                makedirs(f'{path}/raw-images', exist_ok=True)
            
            counter += 1
            #sleep(.5)
            motor_btn.off()
    
    picam2.stop()

[davidplowman]

Hi, I would have expected the ramdisk to be faster, but it can be quite hard to guess about this stuff. Maybe because you're not using full resolution, and there's probably a certain amount of buffering in the file system anyway? The DNG library may be doing quite a lot of CPU work too, you'd have to track down the code for it and take a look. In the end, the only reliable answers come from profiling...!

[Milchgraf]

If I capture at full resolution, then the time doubles or triples. If I have a good cooler, then I clock the CPU higher and see if anything changes.

[cpixip]

@Milchgraf

Hi, thanks again for your reply. Here is my code for my project. It captures 3 frames in a row, then the motor winds to the next frame (from 8mm film) and then it takes 3 frames again. It takes about 1.2 seconds to capture the 3 frames.

If you are interested, consider joining the Kinograph-Forum and describe your system. People there are always interested in new approaches towards film scanning.

[dgalland]

Hi David,
First of all, the trick for quickly capturing a set of exposures works well, thank you. For example, you can capture 5 given exposures in about 15 requests. However, I tried to display the Lux value in the metadata (which is supposed to represent the brightness of the image) and I get inconsistent results. Lux varies inversely with exposure and is almost identical for the three lowest values. I'm not attaching the images, but the first three are really different.
Found: 100 Exposure: 114 Analog: 1.0 Digital: 1.0 Lux: 18642.322265625 Real: 114.0
Found: 400 Exposure: 400 Analog: 1.0 Digital: 1.0 Lux: 18607.49609375 Real: 400.0
Found: 1600 Exposure: 1600 Analog: 1.0 Digital: 1.0 Lux: 18688.97265625 Real: 1600.0
Found: 25600 Exposure: 25600 Analog: 1.0 Digital: 1.0 Lux: 3207.466796875 Real: 25600.0
Found: 6400 Exposure: 6400 Analog: 1.0 Digital: 1.0 Lux: 11556.1181640625 Real: 6400.0

[will-v-pi]

Hi @dgalland, the lux in the metadata is an estimate of the illuminance of the real-world scene rather than the captured image, so if the algorithm is working correctly then it should be the same regardless of the exposure time. However, when the image is very under or over exposed, you will see the lux value varing inversely with exposure due to the large number of saturated pixels, so are your last two images quite over-exposed? That might explain the values you're seeing

[dgalland]

Yes, that's right, the last two images are clearly overexposed.
Thanks for your explanations

[polijn]

@davidplowman, thank you for your code snippet! It worked seamlessly with my IMX219. However, when I try to use it with my IMX708, it can't seem to match the exposure time correctly.

I'm currently on the latest Raspberry Pi OS (Bullseye), and I've made modifications to the tuning file as suggested in issue #622. Additionally, I've implemented a 2-second wait after every configuration change, as discussed in issue #800.

To obtain results, I find myself needing to drastically alter the err_factor, setting it to 1 or even 10. Yet, the results are still far from satisfactory.

Here are some of the results:

wanted 100 got 858
wanted 1000 got 858
wanted 100000 got 77918
wanted 1000000 got 106399
wanted 500000 got 499987
wanted 2000000 got 499987

Why is that?
Am I missing something?

And this is my code:

# #!/usr/bin/python3
from picamera2 import Picamera2

def exposure_captures(exposure_list=[100000, 500000, 1000000, 2000000]):
    with Picamera2() as picam2:
        # config = picam2.create_still_configuration(raw={}, buffer_count=2)
        config = picam2.create_still_configuration(raw={})
        picam2.set_controls({'AfMode': controls.AfModeEnum.Manual, 'LensPosition': 15.0})
        picam2.configure(config)
        picam2.start()
        capture_multiple_exposures(picam2, exposure_list, config, callback_func)
        picam2.stop()
        return [f"{i}.dng" for i in range(len(exposure_list))]


def capture_multiple_exposures(picam2, exp_list, config,callback):

    def match_exp(metadata, indexed_list):
        err_factor = 0.01 # changed it to 1 or 10 to get results
        err_exp_offset = 30
        exp = metadata["ExposureTime"]
        gain = metadata["AnalogueGain"]
        for want in indexed_list:
            want_exp, _ = want
            if abs(gain - 1.0) < err_factor and abs(exp - want_exp) < want_exp * err_factor + err_exp_offset:
                return want
        return None

    indexed_list = [(exp, i) for i, exp in enumerate(exp_list)]
    while indexed_list:
        request = picam2.capture_request()
        match = match_exp(request.get_metadata(), indexed_list)
        if match is not None:
            indexed_list.remove(match)
            exp, i = match
            callback(i, exp, request, picam2, config)
        if indexed_list:
            exp, _ = indexed_list[0]
            picam2.set_controls({"ExposureTime": exp, "AnalogueGain": 1.0})
            indexed_list.append(indexed_list.pop(0))
        request.release()

def callback_func(i, wanted_exp, request, picam2, config):
    print(i, "wanted", wanted_exp, "got", request.get_metadata()["ExposureTime"])
    meta = request.get_metadata()
    exT = request.get_metadata()["ExposureTime"] 
    picam2.helpers.save_dng(request.make_buffer("raw"), meta, config['raw'], f"{i}_{exT}.dng")

[davidplowman]

I would try setting some fixed exposure times "by hand" and see what you actually get back. There will be a minimum and maximum exposure time, and some degree of quantisation, and this will give you a feel for what the sensor does. For example, type

picam2.set_controls({'ExposureTime': my_exposure_time})

and then query

picam2.capture_metadata()['ExposureTime']

to see what it did.

After configuring the camera, also try picam2.camera_controls['ExposureTime']. The first two numbers will be the min and max supported exposures.

[polijn]

Thanks for the quick answer!

The first capture always gives me a min of 0 and a max of 66666 the consecutive and following captures it gives me a min of 9 and a max of 77248381.

When I actually iterate from 0 to 100000 (exposure time) the maximum exposure is 82881.
The step rate (the quantization) is alternating between 18 and 19 (36,55,73,92.....).

Going back to your code snippet the gain was the problem... err_factor = 0.13does the fix.

My results look like this now.

1 wanted: 16000 got: 15984 gain: 1.1228070259094238
3 wanted: 48000 got: 47979 gain: 1.1228070259094238
4 wanted: 64000 got: 63990 gain: 1.1228070259094238
0 wanted: 82880 got: 82867 gain: 1.1228070259094238
2 wanted: 32000 got: 31995 gain: 1.1228070259094238

Thanks for your help!

[davidplowman]

Glad it's working better! Yes, the imx708 has this weird thing where some camera modes have a minimum analogue gain of 1.123.

[juanmf]

The answer @davidplowman
gave me in issues 849 can help wait for new exposure to apply with minimum waste...