tuning automatic lens shading correction (ALSC)

[ballen4705]

I have a couple of questions about adjusting the automatic lens shading correction (ALSC).

My camera sensor is sitting in the focal plane of an instrument. The illumination is green, so I have set picam2.set_controls({"ColourGains":(0.01,0.01)}). This turns off the blue and red pixels, reducing the effects of any stray light.

Because of lens shading, the green illumination is not uniform across the focal plane, but falls off radially from the center. In the subsequent analysis that I do, it would be very helpful to correct for this, so that the average green level is uniform across the entire sensor. I believe that the ALSC part of the image processing pipeline should be able to accomplish this. However the current images, with the stock tuning file, show a distinct radial fall-off in average green levels.

In reference to Section 5.9 of the Raspberry Pi Camera Algorithm and Tuning Guide, the goal should be that after ALSC, each of the 16x12 regions has the same average green value.

Question 1:
According to Table 5.9.5, the default value of luminance_strength is 1.0. So why do I see a falloff in the green levels? Did ALSC get turned off when I set the ColourGains?

Question 2:
How should I set different ALSC parameters? Is the only way to load a modified json tuning file? Or is there a picamera2 call that I could use for this, for example to change the luminance_strength value?

Cheers,
Bruce

[davidplowman]

You don't say what sensor you're referring to, but from your other questions I assume this is the HQ cam?

You're right that you probably need to change the luminance_strength. You'll have to edit this in the tuning file because libcamera has no API for this (it's specific to the Raspberry Pi platform). Generally I've found the stock lenses that we supply for the HQ cam do show some vignetting (though not strong compared to the smaller camera modules), though it will also vary with the lens.

So certainly put the luminance_strength value in the file back up to 1.0, but you may also find you want to measure a table for yourself if this isn't sufficient. I would hope the table is fairly obvious - it's an array of 17x13 gain values, one for each of the corner points in the 16x12 grid.

[ballen4705]

Hi David,

thanks, this is very helpful! Yes, it's a HQ camera. And from your response, I just realised that even though Table 5.9.5 lists 1.0 as the default value of luminance_strength, that is probably not the value in the default json tuning file. I'll play around with it and report back here.

Cheers,
Bruce

PS: since picamera2 does have a function for loading a tuning file, couldn't you also offer a function for "tweaking" individual values by loading the default tuning file then modifying any of the indicated values? Or is that a Bad Idea or simply too ugly to contemplate?

[davidplowman]

You can tweak the tuning file if you want, check out this example.

[ballen4705]

That makes it very simply, thank you!

[EDIT] And a quick check shows that indeed the value of luminance_strength in the
default IMX477 tuning file is 0.5 rather than 1.0:

"rpi.alsc":
            {
                "omega": 1.3,
                "n_iter": 100,
                "luminance_strength": 0.5,
                ...

I'll change this to 1.0 and see if that helps.

[ballen4705]

Hi David,

I had a look at imx477.json and also read Section 5.9 carefully. This left me with a couple of questions, because there is no clear connection between the Section 5.9 documentation and the entries in the rpi.alsc section of the tuning file.

Based on the documentation, the goal is to determine three sets of weights lambda_i, where i is an index over a set of 16 x 12 = 192 points. (By the way, this array is shown in Fig 13, and at 16x12 already extends corner-to-corner. I don't think that it is 17 x 13 in size.)

First question
Does the luminance_lut table correspond to lambda_i, and the calibrations_Cx tables correspond to lamba_i C_i for the red and blue? Or are the calibrations_Cx tables the C_i?

Second question
Do you understand why the tables in the tuning files are not symmetric under reflection about the horizontal and vertical midlines? It seems to me that both the sensor and lens systems have this symmetry, so the tables should reflect that. In other words if we let i=(r,c) where r=0,...,11 is a row index and c=0,...,15 is a column index, why do the three tables T not obey T(r,c) = T(11-r,c) = T(r,15-c)? This would mean that the 192 tables entries are actually defined by the 6 x 8 = 48 entries in the top left rectangular block. The other three blocks are copies, reflected about the horizontal or vertical axis. I'm puzzled that the tables do not show this symmetry.

Cheers,
Bruce

[ballen4705]

Hi David,

I had some time over the weekend, and tried this out. It worked incredibly well.

I build and installed libcamera locally, following your (modified) instructions. Because I was too lazy to figure out how to add another algorithm into the processing pipeline, I just modified libcamera/src/ipa/raspberrypi/controller/rpi/alsc.cpp and then ran
sudo ninja -C build install from the top level to compile and reinstall. That was quick enough to be practical for this.

The modifications are along the lines that you sketched out:

• in copyStats() I update a global "mean brightness" 16x12 array that contains the value of r.val.gSum divided by the average value. I clip this to maximum 8 and minimum 0.125.
• then in doAlsc(), just before addLuminanceToTables() is called, I modify luminanceTable_ based on my global array. For cells where the mean brightness is < 0.99 I fractionally increase the corresponding luminanceTable_ entry, and where the mean brightness is > 1.01 I fractionally decrease it. This implements the IIR filter you suggested, with a bit of hysteresis.Then I scale the table so that minimum is 1 and clip it at 8 (though it normally is smaller).
• to help me understand what was going on, I also added a helper function to print 16x12 float arrays at the libcamera log info level.

It all works very well. I will spend some time tuning the IIR filter so that it's more responsive but not unstable. I can see some minor artifacts of the 16x12 array, but could easily eliminate these if I could get access to the green pixel histograms in the cells, rather than just the totals. Do you know if those are available somewhere?

It's fun to code in C, which was my main language for many years. These days I do most stuff in python, which I'm not very good at and don't know well. Writing in C is refreshing. I don't have to spend any time thinking about how to do what I want, so the only issue is forgetting semicolons at the end of lines.

Cheers,
Bruce

PS: some of what is written in the "Raspberry Pi Camera Algorithm and Tuning Guide" is quite misleading. For example compare ALSC/min_G in the documentation with min_G and minG in alsc.cpp. It's not just the default value that's wrong, but the entire description.

PPS: something that surprises me are the equilibrum values in luminanceTable_. These are the values needed to make stats->awbRegions.get(i).val.gsum the same in the 192 cells. Here is a typical equilibrum luminanceTable_ (unity is fourth row seventh column):

3.43 2.43 2.27 1.84 1.85 1.59 1.56 1.62 1.84 1.66 1.78 1.98 2.26 2.16 2.53 2.41 
2.98 2.14 1.99 1.62 1.69 1.41 1.31 1.35 1.55 1.41 1.53 1.74 2.11 2.12 2.47 2.33 
2.83 2.03 1.91 1.56 1.60 1.28 1.18 1.20 1.35 1.25 1.37 1.60 2.08 2.07 2.48 2.34 
2.60 1.88 1.70 1.36 1.39 1.08 1.00 1.00 1.17 1.04 1.17 1.37 1.80 1.87 2.27 2.21 
2.74 1.93 1.80 1.42 1.38 1.12 1.03 1.04 1.19 1.09 1.19 1.42 1.84 1.98 2.45 2.39 
3.10 2.15 1.98 1.59 1.52 1.28 1.14 1.13 1.29 1.21 1.35 1.54 1.96 2.14 2.66 2.62 
3.19 2.20 2.03 1.58 1.53 1.29 1.19 1.22 1.40 1.30 1.40 1.61 2.03 2.12 2.62 2.56 
2.92 2.01 1.82 1.42 1.39 1.17 1.14 1.14 1.37 1.18 1.29 1.45 1.75 1.89 2.25 2.15 
3.72 2.63 2.34 1.80 1.74 1.46 1.43 1.51 1.72 1.55 1.63 1.84 2.16 2.24 2.63 2.65 
4.09 2.92 2.61 1.98 1.91 1.59 1.55 1.61 2.01 1.74 1.76 1.90 2.25 2.22 2.72 2.83 
4.77 3.49 3.08 2.37 2.22 1.80 1.76 1.90 2.23 1.92 1.95 2.07 2.42 2.38 3.11 3.35 
5.19 3.97 3.54 2.64 2.50 2.03 1.92 1.94 2.24 1.92 2.02 2.17 2.53 2.70 3.47 3.77 

What surprises me is that some of these values are five times as large as the minimum entry (1.0). If I look at the uncorrected image, by replacing this with a table containing only ones, the corners do not look five times as dim as the center. Any idea why this is? Something to do with gamma curves or black levels? Or is it because the corner and edge values are weighted differently?

[davidplowman]

Glad that you're getting something to work. To answer a few of those questions:

• The hardware does not provide histograms for the regions individually, I'm afraid.
• I'll try and revisit the documentation at some point. I don't think that's been done since it was first written, and I honestly suspect no one has ever read it before!
• All this ALSC works on raw pixel values, before the gamma curve is applied. I'm guessing that the pixel differences are getting squashed by the gamma curve (which you could disable if you wanted to see the difference).

[ballen4705]

Hi David,

Regarding the docs, it would be nice to see them in git. If they are in some standard format (LaTex if possible) I suspect you will get a number of useful pull requests to fix/improve them.

I thought that r.val.gSum was produced by some fast (GPU?) hardware, by summing all of the pixels in the 192 different cells. But when I watch those stats, they seem to trickle in almost at random. Could you point me to the code that computes them?

I'll mess around the the gamma curve to see if I can understand what's happening there.

Thanks again for all of your help.

Cheers,
Bruce

PS: will be distracted with other things the next two weeks. Will return here the last week of March.

[davidplowman]

I can talk to the doc team, but it's their call what they want to do!

I'm not totally sure I understood your comment about wanting histograms for the 192 regions. The situation is that for each region you get pixel sums and counts - these are produced by hardware and are therefore "free". But there are no histograms. There is a whole image histogram that AGC/AEC uses, but I don't think it's any use to ALSC.

[ballen4705]

Hi David,

No worries about the docs. But I always keep in mind a comment made by a friend and colleague (around the time that Linux first appeared!). He said, "Computer software can not be better than its documentation. It can be worse, but the documentation is an upper bound." While it might not be strictly correct, there is some truth there.

The main reason I wanted histograms is because some cells contain features that drop the green levels. If I had histograms it would be easy to distinguish these, and to base the green levels on the pixel counts above some threshold. So, since I don't have histograms, I was hoping to read the code that the "hardware" uses to accumulate the current pixel counts. Perhaps there are some thresholds or switches there, that I could play with. Or does "done in hardware" mean that there is no code to read?

Cheers,
Bruce

PS: another comment on the docs, this time The Picamera 2 Library:
Table 2, pg 52 cut and paste mistake, the switch_mode line is repeated four times. There are similar errors on pg 53.

[ballen4705]

Hi David,

I had to focus on other things for a while, but I'm back at work on this code again. I wanted to package my "custom ALSC" in a way that ensures that by default (when picamera2 is used) the stock ALSC algorithm is used, but which permits me to enable "my" ALSC from within picamera2. Could you give me some advice about a good way to go about this?

One approach is to extend the stock ALSC code to include my additions to it, and enable those via a picamera2 control call within my application. But if possible I'd prefer to leave the stock ALSC code in place and simply add my own routines to the .so library via some ALSC_custom version.

Cheers,
Bruce

[davidplowman]

It's certainly possible for people to implement and use their own custom control algorithms, we very deliberately designed it this way. Having said that, if you want to include the standard ALSC as part of that then I'm not sure what the best approach would be. One could just copy the code and add what you want, but the duplication doesn't feel great. Or could our implementation be used directly just by calling the switch_mode/prepare/process methods?

Switching algorithms at runtime is another question. We can't really switch from one self-contained algorithm implementation to another on the fly, so you'd probably need an algorithm that actually incorporates everything, and then allows the switch. For this we'd need some kind of libcamera control to be available, which I'm sure they would agree to if we have good reasons.

Can you remind me again what the scope of your changes is and what happens differently? Thanks!

[ballen4705]

Thanks for the quick reply. I can't use the current ALSC implementation, for example by changing some parameters. New code is needed. The questions are "where to put this new code" and "how to enable this new code without disabling the stock ALSC behaviour for most purposes".

My "custom ALSC code" is designed to equalise and normalise the green intensity in the 12 x 16 = 192 different blocks. If possible, I'd like to be able to enable/disable it while my picamera2 code is running. If that's not possible, then I should be able to enable it at runtime, whereas any other code that uses the camera pipeline should get the stock ALSC. Note that either "my version" or "stock ALSC" should run, but not both at the same time.

(Current status: on my system, I have replaced the stock shared object file with one containing a hacked ALSC. So currently everything that uses the camera produces very weird images! I'd like standard code to run standard ALSC, and my code to run my custom ALSC.)

From what I have understood, there are two options: (a) have my code as a separate routine, called instead of the stock ALSC code or (b) merge my code into the stock ALSC code, enabling it with a switch in the .json file or via a control call from within picamera2.

What would you suggest? To me, it appears that the best thing might be to extend the existing ALSC code by adding a custom switch. The behaviour of ALSC would then be the standard behaviour, UNLESS my custom switch is enabled at runtime. I'm fairly sure that this is possible from within picamera2 via a control call. Is that right? Effectively just passing a new parameter to a (modified) ALSC algorithm.

[davidplowman]

Hi, yes this is a tricky question. In general I'm not terribly in favour of putting in custom behaviour with custom switches, on the grounds that we might find ourselves acquiring ever more of these as time goes on and the whole thing becomes ever harder to maintain.

Making a complete copy is one solution, though duplicating loads of code is never great. I wonder if there's a way forward that allows you to reuse the code that's there, and add to it? For example, one might derive a new ALSC algorithm class from our existing one and then, depending on certain settings, just call the base class methods, or do something different. For a solution like this I'd certainly be happy to tweak things in our implementation to make it possible, perhaps like making private members public, and so on.

What do you think?

[ballen4705]

Hi David,

sorry that this has dragged on - I needed to do some experiments. Unfortunately, these are issues that I don't know much about, so I'm not well equipped to answer them. Good general-purpose solutions might include leaving stubs in the shared object file which can be resolved/overloaded at link time with the user's code.

I think my changes to alsc.cpp are specific enough to my own needs that they shouldn't end up in the general code base. So I'm using a simple solution that works well for me.

There are two things that I want to be able to do dynamically (without restarting the camera or picamera2). The first is to swap back and forth between "green pixels only" and "standard AWB behaviour". I can do this with stock code, as in the following

#!/usr/bin/python3

import time

from picamera2 import Picamera2, Preview
controls={}
picam2 = Picamera2()
preview_config = picam2.create_preview_configuration()
picam2.configure(preview_config)
picam2.start(show_preview=True)
picam2.title_fields = ["ColourTemperature", "ColourGains"]
time.sleep(5)

controls['ColourGains']=(1.0, 1.0)
picam2.set_controls(controls)
time.sleep(5)

controls['ColourGains']=(0.01, 0.01)
picam2.set_controls(controls)
time.sleep(5)

del controls['ColourGains']
controls['AwbEnable']=1
picam2.set_controls(controls)
time.sleep(5)

The second thing I want to do dynamically is to enable/disable my custom ALSC behaviour. To do this I need to pass a flag dynamically into libcamera/src/ipa/raspberrypi/controller/rpi/alsc.cpp, in the same way that the ColourGains are getting passed upstream in the example above.

I could not see a way to do this using the conventional set_controls() mechanism. So I have resorted to the simple method (within the python code) of touching or removing a file to send the information. To receive the information, I use the stat() system call within alsc.cpp to detect if that file exists or not. It's ugly but works fine. However, if there is some way for me to use set_controls(controls) to pass a flag upstream to alsc.cpp, please tell me.

I've attached my modification to alsc.cpp in the form of a patch, so you can see what it looks like. If there is a simple way to communicate a binary flag to this from within picamera2, to enable/disable my special behaviour, it would clean things up for me.
alsc.cpp.patch
In my current code, communication is handled within bool InCustomMode().

Cheers,
Bruce

[davidplowman]

From a pragmatic point of view, what you're doing seems pretty reasonable to me. Switching between green/non-green pixels is fine, I expect you get a slightly weird colour temperature being reported, but I assume that doesn't matter!

With libcamera there is a more general question relating to platform specific or indeed completely custom behaviour. It deliberately doesn't give you easy ways to "tunnel" though the standard API so that you can implement custom things. I think the official position would be to create a more generally meaningful control and then to add it to the full set libcamera controls. But then do we want every custom application in the future to do this? It's not clear to me.

In the past we've "tunnelled" around the standard control API for things like custom tuning files. On that occasion we just defined environment variables, though files (or pipes or sockets, it's not so very different) might be better for things that don't happen only at initialisation time. You could use a memory file (iin /dev/shm) to save going out physically to the SD card (you may be already of course!).

[ballen4705]

Hi David,

yeah, it's reasonable and it works, but it's ugly. I'm planning to use an Overlay File System with the SD card/EMMC only mounted rw for short periods but normally read only. So the writes will go to RAM. I might also modify my code to only stat() the trigger file once per second.

I think it would be good if picamera2.set_controls() could accept ANY dictionary, and passed everything upstream to libcamera. Then libcamera did the same. So any parameter values could be passed upstream to (potentially modified) routines. In the end the C/C++ code could retrieve any parameter by name. I don't imagine it would complicate your code significantly nor the libcamera code, but it would enable easy customizations of the kind that I need, without the "communicate out of band" issues.

Cheers, Bruce

[davidplowman]

We did originally want the libcamera API to be more permissive like this, but they (libcamera) were keen for applications to present a common API. I think they were probably right in that having everyone go off and do their own thing would have been really bad, but it does leave this idea of "tunnelling" application specific commands through in an awkward place. Might be something to try and revisit some day.

On the subject of writing to the file system, it sounds to me like using a pipe might be nicer. Each end would open it just once and keep it open, and you could easily check every frame whether there's a "command" there that requires you to do something - but I take the point that what you have is already good enough, so why change it!

[ballen4705]

I don't believe that my proposal violates the "common API" philosophy. Here, the common API is calls that pass a dictionary of parameters upstream. The library can dictate a common set of parameters that all contributors must support. My proposal is that if the dictionary contains other parameters besides the agreed ones, then those should also be passed upstream in the same structures currently used to communicate, rather than generate "not recognized" errors. This should not require any modifications to existing code, since the upstream modules only look for parameters that they already know about. They should not care if other parameters are also available. This way, if someone like me wants to do a customized processing modules, a simple communications path is available.

Regarding my stat("/a/path") communication method, I'd rather do it with semaphores and shared memory. But under the hood posix shared memory just exploits a memory-mapped file. So in the end, if I adopt an overlay file system, it is doing the same thing.

[davidplowman]

I certainly get where you're coming from, and indeed I'd like to be able to get stuff through the libcamera interface to Raspberry Pi code more easily too. But this would require discussion and agreement with the libcamera folks, and as I explained they also have reasons for how things are, so my reading of the tea leaves is that things there are unlikely to change for the time being.