Alpha-Release of new picamera-lib: "picamera2"

Recently, the Raspberry Pi foundation released an offical alpha-release of a new python library, “picamera2” (alpha = things might still change). There are quite a few film scanner approaches using the HQ camera of the foundation in combination with the old “picamera” library. This old library was based on what was available at that time, namely the propriatary Broadcom-stack. This library hasn’t seen any update since about two years.

The new “picamera2”-library is using the new open-source “libcamera”-stack instead of the Broadcom-stack. It is also an offical library supported by the Raspberry foundation. I will report in the following a few insights for people considering to upgrade their software to the new library.

As the “picamera2”-library is based on the “libcamera”-approach, there now exists a tuning file for several sensors, including the v1/v2 and HQ camera of the Raspberry foundation. In principle, this tuning file allows for fine-tuning of the camera characteristics for the task at hand. Because “libcamera” handles things quite differently from the old “Broadcom”-stack, the python interface is also based on a totally different philosophy.

Accessing the camera with the new software is simple, as a lot of functionality is taken care of behind the scenes. Here’s an example taken directly from the “example”-folder of the new libraries github:

#!/usr/bin/python3
import time

from picamera2.encoders import JpegEncoder
from picamera2 import Picamera2


picam2 = Picamera2()
video_config = picam2.video_configuration(main={"size": (1920, 1080)})
picam2.configure(video_config)

picam2.start_preview()
encoder = JpegEncoder(q=70)

picam2.start_recording(encoder, 'test.mjpeg')
time.sleep(10)
picam2.stop_recording()

After the necessary imports, the camera is created and initialzed. Once that is done, a preview is started (which in this case also silently creates a loop capturing frames from the camera). After this, an encoder (in this case a jpeg-encoder set to a quality level of 70) is created and appended to the the image path (picam2.start_recording(encoder, 'test.mjpeg')). The camera is left running and encoding to the file ‘test.mjpeg’ in the background for 10 seconds, after which the recording is stopped.

For a film scanning application, one probably wants to work closer to the hardware. Central to libcamera’s approach are requests. Once the camera has been started, you can get the current image of the camera by something like:

request = picam2.capture_request()
array  = request.make_array('main')
metadata = request.get_metadata()
request.release()

From aquiring the request until you release it, the data belongs to you. You have to hand it back by request.release() to make it available again for libcamera.

Besides the image data (which you can get as numpy-array (request.make_array('main')) as well as a few other formats) every request has also a lot of metadata related to the current image. This metadata is bascially a dictonary which is retrieved by the request.get_metadata()-line. Libcamera operates with three different camera streams, one of them is ‘main’ (which was used in the above example). These streams feature several different output formats. One called ‘lores’ operates only in YUV420, another one called ‘raw’ delivers just that: the raw sensor data.

Some (but not all) of the data returned in the metadata dictionary can not only be read, but also set. For example, the property AnalogueGain is available in the metadata - this is usually the gain the AGC has selected for you. It can also be set to a desired value by using the command:

picam2.set_controls({'AnalogueGain':1.0})

One notable example of a property which can not be set is the digital gain. More about that later.

Some properties of the camera can also not be set in the way described above, mainly because they are related to requiring a reconfiguration of the sensor. One example is switching on the vertical image flip, which can only be done in the inital configuration step, like so:

video_config["transform"] = libcamera.Transform(hflip=0, vflip=1)
picam2.configure(video_config)

Another important deviation from the old picamera-lib is the handling of digital and analog gain. In the new picamera2-lib, there is actually no way to set digital gain. As the maintainer of the library explained to me, a requested exposure time is realized internally by choosing an exposure time close to the value requested and an appropriate multiplicator to give the user an approximation of the exposure time requested.

An exposure time of, say, 2456 is not realizable with the HQ sensor hardware. The closest exposure time available (due to hardware constraints) is 2400. So the requested exposure would be realized in libcamera/picamera2 by selecting: (digital gain = 1.0233) * (exposure time = 2400) = (approx. exposure time = 2456).

This behaviour spoils a little bit serious HDR-work, as the data from the sensor is only an approximation of what you actually requested. And there exist differences between frames taken for example with exposure time = 2400/ digital gain=2 (a combination the AGC will give you occasionaly when requesting exposure time = 4800) and exposure time = 4800/ digital gain = 1.0 (which is the actual desired setting).

One approach to circumvent this is to choose exposure times which are realizable hardware-wise and request this exposure time repeatably (thanks go to the maintainer of the new picamera2-lib for this suggestion), until the digital gain has settled to 1.0. For example, the sequence 2400, 4800, 9600, etc. should give you in the end a digital gain = 1.0. And usually, it takes between 2 and 5 frames to obtain the desired exposure value.

By the way, the old picamera-lib had a similar issue. With the old lib it took sometimes several frames until the exposure time settled to the requested value.

I am currently experimenting with a few approaches to circumvent this challenge. Based on a suggestion from the maintainer of the library, namely to request a single exposure time until it is delivered, before switching to the next one gets me a new exposure value every 5 frames on average.

With this approach, my current scan speed is about 3 secs per movie frame (2016x1502 px). This includes taking five different exposures and a waiting period of 0.5 sec to settle out mechanical vibrations of the scanner. Not too bad.

As the maintainer of the library indicated, some work might be done to improve the performance of the library with respect to sudden (and large) changes of exposure values - however, as this might require to change things in the library picamera2 is based upon(libcamera), it might take some time.

Finally, here’s a comparision of the color differences with the old and new library. The old picamera-lib/Broadcom-stack delivered for example the following frame:

The new picamera2-lib/libcamera-stack gets the following result from the same frame:

This capture is from Kodachrome film stock. Here’s another capture from Agfa Moviechrome material with the old picamera/Broadcom-stack combination

and this is the result of the same frame, captured with the new picamera2/libcamera-stack:

Two things are immediately noticable: the new picamera2 delivers images which have a better color definition. The yellow range of colors is actually more prominent, which should help the rendering of skin tones. Overall, the new picamera2-lib/libcamera-stack features a stronger saturation, which creates more intense colors. Also, it seems that the image definition in the shadows is improved.

One last note: JPEG/MJPEG encoding is currently done in software, with the simplejpeg-library. So the quality settings of the old library, which used hardware encoding, are not directly transferable. Also, a higher system load is experienced, especially if using the available multiple thread encoder

4 Likes

I also made some tests of Picamera2. Here are some remarks.

The API is quite high level but closely reflects the libcamera API, a bit complicated though. One may regret the simplicity of the original Picamera.

You should not expect any better performance at all and even the opposite.

With Picamera jpeg capture on video port I measure 3fps in full resolution 4056x3040 and 13fps in binned mode 2028x1520.

With Picamera2 the loop on request without copy of the numpy array nor encoding gives 30fps in binned and 9fps in full, the library can practically follow the fps of the sensor, it’s good. But if we get the numpy array and encode it to jpeg we fall to 5fps in binned and 1fps in full mode. On the other hand the possibility to have the metadata without the full frame is interesting.

I don’t really agree with your explanations on the digital gain. It is not a problem of AGC because when setting the exposure we stay in AeEnable = false nor a problem of realizable exposure

Here is a sequence of requests that starts from exposure 571 and that we ask 6514 with AeEnable = false
AeLocked: False Exposure: 571 Analog: 1.0 Digital: 4.0
AeLocked: False Exposure: 571 Analog: 1.0 Digital: 4.0
AeLocked: False Exposure: 571 Analog: 1.0 Digital: 4.0
AeLocked: False Exposure: 571 Analog: 1.0 Digital: 4.0
AeLocked: False Exposure: 6485 Analog: 1.0 Digital: 1.0043612718582153
AeLocked: False Exposure: 6485 Analog: 1.0 Digital: 1.0043612718582153

The value 4 for the DigitalGain is strange. I think that as you ask for an image with exposure 6514 and the library receives from the sensor images with exposure 571 it thinks it is useful to you by applying a digital gain! These frames must be ignored. Then 6514 is not feasible so we stay at 6485 with a digital gain of 1.004, it does not seem really annoying. It seems moreover that this behavior appears only in the increase of the exposure case.

There are two things which push me towards the new library. For one thing, let’s face it: it’s the future. The old picamera-library has not seen an update in two years, and any new Raspberry Pi stuff will be based on the new libcamera-approach. Second, and actually more important for me: the color definition has certainly improved with the new approach (as one can see from the images above).

One current big drawback, especially if you transfer jpeg-compressed images via LAN, is that jpeg-encoding is done in software. With the old Broadcom/picamera combination, this was utilizing a hardware module. With the libcamera/picamera2 combination, it is pure software-based for now. There exists a framework in the picamera2 code which allows multiple encoding threads which will speed up things a little. The drawback is that the system load gets easily close to 100% with 4 encoding threads.

Nevertheless, I do see in my usual client/server setup (server: RP4 with HQ cam, images encoded as jpeg/client: WIN10-PC, Intel Core I7-87000 @ 3.20 GHz) a frame rate of 6.7 fps at a quality level of q=95, and a frame rate of 8.3 fps for a quality level q=60. These numbers are obtained with a sysload of about 26% of the RP4. The HQ sensor is running in these experiments at 4056x3040 px (i.e., not the binned mode), with libcamera scaling down the image to a 2016x1520 resolution (this is the jpeg transferred via LAN to the WIN10-PC). Jpeg-encoding and transfer of the image via LAN is currently done directly in the capture thread, not in separate threads. So there is the possibility that these numbers can be improved. Not so bad for the moment.

The secret to run the HQ camera with highest resolution (mode=3 in the old syntax) but get a non-binned scaled down version of the capture is the following initialization sequence:

picam2 = Picamera2()
video_config = picam2.video_configuration(main={"size": (2028,1520)}, raw={"size": picam2.sensor_resolution})
video_config['main']['format']= 'RGB888'
picam2.configure(video_config)

The second line is the important one: the part raw={"size": picam2.sensor_resolution} switches the sensor to the old mode3 (full sensor resolution, max 10 fps). The “main”-profile (the one which is later used for scanning) is requested by main={"size": (2028,1520)} to be of a size of 2028x1520. Now, picamera2 defaults to image sizes quantitized in x-dimension to 16, for speed reasons. So it will actually silently reconfigure the output to a 2016x1520 px size. Finally, a little further speedup can be achieved by asking the pipeline to drop the alpha-plane by requesting video_config['main']['format']= 'RGB888' for the main profile.

Well, that was what I observed and what the maintainer of the picamera2-lib explained to me. To put it simple: digital gain is gone. There is no way to set it to a desired value within the context of the picamera2-lib.

The reason is the following: as I tried to explain above, not every exposure time can be realized by a given rolling-shutter sensor. Exposure times have to be multiples of the time it takes to scan and transfer a single sensor line. So what to do if a user requests an exposure time of 2456 which is not realizable by the hardware?

Well, the trick is simple: choose an exposure time you can realize with the sensor in question (in the case of the HQ sensor that would be 2400) and make the missing exposure up by ramping up a little bit the digital gain from the default value of 1.0. So if you ask for exposure time = 2456, in reality the sensor will work with exposure time = 2400 and a digital gain = 1.0233:

2400 * 1.0233 = 2456

That is the reason why with the libcamera/picamera2 combo, digital gain is gone. You still can set the analog gain, you still can set the exposure time. But every time the requested exposure time is not compatible with the hardware, you will end up with a digital gain slightly above 1.0.

Now, there is even an additional twist. Let’s assume you ask for an exposure time of 2400 - which can be realized exactly with the HQ sensor.

Now, in order to give you the requested exposure as fast as possible, libcamera might opt to select for a few frames not the desired 2400*1.0-combo, but work instead with 1200*2.0 or even 600*4.0. All these products of (hardware) exposure time times digital gain give you the same, namely 2400.

I must say that in principle this idea of using the digital gain for approximating arbitrary exposure values within the constraints of the hardware is basically not bad. However, the game is spoiled for serious HDR-work, as the images with the above “alternative” realisations do look different from each other.

Anyway, with that understanding what is going on with the digital gain in the new libcamera-lib, the numbers you have cited above from your experiments make perfectly sense to me. For example, the digital gain = 1.0043612718582153 simply indicates that an exposure time of 6485 is not realizable with your hardware.

The maintainer of the picamera2-lib knows about this issue and he promised me to look into this. As it might require changes in libcamera itself, it might take a while.

By the way, from my experiments, it takes about 11 frames until an exposure change has propagated through the picamera2/libcamera pipeline and shows up in the metadata of the current exposure. Even if you asking for exposure values which can be directly realized in hardware, you need to at least request the same exposure value 5 times in a row to reliably obtain the requested exposure time with digital gain = 1.0. Some details can be found here.

2 Likes

Some further insights into the new picamera2/libcamera-approach. The guys developing this do all kinds of fancy stuff - which might be nice for the average user, but troublesome for our purposes. However, as the new approach is more open-minded, you have more influence about the outcome of the whole process.

Let me give you an example. The new libcamera-stack includes something which is called (alsc). This is the short for “Automatic Lens Shading Correction”. These algorithms can be user-tuned with an own tuning file, but by default a tuning file is used which is shipped with the libcamera installation for various sensors.

Now, lens shading is usually used to compensate for deficiancies with a certain lens/sensor combination. Most interestingly, the tuning file for the HQ camera, which is not delivered with specific lens, includes a alsc-section as well! I have no idea what lens was used here during calibration, but certainly it is not the lens I am using in my scanner setup.

Any indeed, if I look closely at a capture of the empty film gate (I have reduced the q-parameter of the image to utilize banding to show better the issue),

you see that the image center is darker than the image egdes. Also noticable is an annoying pink cast, especially in the right image corners.

Continuing our little experiment, I disabled the alsc in the tuning file - which is simple enough, you only need to change the original rpi.alsc on line 207 of the original tuning file to x.rpi.alsc, like shown in the following screen shot:

Screenshot 2022-06-18 100201

After this little surgery I get a much improved flat response:

The bad thing about this is that the color boost observed between the old picamera/Broadcom combination and the picamera2/libcamera compo is gone as well. Here’s a quick example of a scan with the alsc

and one with the alsc turned off:

That looks very similar to results obtained with the old picamera/Broadcom combo. So it seems that the adaptive lens shading is doing more than lens shading correction only. Oh well…

Clearly, it would be perfect if one could come up with an optimized tuning file for film scanning. Theoretically, one could do such a calibration; there is tuning software available. However, this tuning software requires you to take a series of images, flat and with specific color-checkers imaged. At the moment, I have no idea how to do this (specifically: getting the color-checker into the small Super-8 frame for imaging). Any ideas are welcome!

2 Likes

Thank you for sharing these insights. Working with the camera is on my list, and having these postings would certainly save me a great deal of time, since I am hoping to use the new library.

In this article about the Universidad de la República updates of their scanner, they listed an interesting plug-in for creating surface plots from an image for 3D visualization. I have not used any of it, but a similar graphical representation would be a great way to illustrate the alsc issue.

PS. here is a quick comparison of the images above from @cpixip which enhances the visualization of the alsc unintended consequences for film scanning.

This is an excellent tool to visualize and evaluate the flatness of the scanner components chain (light+lens+camera).

2 Likes