Setting up the light source

[verlakasalt]

Following up on Strange encounter in the red channel/RPi HQ Camera, I am struggling to set up my light source in a way that lets me properly illuminate the image area of a Super 8 frame while also avoiding channel clipping in dark areas.

Here’s my current setup, top view (bear with the clutter…).

Schneider → DIY film gate → Lamp.

The setup is two-leveled, with the scanner-parts on top, and the Pi/wiring below. There is a hole at the lamp-area - I went through a couple of iterations… But the hole is relevant, because it results in light being reflected underneath the gate. To avoid having that light reach the lens, I stuck a piece of cardboard below the gate.

The current lamp is two pieces of 95CRI LED strip stuck to a piece of aluminum with a professional grade LED diffuser at the front.

And the DIY film gate up close (it says IKEA somewhere on there).

I like to think that my light source is fairly decent, but maybe I’m setting the bar too low.

DNG of empty film gate with exposure low enough to avoid clipping

Now for the actual issue: DNG of frame captured with above setup

Exposure 10800, red gain 2.16, blue gain 2.46
Darkest parts red 3920, green 4128, blue 3952

Red/blue banding is clearly visible (this is with +2 exposure, the frame still a little overexposed at +0).
If I want to get the channels above the clipping threshold, I need to at least double the exposure. Then there’s nothing left of the sunset.

Now if I go ahead and lift that cardboard flap like so:

I get scattered light, and the banding is much less pronounced(*), most likely because the frame is now slightly illuminated from the front, too. It’s purple.

“Almost there” example of DNG of frame captured with scattered light

Exposure 10800, red gain 2.16, blue gain 2.46
Darkest parts red 4144, green 4432, blue 4096

(*) I had done the same yesterday getting a frame with these “stats” and no banding at all (at least I don’t immediately see it, but blue is below 4096, so I think it’s still there…) It’s hard to reproduce the “flap position”. Sorry for the subpar example.
Exposure 10800, red gain 2.16, blue gain 2.51
Darkest parts are red 4176, green 4384, blue 4080

I’m aware that this is a particularly dark frame, but there has to be a way to expose this thing properly.

Right now the only ideas I have left are to either double up the diffuser in a small strip just where the frame sits (I probably won’t be precise enough), or to modify the ALSC in such a way that the center of the frame is exposed lower than the borders - if that is even a possibility.

So, to any PiHQ camera users that don’t encounter this issue: How are you lighting your frames?

[Manuel_Angel]

Hi @verlakasalt,

Almost from the beginning I have used an inexpensive 3300 K LED lamp (1.3 €), powered by 12 Vdc, not pulsed, i.e. without using PWM or anything similar.

I have never noticed bands in the captured images.
In the link you can see my setup.

Regards

[verlakasalt]

You’re not using a Pi5 either, right?

I’ve downloaded the DNG from this post and the banding I see is negligible, even though the borders are black below 265.
But the image wasn’t taken with a Pi5, which is the only Pi I have.

@PM490 Since you managed to reproduce “my” banding in your reply. Are you also running a Pi5?

Edit: When I roughly hold my hand between lens and gate and capture a photo, upping the exposure reveals rather intense banding.

If I’m closing the lid entirely, it’s this:

Is anyone able to reproduce this?

I’m starting to think I might have a bad camera cable connection. I replaced the cable for a longer one but picked a cheap one, admittedly… Also, there are two camera connectors on the board (I chose 0). But I’m hesitant to poke around without a hint, because it means dismantling Mr. Frankenscan up there and the original cable is just a tad too short .

[PM490]

I’m using RPi4. I believe @Manuel_Angel is using RPi5 now.

I downloaded the file that you posted. Using Rawtherapee only changed the exposure and black sliders.

As you can see there is some very visible horizontal banding…

@PM490

How stable and well-regulated is the current source of your lamp?

I see that you have replaced the bulb, and now are using the LED strip. I believe it is a white LED only light… can you confirm?
If that is the case, consider using a regulator or a better power supply, and that should eliminate the light banding.

In your comments about the reproduction of the red banding due to black clipping:

@verlakasalt

Looks much more uniform, so I’d wager the light source (or there being light in general) does have a play in this.

You were right.

To have a perfect flat light is very difficult, so the biggest concern on the above is the banding, and the large square patch on the lower left area.

For comparison, here is what I am getting.

I would also be concerned by all the light hitting the lens (probably the cause of the square patch). No need for 3D printer, one can use a black heavy stock paper (or flock paper if you have it) and build a cylinder to reduce the undesired light.
Here is an example of what I used on my first scanner version.

Lastly, please keep in mind that the higher the red gain, the more visible the problem will be.

[PM490]

@verlakasalt

If I’m closing the lid entirely, it’s this:

Is anyone able to reproduce this?

No. Using the same exposure and gains you reported…

libcamera-still --gain 1.0 --shutter 10800 --awbgains=2.16,2.46 --awb=daylight --encoding=png --raw --tuning-file=/usr/share/libcamera/ipa/rpi/vc4/imx477_scientific.json --output 20241206_test.png

Suggest you turn off all power supplies and lights around the area, except the RPi one, and do a capture to confirm is the sensor and not some induced noise in the area.

[verlakasalt]

OK. Weird. I turned off everything in the room, plugged the Pi into a separate outlet (was in a distribution plug with the lamp and display before), captured two dark images looking like this and thought I was getting somewhere.

Turned all the things on again and banding was worse. But then I turned everything off again and banding was still bad:

I thought it might be related to sensor temperature, but metadata says 29.0, which is not very hot at all…

Two images (cap on, devices on) right after the other. Temps are 31.0 both times. Guess I can rule that out…

Might try to switch out the cable after all .

[cpixip]

@verlakasalt

I get scattered light, and the banding is much less pronounced(*), most likely because the frame is now slightly illuminated from the front, too.

That should indeed be the case. The frame you are using requires a high dynamic range for capturing. If you expose to the highlights (which you should do) most of the image areas end up really black. And it is only there were the noise stripes tend to be noticable.

Your artifical scattered light onto the front surface of the film reduces the inital contrast, pushing these dark areas above the “noticable noisestripe level”.

The noisestripes were already discussed in the original “jpg vs raw-dng” capture thread. As early as in August 2023 this image was showing them

The testimages displayed above in this thread which have been taken with closed lens lids do show these characteristic stripes as well. This basically rules out any light source as the cause. Simply, because there is no light at all present in these captures (but see below my comment about a camera backplane).

What is interesting is that the statistics of the noise shows different patterns, even in consecutive frames. But overall, everybody is “seeing” these stripes, more or less in a similar fashion. Very weird.

I think any influence of the camera cable on the noise is very remote - afterall, the signals are digital, not analog. Bad transmissions due to a bad cable do show up, but they manifest in a quite different way than these noise stripes.

One thing I have not mentioned so far but what might be important in this context: the PCB of the HQ sensor is slightly transparent. So light entering from the backside of the PCB influences the pixeldata the camera sensor is capturing. That is the reason that one of my future projects is this: “design a camera back for the HQ camera”. In this respect note that the GS camera already does come with such a backplane cover.

I have no idea where the noisestripes are coming from, but I suspect an issue/artifact in the analog portion of the HQ sensor. But in reality, everything is possible. I would rule out an issue with the lightsource.

[verlakasalt]

I’ve been looking through a whole lot of threads trying to find examples of the issue I’m having, and I didn’t find any images with comparable banding. The fish only has those fine lines, I’d say, not the blocky colored ones that are also wildly dancing around when the film is in motion.
Do you still have the dng around?

PM490’s “cap on” frame isn’t half as bad as mine, either. If there’s any way to figure out whether my camera is faulty or something, I’d really like to know…

[cpixip]

@verlakasalt

I’ve been looking through a whole lot of threads trying to find examples of the issue I’m having, and I didn’t find any images with comparable banding. The fish only has those fine lines, I’d say, not the blocky colored ones that are also wildly dancing around when the film is in motion.
Do you still have the dng around?

No, sorry. I usually do not keep files around that long. Here’s another one to download:

This is an actual capture of a black rollout section. Enhancing the red channel, you see the horizontal noise stripes:

Again, I want to argue against the light source being the cause of these horizontal stripes. As already mentioned, they should not be present with a cap over the lens.

Capture your empty film gate (without any film) so that the gate appears medium gray. That is, you do not want any part of your gate to burn out. Note that this requires a shorter exposure than the one you are using when capturing film. If intensity modulations are present in your illumination, you will see them once you increase your contrast level. That is also a good technique to check whether your illumination is even or shows some vignetting. And to check whether your sensor/lens is dirty of dust. I bet you are not going to see any horizontal stripes.

At this point in time, I am tending slightly toward the assumption that maybe the stripes are caused by stray illumination from your room lights through the camera’s PCB. Can you run your lens-cap-on test with all lights in the room switched off? It’s just a shot in the dark, trying to find a black cat in a coal cellar…

[PM490]

@cpixip

Again, I want to argue against the light source being the cause of these horizontal stripes. As already mentioned, they should not be present with a cap over the lens.

For clarity, the horizontal bands that I attributed to the light are these.

@PM490

I downloaded the file that you posted. Using Rawtherapee only changed the exposure and black sliders.

As you can see there is some very visible horizontal banding…

The pattern looks different as the black-clipping bands…

@verlakasalt

If I’m closing the lid entirely, it’s this:

In other words, there may be two issues: The banding on the blacks, and the banding on the light.

For troubleshooting, it is best to work out what is the problem with the cap on first. It is possible that the banding on the whites is caused by the same black shifting.
Once the banding on the dark is addressed (if it can be cleared), then you can confirm if the uniform banding on the white level remains… in that case, it is likely to be the light source.

[verlakasalt]

@cpixip Here’s the link I posted above.

I did change the camera cable and it’s not the issue (surprise… )., but that means I can’t deliver a medium grey image right now. The below one is not overexposed, though, so I think it’s sufficient to tell whether my light source is uniform enough.

DNG of empty film gate with exposure low enough to avoid clipping

I also saw @PM490’s “version” of it… no idea where the smudge could be from. And what sort of dirt would be aperture-shaped?!

I tested taking pictures in full dark, but the results are random. Some have “nice” noise banding, some have “ugly” banding, and I guess it’s just the sensor trying to fill the frame with something when all the pixels are below the threshold…?
@cpixip’s dark frame isn’t devoid of banding either, but it still looks a bit more pleasant than what I get.

Is my light source really that bad?

Edit: I ran your dark frame through the script you posted in the strange encounters thread.

Full Frame Data
_______________
             Minimum red   :  3904.0
             Maximum red   :  31568.0

             Minimum green :  4040.0
             Maximum green :  65520.0

             Minimum blue  :  3840.0
             Maximum blue  :  41856.0

From everything I thought I understood, these values would mean there’d be clipping. But there’s none(?).

It looks like your red and blue gains are lower than mine; what were they set at?

[cpixip]

@PM490

In other words, there may be two issues: The banding on the blacks, and the banding on the light.

Uh… - I missed that one. So you already captured an empty film gate. Let’s have a look at that data, slightly contrast-enhanced:

The darker spots, some approaching in shape a pentagon, are dirt, either on your lens or your sensor. That’s nothing I would be bothered about.

There are some broad bands visible - they are horizontal. Like the noisestripes we are discussing. But they have a totally different spatial extend (coherence). They are much broader than the noisestripes. These are most probably caused by the powersource of your illumination.

Most noteably, there are no noisestripes visible which extend only over a few horizontal lines. Those lines seem to occur only in dark image areas.

Two other, magenta-like broad stripes are also visible. One vertical one, on the left side of the gate, another one horizontal at the bottom. Both are most probably caused by reflexion and shadowing in the path from LED to gate.

All in all, your gate is resonably flat as it should be. But as @PM490 mentioned:

@PM490

In other words, there may be two issues: The banding on the blacks, and the banding on the light.

…

@verlakasalt

From everything I thought I understood, these values would mean there’d be clipping. But there’s none(?).

Yes, there is. Think about the sprocket hole.

@verlakasalt

It looks like your red and blue gains are lower than mine; what were they set at?

The “As Shot Neutral”-tag in the .dng reveals it. It is
As Shot Neutral : 0.3460207612 1 0.476213153
which translates into red gain = 2.89 and blue gain = 2.01.

@verlakasalt

And what sort of dirt would be aperture-shaped?!

Any dirt which is not on your sensor. That is, dirt on the lenses (even internal ones). I would not bother about this. It’s is unnoticable in any scan you are going to do.

@verlakasalt

I tested taking pictures in full dark, but the results are random. Some have “nice” noise banding, some have “ugly” banding, and I guess it’s just the sensor trying to fill the frame with something when all the pixels are below the threshold…?

So we didn’t hit the black cat in this try… Frankly, I do not know what is happening here. What really is hard to understand is why some frames show more noise than others, under otherwise identical capture conditions. Maybe it’s EMI (electromagnetic interference) from an external appliance. I guess that’s a project for another day. Currently, I have no clue what is causing these stripes and especially the variations we are observing over time and different installations.

PS: did you notice this thread? Your lens might not have it, but it might be also a source of interference.

[PM490]

@cpixip

Maybe it’s EMI (electromagnetic interference) from an external appliance. I guess that’s a project for another day.

I suspect so… reason why…

@PM490

Suggest you turn off all power supplies and lights around the area, except the RPi one, and do a capture to confirm is the sensor and not some induced noise in the area.

[cpixip]

@verlakasalt: well, I think it’s about time to summarize all thoughts/results found here on the forum about horizontal stripes in HQ sensors (IMX477).

As @PM490 already mentioned, most probably there are two different causes here at work. One manifests itself with Pablo’s scanner as well as my own scanner. It even is present when there is no artifical illumination present at all. Here’s that example being captured with daylight as only illumination, using a standard HQ lens (not a Schneider Componon-S). @npiegdon does not have that issue, as he’s using a different camera. I did not find any data or comment from other users here on the forum using the HQ camera, like @Manuel_Angel, @jankaiser or @dgalland on that topic. Maybe they did never notice. Importantly, these noisestripes are there even when there is no light at all falling onto the sensor. So they are not caused by any illumination issue.

At this point in time, I think it’s pretty save to assume that these noise stripes are a property of the IMX477 used in the HQ camera.

They are exaggerated by what @PM490 described here long ago

@PM490

I would speculate that what we have/see is noise partial clipping (bottom side), and speculate further that if there is any slight variation in the image black floor intensity, resulting in some black levels reaching the black clipping and some not, it would happen regardless of the light source. When the gain of red is increased, the noise amplitude is greater, and the effect is more visible. But the problem is apparent with all the channels -when their respective levels- reach the black clipping threshold.

That situation is even worsened by the fact that we are regulary seeing pixel values below the blacklevel indicated in the .dng-file. The picamera2 software is working just with a fixed value, not reporting/using the values of special (blackened) pixels on the sensor as other cameras do.

What we are talking about here is a signal variation at the most sensitive level of a 12 bit dynamic range. It’s certainly not correlated with any illumination issues - it might just be EMI from nearby sources inducing noise in the bare PCB of the HQ sensor. That might explain the variations in noise amplitudes you are seeing in your results. Current power supply technology is based on fast switching circuitry - a cheap one might not care at all about EMI regulations. Same goes for any dimmable LED illumination. Stepper motors and their drivers emit also a noticable amount of electromagnetic noise. If any of that is picked up by the HQ PCB, it will be amplified by the clipping action described in @PM490’s quote above.

However, at this point in time, this is pure speculation. Shielding the HQ PCB from external influence would be impossible in my setup and challenging anyway (you do not want to short-cut things when attaching shielding). And it might not even be the real solution, as other possibilities exists on this electronic noise. As already mentioned: use the best power supplies you can afford. Both for your Raspberry Pi (which powers the HQ camera indirectly) as well as for your light source.

Which brings me to the other type of horizontal stripes, nicely displayed by @PM490 in his screenshot above, which I am citing here for reference again:

These stripes should not be there. Compare the above image with the analysis of @PM490 posted here (which explains why the ALSC is turned off in the scientific tuning file).

These wider stripes on your empty film gate capture are certainly caused by a not so great power supply of your LED. These intensity variations will be enhanced by the process @PM490 described in the post I quoted above and that will lead to the appearance of the larger scale horizontal stripes with a noticable color cast in your captures like in this capture of yours:

So, to summarize: the HQ sensor does feature an interesting noise characteristic in raw image data. This noise gets enhanced during the development of the raw. The original cause of this noise is currently unknown, but EMI and/or power supply issues might play a role here. That kind of noise is seen in other installations (mine, @PM490) and I guess we have to live with that noise at the time being.

Besides this, your illumination is not constant over time. You have to address this issue. The intensity variations of your illumination get amplified by the non-linear process (clipping below the blacklevel) described by @PM490, leading to dancing color bands in dark areas of your capture. Of course, this should not happen. Invest in a better power source for your LED and see what happens.

[verlakasalt]

I’ve made a few more test images after removing all the periphals (aside from an USB mouse), switching out the distribution plug and so on, and the one thing I noticed is that the banding gets worse with time. The first couple of images after booting look normal (within the capabilities of the sensor), then after a few minutes the horizontal lines become wider and more irregular.
I don’t know what that could hint at, but I might be ready to switch out the camera entirely. This is really frustrating :(.

Thanks for all the help so far!

[PM490]

@cpixip

The original cause of this noise is currently unknown, but EMI and/or power supply issues might play a role here.

@verlakasalt While not fully, the HQ setup that I use does provide a bit of EMI shielding. The sensor mount and also the T2 tubes are metal. The Rpi4 enclosure is all metal as well.

@verlakasalt

after a few minutes the horizontal lines become wider and more irregular

Given that you replaced the cable, and in the absence of EMI issue, then it may be the sensor. What comes out of the sensor to the RPi is already digital, so the interference or defect would likely be in the sensor conversion itself, or in the power (or power reference) of the A/D.

[cpixip]

@verlakasalt

… the one thing I noticed is that the banding gets worse with time. The first couple of images after booting look normal (within the capabilities of the sensor), then after a few minutes the horizontal lines become wider and more irregular.

Did you try to improve your illumination? Was the illumination running continously “on” during these tests or did you simlutaniously switch off your illumination when rebooting? Or did you not use your illumination source at all, doing your cap-on-lens tests?

All my image captures were done with both the Raspberry Pi as well as my illumination source continously running, usually for hours before the actual capture (as these are intermediate captures from long capturing events). With different HQ cameras and lenses. I have the feeling that the banding you are seeing is more than what I am experiencing with my HQ cameras.

Before jumping to conclusions, I would suggest to make sure to solve your illumination issue. Or at least repeat those tests in the cap-on-lens setting.

With respect to the worsening over time after a reboot: that might indicate a temperature issue.

Now, the HQ camera has three voltage regulators which filter the incoming 3.3 V from the Raspberry Pi. So the voltage regulators of the Raspberry Pi should not cause an issue here.

My HQ sensor chip operates (again: after hours) at less than 30 °C:

That’s only slightly above room temperature - and if you are rebooting your Raspberry Pi, the camera should actually not notice anything, as the 3.3 V voltage from the RP will not get interrupted. In other words: the camera will not cool down or so during reboot, potentially causing a reduction of the noise.

As far as I know (I might be wrong here), the camera is not really initialized during a reboot. That happens only when you create the camera in your picamera2 code or opening the camera in any of the other applications accessing the camera. So does this temporary improvement of your images happen also when you restart your software? Or only after a reboot? I assume the situation is similar to this result you posted above:

with the right side similar to the situation right after a boot, and the left side after some time?

[verlakasalt]

Was the illumination running continously “on” during these tests or did you simlutaniously switch off your illumination when rebooting? Or did you not use your illumination source at all, doing your cap-on-lens tests?

The lamp wasn’t plugged in at all. I tested with the lens cap on. Occasionally I pointed a little flashlight at the lens, but the slightly illuminated images also had banding.

With respect to the worsening over time after a reboot: that might indicate a temperature issue.

I checked the temperature readout from the camera’s metadata, and there was no correlation between banding and temp - started at 25.0, went up a bit, then was at 25.0 again with different banding characteristics over the course of 10-15 minutes, I’d say. Not sure how reliable that value is, though.

So does this temporary improvement of your images happen also when you restart your software? Or only after a reboot?

Interesting question. Will check later.

I assume the situation is similar to this result you posted above:

with the right side similar to the situation right after a boot, and the left side after some time?

That’s what I observed, yes.

[Manuel_Angel]

The banding problem has piqued my curiosity and I decided to do some more tests with my device.

@verlakasalt indeed, the previous images were taken using RPi4. Since the beginning of the year I have been using RPi5 almost exclusively for everything. I usually use it as a desktop computer. Above all I love the silence and the minimum consumption, while maintaining very respectable performance. Unfortunately it is not possible to run DaVinci.

Well, the images I am attaching have been captured with the RPi5 and the HQ camera at the highest resolution.
The following image is of the completely free capture window, with an exposure of 0.1 ms. I don’t know about you, but I still don’t see banding.

This other image corresponds to the one I usually use for tests, I like it, it is a nice image of the Montjuich gardens in Barcelona, ​​taken during the summer of 1975. In my opinion, again without banding.

I have attached jpg images, but at this link you can download the same images in dng format.

If you deem it appropriate, you can tell me to perform any other additional tests.

Kind regards

[jankaiser]

Interesting issue you are bringing up! I thought I’d give my 5 cents as well.

I never really noticed either type of noise in my scans so far, and in checking the flatness of my illumination, I only ever spotted the aperture-shaped dust speckles mentioned above.

However, I went through and found something that looks similar to the effect you describe in an old screenshot, though following the taxonomy of @cpixip’s post above, I’m not sure if its the first or the second kind. In this frame it can be best seen in the dark area on the right.

For context, this is a horribly underexposed roll of film. On a film viewer this is virtually black and all you can make out are the cockpit windows. The underexposure was not caused by the bright window in the otherwise dark environment, but rather by either user error or a technical issue with the camera. Actually, multiple consecutively filmed rolls had this issue as well as white frames in-between scenes. Some of them were filmed outside on a perfectly lit sunny day. Rolls filmed immediately before and after with the same camera did not have this issue. So I never expected these rolls to look good at all and actually figured these stripes could even originate in the emulsion or wherever … I didn’t really suspect the digital capture, but I also didn’t think it an issue. These rolls were not watchable in an analog world and now they are.

Interestingly, the processed video frame does not show this effect, though it is possible I lowered the exposure slightly to hide weird effects … I don’t remember. It also went through a colour noise reduction step.

For technical context, these were captured with the HQ camera on a Raspberry Pi 3B+ with picamerax (i.e. a fork of the original picamera).

I would attach the raw DNG file, but I no longer have it. As previously mentioned, a DNG file from the same capture system can be found in this post.