Evaluating Extended Exposure Fusion

d_fens · January 12, 2024, 7:13am

Hello Everyone,

I’m reaching out to discuss the Extended Exposure Fusion method, a recent development in HDR imaging, particularly for handling exposure series. This method offers several enhancements over the traditional Mertens merge technique and I’m considering its potential integration into OpenCV or C++. Here are the key advantages of Extended Exposure Fusion:

Reduced Artifacts: It effectively minimizes common artifacts such as out-of-range issues and low-frequency halos.
Single Exposure Application: Unlike Mertens merge, this method can also be applied to single exposure images, offering greater flexibility.
Advanced Fusion Techniques: Incorporating advanced methods like Quaternion Factorized Simulated Exposure Fusion (QFSEF), it aims to improve results in challenging lighting conditions.

For a practical demonstration of this method, there’s an online demo available here: Extended Exposure Fusion Demo.

I’m interested in your opinions on two key points:

Do you think the Extended Exposure Fusion method is valuable enough to warrant its porting to OpenCV or C++?
How do you assess the quality of the results obtained through the online demo, especially in comparison with the traditional Mertens merge?

Your insights and experiences would be greatly appreciated!

cpixip · January 12, 2024, 12:00pm

Well, this is a kind of old article from 2019. From what I remember reading it some time ago, the original exposure stack is artificially enlarged by, say, doubling the number of input images. The additional input images are “created” by applying a kind of squashing fct from the original ones. The “normal” Mertens exposure fusion algorithm is than used for merging this artificially extended exposure stack.

Of course, the final result needs to be remapped into a usable exposure range for further processing.

Taking all of this together, you end up with more parameters to tune and potentially better results than usual exposure fusion. Since the input images are squashed before fusion, the proposed algorithm yields also a different look the standard exposure fusion. The details can be found in the paper.

The online-demo is nicely done and I would invite people to run this demo with their own images. This “new” algorithm seems to produce softer (more even) results as the standard one. It’s probably a matter of taste as to which one someone might prefer.

As with every exposure fusion algo I have encountered, the appearance of the material is notably changed by the fusion step. While in original footage you still can sense (with a little expertise) the underlying film stock (Kodachrome vs. AgfaChrome for example), after exposure fusion, in a way “all looks the same”.

The major drawback of the proposed method is the increased running time, as the number of images in the stack is increased on purpose. That will hunt you once you get into processing interesting frame sizes.

Otherwise, note that the implementation uses anyway the original Mertens exposure fusion approach - which is already implemented in openCV. You just need to add the remapping functions for input and output. This should not be too difficult and as these function are a kind of simple ones (only pixel operations), they could be implemented in numPy/Python efficiently with a decent speed.

One caveat - the remarks above are memory items. It’s been at least one or two years since I looked into this paper.

justin · January 12, 2024, 4:58pm

I ran the online demo with the house data set and default settings. It appears the white balance is different for output of the 2 methods. I think the temperature of the extended exposure fusion output is closer to the original bracketed exposures. I can’t tell if the shift is global or local. I’m curious to others opinions because color correction is not a strength of mine.

Others on this forum have criticized exposure fusion for it’s color shifts and have started working with a single raw image for each film frame.

My suggestion would be to use actual bracketed film scans and run the online demo for different values of parameter Beta and compare the results before spending time coding in C++.

Question: would color shift be an issue if the input images were first transformed to a different color space i.e. HSV/HSL or YUV/YIQ? Would it be possible to perform the fusion only on the intensity component and preserve the original color?

Input

Exposure Fusion

Extended Exposure Fusion

cpixip · January 12, 2024, 5:31pm

– note the quite noticable white borders in the window frames looking outward with the EEF algorithm. That is not present in normal exposure fusion. The wall right of the cupboard is too yellowish for my taste in the EEF result.

The EEF result has generally more (local) contrast, and that leads to more saturation in the colors. It is hard to compare color fidelity from such rather arbitrary images. One would need to do some serious here tests, with some color charts in different exposure zones. I planned this for a long time. But…

d_fens · January 12, 2024, 6:53pm

On page one of the archive further down there is a colorlogic color sheet which can be loaded into the demo by clicking the “reconstruct” button.

Initially I was keen on reducing the halo in the merged now I have to fiddle with the settings as any subtle color shift seems to be amplified by the EEF method. Overall the image quality is more vibrant to me but as @cpixip said, it’s a matter of taste which image appearance one prefers.

I tried to contact a guy who ported the SEF (single exposure with generated additional exposure) code to opencv if he could help porting the EEF method also, maybe he has something up his sleeve.

mightimatti · February 16, 2025, 2:47am

Hi,
this is my first time posting but I have been reading this forum from time to time lately and seeing as this is a topic that I have spent quite a bit of thought on over the past year, I thought I may as well sign up and comment. I would like to add that while I am doing a master’s in Computer Science and focussing on vision, I am no expert and my thoughts expressed here stem from my experiments and experience building a film scanner DIY and are not corroborated by scientific sources:

I built a custom scanner (for photographic purposes) over the past year. It uses a monochrome industrial GigE camera and while it has 9MP over it’s CCD’s 14mmx12mm area, it’s dynamic range of ~60dB wasn’t quite satisfactory(especially for color reversal film/slides) for what I was trying to do.

“As film doesn’t change over time, it’s convenient to do do HDR” I thought.

So I started looking into how this can be achieved, thinking that I wanted to do some sort of HDR without a specific idea what this actually means. I soon realized that HDR is actually composed of two steps, exposure fusion(1) and tone mapping(2). Expecting this to be a complex problem and expecting it to require a lot of calibration i started looking for information on how to calibrate this for film, and came out with nothing useful. Eventually, after pondering on this problem for a long time and going back to first principles, I realized the following things:

Exposure fusion is incredibly straightforward, because my sensor is putting out linear data. Under the assumption of good/perfect linearity, this linear nature of the data means that changing the exposure by a know factor F , implies you can multiply the seconds exposure by value 1/F and sum the results. This results in a raw image with values recorded over a wider range. Obviously there will be a little more noise as well. If you’re storing your raw data in a 16-bit container and your sensor doesn’t output 16 bit data, you can do this quite a few times until your raw-container overflows.
My sensor had a 14-bit output, with the maximum ADU value of ~12000(2**14=16384 being the theoretical limit). For photographic slides, I chose to integrate 4 images, bracketed like so:
[-2EV, -1 EV, +/-0EV, +1EV]
Tone mapping is not required! The scene in the image you are capturing is already photochemically compressed by the film. There is no need to map the gathered values, because you won’t store a range of tones which is “unnatural” to the human observer. The limits of the brighest and darkest values recorded can now be determined by the media you are digitzing(and losses in the optical system such as glare and non-image forming light) and your sensor’s noise floor, but not a lack of dynamic range. All I did was apply gamma in the end and the images look great to my eyes.

Here is where I was gonna post example images, but being a new user I am not allowed to :-/

I actually did 2 test scans of IT8 calibration slide that demonstrates the range of tonality I was able to achieve, capturing highlights that would otherwise have been lost with my sensor. Again, I can’t post this.

I went on to scan approx 4K slides with this machine and none of them had coulour that didn’t correspond to their appearance on a light table

I would like to add that this worked incredibly well for me, because my data is very “clean” to begin with. The narrow band LEDs, selected for optimal colour separation, result in little cross-talk between the channels and allow me to fuse every channel individually. I am not sure how this would behave with the wide, overlapping filters used on a convential Bayer-filter’s CFA for colour cameras.

For positive scans with a digital camera of dynamic range inferior to 70dB this is 100% worth it IMO, though it will bring down the frame rate.