OpenCV does all of this. I think all the functions you’d need to pull this off can also use GPU acceleration for free, though I think OpenCV is primarily CUDA, so on a mac, it’d need to be CPU only now that Apple has basically dropped support for GPU acceleration that’s not for their own GPUs or Radeon cards. We’re starting some tests this week to see how OpenCV performs on Mac vs PC, i7 vs i9 vs Xeon, etc. I’m building a simple test cross-platform application to do the frame alignment (from Scanstation scans captured with optical registration turned off, so the film is not stable), and time each step to see where the improvements/bottlenecks are. Hopefully we’ll be running tests in about a week or so.
I’ve used OpenCV in the past. I suppose the bottleneck will be processing speed and the frame rate of the camera doing the capture for the CV imaging. Will it require a separate camera? At what speeds will we be able to scan at?
My guesses are:
-
you could use the same camera for CV of a video stream as you do for still capture, but you’d have to go slower than you would with a separate perf sensor
-
you could use a second camera to speed that up and that will cost a little more (but likely not much) and the speed you can scan it will be dependent on the frame rate of the CV camera and the computational power of the system it’s tied to (which could be totally separate from the rest of the system, as long as it’s capable of outputing a signal for camera trigger).
Overall I think it’s a very promising route. Let me know if you want a little funding for testing and/or if I can send you a reduced rig of parts (a film loop on motorized rollers, for example). I’d love to see us pursue this route.
So I think what we’d be doing may not be applicable to Kinograph, at least not directly, because we’re not working with a continuous motion transport.
Since ours is using intermittent motion, we’re basically snapping the image, then doing all processing before writing out the final to color composited and aligned image to disk. So the rough setup I’m thinking is that a color channel is captured and then a thread is spun off that handles optical alignment. meanwhile, the next color channel is being captured and spun off into its own thread, and then again with the last channel. Then the three aligned channels would be combined into an RGB image and written to disk while the scanner advances to the next frame.
The questions I’m trying to find out in the next week: where is it best to do this optical alignment? OpenCV or something like ImageMagick (my bet is on OpenCV)? What’s the best workflow? Does GPU acceleration help with this?
We’re post processing the image, but the kinograph would likely need to process the image in real time as part of the image processing workflow, and that’s probably going to require GPU or some seriously fast CPUs and clever threading. If they’re GPUs, they’re probably going to need to be pretty substantial, if you’re dealing with any kind of high resolution imaging.
The ScanStation is able to do 6.5k scans at 15fps with continuous motion (7.5fps if it’s HDR). They do something kind of like what I describe above, but instead of CPU threading, they’re sending frame A to GPU 1, frame B to GPU 2, frame C to GPU 1, frame D to GPU 2, and so on – ping-ponging between GPUs to do the heavy lifting (optical alignment, color, etc). That said, it only uses a pair of GTX 1070s, which are not especially expensive or powerful compared to the high end cards…
2 Likes
What kind of speeds are you going to get on the Sasquatch? The Arri’s do I think 3-4fps and the Directors are a bit faster. I don’t know how they work under-the-hood, the original design as we all know was to keep the film as steady as anything in the gate and then you can flash as many times as you need to get a good scan and the image is merged into a 16-bit DPX or TIFF file, that must be still how the merge works because both of them do native micro-scanning and that wouldn’t be possible doing a purely software-based alignment. What a lot of the old machines do to get the film steady is to clamp it while it’s in the gate, but I think the current generation of LG/Arris just have the gate close over the film so that it’s very nearly clamped and it runs through in intermittent motion without any issue similar in design to how the Scanstation’s warped film kits work.
We need a programmer to clone the LG software. 
I don’t know if it works the way you’re describing, they do multiple file-type outputs simultaneously which is probably the only reason they need two GPUs, I know that you can put in a 3rd graphics card and then it can output more simultaneous formats at the same speeds, and I’ll bet if you use just one it can probably still go straight to Prores at the same speed as before but would struggle with an additional output?
It’s only the scanstation, archivist, and the cintel the have a software HDR-merge without intermittent motion, Blackmagic’s one doesn’t work all the time you get misaligned frames, so if someone comes up with an open-source HDR merge it would help a lot. I don’t know how LG’s one works, but if I was designing it I’d use a variable light where you can set most of the scanning image area to the colour you want (i.e. red, green, blue, under-expose, over-expose, infrared) but the sprockets stay the same colour each time (white) so that they photograph exactly the same on each exposure and you can merge more reliably. The Cintels appear to do something like this because their sprocket holes are grey not white. To prevent distortion to the scan if you have a diffused light you’d probably need to use the outside parts of the sprockets for stabilisation not the inside parts. So you’d have 6 or 7 images in black and white that get individually stabilised and then merged into a single HDR frame, and then that frame can be stabilised again to the previous frame.
The other way would be to engineer the scanner so the frame is as steady as it is in a Director.
1 Like
At full resolution, 16bit, 3-4 fps for black and white, 1fps for color, is the goal. The speed is a function of the camera, which is monochrome and runs at about 3.6fps max in 16bit mode. It will likely be faster with lower resolution scans, where we’re using an ROI in the camera rather than the full sensor. But not by much. This machine is not built for speed.
I don’t know about the Arri, but the Director is just using the pressure plate to hold the film flat, not steady. the early Director scanners had registration pins, I believe, which were for steadiness, but they’re sprocketless now. I see no need to have something to hold the film steady beyond the capstan drive motor. If all the tensions are balanced correctly between feed and takeup motors, there’s no reason to believe the film will move between exposures. That’s how we’re building the Sasquatch scanner.
Believe me, that software has its issues, and could use a major rethink. there are some very nice things about it, but it’s rooted in the DI days, where you would set up a project for each reel of film (including a single feature that’s 4-5 reels). If you were to use it the way they designed for a home movie scanning project, we’d have jobs that have 60-80 project files, one for each reel. Instead we use the software a little differently than they imagined people would use it, and that makes some day to day things a bit clunky.
I’ve never heard of this. The software ping-pongs frames between GPUs, which I know for a fact (straight from Lasergraphics). ProRes, in every application I’ve ever used that exports it or reads it, is CPU-bound, so GPUs really wouldn’t help much there. My understanding is that Lasergraphics is doing the color processing on the GPU, and they’re most likely also doing scaling, cropping, and optical pin registration there as well since those are all things that can benefit from the speed a GPU provides. All of that is computationally expensive, so if they can send one frame to GPU A while scanning the next frame and sending it to GPU B, they can speed up the overall process.
But I don’t think it’s designed to scale with more GPUs. Maybe it is, but then you have issues with the motherboard most likely, saturating the PCI bus. This is a single i9 CPU machine, so there aren’t very many PCIe lanes available to begin with, and you have to have a frame grabber board, RAID card, likely a high speed NIC for your network since they don’t use the 10GbE version of the motherboard, and two GPUs in there. Not even sure you could fit a third GPU physically, on the motherboard!
This stuff is built into OpenCV: OpenCV: HDR imaging
Our first version of the scanner will not have HDR, but it’s on the roadmap for future upgrades. It’s just a matter of coding it, really. Obviously, it’ll be a lot slower though. but we’d use the built in HDR tools in OpenCV for this since we’re already using OpenCV for registration, scaling, and color space stuff.
I’m not sure what you mean here. The ScanStation and Director only see the outside of the perfs on small gauges. When you get into 16mm and up, the outer edge of the perfs are cut off by the film gate. They use the inside edges. To do optical registration on the perfs you need to first make the image black and white (as in black and white with no greyscale) and you do the math based on those B/W images, then apply the translation to the actual images. Unless you’re overexposing it terribly, causing some kind of blooming effect at the perf edges, this works well. We have been testing this in OpenCV for the past few weeks.
Oh okay but that’s Prores, your proxy files can be h246 and hvec and those are GPU encoded. That’s why it doesn’t sound logical that they need to “ping pong” frames for stabilisation, but there are two different stabilisations so maybe that’s what they mean. There’s the stabilisation frame-to-frame and the Scanstation has done that from the start (and Lasergraphics I think were years ahead of everyone else with that), and then there’s the HDR merge which they added as an expensive option 4 years ago. With the image stabilisation it’s not essential that each frame is in exactly the same position, as you would know if you go through a splice it can jump a bit and carry on. With an HDR merge, or an R/G/B merge, it is essential they’re exactly aligned.
Yes I know, it looks like this:
I’m just suggesting you’d make it easier (and more reliable) for the software if it’s possible to set the perfs to their own colour when the scan is done as you don’t need them for the final scan. The BMD Cintel produces grey perfs somehow that’s where I’m taking that idea from. Of course their merge doesn’t work great at the moment, but that’s its own issue. If it’s not possible it’s not possible, it’s just an idea. 
If you’ve put an integrating sphere over the light I don’t see how it is possible at least not with just one light, but that’s a problem for an engineer.
The software designer in me really likes this conversation. If I’m following correctly, we want something that does the following. Am I on the right track?
- Writes image to file
- if using CV frame registration, send to GPU for alignment (details TBD)
- (still on GPU) any global processing such as noise reduction, banding masks, color correction
- (still on GPU) transcoding file type, if required, and re-write to disk
Not necessarily. I mean they certainly could be, but they don’t need to be. We did DVD and Blu-ray authoring for 20 years and none of our encoders used GPU. I don’t know one way or another what they’re using to do the encoding, but my guess is that it’s not the GPU. There are a bunch of things they would need to do that would benefit from GPU acceleration, in no particular order:
- locate the perfs
- abstract the image to B/W for easier/faster manipulation
- move the image to align to a predefined position, using the perfs
- if HDR, calculate what parts of the image are kept and discarded from the second flash
- merge the image togehter
In and of itself, each of these operations is done in milliseconds, but there are probably half a dozen more steps here that I haven’t included. that adds up. It also doesn’t include the time it takes to move a 12bit high resolution image across the PCI bus from the frame grabber to the GPU and then back out of the GPU when done. If you want to scan at high speed, you need to split the load the GPU is doing so that you don’t have bottlenecks.
The point of ping-pong operation of the GPUs is to be able to do things in parallel. While a frame is capturing and being loaded onto one GPU, the other one is finishing the processing and getting ready for its next frame. If you didn’t do this, you’d need to do things sequentially, and that’s necessarily slower.
Again, machine vision algorithms take care of this by matching pixels from the two images so they’re perfectly aligned. This is all in OpenCV. Whether Lasergraphics uses OpenCV or not I don’t know, but I haven’t seen any evidence of that in their included libraries with the executable. They probably rolled their own, but it likely does the same thing as OpenCV’s HDR tools do.
Perf detection gets your frames where they need to be frame-to-frame. image alignment does the fine tuning of that. I know for a fact that Lasergraphics sets up calibrations for each gauge and each resolution mode for each gauge, and this is loaded up when the film is loaded. This information tells the scanner where to put the frames once its detected the reference perforation(s). So there’s a fixed position, that works like a mechanical registration pin, only digital. In our brief tests with the northlight (mechanical pin) vs ScanStation (optical pin), we found the ScanStation to be more accurate. I wouldn’t be surprised if that’s good enough to do the HDR stuff with, to be honest.
I don’t know what’s going on with the cintel perfs being Gray. I’ve never used that scanner so I can’t say. But any scanner I have used, the perfs are either white or black unless you’re doing a color correction or base calibration that changes the color in there (this was one of my big beefs with the Fleukiger paper on scanner comparisons - the scanner operators clearly didn’t set the film up right for blue tinted film, calibrating the frames which wiped out the blue color. this in turn turned the perfs pink. their conclusion was that the scanner couldn’t handle tinted film, which is ridiculous. The conclusion should have been - get someone who knows how to scan film, to scan the film). but, I digress…
Can you post a frame from a Cintel with these grey perfs? I’d like to see that.
I would avoid the initial write to disk, but basically yeah. The I/O stuff is really expensive, and will kill your scan time. The way we’re doing it is to grab the frame from the frame grabber’s buffer (which is on the card), into a memory block in our application. From there it’s passed around entirely in RAM from our app to OpenCV and back. We’re using GraphicsMagick to write out the DPX and other file formats, and if we do ProRes, probably something like the ffmpeg API. All of this happens in RAM though, so that we don’t hit the disk.
In my initial tests, most operations are measured in a few milliseconds. Writing a high res file to disk though, could be as much as 3/4 second. Granted, we’re talking 14k, but that number will be faster or slower depending on the disk speed, etc.
Also, you don’t necessarily need to use the GPU for this, but you may if speed is of the essence. In my testing it’s purely CPU for now, and OpenCV is quite fast.
I did some tests scaling a 14192x10640 bitmap down to 640x480. In plain Xojo code (that’s my preferred programming environment, which generates native code for multiple operating systems) this takes 151ms. In OpenCV called from Xojo using our (soon to be open source) library, 18ms. This bodes well for doing a fair bit of OpenCV processing on these images without slowing down the system!
Thanks for reporting back on this @friolator. I’ve been looking into the Jetson Nano and Xavier as a possible computation solution that could help us with OpenCV tasks and keep the price of the PC solution down. Have you had any experience with these?
That’s because (a) the GPU-based encoding isn’t anywhere near as good and (b) they wouldn’t be bluray-certified. For bluray authoring you need to be 100% sure the encode is fully bluray compliant. But it’s good enough for making a quick proxy, after all it’s only the proxy it’s not going directly to bluray. Even x264 I think has a couple of limitations that the commercial h.264 encoders don’t have (other than that it’s equal to them).
We haven’t. Since our camera is tied to a PCIe frame grabber board (it uses CoaxPress to deal with all that data as quickly as possible), we’ve been concentrating on PCs. These are pretty interesting though, especially if you’re working with cameras that dont’ require anything other than standard interfaces like USB3. That said, the high res/high speed cameras all require more than can be reasonably dealt with by USB. That’s why the ScanStation’s 6.5k camera uses 25Gb Ethernet, and many cameras use CameraLink or Coaxpress. USB is good for lower resolutions though.
The last few years we were authoring Blu-rays we only used x264. Hundreds of commercial titles done with that encoder, and the results far surpassed what you could get from any of the commercial encoders, which were overpriced junk. Some of them were $50,000 and delivered softer, less defined encodes (they were murder on film grain), far worse results than the free x264!
It’s a not-so-secret-secret that a lot of the industry (including labels known for their fussiness about picture quality) uses x264 for Blu-ray now, because it’s just better. Getting compliant streams isn’t hard, you just need the right recipe. As for speed - the commercial encoders were all slow, in my experience. It would take between 6-8 hours to encode a 2-pass feature using the Sonic or Cinemacraft encoders, but 3.5 hours or so to do a 2-pass with x264. That’s all CPU.
Honestly, I’m starting to think that the for the purpose of film scanning, GPU is likely overkill unless you’re looking to do realtime or faster at high resolutions. Libraries like OpenCV are really mature even on CPUs, and are focused on getting every last drop of performance from available hardware so you can do a lot more without having to rely on a GPU (that said, I’m really curious to see what happens to our processing times when we do put a real GPU in there).
1 Like
Yes indeed, but it wasn’t an original bluray-certified encoder. If I remember correctly it doesn’t do interlaced encoding right, or something, so it still does have a limitation (not that interlaced encoding matters for film mastering I suppose, but it does mater for material shot on video).
Well, certification is a scam anyway. That’s why the commercial encoders are so expensive, because of the licensing fees they had to pay. The x264 streams are perfectly valid though. If they weren’t there’s no way they’d pass verification at the replication factory.
x264 has a “fake interlace” flag, which works fine. TBH, we never put any interlaced footage on a blu-ray anyway - It doesn’t really make a lot of sense to. TVs are progressive and features are progressive if they’re going to Blu-ray. If it’s shot in HD, it’s likely progressive (unless it’s early HD when there was more 1080i acquisition going on, but that’s a real edge case). Anything that was interlaced, we deinterlaced before encoding, which is the best way to ensure you know what the end result will look like on every TV. So, kind of a moot issue, really.
Not really, they were guaranteed to work on consumer hardware, x264 couldn’t do it originally it was late to party. You can’t release a bluray and have it be incompatible with some hardware.
Ugh, really? That’s like saying “any 35mm we de-grained before encoding”. It may not be at pleasant, but interlaced material should be left as it is. I have a CRT sitting right here I can watch it on, but if you’ve de-interlaced it that’s destructive and it can’t be reversed. Film is film, video is video, each medium should be respected for what it is.
My point is that certification was a scam. What matters is the end result and if it passed verification, which is an incredibly rigorous process. Also, a professional authoring system like Scenarist will choke on files that aren’t compliant. It simply wont let you import files that aren’t to spec. So you have two built-in levels of verification. The fact is, the BD Consortium badly screwed up the format from the get-go. They bungled licensing costs making huge barriers to entry. They enforced copy protection that pretty much everyone outside of hollywood didn’t care about (and as it turns out, was about as easy as DVD to crack anyway, which they clearly knew would happen). And they made manufacturing a lot harder by requiring replicators to buy entirely new pressing and mastering equipment. All of this was to sell expensive licenses and certifications.
HD-DVD, from a creator’s perspective, was a far superior format because it just extended DVD in all the right ways.
No, it’s very different. Blu-ray is not an archival medium. It’s a presentation format. Mixing interlaced and deinterlaced material on a disc (or different resolutions, or frame rates) causes all kinds of weirdness with certain player/TV combos (like HDMI handshaking that causes the screen to flicker, onscreen frame-rate info to display, audio to pop in a receiver, etc). That interrupts the end user’s experience. If we’re capturing a tape for an archive and it’s interlaced, we capture it interlaced. If we captured a tape for a distributor who wanted a seamless experience on a consumer product, it likely would be deinterlaced because at least that way we could control how it looked rather than leaving it up to the end user’s setup. A Teranex comes in really handy for this kind of thing.
Well you’re not wrong there.
I fail to see how putting something like a music video in interlaced format (encoded to MPEG2 since you confirmed x264 doesn’t handle interlaced) on a bluray “interrupts the end user’s experience”? Deinterlacing native video is destructive, I’m surprised there are a publishers that want to do that I thought that they usually insist on leaving it interlaced.
Did anyone notice that Blackmagic released a new version of the their Cintel film scanner? It’s called the Cintel C-Drive HDR. Main differentiating feature: capstans instead of sprockets. (They still retain the previous scanner now called the Cintel Scanner S-Drive HDR.)
Seems better for archival scanning, but it still has the same fixed camera and sub-par resolution. Why did they bother?
Hi Todd, yes that’s their latest improvement since late last year.
By the way you were correct that it’s back in-development. They have a new lead developer, and a friend of mine I believe has spoken to him.
The LG Archivist is now a much better scanner than the HDS+ for the money. The only exception is if you need to scan 8mm at 6.5K, however Lasergraphics have priced it aggressively. It’s not hard to jerry-rig a similar wetgate feature for the Archivist. The HDS+ can drop frames because of the way it advances the frame (using a laser), the Archivist doesn’t drop frames. HDS+ doesn’t have HDR, and doesn’t have warped film gates. The Archivist stops the scan on a failed splice automatically, tells you to fix it, and resumes scanning from the next frame.
There was going to be a 35mm Archivist as well, but it would have killed the Scanstation so it was dropped. The Archivist is basically made from Scanstation parts as it is - for example it has exactly the same 35mm rollers that are on the Scanstation, even though it doesn’t support 35mm or 28mm! But it’s priced a lot cheaper.
On the Cintel, I believe they will have to rewrite the capture software if they change resolution. They haven’t changed the camera in 4 years, but they do need to put in a better one as everyone knows. If they put in a 6.5K camera it would be fine. HDR is also an issue with it, it needs a brighter light so it can do the HDR pass at 24fps, and at the moment HDR forces a rewind followed by the HDR pass it really needs to either scan HDR in a single pass at 12fps or to do the HDR pass in reverse, again either of those options will require a lot of work with the software though so I wouldn’t hold your breath. HDR is a free feature on the Cintel so I can’t see where Blackmagic has the budget to invest in the software development to get it right. The sliding doors are also stupid, it makes the Cintel take up twice the space that it should occupy, it would be better to have hinged doors.
1 Like