H. G. Dietz
http://aggregate.org/hankd/
Department of Electrical and Computer Engineering
Center for Visualization & Virtual Environments
University of Kentucky, Lexington, KY 40506-0046
Initial release: April 8, 2006
This document should be cited using something like the bibtex entry:
@techreport{rgbnir20060408,
author={Henry Gordon Dietz},
title={RGB+NIR Extraction From A Single RAW Image},
month={Apr},
year={2006},
institution={University of Kentucky},
howpublished={Aggregate.Org online technical report},
URL={http://aggregate.org/DIT/RGBNIR/}
}
This is a neat little trick we did back in 2001 to extract Red (R), Green (G), Blue (B), and Near-Infra-Red (NIR) from a single RAW image captured using an unmodified commodity digital camera. The camera does not even need a filter. Back in 2001, we used a Canon G1. We have since used this technique with other cameras, such as the Sony DSC-F828.
The Canon G1 was one of the first "prosumer" digital cameras, with a variety of features allowing serious control over the image capture... including a RAW sensor data file format. Canon did not see fit to make the RAW format, .crw, an open design; however, a guy named Dave Coffin did a huge service to the community by reverse-engineering a method to decrypt these files. His tool, dcraw, has become the basis for a multitude of applications that operate on RAW sensor data files from an ever-expanding variety of digital cameras. It's a great piece of work. It also is a fairly small C program. Looking at the source for this C program, I found that he had done something a bit strange in his color processing.
The Canon G1 doesn't use the typical Bayer pattern of alternating GB and RG lines for its color filter array; instead it uses a pattern of Cyan (C), Magenta (M), Yellow (Y), and Green (G). Why use CMYG instead of RGB? Well, C is sensitive to B+G, M is sensitive to R+B, and Y is sensitive to R+G. The reason the standard Bayer RGB pattern has two G per 2X2 block is that G is very close to luminance for the human eye, so more G pixels means more accurate luminance. Humans barely notice the color being less accurate (or lower resolution) provided the luminance information is of high quality. So, Canon's CMYG pattern means that 3/4 (all but M) instead of 2/4 of the pixels contribute to the G information, potentially yielding significantly better luminance information. As a side benefit, the CMYG filter array also has a lower average density... letting more light pass to the sensor and thus increasing the effective "film speed" of the camera.
Now, most image display devices use RGB, so it is pretty easy to process RGB color filter array images (although display RGB isn't necessarily the same RGB used in the camera filters and JPEGs actually use an entirely different colorspace). CMYG is less obvious, but not much worse. G is directly sensed. Given G, C-G=B, Y-G=R, and M-B=R or M-R=B. Thus, combining a simple subtraction procedure with interpolation can give us RGB data. The odd thing is that's not what dcraw does!
Dave did something rather clever. Converting color filter RGB into display RGB is often accomplished by solving a system of linear equations to get multiplicative factors by which each input color channel should be weighted to determine the value for each output channel. For RGB, that's three equations in three unknowns. For example, display G might be computed from the input color channel values using a formula like -0.2R + 1.1G + 0.1B; very simple and fairly effective. However, solving input CMYG for output RGB is overspecified: you have four equations in three unknows!
I sent Dave an email about this, asking how he got the weightings. Basically, he solved CMY for RGB, CMG for RGB, CYG for RGB, and MYG for RGB. He then averaged the weightings and used them to derive RGB from CMYG. That trick works pretty well... but any way you look at it, the problem is overspecified. What you really want is a fourth output color....
This is a good point to get back to talking about the Canon G1. Among its many cool features is a tiny little wireless remote control. The remote control needs to be pointed at the camera so that a sensor on the G1 can see the Near Infra-red (NIR) signal from the remote, but you can do any of a variety of operations via the remote control... including taking a photo of yourself. The bad news is that when I did that, I usually would see this little bright spot where I was discretely holding the remote control: the camera's imaging sensor was obviously also sensitive to the NIR from the remote control!
Actually, the Canon G1 is not very sensitive to NIR because it has a permanent filter that cuts NIR light falling on the sensor by something like 7 stops. Without that filter, the G1's sensor probably would be significantly more sensitive to NIR than it is to green light. There are plenty of WWW sites that will tell you how to remove NIR-blocking filters from various digital cameras so that you can improve sensitivity (there even is a company that offers this service and sells modified cameras), but the fact that the G1 still saw its NIR remote control told me that NIR data was still making it into my images even with the NIR-blocking filter in place. Modifying the camera is not necessary, but would only help balance the sensitivity between visible and NIR light by increasing NIR sensitivity.
While we're discussing things that show-up in G1 photos that you couldn't see in real life, it's worth mentioning that the G1 also has some sensitivity in the Near Ultra-Violet (NUV). A lot of the "chromatic abberation" that people complain about for the G1 and other digital cameras is really there because the sensor is picking-up light whose wavelength is out of the visible range for which focal-plane corrections were made when the lens was designed. Because ordinary glass is a pretty good UV filter, most lighting has relatively weak UV components, and image sensors are less sensitive to UV in the first place, NUV typically is not as significant an image component as NIR.
Thus, we actually have two additional color bands beyond RGB that the CMYG-filtered G1 can image. Why not take one or both of them as the fourth color for solving the CMYG equations? Solving for RGB+NIR makes the most sense given the low sensitivity to NUV and the fact that bright NUV lights are considered to be a health hazzard.
Fundamentally, to create these equations, we need to know the relative sensitivity of each of the filter-colored pixels to light in each of the output color bands. I did this for the G1 by crude empirical measurements, including measurements taken using a filter to block visible light (passing only NIR). As many people have noted in discussions of NIR photography using filters that block visible light, typical color filter arrays have little impact on the amount of NIR that reaches the imaging sensor. However, even if sensitivity to NIR was identical for all filter colors, that would only convolve NIR with a visible light spectral profile which shared the same property -- which will be quite rare in ordinary picture-taking situations. I used the G1's remote control as the reference NIR source.
Thus, all I needed to do was to replace the weightings used by dcraw with the weightings I computed... and figure-out how to get four color channels into a standard output file format. This was done by modifying the command line to allow specifying that the output should be either the color RGB extracted using the RGB+NIR formulation, or the monochrome NIR extracted using the same formulation.
Because the G1 image sensor is digitized at just 10-bit resolution, the dynamic range was not really sufficient to cover normal room lighting and a modest NIR light source -- such as the G1's remote control -- reduced further by the NIR-blocking filter. Thus, the visible lighting was made very dim (indirect sunlight) and a color chart and the NIR remote control were photographed using the camera at its highest film speed rating (which explains the graininess of the images). Alternatively, filtering could have been used to reduce visible light sensitivity to be comparable to that for the internally-blocked NIR.
The original results:
The six images in the above are all derived from the same single exposure using a Canon G1 without any modifications, not even a filter. The top row show the full frame, the bottom row are close-ups of the area around the infra-red remote control at the top of the frame. The normal processing was accomplished using Canon's RAW conversion software under windows, but is virtually identical to the output of the unmodified dcraw that was available at the time (newer dcraw versions do somewhat better interpolation). Note the indentation in the bottom edge of the remote control -- there is no such indentation visible to the human eye. The RGB and IR extractions were created using a program based on the same old version of dcraw.
The original coefficients for RGB color conversion from GMCY to RGB were:
{ -2.400719, 3.539540, -2.515721, 3.421035 }
{ 4.013642, -1.710916, 0.690795, 0.417247 }
{ -2.345669, 3.385090, 3.521597, -2.249256 }
The RGB-NIR coefficients for RGB color were:
{ -1.382995, 0.249841, -0.159960, 1.289882 },
{ -0.265864, -1.457404, 0.933098, 0.809020 },
{ -1.423661, 0.167074, 1.381977, -0.097471 }
The RGB-NIR coefficients for NIR color were:
{ 15.097626, 12.247668, -9.044717, -8.382289 }
These coefficients were determined based on approximate empirical measurements, and almost certainly could be improved upon even for the G1. Note also that they reflect the approximate ratio between visible light and NIR brightness used for the test images, not the brightness ratios seen under more normal lighting conditions in which the apparent NIR components are far weaker. Our more recent work uses a Genetic Algorithm (GA) and filtered reference images to determine coefficients that give the best overall match to the filtered images.
This technique actually works for nearly any set of K filter colors and K output color bands. The results are not perfect, because most choices of filters and output bands will result in some spectral ambiguity: different spectral patterns having identical sensor response. In general, the narrower the band being resolved, the more distinctive its sensor response is likely to be; this also implies that having more filters with different response curves works better per color band resolved. In any case, the majority of the error probably comes from the underlying assumption of linearity of the sensor response.
Four-color filter arrays have become less common over the past few years, usually with marketing explanations about RGB filter arrays producing more accurate color. Personally, I have seen no evidence to suggest that the marketing claim is true, and suspect the real reason is the somewhat more straightforward processing.
It is worth noting that Sony NightShot removes the NIR-blocking filter from the optical path, so these cameras have the potential to produce a better sensitivity balance between visible and NIR. A Sony camera with a four-color filter array and RAW sensor data file support can work quite well -- which is why I also have used a Sony DSC-F828. The F828 uses an RGBE (E is Emerald) filter array rather than CMYG, but the choice of filter colors is not critical relative to NIR extraction. However, like most Sony NightShot cameras, the F828 has some exposure range restrictions in NightShot mode that make taking RGB+NIR photos difficult in bright daylight. In appropriate lighting, image quality for the F828 is markedly better than using the Canon G1.
The only thing set in stone is our name.