Estimation of the spectral sensitivity functions of unmodified and modified commercial off-the-shelf digital cameras to enable their use as a multispectral imaging system for UAVs

Commercial off-the-shelf (COTS) digital cameras on-board unmanned aerial vehicles (UAVs) have the potential to be used as multispectral imaging systems; however, their spectral sensitivity is usually unknown and needs to be either measured or estimated. This paper details a step by step methodology for identifying the spectral sensitivity of modified (to be response to near infra-red wavelengths) and un-modified COTS digital cameras, showing the results of its application for three different models of camera. Six digital still cameras, which are being used as imaging systems on-board different UAVs, were selected to have their spectral sensitivities measured by a monochromator. Each camera was exposed to monochromatic light ranging from 370 nm to 1100 nm in 10 nm steps, with images of each step recorded in RAW format. The RAW images were converted linearly into TIFF images using DCRaw, an open-source program, before being batch processed through ImageJ (also open-source), which calculated the mean and standard deviation values from each of the red-green-blue (RGB) channels over a fixed central region within each image. These mean values were then related to the relative spectral radiance from the monochromator and its integrating sphere, in order to obtain the relative spectral response (RSR) for each of the cameras colour channels. It was found that different un-modified camera models present very different RSR in some channels, and one of the modified cameras showed a response that was unexpected. This highlights the need to determine the RSR of a camera before using it for any quantitative studies.


INTRODUCTION
A new era of fine-scale remote sensing has emerged with the arrival of unmanned aerial vehicles (UAVs), which have the advantage of being lightweight, low-cost and operationally easy to deploy as safe remote sensing acquisition platforms (Berni et al., 2009;Zhang & Kovacs, 2012).
Commercial off-the-shelf (COTS) digital cameras are typically used as imaging systems on-board UAVs due to their low-cost and researchers often want to make further use of them as multispectral imaging systems due to their ability to detect near infra-red light (Darrodi et al., 2015;Suzuki et al., 2009;Lebourgeois et al., 2008), once modified by removing the hot mirror filter (Rabatel et al., 2014;Verhoeven et al., 2009).
However, this is hampered by a lack of knowledge of the spectral sensitivity of the camera, as COTS camera manufacturers typically do not publish this information.Also when cameras have been modified (with the addition of a long pass or notch filter) by an external party, the wavelengths that could be transmitted through that filter are also not always known (as is the case with two of the modified cameras in this study).Therefore, users need to either measure or estimate their cameras sensitivity (Darrodi et al., 2015;Jiang et al., 2013).
A standard and accurate methodology for measuring the sensor sensitivities is to take photographs of monochromatic light produced by a monochromator (Darrodi et al,. 2015), where the digital numbers (DN) recorded on the images are expected to have a linear response to the radiation output of the monochromator (Verhoeven et al., 2009).
Even though previous studies have measured the spectral function of different digital still cameras using monochromators (Darrodi et al., 2015;Jiang et al., 2013;Suzuki et al., 2009;Verhoeven et al., 2009), there is a lack of a detailed methodology in how to do so, which has motivated us to present an open-source workflow to process the images after image acquisition.In this paper, we aim to identify and to present a step-by-step methodology for identifying the spectral sensitivity of modified and un-modified low-cost digital cameras using open source software, in order for them to be used as multispectral cameras for UAV systems and compare the results for a range of COTS and modified cameras.

Camera types and settings
Six physical cameras (two of each model), which have been used as imaging systems on-board different UAVs, were selected for spectral sensitivity estimation.The cameras were either un-modified and therefore only sensitive to visible light (VIS), or modified (MOD) to also be sensitive to near infra-red (NIR) wavelengths.The modified cameras have had their internal NIR filter removed, thereby turning them into a full spectrum (FS) camera, which was then replaced with either an internal or external long pass or notch filter to alter the cameras spectral sensitivities.
A preliminary image acquisition of monochromatic light was carried out in order to determine the optimal settings for each camera, as presented in Table 1.For each of the cameras, the ISO was set to 100 in order to gain a high signal to noise ratio, and the aperture and shutter speed were altered to ensure that the image was not overexposed.The optimal exposure settings were chosen when the monochromatic light with the highest signal intensity (~540 nm for VIS and between 600~700 nm for NIR) came close too (but not actually) saturating the images digital numbers (DN).
The defined settings (Table 1) were then kept unchanged throughout the acquisition period and the images were recorded in both JPG and RAW formats.RAW format is necessary as it assumes that the cameras sensor detects and stores radiance without applying any processing or compression to it, i.e. the original signal reaching the sensor is not modified (Verhoeven, 2010).Both the Panasonic and Sony cameras were able to record JPG and RAW files natively; however, the Canon camera could not.Therefore modified firmware (CHDK v1.2; CHDK, 2015) was used to allow JPG and RAW (Adobe DNG v1.3 format) images to be produced, as well as to allow the shutter speed to be set at a constant value.

Monochromatic light image acquisition
The spectral sensitivities of each camera were measured by recording their response to monochromatic light produced by a double monochromator (OL 750-M-D Double Grating Monochromator (Additive), Optronic Laboratories, Inc., Orlando, Florida, USA) and reflected by an integrating sphere attached at the monochromators exit slit.The light beam is reflected by the spheres interior surface from where images were acquired through a detection port in the integrating sphere (Figure 1).
Figure 1: The monochromator and camera setup and three example images of the inside of the integrating sphere at different wavelengths.
The monochromator was located in a temperature-controlled dark room and the lens of each camera was positioned touching the integrating spheres detection port.The sphere and camera were also covered by a low reflectance black cloth in order to avoid any external light contaminating the result.
Each camera was exposed to monochromatic light ranging from 370 nm to 1100 nm in 10 nm steps, with two images being captured at each step (most of the cameras were tested across only a portion of this complete range).Therefore, the selected nominal half bandwidth (HBW) was 10 nm (using 1200 g/mm gratings), which is achieved by combining 5 mm slits at the entrance, middle and exit ports (Optronic, 2002).Due to the range of wavelengths being sampled, it was also necessary to vary the lamp voltage and the type of internal filter used.The settings selected on the monochromator and the wavelength interval from which images were acquired to measure the cameras spectral response, are presented in Table 2. Dark images were also acquired for the Panasonic cameras in order to correct for effects of noise create by the dark current signal (Verhoeven et al., 2009).Photographs were taken inside the dark room with the camera lens covered by the black cloth and the room light switched off.

Imaging processing
Each camera recorded its RAW image file using a different format dependant on the camera manufacturer, so the open source software DCRaw (v9.25;DCraw, 2015a) was used to convert the RAW files into TIFF images that maintained a linear relationship with the original RAW data (Gehrke & Greiwe, 2014;Verhoeven et al., 2009).During this step, the Panasonic images were also corrected for the effects of the dark current signal as DCRaw can utilise the dark image taken at the same time as the images of monochromatic light, generating dark-corrected linear TIFF images.The script command used with explanation is present in Appendix 1.
The TIFF image DNs were averaged within a 9 x 9 pixel area in the centre of each image, in order to avoid issues related with the variance of the radiance within the integrating sphere and any differences in the individual pixel response across each camera's image sensor (Darrodi et al., 2015).
To obtain the mean and standard deviation values from the 9x9 pixel window of each image, a macro was created using the open source software ImageJ (v1.49k,Fiji distribution; ImageJ, 2015).Figure 2 details the workflow used to capture these values and a detailed step-by-step guide is presented in Appendix 2 and the ImageJ macro script used is presented in Appendix 3.
RAW and JPG images converted to TIFF using DCRAW TIFF images split into three channels using an ImageJ macro TIFF images cropped to a 9x9 pixel window (at the centre of each image) using an ImageJ macro TIFF images analysed using ImageJ measure function to obtain mean pixel value and standard deviation Figure 2: Post processing steps to convert RAW and JPG images and to obtain mean and standard deviation digital number values for further analysis.

Retrieving RSR from the cameras
The light intensity from the monochromator and its integrating sphere were independently measured using a reference silicon photodiode (OL DH-300C S/N: 12101253, Optronic Laboratories, Inc., Orlando, Florida, USA) in order to identify the relative spectral radiance at each measured wavelength (Figure 3).The settings used mirrored those used at the image acquisition phase (Table 2).
Two lamps voltages were used (Figure 3B), with the 17 V lamp was used in combination with the 345 nm and 599 nm filter, however for the lower wavelengths (<500 nm) the lamp voltage was increased to 19 V in order to generate monochromatic light with an intensity that would be strong enough to be detected by the cameras sensors on wavelengths ~400 nm.This is because the CCD (charge coupled device) or CMOS (complimentary metal oxide semiconductor) sensors which are usually fitted in COTS cameras are likely to sense wavelengths lower than 400 nm (Nijland et al., 2014).
The RSR of the RGB channels for a given wavelength λ were calculated as per Eq. 1 (Suzuki et al,. 2009) and a linear relationship between the input radiance and the output signal is assumed (Darrodi et al. 2015).
, () =  () () (1) Where r, g, and b = the mean spectral response from each camera channel (red, green and blue channels, respectively) given in DN values and calculated from the 9x9 window (output signal); I = the light intensity (monochromator and sphere) given in relative spectral radiance units (input radiance); η = the normalized coefficient.
Instead of using absolute physical units, the η coefficient is applied to generate a relative spectral response, as it results in lower calibration uncertainties (Darrodi et al., 2015).For each camera, the mean spectral response over all of the wavelengths were normalized (η) by the maximum signal detected among the RGB channels, yielding dimensionless units with a peak equal to 1.
Figure 3: The responsivity of the silicon photodiode (A); The relative spectral radiance of the monochromator at given voltages using specific filters (B).

RESULTS
All of the un-modified cameras showed similar ranges within the visual spectrum as would be expected (400~700 nm; Figure 4), with almost all of the RGB channels covering this range but with varying degrees of sensitivity.The green channel for each camera showed the highest peak response due likely to all cameras being using a Bayer colour filter array, which has twice as many green filters compared to red or blue, as this more closely resembles how the human eye sees the world
However a closer analysis on both shape and intensity of the RSR curves reveals differences among the camera models, and most noticeable are the differences of the Sony cameras compared to the other two models (Figure 4).
The red channel of the Sony camera (Figure 4C) has a much lower peak response (0.58) compared to Panasonic (Figure 4A) and Canon (Figure 4B) red channels, which both peak with a response of ~0.8.The green channel curve of the Sony also shows a narrower shape than both Panasonic and Canon.For the blue channel, the Canon and Sony tend to have very low sensitivity after around ~560 nm; however the Panasonic keeps on sensing until around 680 nm.The FS Canon camera (Figure 5) shows the full spectral sensitivity of the Canon sensor, with responses from 370~1020 nm for all of the channels, revealing the expected COTS cameras sensitivity to NIR wavelengths when the internal NIR blocking filter is removed (Verhoeven et al., 2009).The blue channel has a very low response from 560~770 nm with a peak at 820 nm (well within the NIR band), which is accordance with what could be expected from a Bayer filter (Nijland et al., 2014).With a 585 nm long pass filter attached to the FS Canon camera (Figure 6A), the NIR peak in the blue channel could potentially be exploited as the blue channel is now the only channel that is predominately sensitive to NIR wavelengths and is beyond the red edge feature of 700~720 nm (Hunt et al., 2010).This same characteristic is not observed with the blue channel of the modified Panasonic (Figure 6B), as its sensitivity increases rapidly after 670 nm, detecting some wavelengths within the red region of the spectrum and across the red-edge feature.It is noticeable for both cameras (Figure 6) that their blue channels have a much lower intensity than the red channels.
Both modified Panasonic and modified Canon cameras have the red channel as the most sensitive to NIR light, however the Panasonic peaks at 710 nm, within the red-edge feature, meanwhile the Canon peaks at 620 nm, within the red region of the spectrum (Figure 6).The green channels have an intermediate sensitivity between blue and red channels (Figure 6).The red and green channels of the modified Sony camera (Figure 7) have well characterized curves peaking in the green and red wavelengths respectively, that are narrower and of a more equal intensity compared to the same channels in the unmodified Sony camera (Figure 4C) .The blue channel however has a much lower sensitivity than the green and red, with two main peaks at 400 nm and 830 nm, meaning that this channel may not be suitable to be used as the source of a NIR signal.
Figure 7: Spectral response for the modified Sony camera, normailised to the peak of the red channel.

DISCUSSION AND CONCLUSIONS
The image acquisition method employed is a relatively standard technique, variations of which have been employed by numerous studies (Darrodi et al., 2015;Jiang et al., 2013;Suzuki et al., 2009;Verhoeven et al., 2009) and is known to give accurate results but it does require expensive and specialised laboratory facilities and equipment (Bongiorno et al., 2013;Darrodi et al., 2015).Other methods could be employed instead, such as those indicated by Bongiorno et al. (2013), who themselves used a linear variable edge filter to characterise the spectral response of several COTS cameras; However they may not be able to cover the entire range of wavelengths sampled in this study.
In the image processing stage, the use of DCraw (or a distribution/modification of) as a tool to convert RAW images into linear TIFF files is also widely acknowledged in the literature (Lebourgeois et al., 2008;Rabatel et al., 2014;Verhoeven et al., 2009), but often we see the further processing of these converted images using proprietary software such as ENVI (Hunt et al., 2010) or MATLAB (Verhoeven et al., 2009).
Using an open source program such as ImageJ to process the images means that this technique can be used by anyone, and the macro and batch processing options available meant that processing a large volume of images (over 200 for FS Canon camera alone) is both quick and simple.
The data presented here has not gone through any corrections to account for extra noise that may be present, as some of the cameras did not have dark images taken.So the results are representing the response of the sensor to the photons hitting it, plus any dark current signal and bias signal (Verhoeven et al., 2009).The bias signal is likely to be small but the dark current signal could be quite large depending on the temperature at the time of taking the image and the ISO and exposure settings used (Verhoeven et al.,2009).
This unwanted noise reduces the signal to noise ratio (SNR), and can be corrected for by taking dark images at the same time as momochromatic light image acquisition and using DCraw to subtract a dark frame as indicated in Appendix 1.
Other points of possible error that would need to be corrected for are the identification and interpolation of bad pixels within the image (pixels that fail to sense light levels), which can also be corrected though the use of DCraw, and vigenetting.
Vignetting is where the brightness of an image reduces away from the centre of the image and can be caused due to physical effects of the cameras lens as well as the angle of the light source in relation to the lens (Lebourgeois et al., 2008).The reasoning behind having a small and central 9 x 9 pixel sampling window in the image processing phase was partly due to this phenomenon and for imagery that is to be used for remote sensing this would need to be corrected by following a method similar to that of Lebourgeois et al. (2008).
A final source of error that was overlooked at the time of image acquisition is that of reflections off of the camera lenses themselves (and any external filters), back into the integrating sphere (as the lens was positioned so close to the detection port of the integrating sphere).The properties of any anti-reflective coating of the camera lenses or filters used is not known, however it is likely that its effect would be small in the visual region of the spectrum and perhaps more pronounced towards the NIR region, as the cameras were principally designed to detect visual wavelengths of light.
With these potential sources of error acknowledged, the results of the experiment can be investigated and they reveal that the RSR differs between the un-modified camera models and therefore we cannot assume that all COTS digital cameras have a standard Bayer array response or internal NIR filter response, which echoes that of other studies (Darrodi et al., 2015;Jiang et al.;2013).Once the spectral sensitivity of a camera is known, judgments can be made on what the camera could be used for (e.g.vegetation monitoring), how it can be used (e.g.singularly or in combination with another camera) and what filters could be applied in order to capture desired wavelengths of light.
For instance monitoring vegetation with COTS digital cameras usually requires the combination of visible and NIR wavelengths in order to exploit the well-known spectral characteristics of healthy plant leaves, namely the low reflectance of visible wavelengths due to chlorophyll absorption and high reflectance in the NIR wavelengths due to plant cell structure.This behaviour results in a spectral contrast which is the basis for many vegetation indices (VI), such as the Normalized Difference Vegetation Index (NDVI) (Rouse et al., 1973).
In relation to this Hunt et al. (2010) found that certain colour channels (blue and green) of some COTS digital cameras are not very sensitive to NIR wavelengths, allowing a red light blocking filter to be used to create a single camera multispectral sensor that outputs blue, green and NIR bands and could be used to estimate the Green Normalised Difference Vegetation Index (GNDVI) for agricultural crop monitoring purposes.
In contrast to Hunt et al. (2010), a study by Rabatel et al. (2014) used a single COTS camera with all channels sensitive to NIR (as is the case with the cameras of this study) and investigated the use this camera as a multispectral sensor to estimate NDVI.A blue light blocking filter was used to allow NIR wavelengths to be captured in the blue channel and red+NIR wavelengths in the red channel.It utilised an orthogonal projection method to simulate the separation of the red and NIR wavelengths to allow NDVI estimates to be made of agricultural crops.
Alternatively two cameras could be combined, one un-modified and the other modified to capture NIR, so that separate red and NIR bands can be acquired and registered together to allow NDVI estimation (Dare, 2008).This approach can have issues with the geometric alignment of the bands as they come from two separate instruments (Dare, 2008), which of course would not be an issue with a single camera approach (Rabatel et al,. 2014), however the dual camera method has other advantages such as more control of the shape of the NIR band, reduced concerns about SNR and the fact that standard RGB colour images can be produced at the same time and (Rabatel et al., 2014).
From the results displayed we can say that our three models of camera are sensitive to NIR wavelengths of light, and this confirms the common knowledge that COTS digital cameras in general are responsive to NIR and could potentially be used as a remote sensing tool (Lebourgeois et al., 2008;Rabatel et al., 2014;Suzuki et al., 2009;Verhoeven et al., 2009).However, the comparison of the spectral responses shows that there might be some implications for combining different cameras or combining channels from the same camera.
For the Panasonic cameras in this study, the dual camera method could be applied as the red channel from the unmodified Panasonic (Figure 4A) could be used as the red band and the red channel from modified Panasonic as the NIR band (Figure 6B), as it has the highest SNR.However there is some overlap between these bands, in the region of 650-690 nm, which might cause some band correlation (Nijland et al., 2014).Also the red channel from the modified Panasonic peaks within the red-edge feature (710 nm) and so may not show as strong a NIR response compared to longer wavelengths, as green leaves present their maximum reflectance after 740 nm (Brandelero et al., 2012).
Likewise for the Canon cameras, the dual camera method could also be applied as the modified camera with a 585 nm filter shows a peak NIR response at ~820 nm (Figure 6A), which could prove more useful than that of the Panasonic camera if used for vegetation monitoring; however it has a considerably reduced sensitivity (~20% compared to the red channel of the same camera) which could lead to issues with SNR.
Interestingly the single camera method employed by Rabatel et al. (2014) could also be attempted with this modified camera, and in conjunction with a dual camera setup, allowing comparisons between the two methods to be made.
The modified Sony camera has the most interesting spectral response due to the internal notch filter employed, that appears to block wavelengths between 410~510 nm and 690~790 nm.It was expected that this camera would be responsive to red and green wavelengths (in their respective channels) with NIR wavelengths only available in the blue channel.However the blue channel also has a peak response at 400 nm, meaning that it does not have a predominately NIR response and so may not be effective if used with a dual camera or single camera setup.An alternative method will likely have to be devised to make use of this cameras spectral sensitivity.
In conclusion, although this paper made use of expensive and sophisticated laboratory equipment in order to capture spectral images of known spectral radiances, it does also demonstrate a quick and cheap method of processing images using open source software in order to identify a cameras spectral sensitivity.Our results showed that different COTS cameras might have very different sensitivities, which highlights the importance of determining the spectral response function if using COTS cameras for quantitative applications.

Figure 4 :
Figure 4: Spectral responses from the un-modified Panasonic (A), Canon (B) and Sony (C) cameras, normailised to the peak value of each cameras green channel.

Figure 5 :
Figure 5: Spectral response for the modified to full spectrum Canon camera normalised to the peak of the green channel

Figure 6 :
Figure 6: Spectral response for the modified to full spectrum Canon with with a 585 nm long pass filter (A) and for the modified Panansonic camera (B), both normailised to the peak of the red channel.

Table 2 :
Monochromator lamp voltage and internal filter settings used for specific wavelengths being sampled.