HYPERSPECTRAL IMAGE RESOLUTION ENHANCEMENT BASED ON SPECTRAL UNMIXING AND INFORMATION FUSION

Hyperspectral imaging sensors exibit high spectral resolution, but normally low spatial resolution. This leads to spectral signatures of pixels originating from different object types. Such pixels are called mixed pixels. Spectral unmixing methods can be employed to estimate the fractions of reflected light from the different objects within the pixel area. However, spectral unmixing does not provide any spatial information about the sources and therefore additional information is needed to precisely locate the sources. In order to restore the spatial information of hyperspectral images we propose a hyperspectral and multispectral image fusion method based on spectral unmixing. The algorithm is tested with HyMAP image data consisting of 125 spectral bands and a simulated multispectral image consisting of 8 bands.


INTRODUCTION
Each hyperspectral pixel consists of hundreds of contiguous and narrow spectral bands covering typically the visible to short infrared wavelength range.The compactness of channels results in the limitation of the pixel size.Modern hyperspectral spaceborne cameras have normally a ground resolution of about 30 m, which results in pixels consisting of signals from more than one material.Such pixels are called mixed pixels composed of pure spectral signatures, also called endmembers.The mixture of signatures prevents direct material recognition and requires further processing in order to separate the mixed signal.This problem is also known as blind source separation and has been extensively investigated by the signal processing and remote sensing community in the last years.Methods for hyperspectral source separation are known as spectral unmixing and recent spectral unmixing methods allow to extract endmembers with high accuracy, however spatially not located.Unlike hyperspectral data, multispectral data have usually high spatial resolution with only several spectral channels in a relatively wide range, and accurate material recognition is hardly to reach.Therefore fusion of hyperspectral and multispectral images complements the advantages of both modalities and allows to enhance the spatial as well as the spectral resolution, which finally leads to a better understanding e.g.within material studies.In order to localize materials within the pixel area and to enhance the spatial resolution of the hyperspectral image data on subpixel level, there exist several fusion and non-fusion-based methods (Mianji et al., 2009).In non-fusion-based methods the spectral resolution is enhanced using only the hyperspectral image data by analyzing its spectral mixture.Here each pixel is divided to a specified number of subpixels, wich are rearranged according to the endmembers and its fractional abundance using e.g.: spatial regularization tehniques (Villa et al., 2010), total variation minimization (Guo et al., 2009), or Hopfield neural network optimization (Nguyen et al., 2006).The main advantage of these methods is that they do not require any other information as contained in the hyperspectral image itself.These methods can be used for improved data visualization and understanding of hyperspectral images, but is not reliable for detailed image studies or provision of exact spatially located information like for object shape reconstruction.
Fusion-based methods require supplementary information from other sensors, which allow a more accurate restoration of real objects in the image.Eismann and Hardie (Eismann and Hardie, 2005) propose a maximum a posteriori estimation method used to enhance the resolution of hyperspectral images utilizing panchromatic or multispectral images, but with the restriction that the spatial enhancement is limited to several principal components of the hyperspectral image.Multisensor, multiresolution techniques use the spectral unmixing results for reconstructing high resolution hyperspectral image pixel by pixel.Firstly the high resolution image is classified follwed by a window-based spectral unmixing of the hyperspectral image data.Secondly the mean class signal of the unmixing result is assigned to the corresponding pixels of the classified high resolution pixels (Zhukov et al., 1999).
The objective of this paper is to describe a novel method of information fusion of hyper-and multispectral images for spatial resolution enhancement based on multi-modal data fusion and unmixing techniques.This is addressed from the perspective of applicability of the method, quality assessment by pixel based cross correlation in spectral and spatial domain and easy to use of the method.

Spectral unmixing
Most of the pixels in hyperspectral satellite images are mixed i.e. they consist of signals from more than one source.The linear mixing model is the most commonly used to describe the physics foundations of complex hyperspectral mixing phenomena.This model assumes that every pixel xi,j is a linear combination of endmembers organized row-wise in a matrix M , factorized by the fractional abundances of each endmember represented by a vector a, with a second vector u containing the error for each band: xi,j = M ai,j + ui,j Spectral unmixing algorithms aim at recovering endmembers by inverting equation 1 (Keshava, 2003).In our experiments we use an unsupervised unmixing algorithm in two steps.We estimate abundancies with a fully constrained least squared unmixing algorithm (FCLS) (Heinz and Chein-I-Chang, 2001): FCLS detects end-members and also quantifies the abundance fraction for each of them.As unsupervised FCLS requires the mixing matrix M to be known, to estimate the endmembers we perform in a preceding step a vertex component analysis (VCA).The VCA is a fast unsupervised unmixing algorithm which assumes that the endmembers are the vertices of a simplex created on a hyperspectral data cloud (Nascimento and Dias, 2005).

Data resampling
To resample the spectral responses from the hyperspectral to the multispectral sensor we calculate the spectral integration as: where R(wi) is the response of the i-th channel, S is the spectral sensitivity of the channel, R(w) is the hyperspectral sensor's response for the wawelength w, which is considered continuous.We assume the spectral response function to be Gaussian distributed for the multispectral sensor, and we model it as (van der Meer et al., 2001): where w is the continuous spectrum, while µ and σ are respectively the central wavelength and the standard deviation for the i-th channel of the multispectral sensor.

Proposed Methodology
Hyperspectral unmixing provides information about the materials contained within one pixel, but does not help in locating any of these materials at subpixel level.We propose a simple and fast method to fuse a multispectral and a hyperspectral image, in order to combine the high spectral resolution of the hyperspectral sensor with the high spatial resolution of the multispectral sensor.
Suppose to have a hyperspectral scene h with l bands and a multispectral image m with n bands, acquired over the same area on ground and thoroughly coregistered, with n < l.In a first step we estimate the matrix M h related to h, composed of k endmembers signatures, using the VCA algorithm.Subsequently, we resample M h to the multispectral model using equation 2, obtaining a matrix Mm.Finally, the endmembers contained in Mm are used as input to compute the abundances of each endmember using FCLS.The result is an abundance vector am of length k, quantifying the fraction of each endmember within the multispectral pixel.Similarly to 1, the model of the multispectral pixel can be written as: where Mm in a k × n resampled endmembers matrix, while the hyperspectral pixel can be represented as: where Mm in a k × n resampled endmembers matrix, while the hyperspectral pixel can be represented as: To fuse the information from both images we apply once again the linear mixing model.The matrix M h is multiplied by the abundance matrix from the multispectral image, and as a result we obtain a new pixel am with l bands.The fused image has the same spatial resolution as the multispectral scene.A sketch of the algorithm is illustrated in fig. 1.

RESULTS AND DISCUSSION
We tested the proposed algorithm using a 256 × 256 image acquired by the hyperspectral sensor HyMAP with 125 spectral channels, spanning the interval from 0.4 to 2.5 µm, and a spatial resolution of 4 m (fig.To obtain A we applied a Gaussian lowpass filter and subseqently subsampled the HyMAP scene by a factor of 4, while B was obtained by resampling the hyperspectral image to the 8 spectral channels of the multispectral, ranging from 0.4 to 1.04 µm.
Consequently we applied the algorithm described in section 2.3 to fuse the images.The area analyzed contains 7 types of vegetation, soil and water.After applying VCA to the image A we obtain 7 endmembers, which are then resampled to the channels of the image B. For each resampled endmember we then computed its fractional abundance in B.
The result of the fusion algorithm is a 256 × 256 hyperspectral image with 125 spectral channels (fig.3. d, f).
To asses the results we compute the normalized cross correlation between each pixel in the original hyperspectral image and in the fused image as: where m is an original pixel, k is a restored pixel, n is the number of measurements in m and k, m and k are the mean values of m and k, and σm, σ k their standard deviation.The normalized cross correlation was calculated the in spectral domain, i.e. the spectrum of each pixel in the restored image was compared to the spectrum of each pixel in the original image.Figure 4. a).shows the normalized cross correlation for all the pixels, which assumes a satisfactory average value of 0.96.We can observe a very good correlation for regions consisting of vegetation (fig. 4. [1, 2, 4, 6, 7, 10, 11, 12]).The restoration of spectra belonging to soil and roads (fig.4.9.) was not that exact and some features, especially for bands >60, were not correctly restored.In order to measure the similarity between the images for each band, we computed a normalized cross correlation in the spatial domain.A single band from the restored image was compared to the same band from the original image using equation 7. The cross correlation in the spatial domain has a mean value of 0.89 in the spatial domain, where 0 and 1 represent respectively uncorrelated measurements and total correlation.The values are plotted in figure 2. The cross correlation is above 0.9 for bands from 0 to 47, which correspond to a spectral range from 0.4 to 1.1µm and cover the whole spectral range of the multispectral sensor.The correlation for bands 48 -124 is over 0.85, even if these bands are not in the spectral range of the multispectral sensor, with the exception of several noisy bands.The plots from 1 to 9 of figure 4. show the original and restored spectra for selected pixels.From this spectra we can observe that, even when the cross correlation is lower, the main absorption features are preserved and remain clearly distinguishable.

CONCLUSION
In this work a novel method for image fusion of hyperspectral and multispectral data based on spectral unmixing has been proposed.An image exhibiting high spatial (4m) and spectral (125 channels) resolution has been restored from a 16m resolution hyperspectral image with 125 channels and a 4m resolution multispectral image with 8 channels.High values of cross correlation between the images show that both the spectral and the spatial information are well preserved.The proposed algorithm is easy to use and could be employed to increase the spectral information of a multispectral scene, given a thoroughly co-registered hyperspectral image.In the future the proposed algorithm should be tested with real images acquired by different sensors, and an optimal unmixing algorithm should be employed to enhance the endmember restoration in the multispectral image.

Figure 1 :
Figure 1: Schema of the proposed algorithm.In a first step, from hyperspectral image endmembers are selected and unmixed, those retrieved endmembers are resampled to the channels model.The multispectral image is unmixed using the resampled endmembers.Finally a new high resolution hyperspectral image is retrieved using the result of multispectral unmixing and hyperspectral unmixing.

Figure 2 :
Figure 2: Plot of the normalized cross correlation computed for each band.Value 1 for normalized cross correlation means that signals are ideally correlated 0 when there is no correlation between the signals.
3. c).From the HyMAP image we generate a subsampled hyperspectral image A (fig. 3. a) and a simulated multispectral image B (fig. 3. b).

Figure 3 :
Figure 3: Input images used for experiment (a,b,c,e) and result images(d,f).

Figure 4 :
Figure 4: Image above shows result of normalized cross correlation in spectral domain.For each template and restored pixel normalized cross correlation is calculated across all bands.From pixels numbered (1-12) spectral curves are plotted.Reference image (blue lines), restored image (green lines).