REAL TIME DATA PROCESSING FOR OPTICAL REMOTE SENSING PAYLOADS

The application of operational systems for remote sensing requires new approaches for data processing. It has to be the goal to derive user relevant information close the sensor itself and to downlink this information to a ground station or to provide them as input to an actuator of the space-borne platform. A complete automation of data processing is an essential first step for a thematic onboard data processing. In a second step, an appropriate onboard computer system has to be de-signed being able to fulfill the requirements. In this paper, standard data processing steps will be introduced correcting systematic errors during image capturing. A new hardware operating system, which is the interface between FPGA hardware and data processing algorithms, gives the opportunity to implement complex data processing modules in an effective way. As an example the derivation the camera’s orientation based on data of an optical payload is described in detail. The thereby derived absolute or relative orientation is essential for high level data products. This will be illustrated by means of an onboard image matcher


INTRODUCTION
For a large number of possible applications the current capabilities of remote sensing is insufficient: The number of deployed sensors is low, most of them have limited geometrical resolutions, and revisiting periods are long.Moreover, the transfer of new technologies into operational systems is a very resource-consuming process leading to high costs.But the number and the resolution of commercial remote sensing satellite is constantly growing.Programs like GMES are expected to deliver data and information 24 hours / 7 days with a high quality, reliability, availability and integrity.For such systems, no manual interaction is acceptable due to customer and user requirements with respect to throughput, quality assurance, costs, and reproducibility.All processes from commanding, data capturing, data transmission, and data processing have to be automated to the greatest possible extent.Automatic processing on ground is characterized by a very high flexibility with respect to algorithms, software, and hardware technologies.The major weaknesses of ground processing are a) an enormous amount of data to be downlinked and b) the disability for fast, autonomous reactions on situational changes.Whereas the first issue and its drawback are obvious, the second one shall be illustrated by two examples: • An area to be mapped is covered by clouds.An onboard, real time analysis of the situation will allow the selection of an alternative target which can be imaged by a dedicated payload pointing maneuver.

•
Image blur due to de-focusing, jitter or nonadjusted integration times can be detected in real time.If applicable, counteractions can be initiated immediately.Both requirements (reducing downlink data rate and reaction times) give rise to demands for onboard data processing.Relevant information shall be derived from the payload data as close as possible to the sensor itself.This information can be transmitted or can be provided to actuators onboard the satellite.To reach this goal, new concepts for onboard hardware and software have to be developed.Ideally, concepts for onboard hardware should combine a high degree of robustness, high performance, and flexibility.These requirements lead to inherent conflicts.As a result, current onboard computers have a much less performance than mass market home computers.It is a challenge, to develop computer architectures and technologies providing features like onboard re-configuration and limited but sufficient redundancy.As one example a hardware operating system is introduced in Section 2. It was developed in the DLR and allows an efficient implementation of algorithms for onboard hardware.Subsequently, two pre-processing steps are outlined in Section 3. Firstly, the correction of systematic errors as a standard step which should be executed on almost each remote sensing system onboard and real time.More in detail the optical navigation for line sensors is explained, meaning the derivation of orientation information based on data of imaging systems.This processing step is essential for a huge number of applications.One of them is image matching, which relies on precise image co-registration and is a prerequisite for 3D modeling.Section 4 is dedicated to this topic.

HARDWARE DESIGN METHODOLOGY
Real time data processing, high data rates, strict energy and envelop constraints found in space applications usually require custom specified hardware.FPGAs are an ideal choice.On the one hand, FPGAs allow high data processing rates by parallelization.On the other hand great flexibility is guaranteed by reconfiguration.One major cost factor of the integration of customized hardware arises from the complex design process.Investigations on how to optimize the design process have been performed.The aim was to simplify the development process of FPGA designs.This is realized with the concept of an operating system (Krutz, 2006).The re-use of operating system modules reduces the time of development and also increases the reliability.Additionally, shifting the development platform has no influence on application specific modules.In order to apply the concept of an operating system special constraints have to be fulfilled by the hardware description language, which is not supported by VHDL.For that reason a pre-compiler has been developed.The precompiler connects the application program with the operating system modules.The result is a synthesizable VHDL file that can be used as input to standard toolchains.Due to the standard interfaces of the hardware operating system, the development of the communication between a host and the FPGA is simplified, which reduces the development time.This design methodology has been applied to many projects in which its advantages were demonstrated.Furthermore, processing modules, independent of any projects, have been developed, i.e. different algorithms for data compression, decompression, image correction and matching exist in hardware.

Error correction
Using CCD or CMOS detector array, several systematical and statistical errors in an image can be occur.The static components can be partially corrected.Statistical errors must be measured and validated.Typical systematic corrections are related to the dark signal and pixel response nonuniformity (DSNU, PRNU).In this context error or system analyses provides a characterization of the imaging system.Such performance parameters are e.g. the point spread function (PSF) and the signal to noise ratio (SNR), but also the maximum possible dynamics, which is characterized by the maximum number of stored electrons.Typical instruments used for the calibration and characterization the systems are, for example, the integrating sphere and the modulation transfer function (MTF) test bench.The integrating sphere is a spatially homogeneous radiance source.The MTF test bench provides spatial structures (e.g. a point source) from the infinity using collimator optics.With the integration sphere the determination of the PRNU is possible.The above sizes can also be used for a generalized system description (e.g.NIIRS).
The correction of systematic errors has an huge importance for any subsequent processing steps -onboard or on ground.Even if no thematic processing is performed, non-corrected image data will influence data product quality.As an example, if image data is compressed with lossy methods (e.g.JPEG) artifacts will occur which cannot be removed afterwards.For thematic onboard processing real time corrections are essential, otherwise algorithms like classification, image matching will generate sub-optimal results.

Exterior Orientation
Another important part of pre-processing is the geometric correction of images.Especially when operating line cameras, which allow a very economic acquisition of high resolution and wide swath images of the Earth.Beside many other advantages they have the disadvantage that non-uniform motion of the camera can negatively affect the image quality.Due to this, a key requirement for line cameras is that fast orientation changes have to be avoided or measured with a very high rate and precision.Camera stabilization is not always possible and can often only be achieved with very high effort.In such cases it is essential to measure the camera's orientation accurately with a high rate to ensure that the resulting imagery products can be geometrically correctedin real time or in a later processing step.Adequate high-end measurement systems are large and expensive and their angular and temporal resolution can be still too low for many possible applications.Due to these reasons two different approaches have been developed that allow the optical measurement of fast orientation changes for line cameras.In addition to this it is shown how the determined orientation changes can be used to derive the precise absolute orientation of a non-stabilized airborne line camera.Figure 2 shows possible options to combine these approaches.A more detailed description can be found in (Wohlfeil, 2012).

Measurement of fast orientation changes with auxiliary frame sensors
One approach to determine fast orientation changes is based on small auxiliary frame image sensors, mounted on the focal plane of the main camera or behind separate optics (Upper block in Figure 3).These sensors are exposed at a similar rate as the line sensors, which can be several hundred or thou- sands of frames per second.The shift of the frame images' contents (optical flow) provides information about the motion of the imaging system.These shifts have to be determined in real time because it is not reasonable to store or transmit the auxiliary images.Janschek et al. (2001) use special optical correlator hardware to determine these image shifts in real time.Wohlfeil & Börner (2010) presented an approach that allows performing this on a standard CPU (also in real time).It makes use of the finding of Shi, Tomasi, and Kanade (Tomasi & Kanade 1991) (Shi & Tomasi 1994) to select and track small, appropriate image regions (features) with very high computational efficiency and sub-pixel accuracy (Lower Block in Figure 3).Incorrectly and inaccurately tracked features are identified by means of basic statistics.The drift of the features can be reduced drastically by tracking them not only sequentially through all images but also in intervals of tens of images.In a second step, the orientation changes of the whole remote sensing system are derived from the image shifts.With the given trajectory of the camera, the approximate terrain height, and the interior orientation of the auxiliary cameras the unknown change in orientation between the capture of two frame images is determined by the remaining image shifts.Assuming that the auxiliary frame sensors are statically connected to the other sensors of the system, orientation changes of the whole remote sensing system can be measured with a high rate and high precision.

Measurement of fast orientation changes of multispectral line sensors
An alternative approach to measure fast orientation changes uses the optical information from the main line sensors without requiring any additional hardware or bandwidth.The approach takes advantage of the typical geometry of multispectral line cameras with a set of linear sensor arrays for different spectral bands on the focal plane (Wohlfeil, 2012).Each of those sensor arrays is sensitive to light in a different spectral band and has a slightly different angle of view due to its displacement on the focal plane.This means, that if a flat surface is scanned with constant velocity and orientation, the contents of the line images from the different sensor arrays are displaced by a constant shift vector.Additional movements of the camera and elevations on the ground surface result in additional displacements, which are to be determined by matching small image features between the different line images.
There are two major difficulties that have to be handled before the matching of features in such images can be performed.Due to possibly fast, non-systematic orientation changes of the camera the image can be heavily distorted (Figure 5a).Because an object point P is scanned at different times by different sensors, the deformation around the corresponding image point p is different in each spectral band (Figure 4).Especially fast orientation changes around the roll axis can cause lateral shifts between consecutive image lines that heavily impair the tracking process.To reduce these lateral shifts, subsequent pairs of lines are analyzed and preshifted to their approximate position.The second major difficulty is the fact, that the images contain the intensity values of different bands of the light spectrum.Due to this, objects with a non-uniform spectral reflectance appear with different intensities in different images.Although these differences cannot be compensated completely, they can be reduced by locally maximizing the gray value ranges of each image independently.As a result the gray value ranges of large objects (meadows, roofs, streets, trees, etc.) become similar in all spectral bands in most cases.At the same time, the object structure is preserved in all bands and their features can be tracked more reliably.
In the first spectral band of the pre-corrected image a very high number of trackable image features are selected according to Shi & Tomasi (1994).A constant number of features are selected in each line of the image.It is ensured that they are evenly distributed over the full length of the line to enable the determination of its orientation.The selected features are tracked through the images of other spectral bands by using the tracking algorithm of Tomasi & Kanade (1991).The now known positions p ij of a feature in the different spectral bands correspond to the projection of the same object point P i (homologous point) on the focal plane (Figure 4).Hence, each object point observation represents a link between lines captured at a different time.This leads to a connected network of bundles of light rays between all lines of a segment.On the base of this network of observations the orientation parameters of all lines and the positions of all object points are calculated via sparse bundle adjustment (Lourakis & Argyros, 2004).This way the changes of the camera's orientation between the capture of two lines are determined precisely.In order to determine the absolute orientation the remaining offsets and drifts of the optically measured orientation can be determined in the same way as described in the previous Section.Another approach to derive the absolute orientation uses the image contents, as described in the following.

Image-Based Derivation of the Camera's Absolute Orientation
The two presented approaches show how fast and precise orientation changes of a line camera can be determined using optical information received by the camera system.These known orientation changes can be used to pre-correct the line images to compensate short-term distortions, while long-term distortions remain (Figure 5b).On the base of these precorrected line images homologous points can be extracted in areas where two or more images overlap, as shown by Wohlfeil (2010).With their help the absolute orientation of the camera can finally be calculated by a global bundle adjustment.
The pre-corrected images (Figure 5b) have two characteristics, from which the requirements for the matching technique are derived.First, corresponding points can be located in different places in different pre-corrected images.This is due to the rough initial estimate of the images' orientation and the possibly large deviations of the actual terrain height from the virtual planar projection surface.Second, errors of the estimated orientation lead to long-term distortions of the precorrected images, including rotation, shearing and scaling.These requirements are met by the technique of matching scale invariant feature descriptors, such as SURF (Bay at al., 2008), for example, which is used for this task.Again, the matched features are used as homologous points.
Based on the homologous points the position measurements and the parameters of a special orientation correction model can be calculated via bundle adjustment.Starting with a very simple model with only a few parameters the bundle adjustment is repeated iteratively.After every iteration incorrectly matched homologous points are determined and eliminated, the complexity of the model is increased.The iteration is stopped when the full complexity of the model is reached and the drifts in orientation can be compensated completely Figure 5c).This technique can off course also be used to correct remaining drifts and offsets of conventional orientation measurement systems, as shown by Wohlfeil & Bucher (2009).If fast orientation changes have successfully been avoided, their correction by means of optical information may not even be necessary, for example in the case of WorldView I or II imagery.Especially for the generation of high resolution digital surface models (DSM) the high accuracy of image coregistration achievable with this approach very useful.

Experimental Results
The two approaches for the detection of fast orientation changes have been tested separately using offset and drift correction with a simulated low-cost IMU-based navigation system and a simulated attitude and orbiting control system (AOCS), used in most remote sensing satellites.For the tests the DLR's line camera 'MFC3-8K' was used as main imaging sensor with a line rate of about 435 Hz.This camera has three sets of sensors (12° forward, nadir and 12° backward).
Each set consists of a red, a green, and a blue line sensor with an angular offset of 0.077°.The results of both approaches showed that the achieved precision is at least in the range of the high-end reference measurement systems (IGI AEROcontrol / Applanix POS AV).
To test the completely optical orientation determination the above-mentioned approach for multispectral line cameras was used.It is proposed to be the more challenging and comes along without any additional sensor but GPS.For the test, the camera was directly mounted on the floor of the aircraft (Cessna 207) without being stabilized at all, resulting in heavy short-term distortions of the images due to the vibration of the carrier.The accuracy of the absolute orientation achieved after the image-based offset and drift correction was measured by means of ground control points.The RMS of the absolute error (distance from the point to the proposed line of sight) at these points was 35.8 cm and the relative errors (distance between the proposed lines of sight belonging to the same homologous point) 8.4 cm.These results are very promising as the ground sampling distance in the direction of flight was approximately 12.5 cm.Thanks to the very small relative errors overlapping images can be rectified with sub-pixel precision, which is vital for the automatic generation of high resolution DSMs via Semi-Global Matching (SGM).Such a DSM with a horizontal resolution of 15 cm generated with the optically determined orientation is displayed in Figure 6.Such surface models can even be generated in real time, as explained in the following Section.

IMAGE MATCHING
Dense stereo matching generates 3D point clouds as the base information for further 3D modeling tasks.We have chosen the SGM method (Hirschmüller, 2005) for real time and remote sensing applications.This approach can compete with the currently best global stereo methods in quality, at the same time it is much more efficient.Unlike other methods, in practice, SGM appears rather robust and insensitive to the choice of parameters.This makes it suitable for real world applications like tile-wise aerial image matching (Hirschmüller, 2008) (Gehrke et al., 2010) and automotive applications (Steingrube et al., 2009).(Khronos, 2008) SGM version (9x7-Census, 16-bit disparity images with subpixel interpolation) is the possibility to match huge images with many disparity steps in only a few tiles.On the other hand, FPGAs have a low power consumption and high computational power that make them suitable for energy aware embedded systems that demand real time image processing.We built a scalable FPGA implementation of the SGM algorithm with Census matching costs which achieves real-time performance for VGA-resolution images on a midrange board.All hardware components are designed with the hardware description language VHDL, following the hardware operating system concept (HW-OS) based on Krutz described in Section 2. The HW-OS allows for a fast interchange of already deployed and proven hardware modules, at the expense of 5-10 % addition-al logic resource utilization.

CONCLUSIONS
In this article essential techniques and approaches towards highly automated and cost-efficient processing of remote sensing imagery has been presented.New software and hardware development techniques allow smart radiometric and geometric pre-processing solutions on-board.The benefit of thematic processing is a drastic reduction of the amount of data that has to be transmitted or stored.In addition to this, the described approach for the generation of DSMs in real time has a great potential for many different applications including the generation of 3D city models and highly accurate ortho photos.

Figure 1 :
Figure 1: Workflow of the hardware operating system.

Figure 2 :
Figure 2: Options for optical determination of a line camera's exterior orientation

Figure 3 :
Figure 3: Upper: Test flight configuration with the MFC line camera and two auxiliary high-speed cameras mounted on the ground plate.Lower: Selected and tracked features from the auxiliary sensors.The trail of the features indicates their shift vectors relative to their position in the previous image

Figure 4 :
Figure 4: Visualization of the bundle adjustment problem for multi spectral line images.The arrow denotes the flight path during the capture of a line image, while the rotation R and position O of the camera change.The focal plane with three line sensors is shown in grey in front of the center of projection (dot on the flight path).The figure is not true to scale: The focal distance and the angular differences of the spectral bands are extremely oversized.

Figure
Figure 5: a: A part of the green band of an uncorrected line image (lines are just conjoined).b: The same image after pre-correction.c: The image completely corrected with the exterior orientation determined by the presented approach (the image corrected with the reference measurement is almost identical and therefore not included in this figure)

Figure 6 :
Figure 6: A detailed view of the corresponding part of the generated DSM