BUILT-IN LENS CORRECTION PROFILES IN LOW-COST CAMERAS: AN ISSUE FOR PHOTOGRAMMETRIC APPLICATIONS?

: Photogrammetric applications nowadays e n visage the use of more and more low-cost cameras such as those equipped on commer c ial UAV platforms. Typically, these low-grade cameras suffer from extreme radial distortion and strong vignetting among other defects. This, initiated a trend among the low-cost cameras’ manufacturers to try to hide the camera defects by appl y ing software pre-corrections to the images. These Built-In Correction Profiles gets applied to both the JPG files, directly in-camera, and usually to the raw files as well, through the opcode functions of the DNG standard. In this paper we rise this issue that is still under-reported in the literature and further assess the accuracy implication of applying or discarding the Built-In Correction Profile in the scenario of UAV mapping. We tested the commer c ial UAV DJI Phantom 4 Pro v2 in a calibration environment and a field test to compare the performance of pre-corrected versus uncorrected images. In our tests, processing the original uncorrected images led to improved IO calibration and reduced bowing effect in the field test.


INTRODUCTION
Photogrammetry is now a widely adopted digitising technique exploited by various professionals and non-professionals in various fields, from architectural to aerial survey.The employed tools are the most diverse since even non-specialised cameras can be successfully used.In recent years, momentum to the popularisation of photogrammetry has been the growth of the UAV and smartphone sectors, with drones commonly exploited for low-cost aerial surveying purposes and smartphone cameras introducing photogrammetry among non-experts.The advent of commercial Structure from Motion (SfM) software, with the possibility to employ common commercial cameras, has led to a true democratisation of photogrammetry.A key role lies in the ability of SfM software to elaborate a high number of images and to perform the self-calibration of the employed camera from survey images alone.Even among nonexperts, there is a widespread notion that photography introduces unavoidable distortions that are modelled by the software as long as the camera's characteristics remain stable during acquisition.Hence, it is imperative to fix camera settings like focal length and focus, and it is advisable to refrain from modifying the inherent geometric distortion of each image during post-processing, despite the availability of tools for this purpose in all raw image processing software.Nowadays, the most common photogrammetric workflow involves: (i) shooting the images in raw format, keeping camera features stable; (ii) post-processing the images in a "raw processing software", correcting exposure, vignetting, and more, but without correcting the geometric distortion; (iii) processing the photogrammetric block in a SfM software.However, this pipeline is being challenged by manufacturers of low-cost cameras, such as those mounted on smartphones or hobbyist drones or directly sold as point-and-shoots.Indeed, it has recently become the practice among these manufacturers to apply pre-processing to images directly in-camera during shooting.Such Built-In Correction Profile (BICP), as expected, is applied to the JPG files with no possibility of recovering the original data, but, moreover, it is also stored in the raw files as processing instructions that the raw editor software might read and apply automatically.Among other adjustments, the BICP, depending on the applied functions, might also correct the geometric distortion of the lens with a generic model (not tuned for the specific camera).Indeed, the trend among low-cost camera manufacturers is to not show the consumer the original photograph but a pre-processed version to remove and hide its defects or limitations (Figure 1).As mentioned, the BICP is applied both to raw and JPG photographs.Indeed, the two files usually appear identical side by side since a low-resolution JPG preview of the image is usually shown in place of the raw file preview on the Windows viewer.Contrary to the JPG files that are permanently modified, the raw files, though showing modified as well, still retain the original data.The automatic application or not of the BICP to the raw images depends on the software used to read the images.Some raw file processing software, such as Adobe Lightroom, automatically apply the corrections, and do not allow their removal, while others discard it completely.For metrological purposes, it is necessary to be aware of the Built-In Correction Profile and when or whether it is applied to the images, including raw images, since it is the camera manufacturer's interest to pre-correct the images without the user's knowledge.Moreover, the difference between the original and the corrected image can be extreme.The former generally suffers from high vignetting and strong radial distortion.Indeed, it can be much wider than the modified version and therefore discarding the BICP will boost the camera field of view as well.Figure 1 illustrates the difference between original and precorrected version of some images acquired with popular UAV commonly used for photogrammetric applications.

Paper objective
The paper would like to raise this issue that is under-reported in the literature and further assess the accuracy implication of applying or discarding the Built-In Correction Profile in the reallife case scenario of UAV mapping.The objectives are: (i) to analyse the accuracy and systematic residuals obtained during pre-calibration, comparing both sets of images; (ii) to analyse the accuracy implication of processing the corrected or original images in the case study of an extensive UAV aerial survey.Moreover, the paper shows the differences between the original and corrected images and indicates how to discard the BICP before eventually processing the raw images into the chosen "raw processing software".

Related works
Many users report mixed results with UAV photogrammetry, and to match that, the literature offers multiple accounts of metric evaluations of UAVs' cameras, pointing out the instability of the cameras as the source of sub-optimal behaviour.(Hastedt et al., 2015;Cramer et al., 2019).Other authors also pointed out the Built-In Correction Profile as an added issue that could influence the results (James et al., 2020;Hastedt et al., 2021;Martínez-Fernández et al., 2022) In their tests, gently inclined oblique images (<15°) resulted in a correlation of the decentring and radial distortion parameters with a consequently higher doming effect.They pointed out the BICP of commercial drones as a possible source of error, altering the natural physical ratio between radial and decentring distortions and increasing the importance of the latter.Hastedt et al., 2021 tested many UAV platforms and investigated the common residual patterns.They investigated the uses of extended interior orientation models and pointed out that these can be helpful for some of the tested datasets in reducing systematics in interior orientation.They also processed original and pre-corrected images of the DJI Zenmuse X7 and pointed out no significant difference in the calibration residual with the extended models.They also tested the DJI Zenmuse X4S and DJI FC6310 cameras, which have the same camera design, processing the pre-corrected images (with applied BICP) for the former and the original images for the latter.Their tests pointed out different residual patterns obtained by the two datasets but found no significant difference in object space accuracy.They also pointed out similar correlations between Internal Orientation (IO) parameters and hypothesised the source in some identical hardware components.Martínez-Fernández et al., 2022 presented a study conducted with the UAV DJI Mavic 2 Pro, comparing original raw images pre-processed by the BICP and custom-processed images during raw conversion.They pointed out that all datasets can perform well with a strong image network and adequate GCPs placement, although deviations between the models are found.As mentioned in the introduction, the BICP gets applied by the DNG reader, provided that the DNG reader recognises the opcodes.This is the case with the raw processing software Adobe Lightroom Classic (Adobe inc.Development Team, 2022), which applies the opcode functions automatically and does not allow their removal.Other raw processing software may allow the discard of the opcodes at will, and others do not recognise them in the first place, as in the case of the free software Darktable (Darktable Development Team, 2023).Also, the popular SfM software Agisoft Metashape (Agisoft Development Team, 2023) allows processing directly DNG images, not reading the opcodes from them.It follows that, to process the raw images to correct the exposure, attention should be paid to check whether the software in use applies the BICP.Alternatively, the BICP can also be discarded before processing the DNG by clearing the "opcode list 3" tag from the images Exif files.

Datasets preparation
For this investigation, all image datasets were acquired by the PH4 FC6310S camera in DNG format; then, two sub-datasets were derived from each raw image dataset, in which the images were converted into TIFF.For the first sub-dataset, the conversion was done with Adobe Lightroom Classic, which automatically applies the Built-In Correction Profile; for the second sub-dataset, the conversion was done with Darktable, therefore discarding the BICP.
Hereby we refer to the TIFF dataset with applied BICP as "LR" and the TIFF dataset with discarded BICP as "DT" datasets.Figure 3 shows an example image from the LR and DT datasets, i.e., with and without applied BICP.

Test 1 -Camera calibration
In the first test, a laboratory pre-calibration of the FC6310S camera was performed in a controlled test field (Figure 4) to compare the results of the LR and DT datasets.
The calibration test field is set up in a room corner that measures approximately 1.The acquired calibration dataset was subdivided into the LR and DT datasets, as previously described, and the two were processed using Agisoft Metashape Software (v2.0).The DT dataset was further subdivided into the DT-rectilinear and DT-fisheye datasets.Essentially, the DT dataset was processed twice: for the first sub-dataset, a rectilinear lens model was considered, and for the second one, an equidistant fisheye lens model was considered.The distortion is modelled for all processes based on the Brown model (Brown, 1966;Brown, 1971), estimating 8 parameters: c, x0, y0, k1, k2, k3, p1 and p2.A SfM image orientation was processed for each dataset without any target reference.The known coordinates of the photogrammetric targets were introduced only to evaluate the accuracy of the reconstruction performed by comparing the RMSE (Root Mean Squared Error) obtained.The calibration results were compared to the estimated distortion parameters and obtained residual.

Test 2 -Test on the field
In the second test, the PH4 drone was used to survey a large open environment: the archaeological area of Giardini Naxos (ME, Italy) (Figure 6).The Giardini Naxos case study is ideal for comparing the performance of photogrammetric reconstruction in terms of accuracy using original or pre-corrected images.The archaeological garden area measures approximately 780 x 350 m in the longitudinal and transverse directions respectively, and is characterised by a flat profile with no significant elevation changes.The garden area is covered by turf and scattered trees and bushes.Some structures and archaeological finds are also present.No significant tall obstacles are present, making the photogrammetric UAV approach ideal for surveying the area.Regarding the 60 m acquisition, the GSD is 16.5 mm, and the lateral and frontal overlap is around 50 and 75 %, respectively.The accuracy evaluation is performed on GCPs and Check Points (CPs) distributed throughout the archaeological garden.A total of 22 reference points were materialised and measured using a total station, with an estimated accuracy of around 1 cm.As in Test 1, two datasets were derived from the raw DNG images, the LR and the DT datasets, with the BICP pre-applied and discarded, respectively.Initial pre-calibrated IO parameters derived from Test 1 were used.Considering the image-scale differences between the calibration and field test, the initial IO was not constrained but instead adjusted.
Three types of evaluations were performed: E1.
All reference points used as CPs.E2.
All reference points are used as GCPs.
Figure 6.Orthomosaic of the Giardini Naxos Archaeological area with the location of the reference points.For Evaluation 2, yellow points are used as GCPs and white points as CPs.

Test 1
Figure 7 illustrates the calibration test results, and Table 1 reports the CPs' RMSE and reprojection errors.The results show that the three processed datasets produced very different calibrations, with different performances in modelling the IO distortions.
The LR dataset shows how the BICP corrected most of the radial component of the lens distortion.Looking at the first two plots (LR a and b), we can see how only a camera-specific residual distortion has been modelled with a maximum absolute value for the radial distortion of around 5 pixels.On the other hand, the decentering distortion components are much stronger, with a maximum absolute value of around 12 pixels.Such atypical distortion behaviour reveals that corrections have been preapplied.The DT dataset processed with the rectilinear lens model (DT rectilinear) shows a standard distortion behaviour, with an extreme barrel distortion maxing at around 600 pixels (absolute value), as clearly visible in Figure 3 -bottom.The radial distortion of the third test, DT fisheye dataset, processed using the fisheye lens model, shows an opposite-signed radial distortion, i.e., pincushion-type with respect to the equidistant fisheye projection instead.For this dataset, the max radial distortion at around 100 pixels (absolute value) shows that the distortion behaviour of the PH4 camera is closer to the equidistant fisheye projection than to the rectilinear projection.
Regarding the decentering distortion, it is nearly identical for the two DT datasets.Significant differences can also be observed in the residuals' plots (Figure 7    The better performance of the DT fisheye dataset is also evident by looking at the reprojection error and RMSE on CPs of the calibration test field (Table 1).Table 1 also shows how the precorrected images (LR dataset) performed worse with an RMSE of 0.23 mm.The DT rectilinear dataset resulted in a better RMSE but a higher reprojection error, explained by the higher residuals.
On the other hand, the DT fisheye dataset resulted in the lowest RMSE and the lowest reprojection error.
Even though the DT fisheye dataset clearly produced the better results, it is important to note that the residuals of the calibrations do not exceed the pixel value for all datasets.

Test 2
Based on the results of the previous test, specifically the comparison between the two DT datasets highlighting how the fisheye lens model is a better approximation of the PH4 lens projection function than the rectilinear lens model, for Test 2, only the LR and DT fisheye dataset were compared.The comparison is summarised in Table 2 and illustrated in Figure 8.
In the first type of evaluation -E1 -, the two image networks are kept unconstrained, and all reference points are used as CPs to evaluate the accumulated drift error.In this case, the dataset LR (E1) performed better than the dataset DT (E1) with an RMSE on CPs of 4.2 cm and a maximum error of around 7 cm, roughly half the error accumulated by the DT dataset: RMSE of 7.5 cm with a max error of 14.4 cm.For both datasets, the error is mainly distributed along the Z axis, an order of magnitude higher than the error on the XY plane.Figure 8 -on the top -illustrates the magnitude and distribution of the error for E1 along the Z axis for all CPs.The LT dataset (in blue) and the DT dataset (in orange) show a very similar error distribution, i.e., positive at the opposite ends of the archaeological park and negative at the centre, which is an inverse doming effect.
In evaluation 2 -E2 -, 7 out of 22 total reference points were used as GCPs and the rest as CPs.The introduction of constraints in the bundle adjustment resulted in a reduced drift error in both 3D reconstructions.In this case, a higher improvement is shown by the DT dataset (E2) with an RMSE of 2.6 cm compared to an RSME of 3.9 cm for the LR (E2).Most notably, the maximum error of the DT dataset dropped from 14.4 cm in the unconstrained case (E1) to just 4.4 cm in E2.On the other hand, the LR dataset, although showing an improved error overall with the inverse doming effect corrected, resulted in a higher maximum error of 8.2 cm. Figure 8 -on the bottom -illustrates the distribution of the error.Overall, the magnitude of the error decreases for both tests, and the doming effect is corrected.The remaining error picks on the north and south sides, suggesting that more GCPs should be used.
As for Evaluation 3 -E3 -, with all reference points used as GCPs, the drift error of both datasets dropped to the accuracy of the reference coordinates, as expected.However, deviations are expected in the unconstrained areas.
As for the residual of the adjusted calibrations, similarly to the calibration test, the magnitude of the error is higher for the LR dataset (Figure 9 -top), while the DT dataset shows a more accurate calibration (Figure 9 -bottom).Systematism in the residual plot can be observed in both plots at the outer edge of the frame.

Discussion
From the conducted test, it is evident that the Built-In Correction Profile pre-applied by the UAV manufacturers has an influence on both image space and the object space on the calibration residuals and error on CPs, respectively.In the image space, as previously illustrated by Hastedt et al., 2021, we noticed a change in the residual pattern when processing pre-corrected or original images.Contrary to Hastedt et al., 2021, in our calibration experiment, we observed different residual patterns for the same camera design (DJI FC6310S).The original images processed using the rectilinear lens model (DT rectilinear) resulted in stronger systematics than the pre-corrected ones.Using the fisheye lens model improved the calibration, reducing the residuals' systematics.Our findings, which are limited to the PH4 example, indicate that processing the original images is indeed beneficial and results in a superior calibration, proving that the used model is adequate to model the physical behaviour of the camera.In our tests, this is the case for the PH4 using the Brown model implemented in Agisoft Metashape, considering a fisheye lens model.However, looking at the example images in Figure 1, it is clear that the fisheye lens model would not adequately describe most of the original images of the common UAV, and it is therefore suggested that camera-specific tests must be carried out.
The calibration test reveals that the application of BICP to the PH4 images effectively corrects the majority of the radial distortion.However, it exacerbates the decentering component.This confirms that camera manufacturers, in their attempt to correct the prominent radial distortion of inexpensive sensors using a generic distortion model, significantly deviate from the expected physically-based behaviour of the distortion, as indicated by James et al., 2020.Regarding the field test to evaluate the object space consequences of the difference in IO, we can see how the residual plots (Figure 9) indicate a similar pattern to the one obtained during calibration.Especially for the pre-corrected image dataset (LR dataset), the residual error magnitude and distribution overlap almost identically, suggesting that the existing systematism does not depend on the image blocks but indicates that the complex distortion resulting from the application of the BICP cannot be fully modelled by the Brown model.For the original images processed using the fisheye lens model (DT fisheye dataset), the residuals are similar to those resulting from the calibration test.However, residuals in the field test appear more systematic.While in the calibration test they could be attributed to fewer and poorer tie points at the image edges, for the field test it seems that the calibration obtained is not entirely optimal.This suggests that a stronger image block is required to adequately calibrate the raw uncorrected images using the fisheye lens model, such as the one provided during the calibration.
As for the CPs error resulting from the test, it is interesting to notice how the original images perform worse than the precorrected ones with no constraint in the bundle adjustment (Evaluation 1), while performing better when few GCPs are used (Evaluation 2).The higher degree of error resulting from the DT dataset in Evaluation 1 can perhaps be attributed to the high degree of radial distortion of the original uncorrected images, thus more sensitive to suboptimal calibration.Indeed, the significantly stronger improvement of the results for the DT datasets in Evaluation 2 suggests, as shown by the calibration test, that a more accurate calibration of the PH4 camera can be achieved by processing original images using the fisheye lens model, provided that a strong image block and adequate constraint are used.

CONCLUSION AND FUTURE WORKS
This investigation aimed to assess whether the Build-In Correction Profile pre-applied by the manufacturers of low-cost cameras, specifically UAV systems, poses a challenge for photogrammetric application.Regarding the spread of the BICP, we can see from Figure 1 how the most commonly adopted UAV platforms today provide a BICP to pre-correct image vignetting, colours and, most notably, geometric distortion.JPG images are directly pre-corrected in the camera and, to achieve the same, opcode functions are stored in the DNG raw file to usually be applied automatically after demosaicking by the DNG reader.
Because of this, even when raw DNG files are stored, chances are that geometric distortion correction gets applied in most image processing pipelines without the user knowing it.This is an issue when DNG files are explicitly saved with the intent of applying custom adjustments, such as exposure and contrast enhancement during the raw-to-JPG conversion, since some raw processing software then applies the BICP without allowing its removal.It is, therefore, suggested when working with lowcost/low-grade camera sensors to investigate whether automatic corrections are applied at some point along the image handling process.Although some recent commercial UAVs allow the user to turn off automatic correction in the camera settings (Dewarp function for DJI), inspecting the Exif files for any listed opcode function is suggested.Stripping out the opcode functions from the raw file before image processing is the suggested method for removing the BICP, which allows the image processing pipeline of choice to be kept unaltered.
As for the impact of the BICP on photogrammetric applications, our initial tests show how better camera calibration and object reconstruction can be achieved by processing the original uncorrected images and choosing an adequate model to estimate the camera distortion.We tested the widespread commercial UAV Phantom 4 Pro v2 in a laboratory calibration test field and a field case study to survey an extensive flat terrain.The calibration test highlighted how the BICP changes the standard physically-based ratio between the radial and decentering distortion components, as already pointed out by James et al., 2020, and indicated that the equidistant fisheye lens model is more suited than the rectilinear lens model to model IO of the PH4 camera.The field test confirmed the calibration test results by showcasing a lower drift error or doming effect when processing original PH4 images using the fisheye lens model and inserting a few GCPs (DT dataset -Evaluation 2).Our results, when inserting a high number of GCPs (LR dataset -Evaluation 3), as well as other literature (Peppa et al., 2019;Martínez-Fernández et al., 2022), suggest that with more robust image networks and highly constrained solution, the pre-corrected images can be successfully employed.The limitation of this investigation lies on the single tested UAV system and the single conducted field test.For future works, we envisage the analysis of other UAV platforms as well as other low-grade cameras, such as those equipped on smartphones.Moreover, additional case studies for the field tests can be considered to check the consistency of the results with varying scene types (urban areas, forest areas, level changes), image scale and image network type (parallel flight stripes, gridded, circular flight and oblique images).

Figure 1 .
Figure 1.Side-by-side comparison between original images (right) and pre-corrected images (left) of various cameras 1 .
During the investigation, the tested sensor was the camera of the popular UAV DJI Phantom 4 Pro v2 (hereby called PH4) released in 2020.The camera is the same as in the previous Phantom 4 Pro model released in 2016.The PH4 is a quadcopter weighing 1375 g, mounting a dedicated camera unit mounted on a gimbal.The PH4 camera unit, named FC6310S, is equipped with a CMOS sensor measuring 13.22 x 8.81 mm, with a resolution of 5464 x 3640 pixels and a pixel pitch of 2.42 μm.It is equipped with a compact lens with a fixed aperture and adjustable focus, characterised by a focal length of 8.8 mm.The DJI FC6310S camera can capture JPG and raw DNG images.When saving both formats, the images appear the same to the user.The JPG image does not show any visible distortion, precisely as the DNG image that previews as a low-resolution version of the JPG.The PH4 applies a geometric correction to the JPG "in-camera" and stores the correction instruction information in the DNG file.Unlike other UAVs from the same manufacturer released in more recent years, the PH4 does not allow the user to choose whether to apply the geometric distortion correction.The DNG file stores geometric distortion correction information in the opcode functions (Adobe inc.Development Team, 2023).The opcode functions were first introduced in the DNG standard with version 1.3.0.0.These functions consist of predefined processing steps automatically executed by the DNG reader.The most commonly used opcode function corrects the lens' geometric distortion and vignetting.The complete list of available functions can be found in Adobe inc.Development Team, 2023.The opcode functions eventually specified for the image by the manufacturer are stored in the image's metadata and can be consulted by reading the full Exif (Exchangeable image format) file.Figure2shows the opcode function specified for the PH4 camera and the DJI Mini 2 camera (DJI FC7303) under the Exif tag "opcode list 3" that accepts a list of opcode functions to be executed to the image after it has been demosaicked.As evident by the examples in Figure1and Figure3, the BICP stored as opcode functions corrects geometric distortions and vignetting of both UAV cameras.Specifically, the WarpRectilinear function is responsible for correcting geometric distortion and chromatic aberrations, according to generic parameters provided by the camera manufacturer.

Figure 2 .
Figure 2. Opcode functions specified by the camera manufacturer in the Exif tag "opcode list 3".

Figure 4 .
Figure 4. Image acquisition during the camera calibration test.
7 x 2.5x 1.0 m along W x H x D. Robust texture -to aid feature extraction -is provided by a random speckle pattern printed and attached to the corner's wall, and accurate reference points are provided by photogrammetric coded targets.The target coordinates are known from the test field setup calibration with an accuracy of around 0.05 mm.The image acquisition with the PH4 was carried out by slowly moving the UAV by hand in front of the calibration test field while acquiring timed images.The image network is redundant, and attention was placed on varying image scale and roll rotation.Images were acquired following roughly a gridded pattern and repeated four times, varying roll angles at 0, 90, 180 and 270°.Additionally, angled images were acquired closer and further away.Figure5shows the image network followed during calibration.A total of 74 DNG images were acquired.

Figure 5 .
Figure 5. Image network of the calibration dataset.Elevation (left) and top (right) views.
The survey was conducted following a gridded flight path, acquiring only nadiral images from two flying heights.9 UAV flights were conducted, collecting 2906 images during the operations: 2080 images were acquired at 40 m above ground level (AGL) in the transversal direction, and 826 images at 60 m AGL in the longitudinal direction.Regarding the 40 m acquisition, the Ground Sampling Distance (GSD) is 11 mm, the distance between parallel strips is approximately 15 m, and the distance between consecutive images is approximately 8 m (overlap of around 75 and 80 % lateral and frontal, respectively).
-d).Systematic residuals are evident for the results of both the LR and DT rectilinear datasets, indicating that the Brown model cannot fully model neither the complex distortion introduced by applying a generic BICP nor the extremely high radial distortion of the PH4 camera, which image formation model more closely resembles the fisheye projection.Residual errors of the DT fisheye dataset are ideal, showing a systematism only at the frame's border, typically due to the lower tie points count and quality in these areas of the images.Indeed, the residuals plot indicates that a high-quality calibration of the PH4 can be achieved with the Brown model by using the fisheye lens projection function.

Figure 7 .
Figure 7.Comparison of the calibration test results performed on the pre-corrected images (Dataset LR) and the original images processed using the rectilinear lens model (Dataset DT rectilinear) and fisheye lens model (Dataset DT fisheye).Radial distortion plot (a), radial distortion curve (b), decentering distortion plot (c) and residuals plot (d).

Figure 8 .
Figure 8. Field test reconstruction error of Evaluation 1 (top) and Evaluation 2 (bottom).The error bars represent the Z component of the error.LR dataset in blue, DT dataset in orange.

Figure 9 .
Figure 9. Residuals plot of the adjusted camera calibration of the field test.LR dataset (top) and DT dataset fisheye (bottom).

Table 1 .
Comparison of the calibration test results.

Table 2 .
Comparison of the field test results.