MURAL LINEAR FEATURE ENHANCEMENT ALGORITHM COMBINING SEMI-SUPERVISED IMAGE SEGMENTATION AND DIMENSIONALITY REDUCTION

: Due to the influence of natural and human factors, the linear features in the murals are partially blurred, which brings great challenges to the digital preservation and virtual restoration of cultural heritage. Taking the advantages of non-invasive measurement as well as the rich image and spectral information of hyperspectral technology, we proposed a linear feature enhancement method by combining semi-supervised superpixel segmentation with block dimension reduction. The main research work includes: (1) The true color composite image was segmented to obtain the label data by using the local spatial information of the superpixel image and the global feature information extracted by fuzzy c-means (FCM) clustering.(2) According to the label data, the preprocessed hyperspectral data were divided into homogeneous regions, whose dimensionality was reduced by principal component analysis (PCA) and kernel principal component analysis (KPCA). (3) The principal component images with the largest gradient after dimensionality reduction were respectively selected and normalized. The optimal principal component images normalized by the block PCA and block KPCA dimensionality reduction algorithms are superimposed to produce the linear feature enhancement images of murals. The hyperspectral images of some murals in Qutan Temple, Qinghai Province, China were used to verify the method. The results show that the spatial information and the spectral information of different pattern areas in the hyperspectral image can be fully used by combining the superpixel FCM image segmentation algorithm with the dimensionality reduction algorithm. of. It can highlight the linear information in the hyperspectral images of fades murals.


INTRODUCTION
Murals, as an art form that can be displayed intuitively, reflect the ideologies and thoughts of the working people in different eras.It is an important part of China's cultural heritage with a high artistic and research value.Lines help the characters or content of the picture to distinguish the front and back space and express the depth of the picture.When the depth and direction are in a whole or plane, the lines are mainly used to distinguish the boundaries of the figures on the screen.At the same time, it is also one of the important emotional expression elements in the picture.The tough lines in the picture, in the form of different sizes and thicknesses, accurately capture the contradictory and complicated inner world of the protagonist (Yuan, 2018).However, many murals suffer from some damages such as craquelure, flaking, scratch, disruption and so on.Therefore, some line information in the mural may be deteriorated, making it difficult to distinguish with the naked eye or ordinary cameras.How to preserve and protect murals non-destructively has become one of the research hotspots.
Hyperspectral technology has the advantage of combining rich spectral information and maps.It has been widely used in the field of cultural relics protection for its non-destructive nature (Rohani et al., 2016).In terms of information enhancement, Kim used hyperspectral near-infrared bands to visually enhance ink bleeding and ink corrosion in historical documents through band synthesis (Joo Kim et al.,2011).Guo used principal component analysis (PCA) technology to extract the internal information of the painting, excavated the hidden information of the painting and extracted the smear traces around the ancient painting crown (Guo et al., 2017).Shi used short-wave infrared bands in hyperspectral imaging technology to analyze some calligraphy and painting artifacts stored in the Palace Museum, Beijing, China (Shi et al., 2017).They extracted hidden information and draft line information of the calligraphy and painting.Zhang used hyperspectral technology to extract color patterns on colored pottery from the principal component images obtained after PCA (Zhang et al., 2019).Salerno evaluated the performance of PCA and independent component analysis (ICA) technology to extract and enhance Archimedes text from hyperspectral image data (Salerno et al., 2006).Cao proposed a method that combines Minimum Noise Fraction (MNF) with Haar wavelet transform to enhance the linear feature from hyperspectral images of murals (Cao et al., 2021).
At present, the extraction of linear features mostly uses global dimensionality reduction algorithms, such as PCA and MNF.These algorithms cannot fully consider the differences of spectral characteristics corresponding to the spectral information of different pattern areas in the fresco.Moreover, due to the influence of many complex factors, such as the nonlinear scattering of the bidirectional reflection distribution function (BRDF), the inhomogeneity of the target, and the complex effects of electromagnetic waves and the atmosphere.In the imaging environment, linear dimensionality reduction is difficult to mine complex nonlinear features in hyperspectral data.In addition, in image classification, some highly accurate neural network algorithms need a large number of training samples and a large amount of computation.Therefore, we proposed an algorithm that combined superpixels, multi-scale FCM algorithm and classical dimension reduction algorithms to enhance linear features.We used the label data obtained by the superpixel FCM, and then divided the hyperspectral data into different homogeneous regions according to the label data.The PCA and kernel principal component analysis (KPCA) were performed on different homogeneous regions to obtain the reduced-dimensional data, where those principal component image with largest gradient were selected to acquire the linear feature enhanced image.The interference of subjective factors was avoided as much as possible while the information was fully extracted.The local murals of Qutan Temple in Qinghai Province, China was scanned with the hyperspectral imager.The linear feature was enhanced by the method proposed and the result was compared with the PCA linear feature enhancement method.

Data Acquisition
The hyperspectral images of some murals in Qutan Temple, Qinghai Province, China were collected.Qutan Temple was built in the 25th year of Hongwu in the Ming Dynasty (1392).
The main building was Ming Dynasty official style.In 1982, it was included in the second batch of national key cultural relics protection units.The murals of Qutan Temple served as a cultural exchange between Han and Tibet race.There are murals painted in the Ming and Qing dynasties in the temple, which have extremely high historical and artistic value.However, due to their age, the murals urgently need to be protected and restored.
The VNIR400H ground-based hyperspectral imager was used to scan the murals in the fifteenth district of the west corridor of Qutan Temple.The spectral resolution is 2.8 nm, the number of channels is 1040, and the spectral range is 400-1000 nm.In order to reduce interference from external light, the halogen lamp is usually used as an artificial lighting device for indoor collection when acquiring image data because its emission spectrum is similar to that of sunlight.The camera is perpendicular to the target, the shooting distance is about 1.09 meters, and the aperture is 4.0.By scanning and imaging parts of the murals in the fifteenth district of the West Corridor of Qutan Temple, a total of 30 scenes of images were collected, and the images were synthesized and spliced using the hyperspectral RGB band as shown in Figure 1.

Preprocessing Process
Hyperspectral image preprocessing includes reflectance correction and image denoising.The reflectance reconstruction is shown in equation ( 1).
where R is the reflectance image, RData is the original hyperspectral image; RWhite is the standard whiteboard data under the same environment; RDark is the dark current data.
Due to the influence of the instrument, the bands at both ends contain a lot of noise.So, we removed the first 50 bands and the last 50 bands of the hyperspectral data, and the remaining 940 bands are subjected to the MNF forward and inverse transformation.Image data can reduce noise while retaining rich spectral information to achieve the purpose of image denoising.

METHODOLOGY
The band 383 (640 nm), band 241 (550 nm) and band 94 (460 nm) of the preprocessed hyperspectral image were selected as the red, green and blue channels to synthesize a true color image.Multi-scale super-pixel FCM algorithm was used to classify the synthesized true color images to obtain the label data.The preprocessed hyperspectral image and the regions corresponding to the label data are corresponded one by one to obtain the hyperspectral block data.Then the block PCA and the block KPCA are respectively used to acquire the reduced principal components.For the first few principal component images after dimensionality reduction, the optimal principal component image was selected according to the gradient information.The component image with maximum gradient was then normalized and superimposed to get the linear feature enhanced image.The specific process is shown in Figure 2.

Superpixel-based Fast FCM
Since the traditional FCM are sensitive to noise, the spatial information of the pixels is often missing and there is a phenomenon of over-segmentation.In the mural, the adjacent pixels in the pattern often belong to the same category.The use of superpixels can divide the image into homogeneous regions according to local spatial information.Using superpixels, the image can be divided into homogeneous regions according to local spatial information.Since the super-pixel image carries the spatial information of the image, the number of different colors is reduced.Replacing each color pixel in the original image with the average value of the color pixels in the corresponding area in the superpixel image can effectively fuse adaptive local spatial information and global color features.The multi-scale morphological gradient reconstruction was used to select the appropriate segmentation scale to further improve the classification efficiency of FCM.
Lei proposed a superpixel-based fast FCM (SFFCM) clustering algorithm based on superpixels, which requires the definition of multi-scale morphological gradient reconstruction (MMGR) operations to obtain superpixel images with precise contours (Lei et al., 2018).Morphological gradient reconstruction (MGR) removes noise and useless gradient details while retaining the contour details of the target, and effectively avoids over-segmentation.
Multi-scale morphological gradient reconstruction is defined as： where A is the original image, B is the marked image, Z is the structural element, r1<r2, where r1 controls the size of the smallest area, and r2 controls the size of the largest area.G C is denoted as closing reconstructions.
There are 256 gray values in a grayscale image and there are far more than 256 different colors in a color image.Therefore, the average value of the color pixels in the corresponding area in the superpixel image is used to replace each color pixel in the original image, which reduces the number of different colors and effectively improves the effect of image segmentation.
Applying the labeled image with adaptive local spatial information to FCM clustering, the objective function obtained is: where n is the color level, 1 ≤ n ≤ p, p is the number of the superpixel image, n and p are both positive integers, d is the number of clusters, Wn is the number of pixels of Gn in the nth area, and xq is the color pixel in the nth area of the super-pixel image.vnk is the fuzzy membership degree of color level n to the kth clustering center uk.t is the weight factor.

Block Dimension Reduction Algorithm
The traditional PCA is a linear dimensionality reduction method that can perform a global projection transformation on the entire hyperspectral image.However, in the hyperspectral image of the mural, there are obvious differences in the spectral characteristics of different uniform areas.The PCA is difficult to deal with the nonlinear characteristics due to the influence of various complex factors in the imaging process.Therefore, block PCA (Jiang et al., 2018) and block KPCA (Zhang et al., 2019) were used to extract the characteristics of each representative pattern area in the hyperspectral image data of murals.
First, the hyperspectral block data divided according to the label data is subjected to PCA to perform linear dimensionality reduction.In order to further explore the nonlinear features, the method of Gaussian kernel mapping was selected to process the nonlinear data.In this way, the original data is mapped to the feature space through the kernel mapping function and the corresponding linear operation is performed in the feature space.Through nonlinear mapping, the linearly inseparable samples in the original space have a higher probability of linearly separable or linearly separable in the feature space.Then the linear transformation method is used to reduce the dimensionality of the feature space and improve the ability of the linear transformation method to process nonlinear data.

Linear Feature Enhancement
First, the acquired hyperspectral data of the mural in the target area was preprocessed.Then, the superpixel image of the target area is obtained through the superpixel-based fast FCM algorithm.It was used to produce the label data of the classification information in the mural image.The hyperspectral image is segmented to many blocks according to the label data.PCA and KPCA were carried out for data dimensionality reduction on hyperspectral block data to obtain the first several principal component images and then calculate the gradient information of each principal component image, which can be expressed as: where a and b are the width and height of the image, respectively, is the gradient in the x direction, and the gradient in the y direction.
The images with the largest gradient information after the block PCA and block KPCA were selected for normalization to make the grayscale within [0,1].It can be expressed as follows: where is the normalized image; is the image to be normalized, is the minimum value of gray in the image to be normalized; is the maximum value of gray in the image to be normalized.
In order to ensure the integrity of the image information of the murals, the normalized images of the block PCA and block KPCA were superimposed in bands to obtain an enhanced image with linear features.

Semi-supervised Image Segmentation
We select band 383 (640 nm), band 241 (550 nm) and band 94 (460 nm) of the preprocessed hyperspectral image to synthesize true color images.The parameters of SFFCM are set to the maximum number of iterations of 50, the weight index 2, and the convergence condition 10 − 5 .Set the number of clusters according to the type of the color pattern of the synthetic true color image of the target area.When the number of clusters is 6, 7 or 8, the image will be under-segmented, and when the number of clusters is 10、11, the image of the target area will be over-segmented.The segmentation phenomenon is obvious.Therefore, the number of clusters was set to 9, and the segmentation result is shown in Figure 3

Block Dimension Reduction Algorithm
The first few principal component images after the dimensionality reduction of the block PCA and the block KPCA are respectively calculated.Since the image after the seventh principal component contains a lot of noise, the first six principal component images are selected for normalization and the average gradient is calculated respectively.According to the calculation result of the gradient, the optimal principal component images are the first principal component image of the block PCA and the fifth principal component image of the block KPCA, as shown in Figure 4.The proposed method is compared with the PCA image enhancement method (Hou et al., 2014) and the original gray image, and the information entropy of region 1, region 2 and the entire image are calculated respectively.The results are shown in Table 1.It can be found that the information entropy of the enhanced image is better than that of the original grayscale image and the enhanced image obtained by the PCA method, which further proves that the linear features are effectively enhanced.The results show that the use of multi-scale, semi-supervised image segmentation algorithms can make full use of the spatial information in hyperspectral images.Performing PCA and KPCA on hyperspectral homogeneous region data can effectively utilize the spectral characteristics of different pattern regions to better achieve the purpose of information enhancement.
(1) By comparing the original grayscale image and the information-enhanced image, it is found that the linear feature information of the pattern in the real mural is significantly enhanced.The blurred, faded and invisible information is clearly distinguishable after being enhanced by this method.
(2) Compared with the traditional PCA enhancement method, the enhanced image obtained by this method can better achieve the purpose of enhancing linear feature information.

CONCLUSION
Aiming at the problem that the linear features of murals are fading and difficult to identify, a method for enhancing linear features of murals is proposed by combining multi-scale superpixel FCM and a block dimensionality reduction algorithm.The semi-supervised classification is used to obtain the label data of the image, and the optimal image superposition based on the block principal component analysis and the block kernel principal component analysis is used to obtain the linear feature enhanced image.The hyperspectral image of the Qutan Temple murals in Qinghai Province is used as an example.After comparing with the original gray image and the principal component analysis linear feature enhancement method, the results prove the effectiveness of the proposed method.
The research shows that the combination of block dimension reduction and multi-scale super-pixel FCM algorithm can effectively extract linear features in murals and realize the extraction of hidden information.It can provide data reference for the protection and restoration of murals, making the restoration have evidence to rely on, and ensure that the restored murals have a high degree of authenticity and restoration.

Figure 1 .
Figure 1.Mosaic image of murals in the 15th district of the West Corridor of Qutan Temple

Figure 3 .
Figure 3.Comparison of segmentation results

Figure 5 .Figure 6 .
Figure 4. Optimal Principal Component Images for Blocking Dimensionality Reduction

Table 1 .
Information entropyIn order to further verify the proposed method, the structural similarity between the enhanced image and the original gray image is calculated.After normalizing the data, the regions 1 and 2 are intercepted.The structural similarity of the original image, image after PCA enhancement, and image after the method in this work are calculated.The results are shown in Table2.

Table 2 .
Structural similarity index measurement