3D HIGHWAY CURVE RECONSTRUCTION FROM MOBILE LASER SCANNING POINT CLOUDS THROUGH DEEP REINFORCEMENT LEARNING

: Reconstructing the geometric curves of highways holds significant value for various tasks, such as highway design, traffic simulation, and road network planning. An essential step in highway curve reconstruction is fitting highway curve models to the extracted road markings from laser scanning point clouds. Existing methods for highway curve fitting typically handle one curve at a time. However, a stretch of highway often contains multiple curves. In this paper, we introduce a novel method capable of fitting multiple highway curves simultaneously. The method leverages a reinforcement learning (RL) algorithm to achieve this goal. Specifically, we design a unique RL environment that empowers the RL algorithm to fit multiple highway curves. Our experimental results demonstrate the superiority of our method over other methods.


INTRODUCTION
Road construction plays an important role for smart cities.However, due to financial constraints, construction technology limitations, and other restrictions, advanced road network design becomes imperative.Among the crucial elements of road network design, curves hold particular significance (Othman et al., 2014).Rational design of road curves not only enhances road traffic safety but also offers a more comfortable driving experience for people while helping to optimize construction costs.Studies have revealed that curves are accident-prone areas of the road traffic system.When vehicles enter road curves, blind spots and increased centrifugal forces can lead to lateral sliding and, ultimately, collisions (Wang et al., 2019;Chuan et al., 2014).To mitigate such risks and improve highway curve design, it is essential to perform road curve reconstruction.The purpose of road curve reconstruction is to recover the geometric information of highway curves.This process allows for evaluating their performance, identifying potential issues, and implementing improvements or adjustments to enhance road performance.Moreover, it provides accurate data and simulation results for traffic simulation and road network planning, aiding in the avoidance of driving accidents.
There are several types of sensing data that can be used to reconstruct road curves, such as images recorded by the cameras mounted on aircrafts or satellites, point clouds collected by the air-borne or vehicle-borne laser scanners.In particular, the point clouds collected by vehicle-borne laser scanners are called mobile laser scanning (MLS) point clouds.In contrast to other types of data, MLS point clouds contains highly accurate and highly dense 3D geometric information of the scanned highways (Zhang et al., 2020).
A number of methods have been proposed to utilize MLS point clouds to reconstruct road markings (Mi et al., 2021).However, only a few methods have been proposed to reconstruct highway curves from MLS point clouds.The highway curve reconstruction method proposed in (Zhang et al., 2020) consists of two major steps.The first step is extracting the road marking points by using the intensity variance as extraction feature.The second step is fitting the highway curve to the extracted road marking points by the evolutionary optimization algorithm called cuckoo search algorithm (Yang and Deb, 2010).Although the method proposed in (Zhang et al., 2020) can accurately reconstruct highway curves, it is only able to fit one curve at a time.However, a stretch of highway often contains multiple curves.To reconstruct multi-curves, the fitting method proposed in (Zhang et al., 2020) has to be applied multi-times, making it less convenient.
In this paper, we introduce a novel method to fit multiple highway curves simultaneously to increase the convenience of highway curve fitting.The fitting of multiple curves from a highway is a kind of geometric multi-model fitting problem.Zhang et al. (2019b) have formulated the geometric multi-model fitting problem as a sequential decision problem and applied a Deep Reinforcement Learning (DRL) algorithm to solve the sequential decision problem.It is worth noting that DRL has attracted much attention in recent years.Many scientific advances have been made using DRL, such as AlpahGo (Silver et al., 2016).
In (Zhang et al., 2019b), only a very simple case of straight line fitting was presented.However, highway curve fitting is more complex than straight line fitting.That is, a highway curve has more parameters than a line.The DRL method used in (Zhang et al., 2019b) is called Deep Deterministic Policy Gradient (DDPG) (Lillicrap et al., 2016).Although DDPG can handle models with a small number of parameters, such as lines, it is less efficient for DDPG to fit models with a larger number of parameters, such as in the case of highway curve reconstruction.Therefore, this paper proposes to use a more efficient DRL method called Proximal Policy Optimization (PPO) (Schulman et al., 2017) instead of DDPG.
Our main contribution is as follows.We propose a new method for highway curve reconstruction using the PPO algorithm.Specifically, we design a unique DRL environment that empowers the PPO algorithm to fit multiple highway curve.Our experimental results demonstrate that our method is able to reconstruct multiple highway curves simultaneously.
The rest of this paper is organized as follows.Section 2 reviews the relevant research related to the work presented in this paper.Section 3 provides a detailed description of the proposed method.Section 4 introduces the dataset used, presents and discusses the experimental results, and compares them with the results of other experiments.Section 5 concludes the paper.

RELATED WORK
In this section, we review some current studies related to our work.

Road marking extraction
There are many methods for extracting road marking points.Currently, most of the extraction methods are based on geometric features or employ deep learning-based approaches.
2.1.1Geometry-based methods: Geometry-based methods primarily rely on the geometric information derived from point cloud data, such as point coordinates, intensity, and curvature, to extract road markings.In recent years, the most common approach has been to leverage the geometric features of intensity to extract road marking points.Road markings, made of highly reflective materials, exhibit higher reflectance intensity compared to the surrounding road surface.Guan et al. (2014a) employed a point density-related multi-threshold method and adaptively estimated the locally optimal intensity threshold using the Otsu algorithm.Guan et al. (2014b) utilized a road surface segmentation method based on curb-lines to extract the road surface and performed multi-threshold segmentation of the road using Otsu's thresholding algorithm.Cheng et al. (2017) proposed a road marking extraction method based on intensity calibration and high-pass enhancement.The method utilized incident angle to represent the received intensity and employed global Otsu threshold, median filtering, and region-growing algorithms for noise removal and road marking extraction.Yang et al. (2020) introduced a line edge detector that combines intensity gradients with intensity histogram statistics to detect road markings.Ye et al. (2022) utilized local optimal thresholds and distance thresholds determined by road design standards to extract road markings while considering intensity information.The method set minimum and maximum intensity thresholds based on visual interpretation of the test dataset and took into account occlusions caused by large obstacles and erosion of road markings.Other studies have also employed geometric features other than intensity.For example, Yan et al. (2016) used an edge detection and edge constraint method to extract road markings and further refined the extracted points based on segment-based and dimension-based features.
Geometric-based methods are effective in extracting road markings in simple scenes or high-quality point cloud data, but their accuracy decreases in complex scenes.Deep learning-based methods can learn higher-level features from point cloud data, enabling better adaptability and extraction performance in complex scenes.These methods enhance automation when trained with abundant annotated data.However, training and optimizing the models require significant time and computational resources.Limited input data may result in degraded extraction performance.

Road marking reconstruction
Reconstructing road markings provides more accurate, comprehensive, and detailed information compared to methods that only extract road markings.This makes it suitable for complex scenes and dynamic environments.Reconstruction methods have significant value in applications such as road planning and autonomous driving.Soheilian et al. (2010) accurately extracted and reconstructed road markings in dense urban areas using edge point extraction, matching processing, geometric filtering, and theoretical model fitting.Hervieu et al. (2015) defined a road marking pattern library according to national specifications.The method is based on energy minimization using marked point processes, incorporating a data-driven reversible jump Markov chain Monte Carlo sampler and simulated annealing to extract road markings.It addresses the ambiguity of road markings types and transformations using a model-driven kernel.

METHOD
Our method has two major stages.The first stage is road marking extraction.The second stage is highway curve fitting by RL.The details of the two stages are described in Sections 3.1 and 3.2, respectively.

Road marking points extraction
In this paper, the approach for road marking points extraction is similar to (Zhang et al., 2020).We briefly introduce the approach as follows.The dataset used in the approach is comprised of raw MLS point cloud data and its corresponding trajectory data.A raw MLS point cloud data consists of three location coordinates (x, y, z) and an intensity value (i), while the trajectory data is obtained from the GNSS-IMU positioning and orientation system (POS) mounted on the MLS vehicle, providing real-time accurate vehicle trajectory data.The extraction of road marking points consists of four steps: raw data partitioning, road surface detection, road marking detection and clustering, hull detection.
In the first step, the raw MLS point clouds are partitioned based on the vehicle's trajectory data.In the second step, a smoothnessbased region growing algorithm is applied to detect the road surface points from each partition.In the third step, road marking points are detected from the road surface points using a thresholding algorithm based on the intensity variance.These road marking points are then grouped into clusters by calculating the Euclidean distance to the cluster centroids.In the fourth step, hull points are detected from each cluster using the alphashape algorithm.

Raw Data Partitioning
To address the challenges posed by large, unstructured, unevenly distributed MLS point cloud datasets, it is common practice to partition the raw dataset into multiple sub-point sets.The trajectory points of the scanning vehicle, which are organized in the order of acquisition time, are used for partitioning in this paper.This partitioning process helps organize the raw MLS point cloud dataset into smaller and more manageable subsets, enabling further analysis and processing.

Road Surface Detection
In the context of this paper, the focus is on road markings on road surfaces.To facilitate the detection of road markings, it is beneficial to identify road surface points.Highway surfaces are typically smooth, so we use the smoothness-based region growing algorithm (Vosselman et al., 2004) for road surface point detection.As the distance between the laser scanners on the vehicle's top and the road decreases, the density of MLS point clouds increases.By employing region growing, the region with the highest number of points usually corresponds to the road surface.Two parameters are required in this step.The first is a distance threshold for determining neighboring points to calculate surface normal vectors.The second parameter is a smoothness threshold for comparing surface normal vectors.In this paper, we set the smoothness threshold to 2°.

Road Marking Detection and Clustering
On roadway surfaces, road markings are typically brighter than their surroundings.As a result, the intensity of a road marking point is usually higher than that of its surrounding points.However, the intensities of different road marking points can vary significantly due to factors like laser scanner incident angles and ranges (Yu et al., 2015).Therefore, detecting all road marking points is often inadequate with a simple approach like intensity thresholding.To reconstruct highway curves, we focus on detecting the edge points of road markings.In (Zhang et al., 2020), an intensity variance-based approach has been proposed for detecting these edge points.By applying a single threshold to the intensity variances, the edge points can be effectively detected.
The Otsu's algorithm (Yu et al., 2015) is then used to compute a threshold based on the intensity variances, which helps distinguish edge points from other points.Points with variances higher than the threshold are identified as edge points.As there can be multiple road markings on a road surface, the detected points usually do not belong to a single road marking.

Hull Detection
The detected markings often appear wider than the real markings in the proposed approach.Consequently, after detection the road marking points, the alphashape algorithm (Edelsbrunner et al., 1983) is utilized to extract the boundary points of the road markings.This is for providing a more comprehensive description of their shape and obtaining more accurate information about their positions and sizes.In this paper, the alpha parameter in the alpha-shape algorithm is set to 0.1.By adjusting the alpha to a smaller value, a more precise boundary for the road markings can be generated.This process involves connecting the discrete road marking points to form a continuous boundary shape, which accurately represents the morphology of the road markings.
It is worth noted that, we have attempted to extract road marking points using deep learning methods (Hu et al., 2020).However, as shown in Fig. 1, the extracted hull points from the deep learning-based extracted road marking points are contaminated by an uneven distribution of curve edge points, which increases the difficulty of reconstruction and significantly reduces the accuracy of the reconstruction.Therefore, in this paper, we do not use deep learning methods to extract road marking points.

Highway curve fitting by RL
3.2.1 Reinforcement learning As shown in Figure 2, the process of RL is an interactive process between an agent and its environment (Lillicrap et al., 2016).This interaction process is conducted sequentially with the step i ∈ N. At each step i, the agent observes the environment in state si, takes action ai based on the reward ri obtained from the environment, which transitions the environment to a new state si+1 and yields a new reward ri+1.The agent's behavior is determined by a policy.The policy π represents the probability of taking action a in state s: π (a|s) ∈ [0, 1].When action a is taken in state s, the environment provides a reward r(s, a).The cumulative reward starting from a certain state si, known as the return Ri, is defined as follows: Ri = n k=i r(s k , a k ), where n is the step when the final state is reached.It can be seen that the return R depends on a sequence of actions determined by the policy π.The goal of RL is to learn the optimal policy that maximizes the expected return J = E[R1] starting from the initial state.The interaction between the agent and the environment can be represented by a Markov Decision Process (MDP) composed of elements such as state space, action space, state transition function, and reward function (Lillicrap et al., 2016).
The goal of RL is to solve sequential decision problems (i.e., MDP problems).That is, RL aims to find the best sequence of decisions.Therefore, if we can model the process of highway curve fitting as a sequential decision problem, we can apply reinforcement learning methods to perform highway curve fitting.

Mathematical models of highway curves
According to the highway geometric design standard (Bertolazzi and Frego, 2018), there are three elemental types of horizontal curves: line, circle, and spiral.Additionally, two elemental types of vertical curves exist: line and parabola (Zhang et al., 2020).When considering 3D highway curves, they are formed by a combination of horizontal and vertical curves.This results in six elemental types of 3D highway curves, all of which can be represented in parametric forms.Upon observing the parametric forms, it becomes evident that the line and circle curves are special cases of the spiral curve, while the line curve is a special case of the parabola (Zhang et al., 2020).For simplicity, this paper will solely represent an elemental 3D highway curve by the most general combination, i.e., the combination of horizontal spiral and vertical parabola.
The mathematical model of the elemental 3D highway curve can be defined as: where θ = (h, x0, y0, z0, µ0, ξ0, η, κ0, ψ) T represents the parameters determining the model, and fx(u), fy(u), fz(u) are defined as: (2) The parameter u ∈ [0, h], where h is the arc-length of the curve.Including h, the model has 9 parameters.The parameters x0, y 0 and z0 define the start location, µ0 is the start horizontal azimuth, ξ0 is the start slope, η is the vertical curvature, κ0 corresponds to the start horizontal curvature, and ψ is the horizontal curvature change rate.It is worth noting that curvature is the reciprocal of radius.

Design of the RL environment
The objective of multiple highway curve fitting is to find a set of highway curves that collectively best fit the extracted road marking points.This curve fitting problem can be seen as a sequential decision-making process.As mentioned earlier, a highway curve can be represented by a parametric mathematical model.Determining the values of parameters for one curve can be viewed as making a single decision.Extending this to the values of parameters for multiple curves, the problem becomes a series of interrelated decisions.
As shown in Eq. ( 1), for a fixed value of θ, M θ represents an instance of 3D highway curve.The instance actually is a continuous 3D point set (Zhang et al., 2019a), i.e., M θ ⊂ R 3 .Given an extracted road marking point set D ⊂ R 3 , the goal of highway curve fitting is to find a highway curve M that is most similar to D. For multiple highway curve fitting, M is the union set of multiple curves, i.e., where n is the number of curves.
Multiple highway curve fitting now can be formulated as the following maximization problem: (4) where e(•, •) is the geometric similarity estimator defined in (Zhang et al., 2019a).
To utilize RL algorithms to solve the problem as shown in Eq. ( 4), we design the RL environment in the following way.The agent action a is the same as the curve parameter θ: a = θ.The environment state correspond to the parameters of all curves.Given a preset max number of curves n.The environment state s is a vector composed by (θ1, θ2, • • • , θn).Let θ L be the preset lower bound of the curve parameter θ.We define the lower bound of the parameter h (i.e., the arc-length of the curve) as 0, therefore a curve with parameter θ L is just a null set.The initial state of our proposed RL method is defined as The reward in the step i is defined as: (5)

Pseudo-code of our method
The pseudo-code of our method is shown in Algorithm 1, and the PPO policy π used in the algorithm can be found in (Schulman et al., 2017).In each iteration, the algorithm proposes a hypothesis value θi for each curve based on the policy.Subsequently, the hypothesis is verified by computing the reward, which is used to update the policy.The stop criterion for the inner loop is defined as follows: the algorithm stops if a curve is too close to the previously proposed curves or if a curve of insufficient length has been proposed.

Algorithm 1 Our proposed method
Input: a road marking point set D, a preset max number of curves n, a preset max number of iterations jmax, the PPO policy π. Output: Get reward ri according to Eqs. ( 5) and (4) si+1 ← (θ1, θ2,

EXPERIMENTS
We implemented our method using Python for road marking extraction and curve reconstruction, and Java for generating virtual road data (Bechtold and Höfle, 2016).The experiments were conducted on a machine running Ubuntu operating system, equipped with an Intel Core i7-11700K 3.60 GHz CPU and 32 GiB RAM.The experiments included experiments on real datasets, and comparative studies on road curve reconstruction.

Experiments on Real MLS Data
In our experiment, we utilized real MLS data collected from the RIEGL VMX-450 system (Zhang et al., 2020).As shown in Fig. 3, the white portion represents the curve data obtained after hull extraction, while the red portion illustrates the reconstructed curve data using our method.Notably, our approach proves to be effective in handling outliers present in the curve data and successfully fitting multiple curves, encompassing both inner and outer ones.The iterative process resulted in the fitted data converging, ultimately achieving accurate reconstruction of the curve shape of the road, as evident from the red portion in Fig. 3.The fitting process lasted for 12,523,575 iterations and 155,001.635s, and 2,127,875 iterations and 22,131.646s, respectively.
Overall, the experiment using real MLS data provides compelling evidence of the effectiveness of our approach in handling real-world scenarios and accurately reconstructing highway curves.

Comparative experiment
We conducted comparative experiments using a 3D virtual scanning dataset of highways generated by the HELIOS virtual scanning system (Bechtold and Höfle, 2016).Focusing on road curves, we extract a portion of the road for reconstruction.The extracted outer contour of the virtual 3D road curve includes two inner curves (red) and two outer curves (green) as shown in Fig. 4.
In the comparative experiments, we compared our method with Zhang et al. (2020).Our method outperforms the cuckoo search algorithm in reconstructing the inner and outer curves of multiple roads, as shown in Fig. 5 and Fig. 6 using the same virtual road dataset.The curve parameters fitted using our proposed method are close to true curve parameters, as shown in Table 1.

CONCLUSION
In this paper, we introduce a novel approach to reconstruct highway curves utilizing the deep reinforcement learning algorithm PPO.Our method proves to be highly effective in reconstructing multiple highway curves from dense, unstructured, and unevenly sampled MLS point clouds.Through testing on virtual scanning datasets and real MLS datasets, we have successfully demonstrated its capability to accurately reconstruct multiple highway curves, capturing the precise shape of the road curves.
However, it is worth noting that our current method necessitates many iterations to obtain the optimal model, which leads to considerable time costs.As a future research direction, we plan to concentrate on enhancing the efficiency of the reconstruction process.By addressing this aspect, we aim to further refine and optimize our method, making it more practical and accessible for real-world applications.
Overall, our work represents an advancement in highway curve reconstruction, and we look forward to addressing the efficiency challenges to expand the practicality and usability of our approach.

Variable
2.1.2Deep learning-based methods: Deep learning-based methods are commonly employed for extracting road markings, such as using Convolutional Neural Networks (CNNs) for endto-end training and prediction on road point clouds, enabling automatic extraction of road markings.During the current reseach, Liu et al. (2020) introduced a finely tuned image-toimage transformation model, which is based on the pix2pix framework for automatic road marking extraction method.Ma et al. (2021) introduced a deep learning framework based on capsule networks to extract road markings from MLS point clouds, constructing a U-shaped capsule network.Lagahit and Matsuoka (2023) proposed an improved loss function (focal combo loss) to enhance the performance of extracting road markings from sparse point cloud images through training CNN models.

Figure 1 .
Figure 1.The hull extracted after segmentation using deep learning methods.