A CNN-SPEED-BASED GNSS/PDR INTEGRATED SYSTEM FOR SMARTWATCH

: In recent years, wearable devices such as smart bands and smartwatches have gained widespread popularity due to their ability to provide various health and fitness applications by detecting and analyzing the human body and motion information. However, the accuracy of location-based services can be limited, especially in urban areas and indoors. This study proposes a series of smartwatch Pedestrian Dead Reckoning (PDR) improvements based on 9 Degrees of Freedom (DOF) IMU orientation estimation, which includes the heading estimation of human movement and a novel pre-trained velocity regression model. The proposed system holds the potential to enhance positioning accuracy and augment navigation availability for smartwatch users, thus offering potential applications across various fields. This study makes significant contributions to the field of smartwatch navigation by proposing a GNSS/PDR fusion algorithm specifically designed for the consumer-grade IMU, magnetometer, and GNSS receiver built into Apple Watch, tracking varied roll and pitch of the sensor caused by hand swing, and integrating a CNN model to predict the 1-D speed and provide ZUPT information, offering improved accuracy and reliability.


INTRODUCTION
The location-based services of smartwatches have emerged as a highly sought-after concern in recent times.Typically, the determination of a user's outdoor position involves the utilization of a global navigation satellite system (GNSS).Nonetheless, GNSS cannot give a reliable positioning solution all the time, especially in urban areas and indoors.Therefore, alternative methods such as Wi-Fi, Bluetooth, or Inertial Measurement Unit (IMU) are commonly utilized for indoor positioning.In general, WiFi and Bluetooth positioning methods require the installation of a significant number of signal-receiving devices during operation, which can limit their application in certain scenarios.Compared to other sensors, an IMU provides a stable and highfrequency solution that is not affected by environmental conditions.
Pedestrian Dead Reckoning (PDR) is a widely-used IMU-based navigation algorithm for pedestrian navigation and typically involves the use of three types of devices: 1) shoe-mounted sensors, 2) smartphones, and 3) smartwatches.Although shoemounted sensors can provide good PDR results, they are not easy to be deployed and commercialized.Smartphones, as the most common product, do not have a fixed placement, which requires further consideration and processing.In contrast, smartwatches have the advantages of easy installation and fixed position on the wrist, making them the most promising navigation device.Before performing PDR on a smartwatch, it is important to address several factors such as complex hand postures, shaking during walking, and accurate step length estimation in watch mode.
To solve these issues, a series of smartwatch PDR improvements based on the 9 Degrees of Freedom (DOF) IMU orientation estimation is proposed.First, a one-size-fits-all Attitude and Heading Reference System (AHRS) algorithm--VQF (Laidig & Seel, 2022) is adopted, which provides a good solution for estimating IMU motion on the arm.Moreover, the estimated * Corresponding author orientation is used to level the IMU signal and estimate the heading of human movement.
The pedestrian step detection and step length estimation are conducted through the levelled acceleration data.In conventional PDR models, there are many algorithms estimating the step length through pedestrian behaviors analysis and statistical experience.However, these algorithms require determining gain (or other parameters) before being applied to PDR.This gain remains constant throughout the entire PDR application.However, this gain has been shown to be very sensitive to user dynamics and difficult to be globally adjusted.The reinforcement learning methods thus are adopted to address this problem.We decompose step estimation into velocity estimation and propose a novel pre-trained velocity regression model that can be labelled directly through the smartwatch's GNSS, which can not only predict speed but also accurately identify pedestrian stationary status.
By integrating the aforementioned enhancements, our PDR system can yield stable positioning results; however, it is susceptible to drift and necessitates periodic correction to preserve accuracy.Conversely, GNSS offers precise positioning without drift over time.Therefore, a PDR/GNSS integrated system is proposed based on the Extended Kalman Filter (EKF) that synergizes the benefits of GNSS and PDR techniques to overcome their respective limitations.Moreover, we integrate a Convolutional Neural Network (CNN) speed estimation model with static recognition capabilities to provide Zero Velocity Update (ZUPT) reducing system drift.
In summary, a novel GNSS/PDR integration system for smartwatches that synergistically combines a robust 9-DOF AHRS algorithm, an enhanced step detection scheme, and a Convolutional Neural Network (CNN) speed estimation model is introduced.The proposed integration system offers the potential to not only improve positioning accuracy but enhance navigation availability for smartwatch users.This research makes several significant contributions to the field of smartwatch navigation: 1. We propose an algorithm that is specifically designed for the consumer-grade IMU, magnetometer, and GNSS receiver that are built into Apple Watch.
2. Utilizing a reliable attitude estimation, the varied roll and pitch of the sensor caused by hand swing can be tracked, and the heading of human movement can be derived from the heading of the sensor.
3. A CNN model is adopted to predict the 1-D speed and provide ZUPT information, which replaces the previously used empirical model for step length estimation.This new approach offers improved accuracy and reliability.This system represents a significant advancement in smartwatch navigation technology, with potential applications in various fields, such as fitness tracking, healthcare, and emergency services.

RELATED WORKS
PDR is a popular technique for indoor and outdoor positioning, especially in scenarios where GNSS signals are weak or unavailable.PDR utilizes an IMU to estimate the user's position by measuring the acceleration and angular rate signals over time.PDR relies on motion sensors, such as accelerometers, gyroscopes and magnetometers to estimate the user's position by tracking their footsteps.PDR can be implemented on various devices, including smartphones, smartwatches, and footwear, with different motion sensor configurations and algorithms.In this literature review, we will discuss the latest advancements in PDR technology, including conventional PDR algorithms, deep learning-based PDR, and GNSS/PDR integration.We will also review the current challenges and limitations of smartwatch PDR and propose our solution.

Conventional PDR Algorithm
The conventional PDR algorithm typically operates within a horizontal plane (2D) and consists of four main components: step detection, step length estimation, user heading determination, and the final position estimation.The accelerometer measurements are utilized to detect pedestrian steps, and step length estimation.The step length can be estimated by using a variety of methods such as regression-based, biomechanical models, or empirical relationships.User heading determination is accomplished using gyroscope and magnetometer measurements, and accelerometer readings may also be incorporated (Tian et al., 2022).This information is then used to calculate the overall distance travelled and the pedestrian's estimated position.
where   ,   =       =   ℎ   = In order to estimate position, it is necessary to first determine the occurrence of a step.One common method for detecting a step is by employing a peak detection algorithm, which typically involves setting a threshold value for the magnitude of acceleration and a minimum step period.However, relying solely on peak detection methods can be easily influenced by fake peaks.
Consequently, research has emerged that incorporates peak detection and considers more complex pedestrian movements to provide more precise step detection results.Nevertheless, utilizing increasingly complex pedestrian models results in a greater number of parameters that must be considered, making it difficult to achieve generalization (Hancock et al., 2022).Therefore, a compromised algorithm that pairs peaks and valleys is proposed to improve step detection accuracy without the tuning of various parameters in this study.The identification of a step involves identifying the time interval between two consecutive peaks.
The calculation of step length is typically based on pedestrian dynamics, which considers the step frequency and gains adjustment (J.W. Kim et al., 2004;Weinberg, 2002).However, adjusting gains can be challenging and prone to estimation errors.To address this, some studies propose more generalized empirical algorithms that adjust parameters according to the pedestrian's height (Chen et al., 2011).Nevertheless, these algorithms may still not sufficiently describe the complex models of human movement and rely on accurate step frequency estimation.
Consequently, an increasing number of studies use deep learning methods to address the infrequent calculation of pedestrian movement.
Weinberg's step length model is defined as where  =     =      =    Kim's step length model is defined as ) , where ℎ =  ℎℎ f =

Deep Learning-Based PDR
In recent years, it has been shown that the complex dynamics of pedestrians can be described by applying deep learning and simply inputting raw IMU data without the need for additional parameter adjustments, which has become a popular solution.
Deep-learning-based activity recognition models have been developed to classify the placement of smartphones, addressing the limitation of flat placement (Shin et al., 2016).StepNet has also proposed a deep-learning-based regressor to predict step length, which provides a general solution to the challenging parameter problem of pedestrian step length (Klein & Asraf, 2020;Seethi & Bharti, 2020;Sui & Chang, 2021).RoNIN has introduced a deep-learning-based regressor for 2D velocity, which can model pedestrian dynamics completely and achieve remarkable results in predicting changes in pedestrian heading, with the provision of a pre-trained model trained on a large amount of data (Herath et al., 2020; K. S. Kim & Shin, 2021).However, the shaking motion of a smartwatch can lead to inaccurate heading prediction.To address this issue, we employ RoNIN's pre-trained model, reformatting it as a 1D speed regressor to describe pedestrian movement speed, while the pedestrian direction is calculated using the AHRS algorithm specifically designed for pedestrian motion.The heading of pedestrian can be derived and the IMU 6-axis signal can be leveled as the input image to the model.

GNSS/PDR Integration
GNSS/PDR fusion has emerged as a popular technique for both indoor and outdoor positioning.By combining the measurements from GNSS and PDR, this approach offers improved accuracy, availability, and reliability in position estimation.Extended Kalman Filter (EKF) and Factor Graph Optimization (FGO) are two popular algorithms for GNSS/PDR fusion (Jiang et al., 2022;Lan et al., 2015).EKF is a recursive algorithm that estimates the state of a system with a nonlinear model and measurement errors.
It has been widely used in GNSS/PDR fusion due to its simplicity and effectiveness in handling nonlinear models.FGO, on the other hand, is a graphical representation of a probabilistic model that is used for inference in Bayesian networks.In addition to GNSS and PDR, other measurements such as ZUPT (Xie et al., 2022), Bluetooth, and Wi-Fi can also be integrated into GNSS/PDR fusion to further improve positioning accuracy and reliability.ZUPT is a method that utilizes the knowledge of a stationary period to reset the velocity error in PDR.Bluetooth and Wi-Fi are wireless communication technologies that can provide additional range and positioning information.This study proposes a robust integration scheme for GNSS/PDRbased pedestrian positioning in complex environments.The scheme aims to provide reliable, continuous, and accurate positioning results.In terms of algorithm, the integration scheme employs EKF as the algorithmic framework takes into account the gait characteristics of pedestrians, and fuses GNSS positioning results.Furthermore, the scheme incorporates a CNN Speed model to provide ZUPT updates.The proposed scheme is designed for consumer-grade pedestrian positioning, and its effectiveness is evaluated through experiments in a complex outdoor environment.

METHODOLOGY
In this study, a novel algorithm for smartwatch navigation is presented.The proposed algorithm, as shown in Figure 1, includes heading estimation, step detection, a CNN speed model, and a proposed GNSS/PDR integration system.These components are designed to work together seamlessly and provide continuous and accurate pedestrian navigation.In the following sections, we will detail each component of the algorithm and explain how they contribute to the overall performance of the system.

Heading Estimation
A recent research study has conducted a comprehensive review and evaluation of several AHRS algorithms and subsequently proposed an alternative algorithm based on a 9-DOF IMU comprising accelerometers, gyroscopes, and a magnetometer.The new algorithm called the Versatile Quaternion-based Filter (VQF), has been demonstrated to be more accurate and robust than existing solutions (Laidig & Seel, 2022).Specifically, the VQF leverages a low-pass filtering strategy to eliminate updates from high dynamic accelerations, resulting in a stable estimation of roll and pitch.Furthermore, magnetic disturbance rejection techniques have been employed to prevent the distortion of heading correction.The VQF has been utilized in this study to level the IMU signal for step detection and training sequence generation for a CNN-Speed model.The estimated heading of the sensor has been determined using gyros-only (3-DOF), accelerometers-aided (6-DOF), and whole sensor fusion (9-DOF) modes.The improvement in the sensor fusion scheme on the heading has been depicted in Figure 2. As shown in the figure, incorporating a magnetometer into the sensor configuration can effectively suppress heading drift and maintain the approximate trajectory shape, in comparison to other sensor configurations.Consequently, the 9-DOF VQF algorithm has been adopted for subsequent integration.Meanwhile, the heading of VQF estimation may exhibit periodic waveform due to the left-right movement of the hand.This indicates that the heading transformation from the sensor frame to the body frame needs to 0 The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLVIII-1/W1-2023 12th International Symposium on Mobile Mapping Technology (MMT 2023), 24-26 May 2023, Padua, Italy be considered.Therefore, we propose filtering out the periodic variations caused by hand swinging to obtain the actual heading in the body frame.We utilize each step's heading for PDR calculation, as depicted in the lower section of Figure 3.

CNN Speed Model
Since the conventional empirical model has two major shortages: 1) The personal factors should be fine-tuned.
2) The algorithm depends on accurate step detection.Otherwise, the errors (step detection and step length) would be propagated to position error.
To address these issues, we propose a robust and independent CNN-Speed-Model method, which is based on a pre-trained Res-Net18 model of RoNIN (Herath et al., 2020).This method transforms levelled IMU 6 DOF data with a 1-second (50 Hz) window size into a sequence and uses it as input for the ResNet18.The output is modified to convert RoNIN's 2D velocity to 1D speed.Additionally, we use the GNSS speed of the smartwatch in open sky areas as the label.To improve the generalization and stability of the model, we employ data augmentation techniques such as heading rotation and random frame shifts as shown in Figure 4.
These techniques enhance the model's stability and make it more robust against variations in the input data.To evaluate the effectiveness of our proposed method, we collected outdoor walking data from four individuals for training.The testing results demonstrate that the CNN-Speed-Model method outperforms the conventional empirical model in terms of accuracy and stability, as shown in Figure 5. Additionally, by incorporating stationary states in the training data, the CNN-Speed-Model method can accurately predict the stationary state, enabling its application for zero-velocity update (ZUPT) correction.we can overcome these weaknesses and achieve better positioning in complex environments.Algorithmically, we propose a four-state EKF that utilizes the speed predicted by a CNN model and the heading change calculated by each step's heading as prediction input.The detailed process for integrating GNSS and PDR using EKF is as follows: As the prediction component of our algorithm, proposed PDR model utilizes the product of speed and sample interval to estimate the pedestrian's movement distance at each time step.
The change in heading, on the other hand, is only calculated after the completion of a step, as shown in Figure 6.Once the observations, which consist of GNSS positioning results or zero-velocity information, are obtained, the measurement equation can be formulated.
The accuracy of absolute positioning in the system is mainly determined by GNSS positioning, which can effectively suppress the error of the inertial sensor.Therefore, the GNSS measurements (position, speed) are utilized to update the state estimates.
ZUPT is an effective method for mitigating velocity error accumulation.The CNN speed model can provide reliable zerovelocity detection by a simple threshold.Therefore, we utilize the CNN-Speed-Model to detect the stationary state and construct a zero-velocity observation model:   Each method is computed and compared with the reference trajectory.We compared the pure PDR results using Chen's step length empirical model and the CNN speed model, followed by evaluating the fusion algorithm proposed in this paper.As Figure 9 and Figure 10 show, the red track shows the reference trajectory.The blue track represents the fusion solution of the proposed integration method.The green track and the earls green track illustrate the CNN and Empirical model PDR respectively.The white dots represent the GNSS positioning result.The travelled distances are about 670(m) and 1260(m) in route 1 and 2 respectively.Also, the GNSS outage duration time is 100(s) and 120(s) respectively.Our proposed fusion method achieved the best results among the others in both routes, with accuracies of 4 meters and 3 meters, respectively.Moreover, when comparing only PDR accuracy, our PDR trajectories based on the proposed CNN-speed model were more accurate than those based on the empirical model, demonstrating better scale accuracy.In the state of the art, our proposed fusion method performed well in GNSS-denied or challenging environments and effectively reduced PDR drift.In conclusion, integrating CNN-PDR and GNSS provides availability and accuracy advantages.

CONCLUSION
The present study introduces a novel approach for integrating PDR/GNSS with a CNN-speed model to achieve accurate and continuous pedestrian navigation through the fusion of GNSS and IMU sensors.The article outlines the key technologies of the proposed scheme, which include leveraging a reliable attitude estimation, a parameter-free step detection method, an improved CNN-speed model, and a fusion algorithm of GNSS and PDR that includes zero-velocity detection.The accuracy of the proposed method was evaluated through actual experiments using an Apple Watch.The results of the kinematic experiments conducted in outdoor open environments and complex environments demonstrate that the proposed scheme can effectively combine the benefits of GNSS and PDR to achieve a pedestrian navigation position error of less than 5 m by the smartwatch.This system represents a significant technological advancement in smartwatch navigation and has potential applications in various fields, including fitness tracking, healthcare, and emergency services.

Figure 1 .
Figure 1.The proposed algorithm architecture

Figure 2 .
Figure 2. The comparison between 3D, 6D, 9D attitude-based PDR 3.2 Step Detection The levelled acceleration is utilized to detect pedestrian steps by recognizing the periodic signal.In fact, the vertical component (Down) of the acceleration without gravity is used to reflect the motion caused by steps.Since the smartwatch is installed only on one hand, a periodic waveform with a large and a small component can be observed in the vertical axis of the IMU with each left-right stride.We propose a parameter-free step detection algorithm based on vertical component acceleration.The crests and troughs of the signal are detected by slop change.A valid step should meet the following requirements: 1) Each valley must have a corresponding peak.2) The minimum interval of nearby steps conditional is used to remove incorrect steps.First, the lowfrequency noise is filtered out from the vertical acceleration signal, and then peaks and valleys are detected based on the aforementioned principles to identify a single step.During this process, higher amplitude crests are used to identify false crests.To estimate step length, an empirical model (Chen et al., 2011), which only considers the user height and step frequency are used as a comparison with the proposed CNN-Speed-Model.Figure 3 demonstrates the result of step detection (upper) and step length estimation (middle).

Figure 3 .
Figure 3.An illustration of the proposed PDR

Figure 6 .
Figure 6.The schematic diagram of proposed PDR with speed and heading change input

Figure 7 .
Figure 7.The testing result of the proposed CNN-Speed-Model predictionThe experimental setup, as shown in Figure8, involved collecting sensor measurements from the smartwatch (Apple Watch), including the accelerometer, gyroscope, magnetometer, and GNSS.The PwrPak7D-E2 in backpack mode with an antenna placed on the shoulder was used as the reference.The Novatel raw measurements were processed with the Inertial Explorer (IE) software from Novatel, resulting in post-processing position errors of less than 5 cm.In this paper, there are two experiment routes being tested.The two routes represent common outdoor GNSS-challenging environments, where Route 1 represents a high-rise urban environment and Route 2 represents a tree-covered shaded environment.

Figure 8 .
Figure 8. Configuration of the navigation sensor on a pedestrian.The backpack contains our reference system (PwrPak7D-E2)

Figure 9 .
Figure 9.The comparison of trajectory between each method in experiment route 1.

Figure 10 .
Figure 10.The comparison of trajectory between each method in experiment route 2.

Table 1 .
The horizontal accuracy of different methods in both experiment routes.

Table 1
describes the accuracy of each method in different routes.