By offering fast and flexible solutions to create 3D models, handheld scanners are currently under the focus of many research activities in various 3D data processing fields. The real-time constraint is still challenging to achieve especially when it comes with concurrent needs, such as level of accuracy in the data acquisition, easiness of recovering from scanning interruptions or loop closure abilities... Among them, object/scene tracking quality is one of the most critical. In this work, we describe two issues that affects its performance, focusing on the robustness of the process. Specifically, we encounter such issues at to two different steps while moving through the working pipeline of a prototype handheld scanner, <i>i.e.</i> (1) the data pre-processing before running a pairwise alignment between a frame and the model representation, called key-frame, and (2) the temporal and quality criteria that govern key-frame updates. Our approach simply consists in substituting the use of a rigid (uniform) pattern for sampling, with a random distribution of points. We then implement an adaptive statistical method to select suitable timing steps for key-frames refreshing, comparing this solution with a previous static one based on regular updating rate. We run experiments on a dataset created with our own scanner and we show that the adoption of such alternatives reduce the number of tracking failures, consequently increasing the robustness of the system, improving the quality of the alignments and preserving the real-time behavior of the device.