This paper addresses the evaluation of algorithms reconstructing a watertight surface from a point cloud acquired on an open scene. The objective is to set a rigorous protocol measuring the quality of the reconstruction and to propose a quality metric that is informative with respect to the various qualities that such an algorithm should have, and in particular its capacity to interpolate and extrapolate accurately. Our approach aims at being more informative and rigorous than previous works on this topic. In addition, we use publicly available data and our implementation is open-source. We argue that a rigorous evaluation of surface reconstruction of open scenes needs to be performed on synthetic data where a perfect continuous ground truth surface is available, so we developed our own LiDAR simulator of which we give a description in the present paper.