Vision based obstacle detection using stereo images is an essential way for hazard avoidance and path planning in planetary rover missions. However, due to light condition changes and topographic relief, only partial or sparse three-dimensional points may be derived by image matching and triangulation reconstruction, which is not sufficient for recognizing obstacles. In this paper, we developed a strategy to detect obstacles using rover stereo images by combining both image grayscale information and sparse 3D point information. Experiments were carried out using stereo images captured by navigation cameras mounted on the Yutu rover of Chang’e-3 mission. Moreover, how obstacle localization accuracy affected by the parameters are analysed and discussed.