Localization of the rover is critical to support science and engineering operations in planetary rover missions, such as rover traverse planning and hazard avoidance. It is desirable for planetary rover to have visual localization capability with high degree of automation and quick turnaround time. In this research, we developed a visual localization method for lunar rover, which is capable of deriving accurate localization results from cross-site stereo images. Tie points are searched in correspondent areas predicted by initial localization results and determined by ASIFT matching algorithm. Accurate localization results are derived from bundle adjustment based on an image network constructed by the tie points. In order to investigate the performance of proposed method, theoretical accuracy analysis on is implemented by means of error propagation principles. Field experiments were conducted to verify the effectiveness of the proposed method in practical applications. Experiment results prove that the proposed method provides more accurate localization results (1 %~4 %) than dead-reckoning. After more validations and enhancements, the developed rover localization method has been successfully used in Chang'e-3 mission operations.