Attitude estimation is central to a wide range of applications such as robotics, virtual reality and mobile smart devices. With the development of sensor technologies, these application devices are often equipped with gyroscopes and depth sensors. In this paper, we propose a novel method to fuse gyroscope and depth information for drift-free and robust attitude estimation in structured indoor applications. Our method relies on the depth information and the Manhattan world assumption to estimate the absolute orientation, which is then fused to correct the accumulated error of the gyroscope-determined attitude. We first utilize the mean shift algorithm on the unit sphere to align the surface normals from the depth measurements with the orthogonal planar structures of the Manhattan world. Therefore, the orientation estimates are drift-free and absolute with respect to the Manhattan world. We then fuse the orientation estimates with the gyroscope measurements in an error-state Kalman filter manner to further improve the attitude estimation accuracy and robustness. We validate the performance of our method on public datasets, demonstrating the robustness and accuracy of the method for attitude estimation.