There are some problems in the visualization of 3D model of buildings at present, such as redundant texture data and large memory occupation of loading texture data, which pose challenges to the smooth loading of 3D building visualization. In this paper, we propose a method which using fractal quadtrees to simplify texture data and compress management. Firstly, this method uses the fractal self-similarity and the feature that the same texture has the same fractal dimension to screen all the textures, when the fractal dimension of multiple textures is within the threshold range. Radon transform will be performed on these textures to calculate their standard deviation and further simplify the texture. Secondly, we use fractal texture compression to create a multi-resolution texture data structure with quadtree structure, so that the building 3D model can achieve dynamic visualization of different scales. Finally, the parallel texture mapping of all faces of the 3D model of the building is implemented according to the texture calling rules. Experimental results demonstrate that the texture is managed by the method proposed in this paper, which shortens the loading time of texture and reduces the memory occupation by about 48.92%. Therefore, the method of texture simplification and compression proposed in this paper has a significant effect in the process of building 3D model visualization.