Data model is the basis of all the functions of geographic information system. As the land use structure has become more and more complicated in cities, the traditional geometric model are not able to satisfy the increasing demands of precise urban form recognition and space management. Against the shortcomings, we propose to construct a multilevel semantic model for better description of the spatial composition of each building and the relationships among different buildings. Based on the 3D surface models constructed with photogrammetry and remote sensing methods, the semantic model is generated to depict the urban building space hierarchically, from stories, buildings, subareas to the entire city zone. On the one hand, to figure out the stories of each building, the geometric 3D model is segmented vertically with reference to the compositional structures and spatial distributions of the functional features on the surfaces. On the other hand, to determine the subareas of the city, the buildings are grouped into meaningful clusters according to their geometric shape characteristics. Experiments were conducted on a small district with both commercial and residential buildings, and the effectiveness of the proposed approach and usage of the semantic model were demonstrated.