Synthetic aperture radar interferometry (InSAR) has been an established method for long term large area monitoring. Since the launch of meter-resolution spaceborne SAR sensors, the InSAR community has shown that even individual buildings can be monitored in high level of detail. However, the current deformation analysis still remains at a primitive stage of pixel-wise motion parameter inversion and manual identification of the regions of interest. We are aiming at developing an automatic urban infrastructure monitoring approach by combining InSAR and the semantics derived from optical images, so that the deformation analysis can be done systematically in the semantic/object level. This paper explains how we transfer the semantic meaning derived from optical image to the InSAR point clouds, and hence different semantic classes in the InSAR point cloud can be automatically extracted and monitored. Examples on bridges and railway monitoring are demonstrated.