In this paper we present a novel approach for 3D reconstruction of point clouds based on single baseline Multi-Aspect InSAR (MASAR) data. The point clouds represent an intermediate result to achieve a comprehensive building reconstruction framework. The exact determination of scatterers based on SAR data is a non-trivial task since the optimal solution requires the knowledge of the number of scatterers within one range cell. In recent years many methods were proposed addressing this problem but most of them require multiple observations making them inapplicable to our task. We use a Probabilistic Graphical Model (PGM) to combine all aspects into a common framework exploiting all contradictions and redundancy in the data. The model is used iteratively together with local optimizations adjusting the hypothesis of the scatterer within one range cell to the corresponding observation. This makes it possible to find a global solution.