This paper discusses the challenges arising if SAR and optical imagery shall be fused for stereogrammetric 3D analysis of urban areas. In this context, a concept for SAR and optical data fusion is presented, which is meant to enable the reconstruction of urban topography independent of the type of the available data. This fusion is modelled in a voxelized object space, from which 3D hypotheses are projected into the available datasets. Among those hypotheses then the one showing the greatest SAR-optical similarity is chosen to be the reconstructed 3D point. Within first experiments, it is shown that the determination of similarity between high-resolution SAR and optical images is the major challenge within the framework of the proposed concept. After this challenge has been solved, the proposed method is expected to allow 3D reconstruction of urban areas from SAR-optical stereogrammetry for the first time. It is expected to be beneficial, e.g., for rapid mapping tasks in disaster situations where optical images may be available from geodata archives, but instantaneous data can only be provided by daylight- and weather-independent SAR sensors.