Image segmentation is an essential task in many computer vision applications such as object detection and recognition, object tracking, image classification, 3D reconstruction. Most of the current techniques utilise the colour or grayscale information of an image without considering the camera geometry. In this paper, a method is proposed to utilise the camera relative orientation of a pair of images to find a reliable object segmentation. The inputs to the method are a rectified image pair and a disparity map which could be computed from the rectified image pair, the disparity map is used to determine a set of local homographies between planar surfaces in the two images. The planar surfaces are corresponding to image segments despite the inconsistency of the RGB information. Homography based segmentation alone is not reliable due to possible noise in the disparity map and existence of non-planar objects in the scene. Therefore, an RGB technique is used as a complementary approach to enhance the segmentation result. Two colour-based segmentation techniques are used here, the first is the colour edge detector, and the second is Grabcut. Experimental results show the although the colour edge detector is a simpler algorithm than Grabcut, it does not include noisy data in the segmentation results. the This useful for 3D reconstruction, as it is preferable to exclude noisy areas like the sky and window glass. The outcome of the proposed segmentation algorithm is an object-based segmentation of the pair of images as well as a segmented disparity map.