Currently, remote sensing research is focused on developing an automated algorithm that can compete with empirical methods for mapping the contours of individual buildings. Despite facing numerous challenges related to suboptimal imaging conditions, diverse building architecture, and complex backgrounds, creating such an algorithm is essential for monitoring urban and natural areas, generating 3D city models, managing disasters, and estimating population density. Obtaining the polygonal boundary of a building and extracting a vectorized building mask as output for direct use is one of the current challenges in drawing building outlines. This work provides a comprehensive workflow for building extraction and improves their predicted area by regularization the boundaries of buildings. First, a convolutional neural network is used to train the binary semantic segmentation model and then regularization and vectorization processes are performed. The main difference from existing methods is a new regularization method based on compiling a neighborhood matrix for each point belonging to the “building” class. According to the experimental results, the algorithm shows high efficiency: IOU (intersection over union) = 91.2%, AP (average precision) = 64.1% and AR (average recall) = 75.1%, comparable to leading building boundary extraction methods such as PolyWorld.