Tree plantations, characterized by large-scale cultivation of trees with high commercial values, often rely on accurate inventory data to improve their capacity. However, understanding tree plantations with different components on a large scale for growth prediction is still a tricky problem. In this paper, we harness the power of Unmanned Aerial Vehicle (UAV) Light Detection and Ranging (LiDAR) systems to acquire 3D point clouds of tree plantations and investigate the potential of deep learning segmentation for enhanced understanding of plantation UAV LiDAR point clouds, thereby promoting precision forest management. Two datasets from the same plantation without debris on the ground and with harvested debris were tested. Experimental results showed that we were able to process a plantation consisting of 300 trees in 2 min and achieve an overall accuracy of 95% segmentation for this plantation. This research demonstrates the feasibility of the deep learning method in segmenting large-scale tree plantation point clouds, which is able to speed up the inventory of tree plantations.