Early detection of crop pest and disease is very critical for taking up suitable control measures to reduce the loss of economic yield. Coffee is an important commercial crop in India which is affected by pests and diseases every year resulting in major yield loss. White stem borer (<i>Xylotrechus quadripes</i>) is the most serious pest of coffee (<i>Arabica</i> sp.) in India causing substantial loss of yield every year. Detection of the infestation in its early stage is quite challenging. In this regard, image pattern recognition techniques offer cost effective and scalable solutions. An image library was created representing different stages of the plant infestation using camera/mobile devices. Our Convolutional Neural Network (CNN) models use these images of healthy and infested plants for early detection of white stem borer infestation. The overall methodology included image processing, machine learning, supervised transfer learning and unsupervised auto-encoding techniques to solve the problem of early detection and severity of the infestation. Using the Inception v3 transfer learning model, we obtained average accuracy of 85.5% which is quite encouraging with limited image datasets. We explore Unsupervised Autoencoder models, which can work with limited image datasets. In addition, statistical analysis of long-term climatic factors such as temperature, rainfall, humidity and luminescence is explored for reliable detection and diagnosis of the infestation. Based on the encouraging results, a mobile application is proposed for near real time monitoring of WSB infestation to help the coffee planter’s community.