EXPLAINING A DEEP SPATIOTEMPORAL LAND COVER CLASSIFIER WITH ATTENTION AND REDESCRIPTION MINING

Abstract. Deep learning-based land cover classifiers learnt from Satellite Image Time Series (SITS) are known to reach high performances. In order to explain, at least partly, the rationale leading to each one of their decisions, attention-based architectures have been proposed to automatically weight the importance of predefined data components in the classification process. Though generated for each decision separately, the informational content conveyed by such explanations can remain insufficient to end-users because of the complex nature of SITS. Moreover, getting a general perspective about the way a classifier works requires merging all explanations for each class and relating them to its mode of operation, which is not always straightforward. A preliminary and complementary approach for automatically identifying the data features detected by a pixel-wise deep spatiotemporal land cover classifier and explaining its behavior at the class level is therefore proposed in this paper. Classified pixels are first described using interpretable features coming under the form of data mining patterns. A redescription mining technique is then employed to automatically select, for each class, the features matching the different activation level configurations of the layer that is assumed to capture the aforementioned patterns. Experiments based on a Sentinel-2 time series and a deep spatiotemporal neural network implementing a channel-separated processing as well as a channel-based attention mechanism show the interest of such a combined approach.