Sea Ice Drift (SID) is an important parameter in understanding the Arctic climate dynamics and in maintaining navigation safety for the Arctic waters. SID is typically derived from keypoints extracted and matched from Synthetic Aperture Radar (SAR) imagery and is represented as a grid-based field. However, in feature tracking, the widely used Oriented FAST and Rotated BRIEF (ORB) is inherently vulnerable to low contrast, speckle noise, and complex sea ice deformation in feature tracking. To address these limitations, we propose an SID estimation framework that replaces the conventional ORB with deep learning-based methods such as SuperGlue and Local Feature TRansformer (LoFTR). In addition, multi-polarization is applied to exploit complementary information across both the feature tracking and pattern matching stages. Under polarization integration, SuperGlue reduced speed and directional RMSE by 50.8% and 37.3%, respectively, compared to ORB, while LoFTR achieved the best performance with reductions of 70.0% and 41.0%. These results demonstrate that deep learning-based methods can effectively replace the conventional ORB approach for SID estimation in the Arctic environments.