Motion-Classification Using Cnn and Lstm

Abstract

Action recognition is an active research field in computer vision that has gained significant attention due to its practical applications. While deep learning approaches have shown promise, many existing methods fail to effectively capture spatiotemporal features from videos, mainly because they do not adequately consider the diversity of temporal scales. In this paper, we propose a novel approach, termed Long-Short-Term Spatiotemporal Features CNN (LSF CNN), to address this limitation. The LSF CNN consists of two subnetworks: the Long-Term Spatiotemporal Features Extraction Network (LT-Net) and the Short-Term Spatiotemporal Features Extraction Network (ST-Net). The LT-Net extracts long-term spatiotemporal features from stacked RGB images, while the ST-Net utilizes optical flow computed from adjacent frames to extract short-term spatiotemporal features. Additionally, we introduce a novel expression for the optical flow field, which has been empirically shown to outperform traditional methods in action recognition tasks. The spatiotemporal features from both scales are combined in a fully-connected layer and fed into a linear Support Vector Machine (SVM) for classification. By leveraging the two-stream architecture and the improved optical flow expression, our proposed approach enables comprehensive learning of deep features in both spatial and temporal domains. Extensive experiments conducted on the HMDB51 and UCF101 datasets validate the effectiveness of our approach, as it significantly improves action recognition accuracy by effectively incorporating long-short-term spatiotemporal information. This research contributes to advancing the state-of-the-art in human action recognition, opening up new possibilities for enhanced performance in various real-world applications. © 2023 IEEE.