Proceedings of the
The Nineteenth International Conference on Computational Intelligence and Security (CIS 2023)
December 1 – 4, 2023, Haikou, China
Video-based Isolated Chinese Sign Language Recognition via MediaPipe and Time Series Classification
1Sichuan Provincial Key Laboratory of Philosophy and Social Science for Language Intelligence in Special Education, Leshan Normal University, Leshan Sichuan, China.
2School of Computer Science, Southwest Petroleum University, Chengdu Sichuan, China
3Key Laboratory of Internet Natural Language Processing of Sichuan Provincial Education Department, Leshan Normal University, Leshan Sichuan, China
ABSTRACT
Isolated sign language recognition (ISLR) is the foundation and prerequisite for building a large-scale continuous sign language recognition system. In this article, we introduce a new approach for ISLR which is based on time series classification. Firstly, we use MediaPipe which is a state-of-art tool for human skeleton estimation to extract human key points from sign language videos. Secondly, the distance between key points is utilized to transform sign language videos into multivariate time series. Next, we perform data cleaning and normalization processing on multivariate time series. Finally, relevant machine learning algorithms are used to classify multivariate time series to implement ISLR. Through experiments on 100 isolated word datasets collected in the article, it is shown that the method proposed in this article is effective to handle the ISLR task.
Keywords: Multivariate time series, Isolated sign language recognition, MediaPipe, Classification.
Download PDF
1Sichuan Provincial Key Laboratory of Philosophy and Social Science for Language Intelligence in Special Education, Leshan Normal University, Leshan Sichuan, China.
2School of Computer Science, Southwest Petroleum University, Chengdu Sichuan, China 3Key Laboratory of Internet Natural Language Processing of Sichuan Provincial Education Department, Leshan Normal University, Leshan Sichuan, China
ABSTRACT
Isolated sign language recognition (ISLR) is the foundation and prerequisite for building a large-scale continuous sign language recognition system. In this article, we introduce a new approach for ISLR which is based on time series classification. Firstly, we use MediaPipe which is a state-of-art tool for human skeleton estimation to extract human key points from sign language videos. Secondly, the distance between key points is utilized to transform sign language videos into multivariate time series. Next, we perform data cleaning and normalization processing on multivariate time series. Finally, relevant machine learning algorithms are used to classify multivariate time series to implement ISLR. Through experiments on 100 isolated word datasets collected in the article, it is shown that the method proposed in this article is effective to handle the ISLR task.
Keywords: Multivariate time series, Isolated sign language recognition, MediaPipe, Classification.
Download PDF