Translation of Sri Lankan sign language to text using hand keypoints and image processing

Abstract

Sri Lankan sign language (SSL) is a visual-gestural language used by Sri Lankan deaf community for communication. Hearing-impaired people face communication problems due to difficulty in understanding SSL by others. SSL to Sinhala text interpreting technology helps to fill up this communication gap because Sinhala is understandable to the majority of people. Hand gesture recognition can be achieved by using either vision-based or sensor-based approaches. Vision-based approaches use images/videos captured from cameras and are simple and low cost. Sensor-based approaches need complex hardware so are costly. Skeletal based SSL recognition approaches have shown higher accuracy compared to shape-based methods. Scale Invariant Feature Transform (SIFT) performs better as a keypoint extractor robust to scale and is used in the research to develop a vision-based SSL to Sinhala text translation model. Images of 20 static SSL gestures were collected using a web camera as the dataset required for the training. A Support vector machine classifier with SIFT as the feature detector was used in the methodology and reached an accuracy of 70%. The accuracy varied with multi-color backgrounds and the influence of light intensity was not considered for the study. The proposed low-cost model showed stable results with varying distances to the camera compared to some previous research.