MC2 -- American Sign Language Character Recognition with Convolutional Neural Networks
Document Type
Event
Abstract
Introduction American Sign Language (ASL) is a form of communication used between those affected with hearing loss, and those they communicate with. This causes a major disconnect between people in the world who sign and don’t sign, making communication difficult between the hearing impaired and those who don’t know ASL. The importance of learning ASL in this ever-changing world is becoming greater [1]. With ever-increasing importance of this issue, our team is going to create an ASL character detection model to detect ASL characters and turn them to plain-text English for further usage. Related Work Fang et. al. developed DeepASL that detects ASL characters and sent the characters into a text-to-speech program [2]. Garcia and Viesca developed an ASL fingerspelling translator using Convolutional Neural Network [3], and they used a language model to use detected characters and predict a likely word. Development There is an online dataset, on Kaggle [4], of ASL characters being singed from several individuals with over 80,000 images, and we plan to use the dataset to train our deep learning model. We make use of image data augmentation technique to build an even larger dataset by zooming in/out, and shifting the original images. Our model makes use a Convolutional Neural Network (CNN) due to its high accuracy with image classification [5]. The model consists of an input layer, multiple layers of convolutional, and pooling layers, and a flatten layer and a dense layer for an output. In every convolutional and pooling layers, there is a dropout layer for regularization purpose. The model is a classifier that detects alphabet letters from hand images, and the sequence of letters from the model will be submitted to a cloud service to get suggested words. A demo system will be developed as a helpful tool in the world of ASL communication, creating a bridge between non-ASL signers and those who rely on ASL. Resources / Links [1] https://arbiteronline.com/2020/11/13/accessibility-to-all-the-importance-of-american-sign-language-in-everyday-life/ [2] https://www.egr.msu.edu/~mizhang/papers/2017_SenSys_DeepASL.pdf [3] http://cs231n.stanford.edu/reports/2016/pdfs/214_Report.pdf [4] https://www.kaggle.com/grassknoted/asl-alphabet [5] https://analyticsindiamag.com/convolutional-neural-network-image-classification-overview/
Keywords
Math, Computer Science, Informatics
MC2 -- American Sign Language Character Recognition with Convolutional Neural Networks
URC Greatroom
Introduction American Sign Language (ASL) is a form of communication used between those affected with hearing loss, and those they communicate with. This causes a major disconnect between people in the world who sign and don’t sign, making communication difficult between the hearing impaired and those who don’t know ASL. The importance of learning ASL in this ever-changing world is becoming greater [1]. With ever-increasing importance of this issue, our team is going to create an ASL character detection model to detect ASL characters and turn them to plain-text English for further usage. Related Work Fang et. al. developed DeepASL that detects ASL characters and sent the characters into a text-to-speech program [2]. Garcia and Viesca developed an ASL fingerspelling translator using Convolutional Neural Network [3], and they used a language model to use detected characters and predict a likely word. Development There is an online dataset, on Kaggle [4], of ASL characters being singed from several individuals with over 80,000 images, and we plan to use the dataset to train our deep learning model. We make use of image data augmentation technique to build an even larger dataset by zooming in/out, and shifting the original images. Our model makes use a Convolutional Neural Network (CNN) due to its high accuracy with image classification [5]. The model consists of an input layer, multiple layers of convolutional, and pooling layers, and a flatten layer and a dense layer for an output. In every convolutional and pooling layers, there is a dropout layer for regularization purpose. The model is a classifier that detects alphabet letters from hand images, and the sequence of letters from the model will be submitted to a cloud service to get suggested words. A demo system will be developed as a helpful tool in the world of ASL communication, creating a bridge between non-ASL signers and those who rely on ASL. Resources / Links [1] https://arbiteronline.com/2020/11/13/accessibility-to-all-the-importance-of-american-sign-language-in-everyday-life/ [2] https://www.egr.msu.edu/~mizhang/papers/2017_SenSys_DeepASL.pdf [3] http://cs231n.stanford.edu/reports/2016/pdfs/214_Report.pdf [4] https://www.kaggle.com/grassknoted/asl-alphabet [5] https://analyticsindiamag.com/convolutional-neural-network-image-classification-overview/