Auslan Glove
A tool that uses computer vision and machine learning to recognise Australian Sign Language, translate it to written or spoken text and help people learn Auslan.
- Program
- MedTech
- Status
- Research
- Project lead
- Aileen Nguyen
- Stack
- MediaPipe, TouchDesigner
Project motivation
The idea began with a gap in the medical industry: interactive learning devices for Auslan. Many people with deaf family or friends look for ways to learn sign language, but current learning apps are multiple-choice, matching a recorded sign to the correct translation. They never analyse how accurately the user physically signs, so learners can't know whether their signs would be understood in the real world.
Method
The team uses MediaPipe in TouchDesigner to program each sign it wants to teach, with appropriate leniency for accuracy. If the user signs within tolerance, the program confirms it. If joints or movement fall outside tolerance, the user gets feedback and tries again until the sign is correct.
The current phase is a computer-vision prototype using MediaPipe hand-landmark tracking. It captures hand movement and groups it into generalised gestures, forming the foundation for processing data from the wearable sensing glove to come.
Future prospects
Future versions will use a physical glove with flex and touch sensors, paired with facial detection from a camera, for much higher accuracy. The glove would also let users practise communicating with family and friends while receiving real-time feedback.
