Building a game

After all the presentations, I finally returned to the part I was most excited about: actually building the ASL learning game. My first idea was to create a kind of typing game, where a word would appear on screen—like “apple” or “dog”—and the player would have to spell it out using ASL finger spelling. The concept was inspired by classic spelling games, but instead of using a keyboard, your hands would be the input. To build this, I started by sending the hand landmark coordinates from Python (using MediaPipe) to the browser through the WebSocket server I had set up earlier. These keypoints represented the hand’s current pose in real time and formed the raw data needed for gesture detection. On the browser side, I was already visualizing the keypoints with green circles from earlier tests, but now I wanted to build a layer on top that could handle logic for checking whether the correct ASL sign was being made.

For detection, I integrated the same pretrained ASL classification model I had tested earlier. The model would predict a letter from the webcam feed, and I sent that prediction along with the keypoint data to the browser. The browser then compared the detected letter to the next letter in the displayed word. If it matched, the game advanced to the next character. If not, it would wait until the right sign was made. It felt a little like a dance between real-time recognition and waiting for precise hand shapes—which was both satisfying and slightly frustrating to test. The responsiveness of the gesture detection really mattered here; even a slight delay or misclassification could throw off the rhythm of the game. But at its core, this experiment helped me begin thinking of ASL not just as a visual language, but as a form of interaction—and interaction as something that could be both educational and playful.