I wanted to see if I could build a relatively affordable camera based switch that could be triggered off of a facial movement without needing the user to wear anything and without too much calibration. As a first demo I setup an affordable SBC with a webcam and running some Python code to detect when I open my mouth long enough and respond. This is treated as a switch activation. In this demo that activation controls an audio based menu controlling appliances in the room. Below is a video of it in action.

It triggers without me producing any sound. I just need to open my mouth long enough to confirm an intentional activation. This is all running on a cheap SBC (ASUS Tinker Board). It could also run on a Raspberry Pi which is even cheaper, but it’s less powerful and a little less responsive. It’s tolerant of the user not looking directly towards the camera. The menu and outputs could be configured to control any number of functions on multiple devices.  Alternately instead of a menu, the user could indicate their intention to the system with a pattern such as one or two Morse code letters. Also alternately the detection can be directly output as a switch output for use with any switch adapted device.

Below are some pictures of the prototype:

Here’s an alternate version including a heat sink fan and a small USB speaker:

The code I wrote leverages OpenCV and Dlib for the facial landmark detection. Given the estimated position of landmarks such as the standard 68 used in many datasets (pictured below) you can infer the current state of the user’s mouth, eyes, etc.

Once a detection is made then the appropriate action can be taken. This type of setup can give a person with various disabilities more control over their environment even if they are not able to speak recognizably and even if they would find other more conventional switches too difficult or uncomfortable to use.