# 9. Hand Gesture Recognition in ROS

You have done a great job so far by writing both the publisher and subscriber all by yourself. However, I hope you still remember one of the very important reasons we learn ROS: "**Don’t reinvent the wheel"**, meaning there are a lot of existing cool packages (or projects) which can be immediately deployed without much effort of implementation. In this chapter, you will learn how to clone and run an available package that detects hand signs on the published images from your webcam. So instead of just viewing images like in [chapter 8](https://robodev.blog/write-a-ros-subscriber), you can now use them to get really useful information. Let's get started!

# Hand Gesture Recognition

A system that can understand human hand gestures has enormous applications. In robotics, it helps, for example, convert gesture recognitions into control signals making a robot follow users' hands.

%[https://media.giphy.com/media/26AHwLqr8FKVZ9ykU/giphy.gif] 

With the recent developments in artificial intelligence (AI), integrating that system is easier than ever. In this chapter, I want to introduce to you the [Mediapipe Hands](https://google.github.io/mediapipe/solutions/hands.html) which is an open-source machine learning (ML) package from Google that can help track human hands and fingers using images. Basically what it does is combine two ML models: a palm detection model for detecting initial hand locations and a hand landmark model for precisely localizing the 21 3D hand-knuckle (see image below) coordinates inside the detected hand regions. It was trained with about 30k real labeled data and works in a [single-shot manner](https://arxiv.org/abs/1512.02325) which is very fast. A good introduction to this package can be found [here](https://google.github.io/mediapipe/solutions/hands.html).

![hand_landmarks.png](https://mediapipe.dev/images/mobile/hand_landmarks.png align="center")

Now we need to turn the detected key points from Mediapipe into meaningful actions, for instance from the hand landmark image above, our system should tell us the status is "Open" since all the fingers are stretched out. In order to do that, another model for classifying the actions is required. Luckily, there is already an existing program for that. This [hand-gesture-recognition-mediapipe](https://github.com/kinivi/hand-gesture-recognition-mediapipe) repository contains Python scripts that help user train and deploy models to recognize hand signs and finger gestures. I have used it myself and converted it into this [ROS package](https://github.com/TrinhNC/ros_hand_gesture_recognition) so that you can clone it directly. You're welcome!

# Installation

First, install the following dependencies using `pip3` or `conda`:

* Mediapipe 0.8.1
    
* OpenCV 3.4.2 or Later
    
* Tensorflow 2.3.0 or Later
    

For example, installing commands with `pip`:

```bash
pip3 install mediapipe
pip3 install opencv-python
pip3 install tensorflow
```

Then, clone this repository into the folder `catkin_ws/src`:

```bash
cd ~/catkin_ws/src
git clone https://github.com/TrinhNC/ros_hand_gesture_recognition.git
```

Build the package:

```bash
cd ~/catkin_ws
catkin build
```

# Run the package

After building successfully, you can run a demo. Remember to source the workspace by `source ~/catkin_ws/devel/setup.bash`. Open 2 terminals. In the first terminal run the image publisher (from [chapter 7](https://robodev.blog/write-a-ros-publisher) and make sure the webcam is connected to the virtual machine if you are using one):

```bash
roslaunch my_cam my_cam.launch
```

In the second terminal, run the hand pose recognition:

```bash
roslaunch ros_hand_gesture_recognition hand_sign.launch
```

The result should look like this:

![ros-mediapipe](https://user-images.githubusercontent.com/19979949/210186155-c21b0fb2-84ba-430c-94ab-b273f5f36c6c.gif align="center")

In the GIF above, **FPS** stands for Frame Per Second which is the number of images being processed per second, **Right** is the right hand and the words after that (Turn Right, Forward, etc.) are the labels that I assigned to specific hand signs. You can change these labels by following the section below.

# Train Hand Sign Recognition (Optional)

The current package can classify only six signs (classes) and I labeled them: Stop, Go, Forward, Backward, Turn Right, and Turn Left (see the image below) which will be converted to control signals to move a robot later in this series. If you want to change or add gestures, or you find out that my trained model does not perform very well with your hand, you can collect data and train it again by yourself.

![ros-hand-gesture-recognition](https://cdn.hashnode.com/res/hashnode/image/upload/v1678623209549/dd8313bb-8f8e-443e-bc9b-efe214495c50.jpeg align="center")

There are two [jupyter notebooks](https://jupyter.org/) included in the folder [src/notebooks](https://github.com/TrinhNC/ros_hand_gesture_recognition/tree/main/src/notebooks):

* [**keypoint\_classification\_EN.ipynb**](https://github.com/TrinhNC/ros_hand_gesture_recognition/blob/main/src/notebooks/keypoint_classification_EN.ipynb): a model training script for hand sign recognition.
    
* [**point\_history\_classification.ipynb**](https://github.com/TrinhNC/ros_hand_gesture_recognition/blob/main/src/notebooks/point_history_classification.ipynb): a model training script for finger gesture recognition (meaning the model after training can detect the movement of your fingers and not just a static sign like in the keypoint classification).
    

I used only the keypoint classification model in the current ROS package because it is enough for the application but you can feel free to adjust it to match yours.

In the example below, I will show you how to add one more sign to the detection. Let's say we want to add this sign✌️and name it "Hi". **Remember to end all the commands (using Ctrl + C) from the previous parts because this part is completely separate from ROS.**

You may need to install these extra libraries if you have not:

```python
pip3 install -U tf-nightly
pip install -U scikit-learn
pip3 install -U matplotlib
```

![hi_mediapipe](https://cdn.hashnode.com/res/hashnode/image/upload/v1672664044490/a621899e-2321-4311-92e8-4ac8f2ee79ba.png align="center")

First, open the [keypoint\_classifier\_label.csv](https://github.com/TrinhNC/ros_hand_gesture_recognition/blob/main/src/model/keypoint_classifier/keypoint_classifier_label.csv) in the folder *src/model/keypoint\_classifier.* Here you find all the labels (at the moment 6 classes) and you should add 'Hi' to the end like this:

```bash
Stop
Go
Forward
Backward
Turn Right
Turn Left
Hi
```

Next, you need to record data and append it to the file [keypoint.csv](https://github.com/TrinhNC/ros_hand_gesture_recognition/blob/main/src/model/keypoint_classifier/keypoint.csv) in the folder *src/model/keypoint\_classifier*. If you open this file, you will see it contains 6410 lines. The first number in each line is the class ID with respect to the list above, for example, "Go" has ID 0, "Stop" has ID 1, and so on. Then comes 42 numbers or 21 pairs of numbers which represent the coordinates of each keypoint (i.e. the hand knuckle) with respect to the origin which is the wrist. One thing to note is that the IDs in the image below are the key point IDs (0-20) and they are different from the class IDs (0-6).

![keypoint_mediapipe](https://cdn.hashnode.com/res/hashnode/image/upload/v1672665593202/620eaac8-0840-4b8f-9f85-365dc71bf447.png align="center")

In order to record data, run the script **app.py**:

```bash
python3 app.py
```

Press `k` on the keyboard and you should see the line `MODE: Logging Key Point` shows up. Then, use your right hand to make the target sign✌️visible. Press and hold the number 6 (class ID of "Hi") with your left hand. This will continuously append the new data to the file [keypoint.csv](https://github.com/TrinhNC/ros_hand_gesture_recognition/blob/main/src/model/keypoint_classifier/keypoint.csv) until you release the key. You can also try to press & release 6 immediately and check the file. It should have one new line at the end starting with the number 6 and a list of numbers that follow. Also, during the recording, remember to move your right hand to different positions to make the data varied.

![ros-mediapipe-record-dataset](https://user-images.githubusercontent.com/19979949/210239140-cd5998ca-5937-48f4-9f91-8a86ca10da40.gif align="center")

After recording for about 10-15 seconds, the data should be ready and you can stop the program. Open the notebook file [keypoint\_classification\_EN.ipynb](https://github.com/TrinhNC/ros_hand_gesture_recognition/blob/main/src/notebooks/keypoint_classification_EN.ipynb). Edit *dataset*, *model\_save\_path* and *tflite\_save\_path* to match your paths. Change `NUM_OF_CLASSES` to 7 instead of 6: `NUM_CLASSES = 7`. Then run the notebook from beginning to end. The training is executed in the cell \[13\] and takes around 2-3 minutes. After that, you can launch `my_cam.launch` and `hand_sign.launch` like in the Demo to see the result like below.

![](https://user-images.githubusercontent.com/19979949/210258993-de24fcb4-4e1c-44f6-b1c0-6088677c0691.gif align="center")