# 12. Robot Control Using Hand Gestures In this final chapter of the series [ROS 101](https://robodev.blog/series/ros101), we are going to combine everything we have learned so far to finish one last challenge: moving the robot with our hand signs. # The idea The idea is pretty simple and there is actually nothing new. In the [previous chapter](https://robodev.blog/simulate-a-mobile-robot-in-ros-part-2), we used **rqt\_robot\_steering**, a simple GUI that ROS provides to drive mobile robots. It basically lets you change the linear velocity (the vertical slider) to move forward/backward and the angular velocity (the horizontal slider) to turn left/right. These velocity values are then packed into a [Twist message](http://docs.ros.org/en/noetic/api/geometry_msgs/html/msg/Twist.html) and published to the topic in the box, here is **/robot\_diff\_drive\_controller/cmd\_vel**. ![rqt_robot_steering](https://cdn.hashnode.com/res/hashnode/image/upload/v1678057679052/01f4b5f5-85b5-42a0-ac75-0611f47bae78.png align="center") Now instead of using this GUI, we can convert the hand signs detection (from [chapter 9](https://robodev.blog/hand-gesture-recognition-in-ros)) to have the same functionalities. The index finger pointing up/down is for moving forward/backward, the thumb is for turning left/right, and all fingers stretched means stop. All we need is to write a script to convert these signs to velocity commands and publish them as messages to the topic `/robot_diff_drive_controller/cmd_vel`*.* ![hand-sign-robot-steering](https://cdn.hashnode.com/res/hashnode/image/upload/v1678573851195/60cca149-65d3-41c6-bc77-0219722d1a5d.png align="center") If you want to change, add, or remove signs, please follow [this section](https://robodev.blog/hand-gesture-recognition-in-ros#heading-train-hand-sign-recognition-optional) in chapter 9. # The workflow The graph below describes the workflow of the complete system (ellipses are nodes and rectangles are topics). Green parts are what we already implemented: a node `/my_camera` reads images from a webcam and publishes them to the topic `/image_raw` (from chapters [7](https://robodev.blog/write-a-ros-publisher) and [8](https://robodev.blog/write-a-ros-subscriber)); another Node `/hand_sign_recognition` subscribes to `/image_raw`, processes the images, and publishes results to the topic `/gesture/hand_sign` (from [chapter 9](https://robodev.blog/hand-gesture-recognition-in-ros)). We need to implement the blue part which is basically a node, let's call it `/hand_sign_control`, that subscribes the `/gesture/hand_sign` topic, converts the hand signs to control values, and publishes them to the topic `/robot_diff_drive_controller/cmd_vel`. From this topic, the `/gazebo` node (from chapters [10](https://robodev.blog/simulate-a-mobile-robot-in-ros-part-1) and [11](https://robodev.blog/simulate-a-mobile-robot-in-ros-part-2)) will read the velocity values and drive the robot accordingly. ![hand-sign-mobile-robot-control-workflow](https://cdn.hashnode.com/res/hashnode/image/upload/v1678573736267/631ed49f-0467-436e-85a3-dcb2aaf8cf63.png align="center") # The implementation After getting the idea and the workflow, I believe the implementation should be quite straightforward. There are different ways to make it work and I will show you one of them. Let's create a new script named **sign\_to\_controller.py** in the folder *ros\_hand\_gesture\_recognition/src.* This script will basically handle the activities in the blue part above. The source code is located [here](https://github.com/TrinhNC/ros_hand_gesture_recognition/blob/main/src/sign_to_controller.py) and I give below an explanation for the main parts of the code. First, the class GestureController is created. In its `__init__`, the node `hand_sign_control`, the `gesture_subscriber` and the `vel_publisher` are initialized. The `gesture_subscriber` executes the function `callback` whenever it receives a String message from the topic `/gesture/hand_sign` (this topic name is defined in the [launch file](https://github.com/TrinhNC/ros_hand_gesture_recognition/blob/main/launch/sign_control.launch) and is read using the function `rospy.get_param`). The `vel_publisher` will publish a `Twist` message to the topic `/robot_diff_drive_controller/cmd_vel`. ```python class GestureController: def __init__(self): rospy.init_node('hand_sign_control', anonymous=True) # Subscriber for subscribing the hand signs self.gesture_subcriber = rospy.Subscriber(rospy.get_param("hand_sign_recognition/publish_gesture_topic"), String, self.callback) # Publisher for publishing velocities self.vel_publisher = rospy.Publisher("/robot_diff_drive_controller/cmd_vel", Twist, queue_size=10) # Velocity message self.vel_msg = Twist() # Velocity in/decrements self.linear_vel = 0.01 #[m/s] self.angular_vel = 0.1 #[rad/s] ``` [Twist](http://docs.ros.org/en/noetic/api/geometry_msgs/html/msg/Twist.html), one type of ROS [geometries messages](http://docs.ros.org/en/noetic/api/geometry_msgs/html/index-msg.html), expresses velocity in free space broken into its *linear* and *angular* parts. Each part is one vector with 3 components: x, y, and z of type float. In our case, the robot is controlled by two values: the linear velocity along the X-axis for translation and the angular velocity along the Z-axis for rotation, so only the x element in the *linear* and the z element in the *angular* are the control signals. They are derived in the callback function. ![ros-twist-mobile-robot](https://cdn.hashnode.com/res/hashnode/image/upload/v1678614693478/11d894cb-6ae7-48a1-89df-66ae993cc072.png align="center") The callback function is where the conversion and publishing happen. It first checks what the input gesture is. If the gesture is `Forward` or `Backward`, the linear velocity represented by the x component of `vel_msg.linear` is incremented or decremented respectively by `linear_vel` which is set to 0.01 m/s in the `__init__`. If the gesture is `Turn Right` or `Turn Left`, the angular velocity represented by the z component of `vel_msg.angular` is incremented or decremented respectively by `angular_vel` which is set to 0.1 rad/s in the `__init__`. In Forward/Backward cases, this angular velocity is set to 0 to make the robot move without rotating at the same time. If the gesture is either `Stop` or `NONE` (`NONE` means no sign detected), the linear and angular speeds are set to zero making the robot stop. Finally, the Twist message `vel_msg` is published by `vel_publisher`. ```python def callback(self, gesture): if gesture.data == "Forward": self.vel_msg.linear.x += self.linear_vel self.vel_msg.angular.z = 0. elif gesture.data == "Backward": self.vel_msg.linear.x -= self.linear_vel self.vel_msg.angular.z = 0. elif gesture.data == "Turn Right": self.vel_msg.angular.z -= self.angular_vel elif gesture.data == "Turn Left": self.vel_msg.angular.z += self.angular_vel elif gesture.data == "Stop" or gesture.data == "NONE" : self.vel_msg.linear.x = 0 self.vel_msg.angular.z = 0 self.vel_publisher.publish(self.vel_msg) ``` # Let's launch them all! There are 4 files required to be launched/executed: this *sign\_to\_controller.py* script, *my\_cam.launch* from package my\_cam, *hand\_sign.launch* from package *ros\_hand\_gesture\_recognition* and *drive\_robot.launch* from package *ros\_mobile\_robot.* In order to save time, you can group the first 3 in one launch file using the `include` tag. Let's call this file [sign\_control.launch](https://github.com/TrinhNC/ros_hand_gesture_recognition/blob/main/launch/sign_control.launch) and put it in the launch folder of *ros\_hand\_gesture\_recognition.* Following is its content. ```xml ``` Now open 2 terminals. In the first one run: ```bash roslaunch ros_hand_gesture_recognition sign_control.launch ``` In the second: ```bash roslaunch ros_mobile_robot drive_robot.launch ``` After that, you can drive the robot using your hand. To make it more fun, you can add some obstacles with some boxes, cylinders and spheres in Gazebo as I did here: ![add-box-gazebo](https://cdn.hashnode.com/res/hashnode/image/upload/v1678655235688/4c785092-98fa-436b-bc46-451c59ead609.jpeg align="center") This video shows you how it works: %[https://www.youtube.com/watch?v=ZRMiATj_QH4&ab_channel=robodev] I hope you enjoy it! # Now what? You have done a really great job finishing this series. Now you've got to know all the basics of ROS and even created your very first ROS application. From this point on you can, depending on your interests, start to explore other aspects and many awesome packages in ROS. I will put some links below for reference but there are many more out there that I am not aware of. Feel free to leave me a comment if you find any interesting ones. I plan to make more blog posts about making real robots, ROS2, AI, etc. here in Robodev, so remember to subscribe to get updated. Finally, I wish you keep having fun making robots and happy learning! **Resources**: * Awesome robotic repositories on GitHub: * [https://github.com/shannon112/awesome-ros-mobile-robot](https://github.com/shannon112/awesome-ros-mobile-robot) * [https://github.com/jslee02/awesome-robotics-libraries](https://github.com/jslee02/awesome-robotics-libraries) * [https://github.com/fkromer/awesome-ros2](https://github.com/fkromer/awesome-ros2) * DIY Making real mobile robots: * [https://articulatedrobotics.xyz/](https://articulatedrobotics.xyz/) * [https://github.com/ros-mobile-robots/diffbot](https://github.com/ros-mobile-robots/diffbot)