12. Robot Control Using Hand Gestures
5 min read
In this last chapter of the series ROS 101, we are going to combine everything we have learned so far to finish one last challenge: moving the robot with our hand signs.
The idea is pretty simple and there is actually nothing new. In the last chapter, we used rqt_robot_steering, a simple GUI that ROS provides to drive mobile robots. It basically lets you change the linear velocity (the vertical slider) to move forward/backward and the angular velocity (the horizontal slider) to turn left/right. These velocity values are then packed into a Twist message and published to the topic in the box, here is /robot_diff_drive_controller/cmd_vel.
Now instead of using this GUI, we can convert the hand signs detection (from chapter 9) to have the same functionalities. The index finger pointing up/down is for moving forward/backward, the thumb is for turning left/right, and all fingers stretched means stop. All we need to do is write a script to convert these signs to velocities and publish the messages to the topic
If you want to change, add, or remove signs, please follow this section in chapter 9.
The graph below describes the workflow of the complete system (ellipses are nodes and rectangles are topics). Green parts are what we already implemented: a node
/my_camera reads images from a webcam and publishes them to the topic
/image_raw (from chapters 7 and 8); another Node
/hand_sign_recognition subscribes to
/image_raw, processes the images, and publishes results to the topic
/gesture/hand_sign (from chapter 9). We need to implement the blue part which is basically a node, let's call it
/hand_sign_control, that subscribes the
/gesture/hand_sign topic, converts the hand signs to control values, and publishes them to the topic
/robot_diff_drive_controller/cmd_vel. From this topic, the
/gazebo node (from chapters 10 and 11) will read the velocity values and drive the robot accordingly.
After getting the idea and the workflow, I believe the implementation should be quite straightforward. There are different ways to make it work and I will show you one of them. Let's create a new script named sign_to_controller.py in the folder ros_hand_gesture_recognition/src. This script will basically handle the activities in the blue part above. The source code is located here and I give below an explanation for the main parts of the code.
First, the class GestureController is created. In its
__init__, the node
gesture_subscriber and the
vel_publisher are initialized. The
gesture_subscriber executes the function
callback whenever it receives a String message from the topic
/gesture/hand_sign (this topic name is defined in the launch file and is read using the function
vel_publisher will publish a
Twist message to the topic
class GestureController: def __init__(self): rospy.init_node('hand_sign_control', anonymous=True) # Subscriber for subscribing the hand signs self.gesture_subcriber = rospy.Subscriber(rospy.get_param("hand_sign_recognition/publish_gesture_topic"), String, self.callback) # Publisher for publishing velocities self.vel_publisher = rospy.Publisher("/robot_diff_drive_controller/cmd_vel", Twist, queue_size=10) # Velocity message self.vel_msg = Twist() # Velocity in/decrements self.linear_vel = 0.01 #[m/s] self.angular_vel = 0.1 #[rad/s]
Twist, one type of ROS geometries messages, expresses velocity in free space broken into its linear and angular parts. Each part is one vector with 3 components: x, y, and z of type float. In our case, the robot is controlled by two values: the linear velocity along the X-axis for translation and the angular velocity along the Z-axis for rotation, so only the x element in the linear and the z element in the angular are the control signals. They are derived in the callback function.
The callback function is where the conversion and publishing happen. It first checks what the input gesture is. If the gesture is
Backward, the linear velocity represented by the x component of
vel_msg.linear is incremented or decremented respectively by
linear_vel which is set to 0.01 m/s in the
__init__. If the gesture is
Turn Right or
Turn Left, the angular velocity represented by the z component of
vel_msg.angular is incremented or decremented respectively by
angular_vel which is set to 0.1 rad/s in the
__init__. In Forward/Backward cases, this angular velocity is set to 0 to make the robot move without rotating at the same time. If the gesture is either
NONE means no sign detected), the linear and angular speeds are set to zero making the robot stop. Finally, the Twist message
vel_msg is published by
def callback(self, gesture): if gesture.data == "Forward": self.vel_msg.linear.x += self.linear_vel self.vel_msg.angular.z = 0. elif gesture.data == "Backward": self.vel_msg.linear.x -= self.linear_vel self.vel_msg.angular.z = 0. elif gesture.data == "Turn Right": self.vel_msg.angular.z -= self.angular_vel elif gesture.data == "Turn Left": self.vel_msg.angular.z += self.angular_vel elif gesture.data == "Stop" or gesture.data == "NONE" : self.vel_msg.linear.x = 0 self.vel_msg.angular.z = 0 self.vel_publisher.publish(self.vel_msg)
Let's launch them all!
There are 4 files required to be launched/executed: this sign_to_controller.py script, my_cam.launch from package my_cam, hand_sign.launch from package ros_hand_gesture_recognition and drive_robot.launch from package ros_mobile_robot. In order to save time, you can group the first 3 in one launch file using the
include tag. Let's call this file sign_control.launch and put it in the launch folder of ros_hand_gesture_recognition. Following is its content.
<launch> <include file="$(find my_cam)/launch/my_cam.launch" /> <include file="$(find ros_hand_gesture_recognition)/launch/hand_sign.launch" /> <arg name="publish_gesture_topic" default="/gesture/hand_sign"/> <arg name="control_topic" default="/robot_diff_drive_controller/cmd_vel"/> <node name="sign_to_controller" pkg="ros_hand_gesture_recognition" type="sign_to_controller.py" output="screen"> <param name="publish_gesture_topic" type="string" value="$(arg publish_gesture_topic)" /> <param name="control_topic" type="string" value="$(arg control_topic)" /> </node> </launch>
Now open 2 terminals. In the first one run:
roslaunch ros_hand_gesture_recognition sign_control.launch
In the second:
roslaunch ros_mobile_robot drive_robot.launch
After that, you can drive the robot using your hand. To make it more fun, you can add some obstacles with some boxes, cylinders and spheres in Gazebo as I did here:
This video shows you how it works:
I hope you enjoy it!
You have done a really great job finishing this series. Now you knew all the basics of ROS and even created your very first ROS application. From this point on you can, depending on your interests, start to explore other aspects and many awesome packages in ROS. I will put some links below for reference but there are many more out there that I am not aware of. Feel free to leave me a comment if you find any interesting ones. There will be more blog posts about making real robots, ROS2, AI, etc. here in Robodev, so remember to subscribe to get updated. Finally, I wish you keep having fun making robots. Happy learning!
Awesome robotic repositories on GitHub:
DIY Making real mobile robots:
Did you find this article valuable?
Support Trinh Nguyen by becoming a sponsor. Any amount is appreciated!