HMN 2026: How Robot hand approaches human-like dexterity with new visual-tactile training

Human hands are a wonder of nature and unmatched in the animal kingdom. They can twist caps, flick switches, handle tiny objects with ease, and perform thousands of tasks every day. Robot hands struggle to keep up. They typically miss the sense of touch, can’t move many fingers at once, and lose track of what they are holding when their fingers block their camera’s view. Scientists have now developed a smarter way to train a robot’s brain to give its hands human-like dexterity.

Teaching robots dexterity

To improve the way a multi-fingered robotic hand performs complex tasks, researchers from China developed a two-pronged approach using only a basic webcam and low-cost sensors. Their process is outlined in a paper published in the journal Science Robotics.

During the first stage, the robot’s AI brain was pretrained by watching a large library of videos of humans performing tasks using their bare hands and gloves. This taught the robot how visual information (what a hand looks like near an object) and tactile information (when a finger touches a surface) work together.

In the second phase, the robot practiced in a virtual simulation, repeating tasks and learning several skills at once rather than one by one. The only equipment the team used was a standard webcam and low-cost hardware, including basic sensors that signal touch or no touch.

One of the most fascinating aspects of their approach was creating a unit in the AI architecture that acts like a part of the human brain, blending sight and touch. This allowed the robot to keep track of an object even when its fingers were in the way.

Improved handling

The robot was tested on eight different tasks. Five of these were tasks it had practiced in the simulation (such as turning a bottle cap and sliding a lever), and the other three were new (including sharpening a pencil and unfastening a screw). It successfully completed seen tasks 85% of the time and succeeded most of the time with the unseen tasks. The team even tried to trick the robot by changing lighting conditions and swapping out sensors, but in both these situations, the robot kept on working.

“Visual-tactile pretraining from human demonstrations consistently led to superior performance: enhanced learning efficiency, a reduced sim-to-real transfer gap, more human-like manipulation behaviors, and improved generalization to novel objects, varying lighting conditions, and unseen tasks,” the team noted in their paper.

Teaching robots to combine sight and touch, as we do, ultimately allows them to learn faster. Future research will aim to improve how they handle objects by sensing how tightly they are gripping them.

The content is provided for information purposes only.