HMN 2026: How ‘Touch dreaming’ helps humanoid robots handle five tricky tasks with 90.9% higher success

Humanoid robots, robotic systems with a body structure that resembles that of humans, could soon assist humans with various tasks in household environments, manufacturing sites, hospitals and other settings. While some humanoid robots already perform well on basic manual tasks, they often struggle with more complex tasks or with missions that require them to reliably manipulate objects while moving in the space around them.

Researchers at Carnegie Mellon University (CMU) and the Bosch Center for AI recently developed a new artificial intelligence (AI)-based system that could improve the ability of humanoid robots to perform dexterous whole-body manipulation in contact-rich real-world settings. Their proposed AI model, dubbed Humanoid Transformer with Touch Dreaming (HTD), was introduced in a paper published on the arXiv pre-print server.

“Humans are remarkably good at manipulation because they combine whole-body dexterity, hand dexterity, and an intuitive understanding of hand-object interaction dynamics,” said Yaru Niu, first author of the paper and final year Ph.D. candidate at CMU Safe AI Lab.

“When we perform everyday tasks such as folding cloth, inserting an object, scooping, or carrying fragile items, we do not rely only on vision. We continuously coordinate our body posture, hand motions, and predictions about how contact will evolve.

“That human ability was a major inspiration for this paper. We wanted to ask a simple question: if versatile humanoid manipulation in the real world depends on this combination of full-body coordination, dexterous hands, and contact understanding, how can we build a robotic system that begins to capture the same ingredients?”

Improving dexterous manipulation in humanoid robots

When reviewing previous research efforts, Niu and his colleagues realized that most humanoid robots are still unable to coordinate their body movements and tackle more advanced object manipulation tasks. They thus set out to develop a new robotics platform that would enable the execution of whole-body movements and the dexterous manipulation of objects, leveraging data collected by distributed tactile sensors.

“We found that even when demonstrations are available, policies trained only with vision and proprioception often struggle in contact-rich tasks because contact is only partially observed and can change very quickly,” said Ding Zhao, senior author of the paper, Associate Professor at CMU and Director of CMU Safe AI Lab.

“These gaps motivated us to build both an integrated humanoid manipulation system and a learning method that explicitly encourages the policy to become more contact aware.”

To train their AI system, the researchers combined imitation learning with the prediction of future contact-related signals. HTD, the model they developed, was trained not only to predict future actions, but also to anticipate how touch and force will evolve during manipulation, a process that the team dubbed “touch dreaming.”

In addition to action chunks, the model predicts future hand-joint forces and tactile representations, encouraging the policy to become more aware of changing physical interactions while a robot is manipulating an object.

Rather than directly reconstructing raw tactile sensor readings, HTD predicts compact tactile latent representations generated by a slowly updated target network. This helps the model focus on meaningful contact patterns instead of noisy low-level sensor fluctuations, while also allowing touch-aware learning to remain integrated within a single-stage imitation learning framework.

“At a high level, our RL-based whole-body controller is designed to provide a stable execution backbone for humanoid manipulation,” explained Zhao.

“Rather than making a single controller solve every aspect of full-body motion at once, we decouple the problem. The lower-body controller is responsible for tracking base velocity, torso orientation, and height while staying balanced under the disturbances created by upper-body manipulation.

“Upper body pose targets are handled through inverse kinematics, and dexterous hand motion is handled through hand retargeting. This gives us a practical real-world execution stack for humanoid loco-manipulation.”

The AI model responsible for controlling the lower body of humanoid robots was trained in simulations employing a so-called teacher-student approach. In this simulated environment, a “teacher” agent has access to privileged information, while the student plans future actions based on real-world observations, including measurements of base angular velocity, gravity, the positions of lower-body joints and the speed with which they move.

“During training, we replay retargeted arm motions from the AMASS dataset so that the controller learns to remain stable under realistic upper-body disturbances,” said Niu and Zhao. “The final student controller is what we deploy on the real robot.”

Key advantages and initial tests

The system developed by the researchers has various advantages over other controllers introduced in the past. Most notably, it reduces interference between a robot maintaining its balance and its dexterous manipulation skills.

“The lower body focuses on stability-critical execution, while the upper body and hands are handled by separate modules that can more directly track wrist and hand targets,” said Niu and Zhao. “In our experiments, this controller also showed strong tracking performance compared with competitive baselines, and it served as the execution backbone during both teleoperation and autonomous deployment.”

The HTD system could allow robots to manipulate objects in more elaborate ways, all while standing or navigating in their surroundings. The system plans whole-body movements by analyzing a rich pool of information collected by tactile sensors.

“Many prior systems provide some of these ingredients, but few bring them together in one practical real-world platform,” said Niu.

“Our system combines an RL-based whole-body controller, upper-body inverse kinematics, dexterous hand retargeting, VR teleoperation, and distributed tactile sensing. That gives us a practical platform for collecting high-quality demonstrations for contact-rich humanoid manipulation.”

In addition to predicting optimal future actions for robots, the HTD controller estimates forces and tactile information that are crucial for completing complex manual tasks. Unlike many prior tactile-learning systems, HTD learns touch-aware representations directly within the imitation-learning policy, without requiring a separate tactile pre-training stage or an additional tactile world model during inference.

“Our experimental results were strong,” said Zhao. “Across five real-world tasks, namely insert-T, book organization, towel folding, cat litter scooping, and tea serving, HTD achieved a 90.9% relative improvement in average success rate over the stronger ACT baseline.

“Our ablations also showed that simply adding touch as an extra input is not enough. Predicting tactile signals in latent space was more effective than predicting raw tactile signals directly, yielding a 30% relative gain in success rate over raw tactile dreaming.”

Real-world applications and future research directions

In the future, the robotics platform and AI-based controller developed by the researchers could contribute to the advancement of humanoid robots, potentially facilitating their successful deployment in various real-world settings. For instance, it could improve the ability of these robots to tackle household chores, take on service roles or assist humans in stores, and complete tasks at industrial or manufacturing sites.

“The broader lesson of our work is that if we want humanoids to perform useful real-world work, they likely need not just better vision and motion generation, but also better contact-aware learning,” said Jonathan Francis, co-author of the paper, lead of the Robot Learning Lab at the Bosch Center for Artificial Intelligence and is Courtesy Faculty in the Robotics Institute at CMU.

Some of the code underlying the team’s system is open-source and can be accessed by other researchers on GitHub. Niu, Zhao, Francis and their colleagues are now planning new studies aimed at further improving their AI model’s representation-learning capabilities.

“In this paper, tactile latent prediction works as an auxiliary objective, and it works well empirically, but there are still important open questions about how to make the latent space more transferable, more physically interpretable, and potentially more useful for planning or test-time adaptation,” said Zhao.

“Right now, the tactile latent space is effective as a compact training target, but we would like to better understand what kinds of structure make it most useful for downstream robotic behavior. That includes questions such as how to shape the latent space so it captures contact dynamics more explicitly, how to make it generalize across tasks, and whether it can support stronger forms of reasoning beyond action imitation.”

As part of their next studies, the researchers also plan to further scale their learning framework and test it in experiments that entail human-robot collaboration. To do this, they would like to also allow it to draw from visual data and human demonstrations.

“Humans naturally demonstrate whole-body coordination, dexterous hand use, and predictive contact handling at a scale that is hard to match with robot data alone,” said Zhao. “A related goal is to make the pipeline generalize across different robot embodiments, especially robots equipped with different types of dexterous hands and tactile sensor layouts.”

In the longer term, Niu, Zhao, Francis and their colleagues would like to develop AI-based frameworks that can be easily applied to different robots with distinct body structures and sensor configurations. In addition, they would like to create systems that can be easily adapted to tackle a variety of complex manipulation tasks.

“Ultimately, the broader goal is to move from task-specific touch-aware humanoid policies toward more scalable and embodiment-robust humanoid manipulation systems that can learn from both human and robot experience,” added Zhao.

The content is provided for information purposes only.