The Science of Generalizable Robot Manipulation: Perception, Control, and the Physics of Contact

Human manipulation is a dance of contact - carefully choreographed by physics, geometry, perception, and control. Despite extensive theories of contact physics and our advanced learning methods, achieving generalizable robot dexterity in the open world remains elusive. The challenge lies in the “hybrid” nature of contact: interactions are non-smooth and nonlinear, partially observed, and governed by frictional dynamics that are difficult to model analytically and difficult to learn from data alone.

I’ll illustrate these challenges briefly through the problem of extrinsic dexterity. Specifically, I’ll use the example of planar pivoting to highlight the underlying mathematical structure common to most dexterous manipulation tasks. Analytical methods exploit this structure to provide guarantees in controlled environments. However, they are often brittle in the face of real-world uncertainty. This motivates the need for methodologies that can handle the complexity of physical interaction without discarding the insights of physics.

To address this, I introduce a formal framework: “Tactile Intelligence” that integrates high-resolution perception with three distinct methodologies: Model-Predictive Control (MPC) to leverage local models for precise manipulation; Imitation Learning to distill human strategies into robust tactile-driven policies; and Reinforcement Learning to enable the discovery of complex, contact-rich behaviors via sim-to-real tactile transfer.

Speaker: Nima Fazeli, University of Michigan