From humans to robots with the SAHR project
The SAHR project is exploring how humans develop their exceptional dexterity, allowing them to adeptly use tools and apply precise amounts of force to create intricate crafts. By gaining insights from this research, the project aims to develop new learning and control algorithms for robots to replicate similar dexterity. Professor Aude Billard provides more insight into this research.
Today, most robots remain largely preprogrammed to function in a specific way, with manipulation skills limited to simple pick and place object handling. In contrast, humans can handle a wide range of objects, including unfamiliar ones, and exhibit remarkable dexterity in activities such as crafting or playing musical instruments. However, developing such skills requires years of training and relies on advanced cognitive functions that allow us to adapt to different situations and use our fingers in a highly dexterous manner. Aude Billard, Professor in the Learning Algorithms and Systems Laboratory at EPFL in Lausanne, explains that we hold a mental model of what our hands can do, known as a kinematic work space, which changes in size as we move our fingers closer together or further apart. This mental model allows us to manipulate objects in complex ways, such as opening up our hands to create space for an object to rotate, then closing our fingers to grasp it.
SAHR project
As the Principal Investigator of the SAHR (Skill Acquisition in Humans and Robots) project, Professor Billard is investigating how humans learn to manipulate objects dexterously and looking to apply the insights gained to enable robots to learn similar dexterous manipulation. This research involved following a group of apprentices at one of the oldest watchmaking schools in Switzerland over two years to see how they pick up the delicate manipulation skills required to insert tiny elements key to the functioning of the watch. Most amazing is the fact that apprentices receive only general feedback, but no explicit advice on how to place their hands or the tools to handle these small devices. “Understanding how one can
learn to decipher what it takes to achieve the task with so little feedback and in far fewer trials than what machine learning achieves currently is required for robots to act robustly in our daily world.”
The SAHR team is working to model the way that humans unconsciously acquire these skills, applying ideas from a theory called optimal control. This approach involves optimizing multiple factors simultaneously, such as the operator’s comfort and manipulability, in the context of watchmaking. Billard and colleagues find that the key difference between novice and expert watchmakers is their ability to balance these different costs. Experts are willing to tolerate more discomfort to achieve better manipulability, while novices prioritize their own comfort. They further model the way by which one can learn a better balance across these objectives to achieve the task.
Optimal control
The project team translates these insights into improved robot learning and control algorithms. Optimal control is applied to search for the ideal posture of the finger on the tool, taking into account the dimensions of the robot.
“Building a model of the reachability map - the amount of space that each finger has - plays an important role in this respect,” she says. ”but true dexterity is when we go beyond holding objects between the thumb and any of the other fingers.” One can hold objects in-between the fingers or across different phalanxes, which is quite handy for holding several objects in one hand, or to rotate a pen in the fingers. To achieve this, the reachability map is populated with all these different ways in which objects can be held within the fingers, and with a model of the hand’s workspace to avoid self-collisions. To pick up multiple objects in its hand, the robot rapidly computes all possible combinations,
until it finds the right way of holding these multiple objects, balancing discomfort and manipulability, while ensuring feasibility.
“Minimising the costs is much less important than feasibility when it comes to such a complex problem,” continues Professor Billard. This approach holds relevance beyond the realm of hand dexterity and can be applied to obstacle avoidance and full body control, so that a humanoid can bend down and pick up an object without hitting its knee, as well as enabling a robot to navigate its way through a dense crowd. Just as controlling free space is crucial when handling objects with the hands, understanding the dynamics of free space changes is critical to preventing collisions between the arms and knees as the body bends down, or the crowd becomes denser.
The SAHR team draws inspiration from the techniques used by watchmakers to detect subtle changes in force that indicate successful insertion or breakage of a screw. This approach is applied to teach robots how to interpret similar changes in force to manipulate objects skilfully, such as balancing a glass of water without prior knowledge of its contents.
Human and Robotics applications
The project’s work now extends to the study of the acquisition of surgical skills, with the added complexity that the objects are composed of highly deformable tissues, within the framework of a project funded by the Swiss National Science Foundation launched in 2022.
The SAHR project has endowed robots with unique skills to pick up objects never seen before, to hold multiple objects in hand, which holds relevance to different sectors of industry.
Most robotics applications don’t use all of the degrees of freedom in today’s bionic hands, an issue that Professor Billard is working to address. “The goal now is to provide a control system to the robotics community so that they can use all the available degrees of freedom,” she says. This would then allow a robot to take on more complex tasks than is currently possible. “We would like people to get the full dexterity that they used to have with their
SAHR Skill Acquisition in Humans and Robots Project Objectives
The SAHR project aims at gaining insights into the principles of human motor coordination mechanism, so as to guide the development of advanced robot learning and control algorithms.
Project Funding
This project is supported through an ERC Advanced Grant. Project ID: 741945, funded by the European Commission.
Project Partners
own hand from a prosthesis. We hope that our research will lead to improved control systems for future prostheses,” continues Professor Billard. “The problem is to develop prostheses with enough degrees of freedom. One difficulty there is that those prostheses must be light – and the more degrees of freedom, the more motors will need to be included and the heavier they will be.”
This is not an insurmountable problem however and Professor Billard is confident
The project is conducted by the Learning Algorithms and Systems Laboratory (LASA) at the Swiss Federal Institute of Technology Lausanne (EPFL).
Contact Details
Joanna Charlotte Erfani
Administrative Assistant EPFL-LASA
ME A3 495 Station 9 1015 Lausanne Switzerland
T: +41 21 6930939
E: joanna.erfani@epfl.ch W: https://www.epfl.ch/labs/lasa/
that it will be resolved in the coming years, while researchers are continuing to modify and improve the algorithms. A start-up company called AICA Tech has been established to try and translate some of the research conducted in the lab into commercial development, but in terms of the SAHR project itself, at this stage Professor Billard aims to demonstrate the technology and show that robots can adapt to unexpected circumstances. “We want to demonstrate a robot moving in a cocktail party-type environment and avoiding obstacles and hazards. This would show that we can compute an extremely complex task in milli-seconds,” she says.
Prof. Aude Billard is Head of the LASA laboratory in the School of Engineering at EPFL in Lausanne. She holds a B.Sc and M.Sc. in Physics from EPFL (1995) and a Ph.D. in Artificial Intelligence (1998) from the University of Edinburgh. Her research spans the fields of machine learning and robotics.
Dexterity has traditionally been an exclusive ability of humans, but it is now time for robots to catch up and acquire this skill.Prof. Aude Billard