The human hand is amazingly complex. With 34 muscles, more than 30 tendons, and 27 bones, hands can grasp objects, express emotion, create works of art, and accomplish many of the tasks that drive economic progress and help care for those in need.

Imagine the possibilities if we had more of them.

Boston Consulting Group projects that by 2030, the US GDP will take a $290 billion annual hit because of worker shortages in health and social services. Similarly, Deloitte anticipates 2.1 million unfilled US manufacturing jobs by 2030, which could lead to $1 trillion in economic losses.

Fortunately, help is on the way. Northwestern engineers in the new US National Science Foundation Human AugmentatioN via Dexterity Engineering Research Center (HAND ERC) are working to mitigate this impending labor shortfall by developing dexterous, intelligent robot hands with the ability to assist humans with manufacturing, caregiving, handling precious or dangerous materials, and more.

“We’re working toward the day, probably a decade down the road, when a small manufacturing firm can hire people who know its business, are not programmers, but can buy a manufacturing robot and put it to use very quickly and quite differently from the way robots are used today,” says J. Edward Colgate, Walter P. Murphy Professor of Mechanical Engineering at Northwestern Engineering and director of HAND ERC.

Leading a Transformative Era

We’re entering a transformative era where robotics will not only enhance productivity but also will fundamentally change how humans and machines collaborate. HAND ERC represents a bold step toward that future, positioning Northwestern Engineering at the forefront of dexterous robotics research and innovation. Christopher Schuh Dean, McCormick School of Engineering

Funded with $26 million over five years from the NSF—with the ability to renew for another $26 million for an additional five years—HAND ERC is the first Northwestern-led NSF engineering research center. Its core partners include Carnegie Mellon University, Florida A&M University, Texas A&M University, and the Massachusetts Institute of Technology, with additional faculty support from Syracuse University and the University of Wisconsin-Madison.

Leveraging state-of-the-art research testbeds from around the country, the multi-institutional collaboration will advance research in dexterity (versatile robotic manipulation that rivals the human hand) and augmentation (new interfaces enabled by AI to democratize access to robotics) to support a future that accelerates worker and company productivity alike.

“We’re entering a transformative era where robotics will not only enhance productivity but also will fundamentally change how humans and machines collaborate,” says Northwestern Engineering Dean Christopher Schuh. “HAND ERC represents a bold step toward that future, positioning Northwestern Engineering at the forefront of dexterous robotics research and innovation.”

A History of Robotics Innovation at Northwestern

Embodied AI

HAND ERC’s research advances Northwestern Engineering’s leadership in embodied AI. A strategic priority for the McCormick School of Engineering, such research leverages the school’s strengths in robotics, AI and machine learning, and sensor networks to maximize AI’s impact in the physical world. Embodied AI enables the development of intelligent mechanical systems for use in complex networks such as the power grid, autonomous vehicles, smart factories, and, yes, dexterous, intelligent robot hands.

HAND ERC is the latest chapter in the story of robotics innovation at Northwestern Engineering. It began in the 1950s, when Northwestern Engineering professor Dick Hartenberg and PhD student Jacques Denavit created a way to represent mathematically how mechanisms move, developing what became the widely used Denavit–Hartenberg parameters for the kinematics of robots. More recently, in 2019, the school launched the Center for Robotics and Biosystems, a multidisciplinary research center at the intersection of robotic and biological systems.

Colgate’s pioneering work in the field began in the early 1990s when he and fellow mechanical engineering professor Michael Peshkin collaborated with General Motors to develop a new type of robot that could augment worker performance on factory floors safely. Their invention of collaborative robots, known as “cobots,” allowed human workers to operate in concert with machines to manipulate heavy car parts, preserving valuable worker independence and empowerment while the robots performed the heavy lifting.

Cobots transformed the robotics field, ushering in a new paradigm of how humans could work collaboratively with machines. “Cobots were tools for helping people do their jobs better, faster, and more safely, whether through direct assistance, like what you experience in power steering, or completely taking over a task, like a car’s cruise control,” Colgate says. “The same motivation that pushed us to develop cobots drives us today with HAND ERC. How can we leverage our understanding of robotics to improve people’s lives?”

A Vision for the Future

Shifting attention from the large-format work of cobots, HAND ERC will focus on the fine motor skills of hands with several interwoven goals for revolutionizing robotics:

Make It Versatile: Advancing robot hand hardware

At the core of HAND ERC’s work will be expanding robot dexterity capabilities. Using advances in tactile sensing, soft actuators, and improved tendons, center researchers will design robots that can manipulate objects—assemble, twist, tear, and stack—instead of the robot simply moving from place to place.

Manipulation and movement might seem like similar challenges, but they’re worlds apart in complexity. Legged locomotion is relatively straightforward: pick up a foot, put it down; keep moving forward, don’t fall. Robotic hand movements are much more intricate. Each finger has multiple degrees of freedom, and when handling objects, every touchpoint changes the dynamics of movement.

Add tools or multiple objects into the mix and the task becomes exponentially harder. “The challenge is developing robot hands that can perform everything from fine in-hand manipulation, such as tying shoelaces or using chopsticks, to power grasps that can open a sealed jar,” says Kevin Lynch, professor of mechanical engineering and HAND ERC research director.

Make It Smart: Augmenting robot learning with embodied AI

Equipping robots with the mechanical and sensing capabilities to manipulate dexterously, however, is only one of the challenges HAND ERC researchers are working to overcome. Another is providing robots with human-like reflexes and control policies that map sensor feedback to hand motor controls. Although AI and machine learning have supported intelligent control of relatively simple robots, applying machine learning systems to control policies for robot hands is far more challenging due to the robot’s large number of actuators, joints, and tactile sensing channels. The amount of data needed to learn these control policies is significant, and it does not currently exist.

To overcome this challenge, HAND ERC researchers are developing a powerful system that allows an operator to intuitively teleoperate hand-equipped robot arms via a virtual reality headset, allowing the user to see what the robot sees, and haptic gloves, which let the user feel what the hands’ tactile sensors feel. Data collected during teleoperated dexterous tasks can then be used to train the robot’s autonomous skills. In addition, the experimental data will be augmented by large-scale synthetic data from simulation.

These methods will help robots become better prepared to seamlessly integrate with human workers in a variety of settings, from manufacturing facilities to hospitals to homes. They will provide even more effective support the longer they operate in those environments and collect data.

“Easy-to-use AI-enabled robot hands will become facilitators of creativity,” Lynch says. “They will allow people to do things that we would not have imagined.”

Make It Accessible: Moving the technology out into society

Fifty years ago, owning a computer was only practical for the largest and richest companies. Today, computers are more affordable and used by nearly everyone.

Through HAND ERC, Colgate envisions a similar future for robotic hands—inexpensive, easy to operate, robust, durable, and massmanufacturable. “We have economists on our team,” Colgate says. “What we’ve learned is that when you drive productivity, you drive opportunity and wages and make things better for the economy and for people. That is the big vision.”

Colgate is confident in his collaborators. He is passionate about translating work from the lab to the marketplace and sees HAND ERC as an opportunity to make a major impact on countless fields while creating new jobs fit for the 21st century. “Translation is vitally important in robotics,” Colgate says. “We’ve worked hard to build a strong robotics community at Northwestern, one that is interested in how we can have the greatest impact and add value to the world.”

Assembling the Northwestern Team

To achieve HAND ERC’s mission to democratize access to robotic manipulation, the center has convened leading robotics minds from five core universities across the United States. Eight faculty from Northwestern Engineering will play pivotal roles advancing research and industry collaboration.

The Research Path to Robotic Hands

While robots already play an important role in manufacturing and can improve workers’ job quality, Northwestern Engineering researchers at HAND ERC believe the full potential of current models has been limited. Developing robotic hands as versatile and dexterous as human hands will enable robots to expand human capabilities and boost industry competitiveness.

HAND ERC’s fundamental research will lead to robots with dexterous and versatile hands, manual skills, and intuitive interfaces that anyone can learn to use. J. Edward Colgate Walter P. Murphy Professor of Mechanical Engineering, HAND ERC Director

“Rapid advances in AI have created an incredible opportunity to make robot manipulators accessible to small and medium enter-prises, people with motor impairments, and many others who might benefit,” Colgate says. “A huge challenge, however, is what we put at the end of the robot’s arm. Today’s two-jaw grippers are far too limited. HAND ERC’s fundamental research will lead to robots with dexterous and versatile hands, manual skills, and intuitive interfaces that anyone can learn to use.”

To achieve its goal, HAND ERC will focus on three research thrusts.

Testbeds

To evaluate its research advances on high-impact applications, such as manufacturing, food preparation, and caregiving, HAND ERC will maintain five testbeds throughout its partner ecosystem.

1: High-Mix Manufacturing

Institution: Berkshire Innovation Center

2: Assembly/Disassembly

Institution: Manufacturing Futures Institute

3: Assistance and Caregiving

Institution: Shirley Ryan AbilityLab

4: High-Consequence Material Handling

Institution: Texas A&M University

5: Dexterity Education

Institution: Florida A&M University

Rise of the Cobots

Nearly 30 years before Northwestern Engineering was selected to lead HAND ERC, Professors Ed Colgate and Michael Peshkin revolutionized human-robot collaboration with their invention of “cobots.”

Supported by a grant from General Motors, Colgate and Peshkin were tasked with developing a new robotics system to function in concert with—not separately from—human employees working on GM’s automobile assembly lines.

To understand the challenge, the team observed and worked with GM’s assembly line workers as they guided bulky, awkward parts, such as dashboards or heating and cooling units, into cabs along a moving assembly line. They learned that coaxing parts into tight spaces at odd angles sometimes left workers feeling tired, even in pain.

When Ed and I started at Northwestern, the primary goal of robotics research was autonomy—could robots do what people can do? We embarked into cobots looking instead for how robots and people could best work together, each bringing their own strengths.Michael Peshkin Allen K. and Johnnie Cordell Breed Senior Professor in Design and Professor of Mechanical Engineering

Colgate and Peshkin designed a safe robot that could work closely with humans and make their work easier. These cobots used computer-controlled pathways to support and guide a payload into its desired location smoothly while remaining under the immediate physical control of the worker.

Colgate and Peshkin translated their invention into a company called Cobotics, which was eventually purchased by the Stanley Works. Today, the term cobot is used generically for all collaborative robots.

Northwestern remains a hub for cobot innovation today.

In 2022, researchers from the Center for Robotics and Biosystems unveiled mobile cobots, or “mocobots.” These mobile manipulators act as assistants for humans, collaborating with each other and supporting a payload at multiple points of contact, rendering heavy objects weightless and easily maneuverable. The technology could help usher in a new era of human-robot interaction in environments such as warehouses, construction zones, manufacturing plants, and even Mars.

Advancing Collaborative Robotics Through Bio-Inspired Science

Since its founding in 2019, the Center for Robotics and Biosystems (CRB) has had one mission: develop robotic systems that complement and augment human behavior, not replace it.

We take pride in our interdisciplinary research, innovation, and collaboration.Kevin Lynch Professor of Mechanical Engineering, HAND ERC Research Director

Directed by Professor Kevin Lynch, the interdisciplinary center brings together faculty from mechanical engineering, biomedical engineering, and computer science as well as Northwestern’s Feinberg School of Medicine and the Shirley Ryan AbilityLab. Here, researchers are advancing the science and engineering of embodied AI; AI-supported sensors, actuators, and controllers are inspired by biological principles; and robotics and computational modeling come together to create breakthroughs in human and animal neuromechanics.

“We take pride in our interdisciplinary research, innovation, and collaboration,” Lynch says. “By integrating advances in autonomy, human-machine interaction, and bio-integrated technologies, we’re preparing the next generation of researchers and engineers to tackle the world’s most pressing challenges.”

The CRB emphasizes five areas of robotic research:

Translating Robotics Research

Northwestern Engineering robotics researchers aren’t content to keep their innovations as proofs of concept in their labs. They consider it part of their mission to help people—industrial workers, physical therapists, surgeons, consumers, and others—perform tasks and live better lives.

A few examples of faculty-founded startup companies and commercialization efforts based on robotics research at Northwestern include: