Rendering of a proposed apparatus for standardized testing of humanoid robot capabilities. Source: NIST and ChatGPT
As the global race to a commercially successful humanoid robot continues, the need to be able to evaluate them is growing. The National Institute for Standards and Technology, or NIST, last month proposed a “comprehensive method to evaluate minimum expected physical capabilities for humanoid robots.”
“NIST is proposing the first standardized performance benchmark for humanoid robots since the 2015 DARPA Robotics Challenge,” noted Aaron Prather, director of the Robotics & Autonomous Systems Program at ASTM International, on LinkedIn. “In a decade that’s seen [Tesla‘s] Optimus, Figure, Agility, Apptronik, Unitree, and a dozen other humanoid platforms attract billions in investment, there is still no agreed-upon way to measure what any of them can actually do. Marketing videos have filled the gap.”
Bethesda, Md.-based NIST’s Intelligent Systems Division said it built the proposal on its previous collaboration with the Defense Advanced Research Projects Agency (DARPA) to evaluate humanoid capabilities across industry and academia and to provide direction for future development. The institute is part of the U.S. Department of Commerce.
Earlier this month, Fraunhofer IPA shared its own benchmark for humanoid safety and development with six criteria.
NIST to test humanoid locomotion and manipulation
NIST described its proposed baseline performance benchmark as “a low-footprint set of locomotion and manipulation tasks.” They use previously defined and standardized test methods and performance metrics.
The tasks are intended to represent the minimum capabilities for commercially available humanoid robots in industrial, household, healthcare, or other applications. NIST said the results will establish capability measurements for the current industry-leading robots, as well as a common set of tasks for researchers and manufacturers.
The organization added that the tasks will provide a baseline for performance while also testing:
- Domain-agnostic, basic humanoid robot mobility and manipulation/dexterity capabilities
- Coordinated capabilities combining locomotion and manipulation tasks
- Whole-body awareness and control through confined-space manipulation tasks
- Minimal reasoning, task and scene understanding, and decision making
Benchmarking process the result of collaboration
NIST said it has designed the baseline benchmark apparatus in collaboration with industry and the research community. It is seeking participants as it develops a consensus on what tasks should be included.
The institute plans to build a limited number of these testing apparatuses for free distribution to U.S. humanoid robot manufacturers and established regional testing facilities. NIST also plans to publish the designs and 3D models of the apparatus for use as a physical and/or virtual testbed for robot training and control development.
Robot manufacturers that choose to test their robots could receive an apparatus (pending availability) to run their own tests, or they can test their robots at NIST or a participating facility. The humanoid makers and NIST will collect the test results under pre-approved data-sharing agreements to protect intellectual property and any attribution if needed.
NIST plans to aggregate results to showcase the state of the art of humanoid robot capabilities.
Considerations for prospective participants
- Do these tasks sufficiently exercise minimum humanoid capabilities?
- Are there tasks that would better exercise whole-body-control and/or loco-manipulation?
- What additional constraints would you need for testing your robots in such a suite of tasks?
- Are you willing to help design the benchmark and/or have your robots tested?
- Are you interested in being a participating test facility?
To learn more or to participate, NIST has a contact form. Interested parties can also contact Dr. Benjamin Beiter at [email protected] and Dr. Kamel Saidi at [email protected].