How Do You Design a New Robot Test Method?

The National Institute of Standards and Technology (NIST) has a long and distinguished history of developing standard robot test methods for ground, air, and water based robot systems.

No alt text provided for this image

NIST works collaboratively with public safety agencies from around the world and generously shares their work for the purpose of helping to make the world a safer place.

Having worked with NIST on air, ground, and water based test methods for many years, I jumped at the opportunity this past year to help them explore how test methods might look for a new class of ground robots that use legs for mobility in place of wheels and tracks. It was rewarding to be able to provide this feedback, given what NIST provides in support of public safety agencies around the world.

The new mobility capabilities of these emerging systems will have a dramatic effect on both business and public safety. But how do you develop new test methods for something that operates so differently from current systems?

A new test method must do several things. It must have an indentified focus which in this case is the systems mobility and ability to sense its surroundings. Other requirements for NIST test methods include the following criteria: measurability (evaluate a robots function), the testing must be replicable, and the methods must be as efficient and affordable to ensure it can be used as widely as possible.

The test methods developed here used plastic crates and wooden boards in a simple setup to test the systems, proving that effective test methods can be easily replicated in a cost effective way.

The first trials proved very effective in evaluating not only the robot but as a means to develop new operator training too. They will be the start of a continous series of adjustments and modifications as our knowledge of these systems and their capabilities continues to expand.

Lastly, I would be remiss to not thank RMUS Canada for all the support on this project.

To learn more about NIST Standard Robot Test methods go to their web page.

Below are photos of the trials from a year ago and a recent set up of a course

No alt text provided for this image
No alt text provided for this image
No alt text provided for this image

Setting up the first course

Putting the robot through its paces

ICOR’s Purple Gear award

ICOR Technologies presents the Purple Gear award to customers show robots have been involved in high risk or life threatening situations

Recipients to date have included :

the Saskatoon police whose ICOR robot suffered major damage from a sledgehammer during the unit’s attempt to communicate with a person barricaded in their home .

New Brunswick RCMP with two ICOR robots one of which was damaged by fragmentation from a pipe bomb during a render safe procedure and the second while locating an armed and barricaded subject inside a residence

CETA is proud to recognize all recipients of the award

For details contact

What is the NIST Response Robot Method?

The Need for Response Robot Standards.

DHS‐NIST‐ASTM International Standard Test Methods for Response Robots   NIST has been developing the measurements and standards infrastructure necessary to evaluate robotic capabilities for public safety emergency responders, military organizations, and other critical national needs. NIST leads an international collaboration that has generated more than 50 test methods for robotic ground systems, aquatic systems, and micro aerial systems (FAA Group I under 2 kg (4.4 lbs)). These test methods measure robot maneuvering, mobility, dexterity, sensing, endurance, communication, durability, autonomy, logistics, and safety. They produce objective, quantitative results that facilitate comparisons of different robot configurations and highlight best-in-class implementations. A variety of civilian and military responder communities have used them to understand deployment capabilities and guide

This suite of standard test methods is now being used to focus operator training and provide standard measures of operator proficiency that can help evaluate very perishable skills. The objective is to improve the effectiveness of remote operators and ensure they can reliably perform hazardous operational tasks from safer standoff distances. Using Standard Test Methods to Evaluate Robots and Train Bomb Technicians  In 2015, the Joint Program Office for Counter-Improvised Explosive Devices (C-IED) hosted 4 Interoperability and Training Exercises, aka the Raven’s Challenges. These exercises provided an opportunity to validate 30 standard and draft standard test methods for C-IED missions. More than 200 bomb technicians from civilian and military response organizations trained in the test methods with more than 100 robots. Most used their own organization’s robots, but commercial robot manufacturers also participated by providing robots for domestic and international responders who could not bring their own. The exercises demonstrated how to use repeatable tasks and inherent measures of proficiency to form a “circuit training” model for robots and operators. They included 10 basic skills test methods for maneuvering, mobility, dexterity, and camera pointing; 15 C-IED test methods for tasks involving

Summary of rules for standardized training:

1. Operate from remote stations. The operator should be out of sight of the test apparatus and preferably out of sound too. This ensures that the operator gains all situational awareness through the system interface, as they would during operational deployments downrange. Ample practice time in each test method is encouraged, even with eyes-on operation. This enables operators to develop safe, effective techniques before attempting them from the remote operator station during timed trails.

2. Perform basic skills tests first, every time. The basic skills suite of tests is intended as an exercise or warm-up level of difficulty. It ensures that all systems are functioning on the robot, and on the operator! They can be considered “readiness” tests that should be done before both training and deployment to warm up and identify issues.

3. Conduct the maneuvering suite in FORWARD and REVERSE. The basis skills maneuvering tests require the robots to negotiate the mid-range obstacles driving FORWARD to the far end zone and then driving back through the mid-range obstacle in REVERSE to return to the start point. This encourages omni-directional situational awareness, teaches operators to back out of situations effectively, and equalizes performance between robots training with and without tethers.

4. Time limit training trials. Typically 10 minutes for each test method in the suite depending on complexity. Use enough time to establish a measurable rate of completion that can be noted and tracked over time. The time limits ensure that novices and expert operators work the same overall time across a set of 10 related test methods (100 minutes or so).

5. Repeated trials are required to measure proficiency. A series of successfully completed trial repetitions begins to measure operator proficiency. A few successful repetitions are meaningless indicators of performance. This applies to individual test methods, and to overall suites of tests reflecting mission task combinations. Repeat tests often to establish a body of data over time. Each test method is designed to be quick in order to minimize overall time, effort, and costs.

6. Use all applicable test methods for a given robot configuration. Capturing inherent capability trade-offs across 20-30 test methods is essential to characterize that particular robot configuration. Any changes to the robot configuration, especially changes that affect the distribution of weight or center of gravity, must be re-tested across the entire suite of test methods to see how the changes affect performance. But even changes as simple as tracks vs. wheels can constitute a configuration change.

7. Training partners are encouraged. Safe training for people and robots is the objective. Training partners watching the robot within the lane can ensure no damage happens to the robot or its tether when remote operators make mistakes. Test methods that can result in significant damage to the robot, like stairs, are equipped with a rope belay that the training partner can attend from outside the apparatus, preventing a falling robot before it starts, and without hands-on risk.

Standard Test Methods For Response Robots ASTM International Standards Committee on Homeland Security Applications; Operational Equipment; Robots (E54.08.01)
Collaborating Test Facilities Hosting Standard Test Methods  Two of these regional training facilities remain as year-round venues. They join a growing roster of robot standard test facilities around the world that are interested in supporting regional responder organizations.

Collaborating facilities helping to generate, validate, and standardize a growing suite of test methods:  National Institute of Standards and Technology, Gaithersburg, MD (opened 2006)  Southwest Research Institute, San Antonio, TX (opened 2010)  International Rescue Systems Institute, Kobe/Sendai, Japan (opened 2011)  Bundeswehr (BAAINBw), Koblenz, Germany (opened 2012)  University of Massachusetts – Lowell (opened 2013)  Army National Guard- Camp Dawson, WV (opened 2013)  San Diego Fire Rescue Department Training Facility for Bomb Squads/US&R (opened 2014)  State Dept. Anti-Terrorism Assistance Training Facility, Kabul, Afghanistan (Feb. 2015)  Port of Seattle Police Department, Seattle, WA (June 2015) **Raven’s Challenge**  New York State Division of Homeland Security and Emergency Services, State Preparedness Training Center, Oriskany, NY (June 2015) **Raven’s Challenge**  Air Force – Tyndall Air Force Base, Panama City, FL (2015)  Navy – SPAWAR, San Diego, CA (2016)  Navy – NAVEODTECHDIV, Indianhead, MD (2016)  Japanese Atomic Energy Agency, Remote Test Facility, Nahara, Japan (near Fukushima) (2016)  Korean Atomic Energy Research Institute, Seoul, South Korea (2016) Downloadable Documents

 Guide for Evaluating, Purchasing, and Training with Response Robots using DHS-NIST-ASTM International Standard Test Methods

 Counter-Improvised Explosive Device (C-IED) Applications

 Assembly Guide for Counter-Improvised Explosive Device (C-IED) Applications (if you have problems registering, please email

If you are having problems downloading a document, send an email to with your name and organization. You will receive a pointer to download the document. For More Information