The Need for Response Robot Standards.

DHS‐NIST‐ASTM International Standard Test Methods for Response Robots   NIST has been developing the measurements and standards infrastructure necessary to evaluate robotic capabilities for public safety emergency responders, military organizations, and other critical national needs. NIST leads an international collaboration that has generated more than 50 test methods for robotic ground systems, aquatic systems, and micro aerial systems (FAA Group I under 2 kg (4.4 lbs)). These test methods measure robot maneuvering, mobility, dexterity, sensing, endurance, communication, durability, autonomy, logistics, and safety. They produce objective, quantitative results that facilitate comparisons of different robot configurations and highlight best-in-class implementations. A variety of civilian and military responder communities have used them to understand deployment capabilities and guide

This suite of standard test methods is now being used to focus operator training and provide standard measures of operator proficiency that can help evaluate very perishable skills. The objective is to improve the effectiveness of remote operators and ensure they can reliably perform hazardous operational tasks from safer standoff distances. Using Standard Test Methods to Evaluate Robots and Train Bomb Technicians  In 2015, the Joint Program Office for Counter-Improvised Explosive Devices (C-IED) hosted 4 Interoperability and Training Exercises, aka the Raven’s Challenges. These exercises provided an opportunity to validate 30 standard and draft standard test methods for C-IED missions. More than 200 bomb technicians from civilian and military response organizations trained in the test methods with more than 100 robots. Most used their own organization’s robots, but commercial robot manufacturers also participated by providing robots for domestic and international responders who could not bring their own. The exercises demonstrated how to use repeatable tasks and inherent measures of proficiency to form a “circuit training” model for robots and operators. They included 10 basic skills test methods for maneuvering, mobility, dexterity, and camera pointing; 15 C-IED test methods for tasks involving

Summary of rules for standardized training:

1. Operate from remote stations. The operator should be out of sight of the test apparatus and preferably out of sound too. This ensures that the operator gains all situational awareness through the system interface, as they would during operational deployments downrange. Ample practice time in each test method is encouraged, even with eyes-on operation. This enables operators to develop safe, effective techniques before attempting them from the remote operator station during timed trails.

2. Perform basic skills tests first, every time. The basic skills suite of tests is intended as an exercise or warm-up level of difficulty. It ensures that all systems are functioning on the robot, and on the operator! They can be considered “readiness” tests that should be done before both training and deployment to warm up and identify issues.

3. Conduct the maneuvering suite in FORWARD and REVERSE. The basis skills maneuvering tests require the robots to negotiate the mid-range obstacles driving FORWARD to the far end zone and then driving back through the mid-range obstacle in REVERSE to return to the start point. This encourages omni-directional situational awareness, teaches operators to back out of situations effectively, and equalizes performance between robots training with and without tethers.

4. Time limit training trials. Typically 10 minutes for each test method in the suite depending on complexity. Use enough time to establish a measurable rate of completion that can be noted and tracked over time. The time limits ensure that novices and expert operators work the same overall time across a set of 10 related test methods (100 minutes or so).

5. Repeated trials are required to measure proficiency. A series of successfully completed trial repetitions begins to measure operator proficiency. A few successful repetitions are meaningless indicators of performance. This applies to individual test methods, and to overall suites of tests reflecting mission task combinations. Repeat tests often to establish a body of data over time. Each test method is designed to be quick in order to minimize overall time, effort, and costs.

6. Use all applicable test methods for a given robot configuration. Capturing inherent capability trade-offs across 20-30 test methods is essential to characterize that particular robot configuration. Any changes to the robot configuration, especially changes that affect the distribution of weight or center of gravity, must be re-tested across the entire suite of test methods to see how the changes affect performance. But even changes as simple as tracks vs. wheels can constitute a configuration change.

7. Training partners are encouraged. Safe training for people and robots is the objective. Training partners watching the robot within the lane can ensure no damage happens to the robot or its tether when remote operators make mistakes. Test methods that can result in significant damage to the robot, like stairs, are equipped with a rope belay that the training partner can attend from outside the apparatus, preventing a falling robot before it starts, and without hands-on risk.

Standard Test Methods For Response Robots ASTM International Standards Committee on Homeland Security Applications; Operational Equipment; Robots (E54.08.01)
Collaborating Test Facilities Hosting Standard Test Methods  Two of these regional training facilities remain as year-round venues. They join a growing roster of robot standard test facilities around the world that are interested in supporting regional responder organizations.

Collaborating facilities helping to generate, validate, and standardize a growing suite of test methods:  National Institute of Standards and Technology, Gaithersburg, MD (opened 2006)  Southwest Research Institute, San Antonio, TX (opened 2010)  International Rescue Systems Institute, Kobe/Sendai, Japan (opened 2011)  Bundeswehr (BAAINBw), Koblenz, Germany (opened 2012)  University of Massachusetts – Lowell (opened 2013)  Army National Guard- Camp Dawson, WV (opened 2013)  San Diego Fire Rescue Department Training Facility for Bomb Squads/US&R (opened 2014)  State Dept. Anti-Terrorism Assistance Training Facility, Kabul, Afghanistan (Feb. 2015)  Port of Seattle Police Department, Seattle, WA (June 2015) **Raven’s Challenge**  New York State Division of Homeland Security and Emergency Services, State Preparedness Training Center, Oriskany, NY (June 2015) **Raven’s Challenge**  Air Force – Tyndall Air Force Base, Panama City, FL (2015)  Navy – SPAWAR, San Diego, CA (2016)  Navy – NAVEODTECHDIV, Indianhead, MD (2016)  Japanese Atomic Energy Agency, Remote Test Facility, Nahara, Japan (near Fukushima) (2016)  Korean Atomic Energy Research Institute, Seoul, South Korea (2016) Downloadable Documents

 Guide for Evaluating, Purchasing, and Training with Response Robots using DHS-NIST-ASTM International Standard Test Methods http://www.nist.gov/el/isd/ms/upload/DHS_NIST_ASTM_Robot_Test_Methods-2.pdf

 Counter-Improvised Explosive Device (C-IED) Applications http://www.nist.gov/el/isd/ms/upload/C-IED-Training-Using-Standard-Test-Methods-forResponse-Robots-2015.pdf

 Assembly Guide for Counter-Improvised Explosive Device (C-IED) Applications https://robottestmethods.nist.gov/register (if you have problems registering, please email RobotTestMethods@nist.gov)

If you are having problems downloading a document, send an email to RobotTestMethods@nist.gov with your name and organization. You will receive a pointer to download the document. For More Information

Visit http://nist.gov/el/isd/ms/robottestmethods.cfm