In 2017 CETA invited the National Institute of Standards and Technology to bring their Robot Response program to the Canadian Law Enforcement Community.

Along with NIST, CETA is working with other partners to expand the benefits of the program to our members.

Below is an overview of the program.

Project Report

Date: 2017-02-03
Event: ASTM E54.09 Response Robots Committee Meeting and Exercise Summary Location: Virginia Beach Fire Department, Virginia Beach, VA USA
Emphasis: Validated 50 test methods for Ground, Aerial, and Aquatic Response Robots Distribution: General
Prepared By: Adam Jacoff
Intelligent Systems Division
National Institute of Standards and Technology
U.S. Department of Commerce Gaithersburg, MD 20899j | | 301-975-4235

Event Overview:
Remotely operated robots enable emergency responders to perform extremely hazardous tasks from safer stand-off distances. Standard test methods help robot manufacturers and users objectively evaluate system capabilities to align with mission requirements.
This improves the safety and effectiveness of emergency responders as they attempt to save lives and protect property in our communities.

Figure 1: A) For example, “bomb suits” consist essentially of a helmet, Kevlar-reinforced padding, minimal gloves to maintain dexterity, and constant communications to other people at a safe standoff distance. This form of personal protective equipment is designed to mitigate blast effects and fragments from small explosive devices such as pipes and packages. B) Older robots with limited dexterity often cannot disable the explosives quickly enough, so bomb technicians often do it manually (see the robot in the background). C) Emerging threats from cars and trucks packed with explosives pose the greatest danger, far exceeding the protection of a bomb suit. Examples include failed attempts at the World Trade Center and more recently in Times Square in New York City. The rental truck bomb that was detonated in front of the Murrah Federal Building in Oklahoma City had catastrophic results. (Images are from a Raven’s Challenge training exercise in 2016).

Figure 2: A) More than 100 robots have been tested ranging in size from small throwable systems up to 400kg (900lbs) systems. Many of the robots can perform certain tasks associated with various size improvised explosive devices, but in different combinations and with varying reliability. B) Quantitative comparisons of robot capabilities help compare robots and align them with mission requirements. The bar chart shows eight different robot in a defined class being compared by their rates of traversal within increasingly complex standard terrain tests developed by NIST. C) NIST’s test methods are always conducted with operators at a remote standoff distance to ensure the systems perform as intended during deployment – no eyes-on operation allowe

There is an obvious need for these robots to keep bomb technicians and soldiers
safe. Many organizations already own robots but have difficulty deploying their outdated systems effectively. New robots are promising advanced capabilities and more intuitive operator interfaces, but it is hard for bomb technicians and soldiers to sort through the marketing and compare actual system capabilities. Meanwhile, robot manufacturers need quantitative ways to measure the capabilities they develop, make informed design decisions, and demonstrate best-in-class performance and/or reliability.

The ASTM International Standards Committee on Response Robots for Homeland Security Applications (E54.09) is developing the measurements and standards
infrastructure necessary to inspire innovation, inform purchasing decisions, and focus training with measures of operator proficiency. The committee is chaired by Adam Jacoff from the Intelligent Systems Division of the National Institute of Standards and Technology (NIST). His team of robotics researchers are developing the underlying measurement science necessary to quantitatively evaluate and compare robotic system capabilities. A network of international test facilities help refine and validate the test methods for use in their own regions.

Along with the standards committee meeting, they organize comprehensive test method validation exercises with sponsorship from the Department of Homeland Security, Science & Technology Directorate, Capability Development Support Group, Office of Standards. This provides an opportunity for robot manufacturers, emergency responders, and international collaborators to evaluate their systems while validating the 50 test methods for ground, aerial, and aquatic systems under development.

The most recent event was a five day meeting and exercise hosted at the Virginia Beach Fire Department Training Facility, home to the Federal Emergency Management Agency’s (FEMA’s) Virginia Task Force 2 Urban Search and Rescue (US&R) Team. Roughly 100 people participated each day from Canada, Germany, Israel, Japan, Korea, UAE, UK, and the USA. More than 30 robots provided by manufacturers, emergency responders, and military organizations evaluated their robotic capabilities and operator proficiencies within the test methods. Test administrators from collaborating facilities around the world conducted trials to learn about and validate the test methods. They also helped devise new tests to address emerging requirements articulated by responders at the meeting.

Key User Communities
FEMA’s new US&R Robotics Standards Committee met for the first time. They are tasked with selecting suites of standard test methods to evaluate ground, aerial, and aquatic systems to purchase for US&R Task Force equipment caches nationwide. The Virginia Beach Fire Department Training Facility has several US&R training scenarios that were embedded with standard test methods to enable quantitative evaluation of robots even within the uncontrolled and very difficult environmental variables. US&R teams are particularly interested in using the suite of 60cm (24in) wide ground robot test methods to evaluate confined space capabilities to search for and rescue people from compromised and collapsed structures, tunnels, etc. They also want aerial systems for wide area surveys and aquatic systems to inspect the structural integrity of bridges after large scale disasters.

Ground System Testing and Training
Dozens of civilian and military bomb squad technicians in the region participated with their own ground robots to validate new approaches toward standardized robot operator training for countering improvised explosive devices (C-IED). This suite of 120cm (48in) wide ground robot test methods have been used across the country at several such events. The test methods have been replicated in dozens of responder training facilities across the country enabling them to measure and compare their proficiency against “expert” operators provided by the manufacturers, or regional/national averages. These test methods will help identify deficiencies in equipment and training across the country.

Underwater System Testing and Training
Underwater explosive ordinance disposal (EOD) teams from the local Navy base collaborated with manufacturers demonstrating the newest available remotely operated vehicles (ROVs) in an 18,000 gallon water tank with more than a dozen test methods. The repeatable tasks are conducted in five different fixed orientations, floating on the surface, and on the tank bottom. The clear water and resulting videos of the tasks being performed provided key insights into the capabilities of the systems. Especially slow-motion videos of instantaneous actions. Multiple submerged pumps surrounding the test apparatuses provide constant turbulence while performing the tasks, encouraging automatic holding of position and orientation to improve performance. A blackout tarp over the top of the tank provides darkened conditions. Beyond the obvious training benefits, these test methods

help measure important upgrades implemented by manufacturers and check the readiness of systems after repairs.

Aerial System Testing and Training
Indoor and outdoor netted aviaries contained more than a dozen test methods to evaluate small unmanned aircraft systems (sUAS). These systems were provided and operated by manufacturers and responder organizations leading the effort to safely and effectively incorporate these emerging technologies into their fire departments and emergency response operations. These test methods quantify essential safety features and airworthiness while providing useful practice tasks that measure proficiency for responders or hobbyists prior to flying in the national airspace. A wall of fans provides turbulence for the aerial systems, encouraging automatic holding of position and orientation to improve performance of tasks.

Inspiring the Next Generation of Robots
Many of the local Virginia Beach fire fighters brought their kids to expose them to the latest emerging technologies. The smallest 30cm (12in) wide ground robot test methods were their main focus. They learned about rapidly manufactured robot kits they could build on their own using 3D printers and laser cut parts to create truly disposable robots for the most hazardous tasks. Many of the kids managed to get ample time operating the larger commercial systems within the standard test methods, safe for kids and robots. There must have been many “sick” days called in to local schools, but it appeared well worth it for the parents to inspire their young engineers of the future.

Thanks to All Participants
The NIST team included: Adam Jacoff, Ann Virts, Kamel Saidi, Hui-Min Huang, Anthony Downs, Kenny Kimble, Rick Candell, and Murat Aksu.

International test administrators:
• USA UMASS Adam Norton
• USA PSU Karl Reichard
• USA PSU Jesse Pentzer
• USA SWRI Andy Moore
• CAN Alex Ferworn, Reyerson Univ.
• AUS Curtin Univ. Raymond Sheh
• DEU Johannes Pellenz, Germany (bring ground truth generator!)
• JPN Tetsuya Kimura, University of
• JPN Kuniaki Kawabata, Japan Atomic Energy Agency
• KOR JW Park, Korean Atomic Energy Research Institute (KAERI)
• KOR SeungSub Oh, Korean Institute for Robotics and Convergence (KIRO)
• GBR Martin Townsend, United Kingdom Atomic Energy Authority (UKAEA)
• GBR Steve Wheeler, United Kingdom Atomic Energy Authority (UKAEA)
• AFG Sam Hsu, Dept. of State Bureau of Diplomatic Secrurity, Anti-Terrorism Assistance


Key visitors:
• Kathie Morgan, President of ASTM International
• Satoshi Tadokoro, President of IEEE Robotics and Automation Society

Media Pointers
• Virginia Beach Television:

• ASTM’s President tours our meeting and Exercise:

• Channel 13 news: responders-gather-to-test-robots/396034755?client=safari

• emergency-response-tech/amp/?client=safari


• becomes-president-astm-international/amp/?client=safari

Overview Pictures
Here are some pictures to try to convey the enormity of the effort… what’s not shown are the scenarios embedded with standard test methods to capture quantitative scores within the uncontrolled environmental variables.

These included:
• The “smoke house” with random maze test method and several other thermal and laser tests investigating degradation due to obscurants.

• The vehicle-borne improvised explosive device (truck bomb) similar to what we set up here at NIST for local bomb squads to train against.

• The rubble pile and surrounding wide area survey embedded with visual, thermal, and mapping targets. The responders intend to search and map such areas to identify road closures and targeted search areas after natural disasters using aerial systems.

• Ground Test Methods (120cm scale in foreground, an array of 60cm scale behind)

More information is available on our website: see Meetings/Exercises.

Figure 3: Ground test methods in two scales included 120cm (48in) scale in the foreground and 60cm (24in) scale in the background.

Figure 4: The 60cm (24in) scale tests for A) Crossover Aggregates and B) Crossing Ramp Terrain.

Figure 5: Dexterity test methods included Inspect, Touch, Rotate, and Extract test apparatuses on both inclined and crossing ramp terrains.Figure 5: Aerial test methods in indoor and outdoor netted aviaries. Indoor we use the fan wall to inject turbulence at each test method (auto station keeping is a must). The outdoor aviary was near the rubble pile to embed thermal targets.

Figure 6: Outdoor aviary near the rubble pile for GPS enabled testing and embedded targets in the nearby rubble.

Figure 7: The underwater test methods were performed in a rented frac tank. A) The underwater tests include: Inspect, Grab, Cut, Hook, and Aim Disruptors in multiple orientations fixed and B) in floating versions.

Figure 8: A) Submerged inject turbulence at each test method, so auto station keeping improves performance. B) Other tests are performed free swimming.