Parallel testing of vehicle intelligence via virtual-real interaction

Although researchers and automobile manufacturers have built several proving grounds (1) and testing datasets (2) dedicated to autonomous driving, tests for intelligent vehicles remain time-consuming, inefficient, and sometimes dangerous for people who use the same roads.

According to Turing (3), a system could be said to be intelligent enough for special kind of tasks if and only if it could finish all the possible tasks of its kind. Therefore, we can begin to achieve safe and reliable artificial intelligence (AI) systems if and only if the tests have clear definitions of tasks and efficient methods to generate abundant data for tests. As a result, appropriate AI testing methods should be task-driven and data-centric.

Many existing test systems for autonomous vehicles do not provide a systematic, standard, and practical way to describe the driving tasks so that we can equivalently translate, modify, and reuse the tasks of both field and simulation tests (4–6). We can neither sufficiently sample the driving scenarios that we may encounter in practice nor learn to generate the challenging testing tasks to promote the capability of autonomous vehicles.

Many autonomous vehicle companies resort to simulation-based tests to save time and money. The simulation-based testing system can handle thousands of quantitative judgments in a short time and be more objective than a human expert. However, such a system relies heavily on human knowledge to properly design the scenarios. Some scenarios tested by simulations should also be re-evaluated and verified in field tests to validate the effectiveness of the simulation systems and the reliability of the hardware of autonomous vehicles.

Therefore, a human-in-loop simulation system is useful to evaluate the performance of vehicles (5–7) efficiently. A human expert can first vaguely define the tasks and perform qualitative judgments, and then the simulation-based system can make more precise task definitions, generating more tests, and receive feedback from humans to validate the test results.

We built a closed-loop testing system that focuses on implementing more challenging tests to accelerate the building and testing of autonomous vehicles. As shown in the system overview flowchart of Fig. 1, there are three parts in this system.

• Download high-res image
• Open in new tab
• Download Powerpoint

The first part establishes a set of semantic definitions to characterize the tasks that should be finished by autonomous vehicles (5, 6). Each semantic entity of the tested driving scenario will be retrieved and reproduced in the semantic task space. The semantic task atoms for each entity will be labeled, with a special focus on the spatiotemporal range of each task atom. The merit of semantic analysis lies in its ability to capture the abstract attributes of a special task and discard the unnecessary details. The complexity of the abstracted semantic tasks also provides harmonized classification levels that describe the capabilities of autonomous vehicles.

We can rearrange the spatiotemporal ranges of task rectangles to sample different driving scenarios that belong to the same category so as to ensure that autonomous vehicles could work for these driving scenarios. Adding more semantic task atoms over time permits us to increase the complexity of tests and expand the range of scenarios in which autonomous vehicles can be tested.

The second part implements the tests for the specified task instances. The field test and simulation test are tightly integrated to ensure test safety and accelerate testing speed. Given the tight integration, we call this a “parallel testing system.” Unlike many systems in which the behaviors of all the agents are manually coded and rigid, the parallel testing system keeps collecting new field data to update the simulation system. Moreover, we can carry out field tests that exactly correspond to the simulation tests held in the simulation system and compare their outputs to update the simulation system from time to time. To this end, we built an integration system to automatically and accurately collect various measurements of vehicles in a real-time manner.

We had developed various methods to mix both real scenario data and virtual scenario data generated by simulation engines to provide diverse scenarios for testing autonomous vehicles. Especially, we used the parallel vision techniques to transfer the real-world sensing data collected in the normal daytime to virtual-world sensing data in less frequently encountered situations (e.g., adverse weather and emergency events) (8).

The third part evaluates both vehicle performances and task difficulties to seek the most challenging new tasks. We set up several quantitative performance indices associated with each particular semantic task to fairly and quickly evaluate the performance of an autonomous vehicle (9, 10). We designed a statistical learning model to simultaneously determine the relative difficulties of different tasks and the relative capabilities (ranks) of different autonomous driving systems under tests. Such ranking models help us not only find the challenging tasks, but also understand the real capability levels of the tested autonomous vehicles (10).

We also designed an adversarial learning model to automatically generate new task instances that may be harder than existing task instances based on the past testing results, aiming to push the autonomous vehicles to improve its capability. From this viewpoint, the test in our systems is a self-upgrading process occasionally guided by human experts. Such designs make the test of vehicle intelligence more quantifiable and automatic. We believe that this design philosophy may also be useful for building and testing other intelligent systems.

Our integrated parallel testing system successfully supported the Intelligent Vehicle Future Challenge of China (IVFC), which is the longest-lasting autonomous driving competition (6). Along with IVFC held from 2009 to 2018, our testing system was upgraded to implement systematic, quantitative, automatic, and safe tests for industrial vehicles. Results show that our testing system significantly reduced the burden of competition organizers and test engineers. For more details of the applications, see the Supplementary Materials.

• Copyright © 2019 The Authors, some rights reserved; exclusive licensee American Association for the Advancement of Science. No claim to original U.S. Government Works