Why are standard metrics for evaluating autonomous driving insufficient?
Current benchmarks for evaluating autonomous systems mainly rely on aggregated metrics, such as the effectiveness of reaching a destination, the number of collisions, or the overall driving score. However, these metrics do not take into account specific violations of traffic rules. As a result, planners may achieve high scores while exceeding speed limits, ignoring prohibitive signs, or performing maneuvers that are illegal. The study shows that this approach leads to an illusion of safety - the system may operate "efficiently," but dangerously.
The results of the research indicate a significant gap in current evaluation methods: there is no mechanism to clearly detect violations of specific rules. This limitation prevents a reliable assessment of the readiness of systems for actual use on public roads, where compliance with traffic rules is not only a matter of safety but also legality.
How does the platform for evaluating compliance with traffic signs in autonomous vehicles work?
TrafficSignBench is a simulation-based testing platform that uses real road maps and procedurally generated scenarios focused on traffic rules. The designers have created a system that not only simulates various road conditions but also automatically checks whether the vehicle planner complies with 34 different traffic signs - from prohibiting driving to requiring yielding. More context is provided in the material about autonomous vehicles.
Each of these rules has been equipped with an automatic violation checker operating in real time during the closed-loop testing. In this way, researchers can accurately track when and how the system violates a specific rule - which allows for analysis of causes and improvement of the model.
Test results: high scores, but poor compliance with rules
Enlarged imageClose zoomPrevious imageThe researchers created 5800 test scenarios and used them to evaluate existing planners. The results were surprising: despite the high overall score in standard benchmarks, many systems showed serious problems with complying with specific rules - for example, they ignored prohibitive signs or did not yield the right of way at intersections.
This shows that current evaluation metrics are unable to detect significant gaps in the behavior of autonomous systems. A system may function "correctly" from the perspective of an overall goal, but still operate illegally or dangerously - which poses a serious risk to traffic safety.
New approach: from evaluating compliance with traffic signs to improving models
Instead of just evaluating, TrafficSignBench also enables improvement of systems. Researchers have shown that by adding explicit constraints related to traffic signs, existing planners can be transformed into trajectory experts compliant with regulations. This allows for scalable generation of "oracle" trajectories - i.e., ideal, legally compliant paths - which can be used to fine-tune models in the learning process.
Such an approach is crucial for the development of autonomous systems: it not only evaluates their performance but also gives them the opportunity to learn correct behavior. This is a step forward towards creating systems that are not only effective but also compliant.
TrafficSignBench not only reveals weaknesses in existing systems, but also opens up new possibilities in the process of developing autonomous models. Thanks to the integrated real-time violation checking mechanism, researchers can identify a specific rule that the planner systematically ignores - e.g., a prohibition of turning left at a given intersection or the obligation to yield to vehicles from the right. This allows for precise model tuning, rather than just general performance improvement. In practice, this means that the system can be trained not only on the basis of its effectiveness in reaching a goal, but also on the legal and ethical principles of road traffic.
Such an approach is crucial for building public trust and meeting regulatory requirements in countries where liability for errors in autonomous driving may have legal consequences. In the long term, TrafficSignBench can become a standard in validating systems before their deployment on real roads, ensuring not only higher efficiency but also full compliance with the law.