Why are standard metrics for evaluating autonomous driving insufficient?
Current benchmarks for evaluating autonomous systems are mainly based on aggregated indicators, such as the effectiveness of reaching a destination, the number of collisions, or an overall driving score. However, these metrics do not take into account specific violations of traffic rules. As a result, planners can achieve high scores while exceeding speed limits, ignoring prohibition signs, or performing maneuvers that violate the law. The study shows that this approach leads to an illusion of safety - the system may operate "efficiently" but dangerously.
The research results indicate a significant gap in current evaluation methods: there is no mechanism to clearly detect violations of specific rules. This limitation prevents reliable assessment of the readiness of systems for real-world use on public roads, where compliance with traffic rules is not only a matter of safety but also legality.
How does the platform for evaluating compliance with traffic signs in autonomous vehicles work?
TrafficSignBench is a testing platform based on simulation using real road maps and procedurally generated scenarios focused on traffic rules. The designers have created a system that not only simulates various road conditions but also automatically checks whether the vehicle planner complies with 34 different traffic signs - from prohibiting driving to requiring yielding the right of way. More context is provided in the material concerning autonomous vehicles.
Each of these rules is equipped with an automatic violation checker that operates in real time during a closed-loop test. This allows researchers to accurately track when and how the system violates a specific rule - which enables analysis of the causes and improvement of the model.
Test results: high scores, but poor compliance with rules
Enlarged imageClose zoomPrevious imageThe researchers created 5800 test scenarios and used them to evaluate existing planners. The results were surprising: despite high overall scores in standard benchmarks, many systems showed serious problems with complying with specific rules - for example, they ignored no-driving signs or failed to yield the right of way at intersections.
This shows that current evaluation metrics are unable to detect significant gaps in the behavior of autonomous systems. A system may operate "correctly" from the point of view of the overall goal, but still operate illegally or dangerously - which poses a serious risk to traffic safety.
New approach: from evaluating compliance with traffic signs to improving models
Instead of just evaluating, TrafficSignBench also enables improvement of systems. Researchers have shown that by adding explicit constraints related to traffic signs, existing planners can be transformed into trajectory experts compliant with regulations. This allows for scalable generation of "oracle" trajectories - i.e., ideal, legally compliant paths - which can be used to fine-tune models in the learning process.
Such an approach is crucial for the development of autonomous systems: it not only evaluates their performance but also gives them the opportunity to learn correct behavior. This is a step forward in creating systems that are not only effective but also correct.
TrafficSignBench not only reveals the weaknesses of existing systems, but also opens up new possibilities in the process of developing autonomous models. Thanks to the integrated real-time violation checking mechanism, researchers can identify a specific rule that the planner systematically ignores - for example, a prohibition of turning left at a given intersection or the obligation to yield to vehicles from the right. This allows for precise fine-tuning of the model, and not just general improvement of results. In practice, this means that the system can be trained not only on the basis of the effectiveness of reaching the goal, but also on the legal and ethical principles of road traffic.
Such an approach is crucial for building public trust and meeting regulatory requirements in countries where liability for errors in autonomous driving may have legal consequences. In the long term, TrafficSignBench may become a standard in validating systems before their deployment on real roads, ensuring not only higher efficiency but also full compliance with the law.