Category 3: Monitoring & evaluation data considerations

Given many road safety M&E efforts rely on injury and crash data, it is essential to consider data limitations. In the following section, some of these limitations and how they can be addressed or considered in M&E systems are highlighted.

• Incomplete reporting of crashes, injuries, and deaths: As noted, most studies of road safety measures are based on existing databases, including police-reported crashes or data from hospitals or insurance companies. One of the limitations of this is that the databases may be subject to error and data loss, resulting in inaccurate and incomplete information. This may happen in a variety of ways, including that the municipality, state, province, or country do not require the reporting of certain outcomes (e.g., property damage only crashes), or that people do not alert the police or insurance for minor collisions or do not report to the hospital. Certain road users (e.g., cyclists) may be less likely to report crashes as they do not have insurance and benefits after a collision like vehicle drivers. This same limitation is true for other factors related to crash occurrence or injury severity, such as drinking and driving and excessive speeding. Given that no one source is complete, it is best practice to collect and compare several sources. It is impossible to know the true number of road safety outcomes, but combining data may be a more accurate measure.
• Differences in data collection approaches and data quality: When relying on existing data, there may be substantial differences in the quality and completeness of the data between geographic areas or agencies. Some agencies use advanced methods to collect comprehensive data about the crash and the associated injuries, whereas others rely on rudimentary methods such as aggregating the numbers of injuries or deaths at the district/city level. Relatedly, agencies may use different definitions for outcomes. The recommended definition of a road traffic fatality is a death that occurs within 30 days of the crash. However, this definition is not universally applied; such differences are critical to consider when conducting M&E. To address this limitation, a research team can collect detailed information about data collection procedures to assess if this may influence the ability for an evaluation to adequately address and provide understanding of the effects of the intervention.
• Natural variation in crash data: This concept refers to the idea that crashes, from a statistical perspective, are rare and random occurrences. When they are described as rare, it means that crashes comprise only a small number of the total events in a transportation system. When they are described as random, it means that crashes occur due to a set of factors, and there are some factors which are known and can be controlled (e.g., speed) whereas other factors are random and unpredictable (e.g., inevitable human error). The set of factors may lead to a crash one time, but not necessarily each time. The rare and random nature means there are natural fluctuations in crash data. As such, when conducting M&E and using short-term crash frequency, it is important to note it may not necessarily be an impact of an intervention, but rather a natural fluctuation that is being measured. Similarly, when an area has a high crash frequency, it is statistically likely that it will be followed by a comparatively low crash period even in the absence of any efforts to reduce crashes (a term coined regression to the mean in research).

Although these limitations cannot always be measured or completely understood, it is important to consider their influence as it could substantially affect the reported results, interpretation, and subsequent decision-making efforts.

Back to top