There are two general categories of evaluation studies in road safety: 1) before-after studies, and 2) cross-sectional studies.
A before-after study involves collecting and comparing data before and after an intervention from the locations where that intervention was applied. Any observed differences are then attributed to the intervention. These studies may also include a comparison group of similar locations that were not subject to the intervention to account for factors influencing the outcome measure that have nothing to do with the intervention itself, for example, weather.
Cross-sectional studies involve comparing data from sites with a particular safety feature to sites without that safety feature and then attributing the difference in safety to that feature. These studies are often employed when before and after intervention data is not available. Often, it is not feasible for agencies to collect data before an intervention, or they may not have access to the precise intervention dates and details. These types of studies are not preferred because there may be differences which are not immediately obvious or recognized between sites that affect safety that cannot be accounted for when estimating the effect of the intervention.
For both before-after and cross-sectional evaluations, studies can be either experimental or observational. An experimental study is planned, whereby sites are identified before an intervention is implemented and randomly assigned to be a treatment or non-treated site. An observational study is one in which data is collected retroactively and there was no consideration of an evaluation study when selecting those sites. Although both types of studies provide an effectiveness estimate, experimental studies are of higher quality than observational studies because the randomized selection of treatment and comparison sites ensures the study locations and populations are identical with the only difference being whether the treatment was applied. The vast majority of road safety evaluations are observational, as many agencies hesitate to randomly assign a treatment to one population or geography and not another for ethical or political reasons.
For both types of evaluations, there are various methods for analyzing the data ranging from those that are simple to those that are complex and utilize statistical models.
With each evaluation study type, there are tradeoffs between cost, rigour, complexity, and other measures. The following table summarizes some key features to consider when selecting a study design.
Table 1: Considerations for each study type
| Study type | Data availability | Cost | Rigour | Other |
|---|---|---|---|---|
| Randomized controlled trial | Data about indicators must be collected before and after the intervention | This is a very expensive design as it requires complex implementation, technical research skills, and substantial time | This design is the most rigorous, high-quality, and trustworthy in measuring the true impact of an intervention | Some agencies may be reluctant to randomize and exclude a community or site from potential road safety benefits |
| Before-after with a comparison group | Data about indicators must be collected before and after the intervention | This is also a very expensive design due to the requirements for matching and collecting data in treatment and comparison sites | This design has the second highest level of rigour given the temporal (before and after) nature and comparison group | Selecting an appropriate comparison group is critical to estimating the intervention effect |
| Before-after without a comparison group | Data about indicators must be collected before and after the intervention | This is an inexpensive approach as it collects data at a few points in time but does not involve a comparison group | This study type is not as rigorous and may not accurately estimate an intervention’s effect given the lack of a comparison group. It may be capturing trends that would have occurred, irrespective of the intervention. | |
| Cross-sectional | Data collected at only one point in time | This is the cheapest study option as it collects data at only one point in time | This study type is not as rigorous and may not accurately estimate an intervention’s effect because there will be differences between sites that cannot be controlled for in the analysis. | Selecting an appropriate comparison is critical to estimating the intervention effect |
In the following table, examples of each study type are provided.
Table 2: Examples by study type
| Study type | Title and link | Description of study and intervention |
|---|---|---|
| Randomized controlled trial | Traffic safety education for child pedestrians: A randomized controlled trial with active learning approach to develop street-crossing behaviors | In Mehriz City, Iran, 7-year-old children were randomly assigned to three groups in a study to improve street crossing behaviour: 1) an active learning group; 2) active learning plus parent involvement; 3) no training. The study includes 53 children in the active learning group, 51 in the active learning plus parent group, and 46 in the control group. The researchers observed the children’s street crossing behaviour in traffic at three time points: one week before the intervention, one week after, and 6 months after. The researchers found that the groups assigned to training had better outcomes related to looking for vehicles on the street and crossing from safe places one week and six months after the intervention. |
| Before-after with a comparison group | Impact evaluation of camera enforcement for traffic violations in Cali, Colombia, 2008–2014 | In Cali, Colombia, enforcement cameras were set up to detect traffic violations (speeding, running a red light). The study collected data from 38 intervention sites (250-metre radius around the camera), and 50 comparison sites. The researchers selected comparison sites on road and urban characteristics such as the number of lanes and land use (e.g., residential, commercial). Data on crashes was collected 42 months before the installation of cameras and 34 months after. Researchers calculated the number of crashes per month before and after the intervention and comparison sites (termed a “difference-in-difference” study). Intervention areas were found to have a reduction of 19.2% of crashes 12 months after the cameras were installed. Comparison areas were found to have a reduction of 15% in crashes in the same period, indicating that the cameras led to a 5.2% additional decrease in crashes. |
| Before-after without a comparison group | An educational intervention in road safety among children and teenagers in Mexico | In Mexico, a public school-based intervention was implemented to increase knowledge and improve attitudes and practices related to road safety. The study included 219 students across the intervention schools without a comparison group. Both quantitative and qualitative data were collected. For the quantitative data, surveys on road safety experiences, knowledge, attitudes, and practices were conducted. For the qualitative component, the study team conducted focus groups, community mapping, and observations. The students showed changes in their knowledge, practices, and attitudes after the intervention. They were more likely to report road traffic crashes as the leading cause of death and had a greater understanding of the danger of failing to wear seatbelts, crossing the street while using a cell phone, and being in a vehicle with a drunk driver. |
| Cross-sectional with comparison | Accident Relationships of Roadway Width, on Low-Volume Roads | This study focuses on understanding the impacts of lane and shoulder widths on rural roads (i.e., those carrying fewer than 2,000 vehicles a day) in seven US states. Researchers used a database which contained crash and roadway data for 4,000 miles of roadway. They compared the crash rate for 10-foot lanes to 9-foot lanes and found the narrower roads (9 feet) were associated with a lower crash rate. They also noted that for roads with fewer than 250 vehicles per day, the crash rate was similar for paved and unpaved roads. |
Several resources can provide guidance with respect to the methodological design of evaluation studies.
PIARC Road Safety Manual: A Manual for Practitioners and Decision Makers on Implementing Safe System Infrastructure. This manual introduces various aspects of road safety, including planning and designing, identifying issues and risks, selecting interventions, and monitoring and evaluation. World Road Association (PIARC), https://roadsafety.piarc.org/en
The Highway Safety Manual. This comprehensive resource provides knowledge and resources aimed at promoting evidence-based decision-making in road safety. Chapter 9 on Safety Effectiveness Evaluation provides guidance on performing evaluations of road safety countermeasures. National Research Council (US). Transportation Research Board. Task Force on Development of the Highway Safety Manual, & Transportation Officials. Joint Task Force on the Highway Safety Manual. (2010). Highway safety manual (Vol. 1). AASHTO, https://www.highwaysafetymanual.org/Pages/default.aspx
Why are there so few experimental road safety evaluation studies: Could their findings explain it? This research article provides an overview of some of the challenges of conducting experimental/randomized evaluations of road safety interventions. Elvik, R. (2021). Why are there so few experimental road safety evaluation studies: Could their findings explain it?. Accident Analysis & Prevention, 163, 106467. https://www.sciencedirect.com/science/article/pii/S000145752100498X#b0085
Integrated system for monitoring road safety performance in cities. This research article provides an overview of monitoring as it fits into a road safety program. Al-Haji, G. (2014). Integrated system for monitoring road safety performance in cities. July. https://www.witpress.com/elibrary/wit-transactions-on-the-built-environment/116/22278
A Guide to Developing Crash Modification Factors. This report provides information on developing crash modification factors, including the process of selecting an evaluation based on data considerations and evaluation methods. Gross, F., Persaud, B. N., & Lyon, C. (2010). A guide to developing quality crash modification factors (No. FHWA-SA-10-032). United States. Federal Highway Administration. Office of Safety. https://cmfclearinghouse.fhwa.dot.gov/collateral/CMF_Guide.pdf
