WHITE PAPER
A Transformative Data Collection Solution An Independent Review of StreetLight Data’s Intersection Turning Movement Volume Estimates
A TRANSFORMATIVE DATA COLLECTION SOLUTION An Independent Review of StreetLight Data’s Intersection Turning Movement Volume Estimates AUTHORS: Ian Barnes, PE Ronald T. Milam, PTP Ronald Ramos, PE Marissa Milam Jackie Zielstorff Presented & Designed by FEHR & PEERS 100 Pringle Avenue, Suite 600 Walnut Creek, CA 94596 T: .925.930.7100 Copyright © August 1, 2020 Fehr & Peers
A Transformative Data Collection Solution An Independent Review of StreetLight Data’s Intersection Turning Movement Volume Estimates
Executive Summary The traditional state of the practice of using one-day or two-day traffic counts has always limited practitioners’ understanding of traffic operations and the fluctuations that occur throughout the year. COVID-19 conditions have exposed this limitation to a much greater degree, while also creating an opportunity to transform the approach to current year traffic volume estimation. In response to client concerns, Fehr & Peers completed an independent review of StreetLight Data’s intersection turning movement volume estimation product by comparing StreetLight’s estimates against 2019 observed traffic counts at over 70 intersections in the western United States. Nearly 90 percent of the intersections in our sample had counts that fell within our reasonableness range based on the StreetLight estimates. The reasonableness range included locations where the count was within two standard deviations of the StreetLight estimate (almost 70 percent) or over-estimated the count in a consistent and repeatable manner across the sample, such that it could be corrected with calibration adjustments.
Intersections in which StreetLight estimates were consistently higher than the one-day or two-day counts typically occurred in areas with high mobile device concentration. Highdensity urban areas with substantial transit service, walking, and bicycling are characteristics of these areas. We hypothesized that StreetLight scaling algorithms that convert device trips to vehicle trips do not fully account for device concentration in higher-density areas. Further adjustments (both inside and outside the StreetLight portal) may be warranted in these areas based on local context. Users are cautioned to carefully consider the context of the intersections to be studied,
and to apply their knowledge of the areas in which they work to assess the reasonableness of the data. The data source is a valid replacement for counts or a valid source for factoring older counts, providing up to 90 days of observations for the price of two to three days of typical turning movement counts. The StreetLight portal includes a module where users can upload local count data to improve the performance of the algorithms at the local level. Other improvements to the algorithm will be released by end of 2020.
Would the conclusions of a project change if we had 90 sample days instead of a single sample day?
1
FEHR & PEERS WHITE PAPER • A TRANSFORMATIVE DATA COLLECTION SOLUTION
Background Observed data is the backbone of transportation engineering and planning analyses, providing key information to practitioners and decision-makers about the current conditions. One of these data sources, intersection turning movement (ITM) counts, provides an understanding about the volume of vehicles (and often bicycle and pedestrians) traveling through the intersection during peak analysis hours. ITM counts can inform decisions regarding the number of lanes at an intersection,
Traditional count methods can require weeks of schedule to collect, process, analyze and QA/QC the data, and generally requires waiting for periods of good weather with local schools in session. These limitations can delay projects for long periods of time (e.g. over school summer and winter break, during rainy seasons, etc.). The COVID-19 pandemic has brought these limitations to the forefront, and alternate data collection methods would allow transportation engineers and planners a better path forward during and after the COVID-19 pandemic.
2
what signal timing parameters should be chosen, and what treatments may be needed to accommodate pedestrian and bicycle travel. Frequently, the ITM counts form the foundation for future year volume forecasts, which further provide information as to the future design and operational characteristics of the multimodal transportation system. Generally, ITM counts are performed when schools are in session, in periods of good weather, and in
periods of typical travel activity (e.g., avoiding holidays or other periods of suppressed economic activity). In many urban areas, this combination of characteristics is becoming rare and does not provide any insights into how much traffic volume vary across months and seasons. ITM counts that do not capture more than one or two days will limit the ability of analysts, decision-makers, and the public to understand the complete existing conditions picture.
A single day of counts may cost less, but it does not account for the fluctuations found in a larger sample size.
The typical state of the practice is to collect ITM counts over just one to two days to serve as a basis for volumes under current conditions. While an observed count is both accurate and precise in terms of representing the day when the count was performed, it provides no information about the variation in volumes across other days, weeks, months, and seasons. Collection of additional days of data has always been advised, but comes at additional costs that many projects have not be able or willing to absorb.
The COVID-19 pandemic has resulted in unprecedented changes to human behavior, including a focus on social distancing, the avoidance or prohibition of events or business activities that involve large gatherings of people, working from home, online schooling, and online retail. These changes have resulted in a historic economic downturn, in addition to a substantial suppression of travel activity. It is uncertain how long (or if ) travel activity will take to return to pre-pandemic conditions, but it is certain that traditional data collection methods would only be able to measure the traffic volumes associated with this level of suppressed travel activity. In light of this, Fehr & Peers’ clients have indicated that a new data collection solution would be needed to (1) allow projects to continue forward on schedule, (2) reflect traffic volumes in a pre-pandemic condition until more is known about a return to a “new normal” level travel activity, and (3) provide a tool to track the rebound in travel associated with the “new normal.”
COVID-19 adds additional uncertainty to traffic volume data, and travel patterns may not stabilize until a vaccine is found.
The StreetLight Data ITM volume estimates have the potential to satisfy these requirements, including the ability to reflect pre-pandemic traffic volumes. To verify the reasonableness of using new data, Fehr & Peers routinely performs independent validation tests, and applied our process to the StreetLight ITM volume estimates in an accelerated manner due to client interest caused by COVID-19 conditions. Our independent validation was self-funded and relied on traffic count data we obtained through various projects in California, Utah, and Washington, as well as the standard ITM volume estimates available through the StreetLight data portal.
3
FEHR & PEERS WHITE PAPER • A TRANSFORMATIVE DATA COLLECTION SOLUTION
Literature Review In December of 2018, StreetLight Data released a white paper detailing its internal ITM volume estimate validation analysis. Completed prior to the release of the StreetLight ITM volume estimate product, the validation used StreetLight Index, which is a normalized index describing the relative volume of trips for each Origin-Destination pair. StreetLight ran an Origin-Destination analysis on 15 intersections in Illinois, and seven intersections in Maryland that had publicly available hourly turning movement counts. The analysis used the ratios of turning movements by approach to compare between StreetLight Index values and the count data. This is described by the equation:
Ratio =
Tl ΣTl,r,s
Where T is the indexed value of the vehicles making each turn, and the subscripts “l, r, s” designate all possible leftturn, right-turn, and straight-ahead (through) movements. With multiple hours of data and 12 turning movements per intersection, a total of 2,480 data points were included in the comparison, resulting in an R2 of 0.9, and showing a strong, linear correlation between the StreetLight Index ratios and the count data ratios.
4
In the fall of 2019, StreetLight Data published a white paper detailing the methodology and validation of the new StreetLight Volume product, which provides an estimate of average daily traffic and can be used within multiple analyses to provide estimated vehicle trips. The analysis relies on the StreetLight AADT model and 526 permanent traffic counters, mostly located in the eastern United States and western Canada. This was followed by a second turning movement validation study comparing the StreetLight ITM volume estimates with five intersections in Minnesota that had 12-hour turning movement count data, for a total of 60 data points of comparison. The StreetLight OriginDestination analyses produced hourly turning movement volume estimates for the average weekday during the month the counts were collected. This direct volume comparison yielded an R2 of 0.95 for the 60 data points used. After the analysis described in this white paper was completed, StreetLight staff informed Fehr & Peers that an updated algorithm was developed for release in May 2020. This new algorithm includes data from over 10,000 permanent count stations located in all 48 mainland U.S. states. The updated algorithm is projected to be implemented in the ITM volume estimates product process by the end of 2020.
Analysis Methodology As noted previously, single-day ITM counts fail to account for the day-today, month-to-month, or season-toseason variation in traffic volumes. The StreetLight ITM volume estimates provide access to several months of data, thus allowing an assessment of variation in travel patterns over the course of a year. Therefore, our approach to comparing these two independent estimates of ITMs considered multiple factors. First, the ITM counts were not simply used as the ‘benchmark,’ since the sample size of one or two days is too small to be meaningful for reasonableness checking the much larger StreetLight sample, consisting of 90 sample days. Second, we were interested in
where the pre-COVID-19 ITM counts fell within the range of StreetLight estimates. Finally, we wanted to understand the risk of under- and overestimates and whether any consistent biases existed in the StreetLight estimates that required calibration or post-processing adjustments. We were less concerned about over-estimates that were due to local context and device concentration arrive at a data set that reflects the local seven typical months of counts. If this bias was consistently present in the data set, calibration adjustments can easily be made to correct for it. This combination of factors established our assessment of reasonableness for the StreetLight ITM volume estimates.
Fehr & Peers completed the following specific reasonableness tests. Confidence Interval test: Does the count value (by movement by hour) fall within a confidence interval (two standard deviations) constructed using data from seven months of StreetLight data? Sub-Test 1: For movements not meeting the Confidence Interval test criteria, does the count value fall below the lower-bound of the confidence interval, or does the count fall above the upper-bound of the confidence interval?
The tests noted below were conducted at 73 intersections in the San Francisco Bay Area, Sacramento, Seattle/Tacoma, Salt Lake City, and Southern California areas. Data at most intersections were comprised of single-day ITM counts taken in 2019. Generally, these counts included the 7:00 AM to 8:00 AM, 8:00 AM to 9:00 AM, 4:00 PM to 5:00 PM and 5:00 PM to 6:00 PM periods. For the Confidence Interval test, each hour of data was tested individually by movement, which was then aggregated to totals by approach, and total entering volume at the intersection. Nearly 2,670 hourly movement volumes, 975 hourly approach volumes, and 266 hourly total entering volumes were included in the analysis. The seven months of data aggregated from the StreetLight portal typically included data from February, March, April, May, September, October, and November; however, based on local school schedules and local practices, data from other months were included in the analysis to arrive at a data set that reflects the local seven typical months of counts.
Sub-Test 2: Grouping by turning movement (left turn, right turn, or through), does the Confidence Interval test reveal any bias in the data?
5
FEHR & PEERS WHITE PAPER • A TRANSFORMATIVE DATA COLLECTION SOLUTION
Results of Analysis and Findings Summary outputs of the analysis are included in the following tables present the results of the analysis. Table 1 presents the results of the Confidence Interval analysis by hourly movements. Table 2 presents the results of the Confidence Interval analysis by total approach volume and total entering volume. It is important to stress that the results presented in these tables represent unadjusted data for all locations, including locations where the StreetLight data has known limitations due to contextual factors. These factors include high pedestrian and transit activity, which tends to result in high mobile device concentration. When device concentration is high, the StreetLight volume estimates may overestimate vehicle trips. StreetLight has calibration options available to correct for device concentration that are discussed later, but those were not used in this study.
The results of the analysis yield the following findings: Depending on the type of movement (left turn, through, or right turn), counts were within the unadjusted StreetLight estimates confidence or were higher for 88 percent of locations when considering all movements and 87-91
percent for individual movements. Unadjusted StreetLight data generally produced higher volumes than counts on through movements more often than for left turns and right turns, indicating that there may be some bias in the data. Unadjusted StreetLight data also generally produced higher volumes than counts in locations with high pedestrian activity or high transit activity as expected given the data limitations noted above. Based on feedback from StreetLight Data staff (discussed later in this white paper), the through movement bias is correctable within the StreetLight portal. The high pedestrian activity and high transit activity bias is correctable by (1) developing local calibration factors, or (2) seeding the StreetLight portal with local permanent count data to better calibrate the StreetLight machine learning algorithm. Thus, the opinion of Fehr & Peers is that the locations where StreetLight volumes are higher are due to a consistent set of circumstances, and an analysis solution does in fact exist to reduce these occurrences. Thus, Fehr & Peers found that nearly 90 percent of our tests showed that StreetLight data fell within our reasonableness range. At the approach and total entering volume levels, counts were within the StreetLight data confidence interval or were lower than the lower bound of the StreetLight Data confidence interval 84 percent of the time and 86 percent of
The StreetLight Data product provides critical information about the variation in volumes throughout the year, something that a single-day count does not provide.
6
the time, respectively. Higher through movement volume estimates tend to result in approach volumes and total entering volumes higher than the counts, which is consistent with through volumes generally being greater than left turn or right turn volumes. As noted previously, the through movement bias is correctable within the StreetLight portal.
Based on the Fehr & Peers knowledge of local context at the counted intersections, the StreetLight data product tends to result in higher volume estimates in high pedestrian areas (such as CBDs) and areas of high transit ridership or very high vehicle occupancies , which coincide with high device concentrations. The algorithms also tend to overestimate low-volume movements of 25 vehicles per hour or less.
TABLE 1: Confidence Interval Test Results by Movement (Unadjusted Data) ALL MOVEMENTS
TEST OUTCOME
BY MOVEMENT TYPE Left Turn
Through
Right Turn
Tests Conducted by Movement by Hour (2,669 Hourly Movements) Count within Confidence Interval
66%
70%
57%
71%
Count outside Confidence Interval lower bound (StreetLight higher than count)
22%
21%
30%
17%
Count outside Confidence Interval higher bound (StreetLight lower than count)
11%
9%
13%
12%
Count within Confidence Interval or StreetLight higher than count
88%
91%
87%
88%
Notes: Totals may not equal 100% due to rounding. Source: Fehr & Peers, 2020.
TABLE 2: Confidence Interval Test Results by Approach
and Total Entering Volume (Unadjusted Data)
TEST OUTCOME
BY APPROACH
BY TOTAL ENTERING VOLUME
Tests Conducted by Hour (975 Hourly Approaches and 266 Hourly Total Entering Volumes) Count within Confidence Interval
51%
47%
Count outside Confidence Interval lower bound (StreetLight higher than count)
33%
39%
Count outside Confidence Interval higher bound (StreetLight lower than count) or Inconclusive*
15%
14%
Count within Confidence Interval or StreetLight higher than count
84%
86%
Notes: *An inconclusive result occurs at 2% of total entering volume tests (five total tests) where there was a very high variation in the StreetLight data estimates after summing to the total entering volume level. Totals may not equal 100% due to rounding. Source: Fehr & Peers, 2020.
7
FEHR & PEERS WHITE PAPER • A TRANSFORMATIVE DATA COLLECTION SOLUTION
Potential Corrective Actions After discussion about the overestimate of through movements produced by the StreetLight Volume estimates, StreetLight staff recommended filtering out trips with a circuity greater than 2.0 to correct this overestimate. StreetLight’s trip algorithm can assign the same trip to multiple Origin-Destination pairs within an intersection if the trip does not break before (e.g., have a defined stop) passing through the intersection again. This is most common at freeway interchanges, or at intersections near destinations such as gas stations, drive-throughs, or with high pass-by rates. Filtering the percentage of trips with a circuity greater than 2.0 is key reducing the instances of locations where StreetLight volume estimates are higher than counts.
As with most data products, some level of local calibration of data analysis methods should be performed to assess if factors should be applied to the StreetLight ITM volume estimates to account for local conditions. This calibration factor review would be critical to perform in high pedestrian and high transit ridership areas to account for higher number of devices per vehicle or a higher number of nonautomobile devices captured in the analysis. Additionally, the StreetLight portal allows for the input of local data to seed the StreetLight algorithms with local information. Recent applications (June 2020) of the locally-calibrated StreetLight ITM data product by Fehr & Peers in California (after development of the research in this white paper) have shown that the product exceeds the performance of the unadjusted product.
High pedestrian and high transit ridership areas will require additional calibration prior to use of the data in studies.
8
Conclusions In summary, the StreetLight ITM volume estimates are a reasonable replacement for traditional traffic counts in most applications, or as a source for factors to adjust previously collected counts. For a cost similar to two days of traditional counts, StreetLight offers hourly volumes estimates for all 365 days of the year. Some adjustments may be required to reduce the potential for StreetLight data to overestimate volumes, particularly in areas with high pedestrian or high transit activity. Other parameters in the StreetLight portal (i.e., circuity factor) should also be carefully reviewed for all locations to assess their effects on the estimates provided. Practitioners are cautioned to carefully consider the contexts of study areas, and to review the data for reasonableness with appropriate agency staff.
Acknowledgments We would like to acknowledge StreetLight Data for their support in responding to questions and confirming approaches to corrective actions.
9
Copyright © Fehr & Peers 2020. All Rights Reserved.