APPLICATION OF US DOT ITS EVALUATION GUIDELINES

Disclaimer

The contents of this report reflect the views of the authors, who are responsible for the facts and the accuracy of the information presented herein. This document is disseminated under the sponsorship of the Department of Transportation, University Transportation Centers Program, in the interest of information exchange. The U.S. Government assumes no liability for the contents or use thereof.

Intelligent transportation systems (ITS) represent the application of information processing, communications technologies, advanced control strategies, and electronics to the field of transportation (12). In other words, ITS means electronics, communications, or information processing used singly or in combination to improve the efficiency or safety of a surface transportation system (3).

Integration of ITS into the transportation system requires justification along with other infrastructure improvements. To accomplish this, the basic level of ITS at which benefits/costs have been established is "the market package", a collection of equipment or technologies that work together to deliver a particular ITS service (8). The market packages are to be listed in Table 1. Another concept, "the market area", is a collection of market packages that serves a group of buyers and users of ITS who have similar needs or objectives. The world of ITS is seen to be comprised of nine market areas (Figure 1).

ITS strategies differ from traditional transportation improvements in that they are operationally and information oriented, and aimed at events and unusual conditions. Because of the differences between the impacts of ITS and those of traditional transportation improvements, there is a growing need for new systematic evaluation methods to show the impacts of ITS projects.

(Source: Intelligent Transportation Systems Architectures, Judy McQueen, 1999, Bob McQueen, Artech House, Inc.)

According to the TEA-21 ITS Evaluation Guidelines, evaluations are critical to ensuring progress to achieve ITS deployment goals, assisting in understanding the impacts of the ITS Program activities, and allowing for the program's continued refinement (9). Evaluations can be both qualitative and quantitative; however, it is often desirable to employ a combination of qualitative and quantitative assessments for a complete evaluation.

There are two major types of evaluation recommended for ITS deployment, formative and summative evaluation (9). Formative evaluation is carried out during the course of the development work in order that the objectives of the project be attained. This evaluation is, therefore, designed to provide useful short-term feedback into the deployment process. Summative evaluation is a retrospective assessment of the whole deployment with the aim of justifying the finished work and identifying lessons to be learnt for the next similar projects.

One example of formative evaluation is the evaluation of the first Phase of an Alternate Bus Routing project. The Alternate Bus Routing project provides real-time alternate routing information to the New Jersey Transit buses traveling north bound on the Garden State Parkway. The primary purpose is to determine the feasibility of the technology and make sure that the project be implemented properly. Problems identified in the formative evaluation, such as sensor inability to detect a tagged vehicle, incorrect route assignment, and inability to compute travel time for a tagged vehicle traveling on the network, are to be solved in future (18).

Evaluation of the CHART Program (Chesapeake Highway Advisories Routing Traffic) is an example the summative evaluation. CHART focuses on improving traffic conditions on the interstate highways and state highway arterials in the area of Washington, D.C., Baltimore, Annapolis and Frederick, MD. The final evaluation concluded that the overall benefits exceed the system capital and costs and that the program has a direct impact on delay reduction and fuel saving (17).

The ITS Joint Program Office recommends employing the following six-step process for ITS project evaluation (9).

• Form the Evaluation Team

Team members are designated by each of the project partners and stakeholders. In order to conduct an effective evaluation, an independent evaluator should participate in the process periodically.

• Develop the Evaluation Strategy

The Evaluation Strategy relates the purpose of the project to the overall ITS goal areas. Currently, ITS goal areas include traveler safety, traveler mobility, transportation system efficiency, productivity of transportation providers, and conservation of energy and protection of the environment. At this stage, those goal areas which have the highest priority for the specific ITS project are identified. Appropriate measures of effectiveness are also determined to investigate the impacts of the ITS deployment. This process gives partners insights regarding areas of agreement and disagreement and assists them in obtaining consensus.

• Develop the Evaluation Plan

The next step is to refine the evaluation approach by formulating statistical hypotheses where they apply. In addition, the Evaluation Plan identifies qualitative studies that will be performed. A special emphasis should be placed on the non-technical factors, such as institutional issues, that influence project performance.

• Develop One or More Test Plans

A Test Plan specifies all of the details about how the test will be conducted, and identifies the number of evaluation personnel, equipment, supplies, procedures, schedule, and resources that are required to complete the test.

• Collect and Analyze Data and Information

This step involves the implementation of each Test Plan. Analysis of the data collected in this stage forms the basis of the final conclusion.

• Prepare the Final Report

The evaluation strategy, plans, results, conclusions, and recommendations should be documented in a Final Report.

The TEA-21 ITS Evaluation Guidelines suggests that the evaluation of ITS projects should encompass five major goals (9). Also, several related measures have been identified as useful to capture the impacts of ITS projects. Table 2 indicates the goal areas with key measures, which is followed, with detailed explanations and recommendations for evaluating ITS projects.

(Source: http://www.its.dot.gov/eval/ResourceGuide/EvalGuidelines, Transportation Equity Act For The 21st Century; Guidelines For The Evaluation Of Operational Tests And Deployment Projects For Intelligent Transportation Systems (ITS))

Safety benefits are mainly associated with the reduction of transportation related accidents or a decrease in the severity of accidents. Accident rate reduction constitutes one of the most direct measures and improvements are typically measured by a comparison of ‘before’ and ‘after’ accident studies. It is important to ensure that safety improvements are over the whole region rather than moving the accidents between different locations. It is also critical to isolate other external influences from the measurement, like reduction in kilometers traveled or better law enforcement.

In addition, since data collection of automobile accidents is not a perfect science and the details describing the accidents are often approximate and incomplete, some proxies may be used for the data required. Popular examples include the reduction in the overall number of recorded incidents on a stretch of highway, and reduction in the incident response time. It might also be helpful to relate the safety measurement to the insurance industry statistics.

• Mobility

Delay to a user of a system is typically measured in seconds per vehicle or minutes per vehicle of delay. Delay can be measured in many different ways. For example, the "floating car" method can be used to measure the delay experienced before and after installation of the system. Delay can also be measured by comparing the number of stops experienced by the drivers before and after the introduction of an ITS project.

Travel time variability indicates the fluctuation in overall travel time of trips between an origin and destination pairs via the transportation network. By improving response time to incidents, and providing information on delays, ITS services can reduce the variability of travel time. In turn, the increased reliability can help travelers and freight companies make planning and scheduling decisions. Several types of statistics, like the standard deviation or variance around the mean, the range of travel time values, can be computed to indicate the variability of travel time. Travel time variability can be calculated under different time horizons, such as within day and day-to-day variability of a given trip or goods movement from an origin to a destination.

• Efficiency

A major goal of ITS projects is the optimization of flow on existing facilities. One way to accomplish this goal is to increase the effective capacity. A frequently used observable measure is the ‘throughput’, which involves taking volume counts of the number of persons or vehicles traversing a roadway section or network per unit time.

• Productivity

Cost savings is used to quantify improvements in productivity. The components of the costs include the acquisition cost (capital cost), operating/maintenance cost, income from revenue-generating transportation facilities, and user costs. Cost savings can be either the difference in costs before and after installation of a system or the difference between the cost of an Intelligent Transportation System and traditional transportation improvements that are designed to address the same problem.

• Energy and Environment

The air quality and energy impacts of ITS services are of particular importance, especially for the areas where the air quality standards specified in the Clean Air Act Amendments of 1990 have not been met. Typical pollutants to be measured include carbon monoxide (CO), nitrogen oxides (NOx), and volatile organic compounds (VOC’s) like hydrocarbons (HC). Simulation models, such as CORSIM, INTEGRATION, are often used to estimate the resulting changes in emission levels and energy consumption before and after installation of an ITS product.

There are many challenges to evaluating the environmental impacts of ITS projects. First of all, the impact of an individual ITS project is very small, especially when compared to the environmental conditions of the larger geographic region. In addition, many external variables, such as weather conditions, pollutants emitted by non-mobile sources and even pollutants carried from other metropolitan areas to the study area, make it more difficult to explicitly and accurately evaluate the impacts of an ITS project.

Case 1 - By applying a variety of advanced technologies including adaptive ramp metering, adaptive, traffic signals, motorist information, and surveillance systems, ICTM (Integrated Corridor Traffic Management) aims at optimizing corridor capacity, traffic operations, and safety (16). The selected corridor was an 8-mile section of the I-494 in the south of the Twin Cities. Besides I-494, the whole section includes four parallel and seven perpendicular arterial streets. Table 3 lists the measures employed in their evaluation and meanwhile relevant to the goal areas of mobility and efficiency.

(Source: http://www.itsdocs.fhwa.dot.gov/jpodocs/repts_te/9xb01!.pdf, Integrated Corridor Traffic Management Final Evaluation Report, Booz. Allen, Hamilton, April 2000)

The evaluation derived its conclusion from a variety of quantitative and qualitative data sources. These sources included travel time runs, screenline traffic counts, automated databases from loops detectors, written surveys and interviews with local project stakeholders and corridor motorists. The evaluation came to the following conclusion:

• During nonincident conditions, traffic patterns were changed through improved use of corridor capacity because trips were diverted from the corridor to local street. This conclusion was drawn through a comparison of the traffic counts for the year 1996 and 1999. Appendix A gives the traffic flow collected at the screenlines. Meanwhile, surveys to the motorists also support this conclusion.
• Traffic operations experienced improvement during non-incident conditions. The MOEs, including travel time, frequency of stops, space mean speed, and overall delay, were analyzed in the before-and-after cases. Please refer to Appendix A for the comparison results for various analysis periods and link orientations.
• Motorists made more intelligent route choices during incidents because the surveyed motorists were generally satisfied with the advisory system and indicated a preference to alter their travel behavior.

Case 2 - CHART (Chesapeake Highway Advisories Routing Traffic) is a highway ITS program initiated by the Maryland State Highway Administration. CHART focuses on the areas of Washington, D.C., Baltimore, Annapolis and Frederick, MD and encompasses about 375 miles of interstate highways and 170 miles of state highway arterials. The CHART program is composed of four major components: traffic monitoring, incident response, traveler information and traffic management (17). The evaluation of the performance of the CHART program was implemented through the application of a freeway corridor simulation model (FREQ). Table 4 summarizes the MOEs employed in this study.

(Source: http://www.itsdocs.fhwa.dot.gov/jpodocs/repts_te/7pw01!.pdf, CHART Incident Response Evaluation Final Report, CORSIM Corporation, May 1996)

The most significant finding of the evaluation is that the benefits of the CHART incident response program, including the estimated reduction in delay, fuel consumption and secondary incidents, exceed the system's capital investment, operating and maintenance costs by a ratio of over 7 to 1. Please refer to Appendix B for a detailed review of the evaluation results.

Table 1 gave an index of goals and measures to evaluate overall impacts of different ITS projects. However, as introduced in the section "Introduction", "market package" is the basic level of ITS at which benefits/costs can be evaluated. It has also been advocated to incorporate some MOEs which are sensitive to each specific ITS strategy (7) (here, 'strategy' refers to a similar concept to 'market package'). For this purpose, Table 5 relates certain measures with some popular ITS strategies.

Based upon the Evaluation Guidelines and a review of recent research papers and reports, the following matrix (Table 6) is suggested to evaluate ITS projects. This table encompasses all the five areas that have been identified as the major goals by the Evaluation Guidelines. Because different ITS strategies often address different traffic problems and are the basic level for evaluation, the table classifies the measures according their sensitivity to the ITS strategies.

Data availability is a critical factor in determining whether the evaluation can be conducted successfully. For example, in order to assess whether safety has been improved resulting from an ITS project, it is often desirable to collect accident data over several years. Therefore, during the beginning years of an ITS project, we cannot get sufficient data to evaluate the level of safety. For another instance, unlike certain data that can be obtained electrically (e.g. volume, speed), some data acquisition requires manual work (e.g. number of stops, time waiting for service). In such cases, it might become economically inefficient to get these accurate data only for an evaluation purpose. In fact, survey of the system operators or users often turns out to be a fast and economic way to collect data. This method has been widely accepted nowadays. Another approach to solve the problem of insufficient data is to employ traffic simulation models. Actual detector counted value of traffic volume can be input into the models to estimate the benefits in the area of safety, mobility, pollution and energy.

Federal Register (23 CFR part 1410) published on May 25, 2000 includes a notice of proposed rule making (NPRM) for statewide or metropolitan transportation planning which includes provisions for incorporating ITS strategies and investments into the statewide and metropolitan planning and programming process. Since the success of ITS integration depends greatly on the two fundamental issues – technical and institutional integration, the new regulations emphasize that 1) agreement should be reached among the MPOs, State DOTs, transit operators and other agencies addressing policy and operational issues; 2) the ITS project should be interoperable and should utilize ITS related standards (e.g. the National and Regional ITS Architecture), and the routine operation of the projects (12).

The transportation planning processes calls for a coordinated approach to assess transportation needs, evaluate a range of solutions, and produce an agreement among relevant agencies (12). The new regulation no longer requires the Major Investment Study to appear as a separate process; instead it will be integrated into the planning process. A key step in the planning process is to estimate the impacts of alternatives. Methods that have been used to estimate the benefits/costs of various strategies includes the 'Combination of Planning and Simulation Models' and the 'ITS Deployment Analysis System' (IDAS).

By providing improved information to travelers and adjusting traffic control policies in real-time, ITS allows travelers and transportation managers to react to changing conditions and to more effectively use transportation capacity. In order to model time-varying conditions and demands, as well as individual vehicle-level capabilities and routing decisions, a model framework, which is comprised of a set of transportation models, has been proposed and utilized (6). The structure for this analytical framework is represented in Figure 2. In the framework, planning and simulation analysis is executed as an iterative process. Estimates of mode split and assigned traffic volumes are produced by the planning model and input to the simulation models. The revised speeds from the simulation models are then fed back into the planning model. The process is repeated until travel speed and volumes converge. This evaluation framework will be explained in detail through a presentation of Seattle Case Study in the following.

The area of the project is located north of Seattle, Washington. It is one of the densest areas in the Seattle region and serves as a major origin/destination and through commuter corridor into the Seattle activity centers. Interstate I-5 and State Route 99 are the major north-south transportation facilities serving the corridor.

The project intends to alleviate north-south congestion and improve mobility throughout the area. Based on this objective, the set of MOEs shown as Table 7 was defined.

In all, six alternatives were introduced for alleviating congestion in the area. The alternatives encompass both traditional transportation improvements and traditional transportation improvements plus ITS solutions. The alternatives include: do-nothing, ITS rich (only ITS improvements), SOV capacity expansion without ITS enhancements, SOV capacity expansion with ITS enhancements, HOV/busway without ITS enhancements, and HOV/busway with ITS enhancements. Figure 3 shows these alternatives.

(Source: Incorporating ITS into corridor Planning: Seattle Case Study – Executive Summary, August 1999)

In order to fully capture the ITS/operational improvements, especially the responses to time-variant conditions (recurrent and non-recurrent), a two-level modeling process was adopted. At the higher (regional) level, the overall travel patterns and the system’s responses to average/expected conditions are analyzed using a traditional regional planning model (EMME/2). Output from this analysis is then fed into a more detailed sub-area simulation model capable of modeling time-varying conditions and demands, as well as individual vehicle-level capabilities and routing decisions (INTEGRATION). At the lower level, the detailed traffic operations, queuing, and buildup/dispersion of demand are captured, as well as the real-time response of travelers to information. Feedback is then carried out to ensure that the impacts to expected conditions, estimated in the sub-area model, are reflected in the regional analysis.

Another key element in capturing the impacts of ITS is the use of a representative day scenario analysis to address non-recurrent conditions. Each scenario, or representative day, is selected to capture a type of incident or occurrence that may produce a significant influence on the traveler and his/her choices of route. The variables in the scenario include: incident/accidents, overall travel demand, and weather conditions. In all, 30 scenarios were defined. Their corresponding probability of occurrence in the analysis periods were also identified. In the end, the results from the representative day simulations were combined to estimate the impacts and fed back to the regional travel forecasting process. Figure 4 describes the analysis procedure employed in this case study.

The overall results indicated that, when comparing ITS Rich to the Baseline, the studied corridor experienced a 4.3% increase in average daily vehicle, 14.6% reduction of annualized delay, 30% reduction in travel time variability and 25% reduction in the traveler risk of a significant delay.

IDAS systematically and quantitatively estimates costs and benefits associated with the deployment of ITS as well as other transportation options. It has been designed as a post-processor to travel demand models in transportation planning purposes. IDAS consists of five major analysis modules: an input/output interface module, an alternative generator module, a benefits module, a cost module and an alternatives comparison module. Figure 5 shows the general structure of the IDAS modules and sub-modules (5).

The development of IDAS has experienced two phases: Build 1 and Build 2. The final version include the following abilities 1) to specify ITS deployments and their characteristics in transportation planning networks; 2) to analyze the impacts of transportation infrastructure improvement alternatives; 3) to provide life-cycle cost estimates; 4) to compare the results of alternative ITS deployment (14). Table 8 lists the ITS components in Build 2.

IDAS is designed to help planners compare the performance of several ITS options against ‘control’ alternatives. Data input into IDAS is prepared by a typical regional travel demand model to construct the basic supply and demand characteristics of the system. Reading data from the travel demand model is the first step to run IDAS.

Basic components in the analysis hierarchy of IDAS are projects, alternatives, and ITS options. Unlike traditional sketch planning tools, where control alternatives refer to no-build choice, IDAS defines control alternatives as those that serve as the baseline for comparing ITS options. That is to say, the control alternative does not contain any ITS component (5). Figure 6 explains how the analysis hierarchy can be used in evaluating scenarios where ITS has been incorporated into transportation planning process.

• Users can use the deployment screen to deploy the ITS elements, examine and modify their parameters, properties and locations directly on maps.
• IDAS includes a large database describing the costs, anticipated useful life and direct benefits of a wide variety of ITS elements. This data information can also be accessed by two separate files, Equip and DirectBenefits, through Excel 97.
• The analysis hierarchy of IDAS has made itself especially suitable for evaluating projects which have incorporated ITS elements in the alternatives.
• As a sketch planning tool, IDAS has a less restrictive requirement of data input. Moreover, IDAS provides default values for the parameters. As a result, it is an inexpensive but efficient tool to help planners get a general idea of the performance of each alternative.
• The evaluation matrix produced by IDAS conforms to the requirements of the TEA-21 ITS Evaluation Guidelines. The following table lists the measures employed by IDAS for evaluating alternatives as well as the goal areas the measures are addressing to.

Safety, mobility, efficiency, productivity, and protection of environment and energy, are the major goal areas in ITS deployment. As a result, the evaluation team should relate the purpose of its projects to these overall goal areas and develop a table of MOEs to address them. The table of MOEs is the key to a successful evaluation of ITS deployment. The MOEs should be easy to measure and calculate, should be sensitive to the specific goals of the ITS project, and should consider the requirements by the TEA-21 Evaluation Guidelines. The index of MOEs given in this paper can be referred to start an evaluation process.

Qualitative methods are also used to evaluate ITS deployments. Surveys and interviews of project partners and travelers are major data sources to qualitative evaluation, while quantitative evaluation involves statistical tests between before- and after- the ITS deployments or between non-ITS and ITS rich alternatives that address the same problem. In most cases, a combination of the two approaches is employed to achieve a comprehensive evaluation.

Federal Register (23 CFR part 1410) emphasized the importance of incorporating ITS elements into the statewide and metropolitan planning processes. "Combination of Planning and Simulation Models" and IDAS are the two major methods used to estimate the benefits/costs of ITS alternatives in the planning processes. Both methods require input from travel demand models and make estimations in terms of traffic performance, safety, and emissions and energy. The first method is more of a microscopic level and requires more detailed and extensive data input, and hence, is expected to produce more accurate results.

1. Bob McQueen, Intelligent Transportation Systems Architectures, Judy McQueen, 1999, Artech House, Inc.
2. Daniel Brand, Criteria And Methods For Evaluating Intelligent Transportation System Plans And Operational Tests, Transportation Research Record 1453
3. Steven E. Underwood and Stephen G. Gehring, Framework For Evaluating Intelligent Vehicle-Highway Systems, Transportation Research Record 1453
4. S. Gregory Hatcher, James A. Bunch, and Donald L. Roberts, Incorporating Intelligent Transportation Systems into Major Investment Studies – Conceptual Issues and Challenges, Transportation Research Record 1651
5. ITS Deployment Analysis System, Build 1, Cambridge Systematics, Inc. December, 1998
6. Incorporating ITS into Corridor Planning: Seattle Case Study – Executive Summary, August 1999
7. Integrating Intelligent Transportation Systems within the Transportation Planning Process: an Interim Handbook, January 1998, TransCore
8. Intelligent Transportation Systems Impact Assessment Framework: Final Report, Volpe National Transportation Systems Center, September, 1995
9. http://www.its.dot.gov/eval/ResourceGuide/EvalGuidelines_EvaluationGuidelines.htm, Transportation Equity Act for The 21st Century; Guidelines for The Evaluation of Operational Tests And Deployment Projects for Intelligent Transportation Systems (ITS)
10. http://www.its.dot.gov/piarc/itshandbk1.htm, ITS Handbook
11. http://www.fhwa.dot.gov///////environment/plng_fr.pdf, Federal Register, May 25, 2000, Part III, 23 CFR Parts 450 and 1410, 49 CFR Parts 613 and 621
12. http://frwebgate.access.gpo.gov/cgi-bin/getdoc.cgi?dbname=2000_register&docid=00-13023-filed.pdf, Federal Register, May 25, 2000, Part V, 23 CFR Parts 655 and 940, Department of Transportation
13. http://toolkit.virginia.edu/cgi-local/tk/UVa_SEAS_2000_Fall_C_E737-1, CE737 Intelligent Transportation Systems, syllabus
14. http://www-cta.ornl.gov/cta/research/idas/sowindex.html, ITS Deployment Analysis System, Statement of Work, Subcontract No. 1DX-SY245, November 7, 1997
15. http://www-cta.ornl.gov/cta/research/idas/kickoff.PDF, IDAS Kick-off Meeting, Vassili Alexiadis Vassili Alexiadis, Cambridge Systematics, Inc., December 4, 1997 December 4, 199
16. http://www.itsdocs.fhwa.dot.gov/jpodocs/repts_te/9xb01!.pdf, Integrated Corridor Traffic Management Final Evaluation Report, Booz·Allen, Hamilton, April 2000
17. http://www.itsdocs.fhwa.dot.gov/jpodocs/repts_te/7pw01!.pdf, CHART Incident Response Evaluation Final Report, CORSIM Corporation, May 1996
18. http://www.itsdocs.fhwa.dot.gov/jpodocs/repts_te/7c401!.pdf, Evaluation Of Alternate Bus Routing Project - Phase I, Kaan Ozbay, Mohsen Jafari, Diogenes Feldhaus, Tilanka Karunaratne, Trefor Williams, FRRITS- 1-6-98, June 1998
19. http://www.itsdocs.fhwa.dot.gov/jpodocs/repts_te/6wg01!.pdf, Seattle MIS Executive Summary: Incorporating ITS into Transportation Improvement Planning, the Seattle Case Study using PRUEVIIN, Interim Report (methodology), March 1998
20. http://www.odetics.com/itsarch/ National ITS Architecture