Executive Summary

The purpose of this international scan was to investigate the use of performance measures in transportation planning and decision- making in selected countries. Performance measures can relate to many different aspects of and be applied at different levels of decision- making, so the scan panel represented a diverse set of interests and concerns for both national and State-level decision-making.

The Federal Highway Administration ( FHWA ) and the American Association of State Highway and Transportation Officials ( AASHTO ) jointly sponsored this scan. In addition to FHWA and AASHTO officials, the panel included representatives of State transportation agencies for the States of Maryland, Minnesota, Virginia, and Washington; representatives of the International Bridge, Tunnel, and Turnpike Association and the American Council of Engineering Companies; and two university professors of transportation engineering and policy. These panel members represented diverse interests and expertise in engineering, intelligent transportation systems, policy, planning, safety, and system operations.

The scan team met with the following types of representatives during its 2.5- week study:

• National ministry of transportation and other national transportation agencies in Canada, Japan, and New Zealand.
• Provincial or state departments of transportation in British Columbia; Japan; and Queensland, New South Wales, and Victoria, Australia.
• State department of infrastructure in Victoria.
• Rail or transit planning organization in Vancouver, Victoria, Queensland, and Japan.
• Police in Victoria.

GENERAL OBSERVATIONS

The scan team identified 23 observations on the use of performance measures in the countries visited that would be of interest to officials in the United States. Many of these observations are of broad interest to transportation professionals, while others might be of interest in a specific context. These observations include the following:

1. The scan team found examples in which the processes of setting priorities and making planning, investment, and management decisions are based on, or use, performance measures to a much greater extent than is typical in the United States. In those cases where performance measures were used as input to priority setting, the process represented a new level of organizational behavior. The vertical integration of information flow through the use of performance measures characterized the better examples of performance measurement.
2. Perhaps the most impressive application of performance measurement, in terms of showing how the process can influence governmental policy and budget determinations, was in the area of road safety. Impressive results in reducing fatalities and injuries have occurred in some of the sites the scan team visited through a comprehensive program of engineering, enforcement, and education.
3. A common framework to performance measurement appeared to be present in all cases the team examined. Each effort was related to a broader set of goals and objectives defined either by a legislative body or through a public visioning process. These goals and objectives led to identification of transportation system- specific performance measures, often tied to target values to be achieved in a future year.
4. Transportation officials appeared to have a general understanding of the distinction between the concepts of outcomes and outputs. Outcomes were viewed as the ultimate characteristic of transportation system performance, while outputs were the products and services of the organizations that led to these outcomes.
5. In each of the cases the team observed, transportation officials explained the initial movement toward performance measurement as a means of providing greater accountability and visibility to the public of their agency's activities.
6. In addition to accountability and public visibility, many officials commented that performance measurement can be used to educate elected decision makers and the general public on the role that transportation plays in society and/ or the need for additional investment.
7. Performance measurement was applied at several different levels of planning and decision-making. For example, many measures were targeted at strategic investment decisions relating to long-term improvements to the transportation system. In other cases, performance measures were used to manage network operations.
8. In Japan, the national ministry established a core set of 17 performance measures as a guide for all regional offices and prefectures (states) in the country. This was an example of a phenomenon common to all applications—the need and desire to determine a small set of measures that were really important to an agency.
9. All of the sites visited used measures commonly used in the United States. Road network congestion was a transportation system measure found in all of the performance measurement applications examined. Officials often viewed this as one of the important issues facing their region and agency. Other network measures related toaccessibility and mobility provided by the transportation system, road safety, travel time, and trip reliability.
10. Environmental measureswere also present in all of the performance measurement efforts examined. The scan team noted that of all the performance measurement categories it examined, the environmental measurement category created the greatest challenge for transportation agencies.
11. Measures of customer satisfaction were found in all of the sites visited. The measures most often related to the average scores obtained from public surveys. Also, New Zealand's approach to customer satisfaction measures focused on identifying issues causing customer dissatisfaction and targeting organizational action to deal with the issues.
12. Measures relating to transportation system security were not found in any performance measurement examples. In many ways, this was viewed as an American issue, although transportation officials in Queensland and New South Wales did say that security was becoming a more important issue and that some form of a security indicator most likely would be incorporated into their performance management regime in the future.
13. Many performance measurement applications targeted rural transportation network performance as a specific category for measurement. This was particularly true in Australia, where each state has one major urban area dominating the economy. This was viewed primarily as a regional equity issue.
14. The level of integrated data collection strategies as they related to performance measurement varied from one site to another. Some of the more successful performance measurement programs occurred in data- rich environments with a history of strong data collection and analysis.
15. Before-and-after studies were important elements of the performance measurement systems in Japan and Australia. Each performance measurement case in the scan countries paid considerable attention to discovering what impacts adopted or implemented measures have had on selected performance measures. The results of these studies act as feedback to the decision- making process and help determine likely results to similar actions in other areas.
16. Graphic (and visual) presentation of performance results was viewed as a key component of the performance management process, as was identification of measures that the public can relate to. Most officials involved in managing the performance measurement efforts stated that unless top decision makers understand the information presented to them, the efforts will be ineffective. Visualization of information is thus critical to successful performance measurement.
17. Benchmarking against peer organizations and jurisdictions was used in Japan, Australia, and New Zealand to understand key factors that distinguished economic and transportation performance among states. This benchmarking was used as a screening tool to identify differences (in both quantity per capita and relative ranking), which then led to a more detailed examination of why those differences occurred.
18. The key measure of success for performance measurement itself is the degree to which it influences decisions and budget allocation. This was a difficult linkage to pin down during the scan study. The most advanced application appeared to be in Queensland and Victoria, where the performance measurement process was important for determining program priorities for safety actions.
19. The team found few examples in which performance measurement resulted in multimodal investment tradeoffs. In most cases, performance measurement is implemented within a modally focused agency, so performance measures were targeted at decisions relating to the performance of that modal network.
20. Measures relating to freight movement were found in many performance measurement efforts. Examples include commercial trucking travel time between economic gateways (British Columbia); mode split for goods movement (Vancouver); increase in average annual truck payload capacity, percent of illegally overloaded trucks, and freight rail delays and travel time variability (Queensland); and freight productivity, access to ports, freight rates, freight-related infrastructure condition, and effects of congestion on freight movement (Victoria).
21. Organizational productivity measures were found in all of the performance measurement applications. Agency managers viewed them as a critical element of the performance measurement exercise in that they indicated the degree to which their agency was delivering the products and services needed to meet other performance objectives.
22. Monitoring project delivery through the use of performance measures was also a common approach at most of the sites visited. An interesting aspect of this project delivery monitoring was the effort to assess project performance against project expectations.
23. In cases where performance management was most successfully institutionalized within agency operations, top management leadership and commitment to the process of development and continuous use was essential to get performance measurement past its infancy. This was particularly important for agencies where the organizational culture and even the culture of the society itself were not conducive to the concept of open and responsive performance- based planning and decision-making.

LESSONS FOR THE UNITED STATES

Similar to the scan team observations, a large number of "lessons learned" came out of this scan. The most significant are described below:

1. Safety was viewed as a strategic use of performance measurementthat has resulted in a significant decline in fatalities. A great deal can be learned from this application of performance measurement, especially as it relates to the identification of strategies and actions that need to be put in place to achieve reductions in road fatalities.
2. Meaningful performance measurement is a product of extensive outreach, discussion, and collaboration with partners. Performance measures are readily available and easy to create, but without a comprehensive (internal and external) outreach process their value as a behavioral influencer is limited. Open, inclusive planning processes are fundamental to good performance measurement.
3. In the best examples of performance measurement, officials were still refining what measures to use, and how to make sense of the political guidance they received. This suggested that performance- based planning and decision-making are never-ending processes, and must be viewed as such by top agency officials. This also implied that an agency should not measure too many objectives; it should measure only what is needed to make business decisions. Too many measures can be a burden on staff.
4. The key to success for measurement is accountability. This translates into the need for understandable measures and systematic follow- through to determine the level of failure or success in meeting the objectives defined by the measures. This issue is important across all transportation areas, but has particular short-term relevance and application in the safety area.
5. Performance measurement is most relevant when linked to decision- making, especially resource allocation. Elements to consider include allocation of funding at the program and project levels, multimodal tradeoffs, and distributional equity. A multimodal approach is best, although the scan team found few examples of such an approach. Performance measures position an agency well to engage in debate, but are not necessarily the determining factor in a decision, especially in the legislative arena. Measures sharpen and focus the debate, and help clarify organizational direction.
6. Quality systems have been put in place and appear to be replacing externally defined quality criteria (e.g., ISO 9000). Many are redefining quality and finding that the International Organization for Standardization ( ISO ) process does not meet their needs.
7. For safety, enforcement strategies were the key to changing driving behavior and affecting overall success. The best-case examples go beyond the mandatory use of seat belts (alcohol testing and speed enforcement) and are succeeding at reducing fatalities. In addition, engineering strategies are an important element of an overall safety program, resulting in significant short-and long-term improvements.
8. Analysis is an important underpinning for setting targets and determining the effectiveness of actions to reach these targets. The scan team found a much greater use of benefit-cost analysis to determine the desirability of projects, and a desire to determine after the project was implemented whether the initial analysis was close to the resulting performance.
9. Customer surveys are an important element of determining organizational performance, if done appropriately and in a valid manner. Many countries appeared to be most successful at getting useful data from customer surveys by asking more focused questions on specific aspects of transportation system performance. Many survey efforts focused on getting feedback from users on what should be changed and how it should be changed.
10. A strong linkage exists between performance measurement and asset management in Australia and New Zealand. In both countries, scan participants found that transportation officials have a much better handle on the changing value of their asset base than most transportation agencies in the United States. The team believes the U.S. transportation community could learn from these examples of how to better link asset value to decision-making via performance measurement.
11. Although all of the sites the team visited professed concern about environmental quality, all found it difficult to come up with area- wide environmental measures. This was the most disappointing aspect of the site visits for the team. New Zealand faces a significant challenge because recent legislation requires the national transportation agency to redefine its mission in the context of sustainability, or how its actions will contribute to a more sustainable land transport system for the country. The resulting actions over the next six months will be worth watching.
12. Do not measure too many things. The most important measures are those needed to influence budget allocations and investment decision- making. In situations where large numbers of performance measures were considered, lack of focus resulted in little influence on the decision-making process.
13. Post evaluations/ assessments should be part of performance measurement. In many cases in the United States, little effort is made to determine the aftereffects of transportation investment. One of the surprising results of this scan was the widespread use of before-and-after studies as a means of determining the effectiveness of implemented actions.
14. Performance measurement is a complex, evolving area of opportunity. The U.S. transportation industry is advanced in many areas of performance measurement, but till has much to learn. The scan team believes a coordinated, structured approach to sharing and advancing in the performance measurement area would serve the U.S. transportation industry well. This approach should emphasize the business model approach to assessing performance.

IMPLEMENTATION STRATEGIES AND RECOMMENDATIONS

The timing of this scan is most conducive to implementation of the scan results because many opportunities exist for disseminating them. For example, the results were reported to several AASHTO committees during 2004, including the Standing Committee on Quality in April, the Standing Committee on Planning in May, and others at the annual meeting in September.

The team has developed the following preliminary recommendations on further activities that should follow from the scan:

National emphasis area demonstration on safety ( FHWA ).The most integrated and impressive application of a performance measurement framework the scan team observed was in the area of safety. The team believes that the Australian model and the significant results achieved in the safety area are worthy of sharing and ultimately implementing in the United States. Two safety implementation strategies are recommended:

• Bring Australian safety leaders to the United States to tell their story to key groups.
• Encourage States to implement the best practices learned.

Data exchange and warehousing consortium for benchmarking ( AASHTO ). Develop an action plan to initiate a data exchange and warehousing consortium for benchmarking performance among participating States. The goal is to export State-level performance data to an external source for the purpose of comparing performance in a variety of service areas.

Performance measure ( PM ) research. Initiate research and disseminate findings (through FHWA and AASHTO ) on several performance measurement-related topics:

• Monetizing PM s—initiate a study on converting measures into cost-based numbers and targets by identifying the cost and/ or the benefit of providing the improvement on a monetary basis.
• Interrelationship of PM s—Initiate a study on the cause-and-effect impacts between PM s that examines the correlation between improving possibly competing transportation attributes.
• Multimodal tradeoffs—Initiate a study on the interrelationship between modal PM s. The study would expand on the "interrelationship" issue to include tradeoffs between modes.
• Outcome/ output empirical relationships—initiate a study on translating output indicators to outcome consequences.

Training (National Highway Institute) or guidance papers ( AASHTO ).Develop performance measurement courses and instruction aimed at executive and mid level leadership. Topics could include the following:

• Development and use of performance measures
• Outsourcing of products and services
• Best practices

Stand-alone overview document on the scan team's findings related to congestion and reliability performance measures. Develop a resource document on international and domestic practices on measuring congestion and reliability. Related publications could include an abbreviated pamphlet for distribution at conferences and meetings.

Conferences and meetings. Plan, develop, and implement conferences and meetings specifically oriented to presenting the scan findings and recommendations to a variety of transportation professionals.

Technical guidance. Develop and distribute through AASHTO, FHWA, and other stakeholder organizations guidance on various secondary PM topics:

• Auditable PM s—Provide PM design and tracking methodology for developing verifiable measure numbers and associated impacts.
• Before- and- after analysis—Prepare best-practice materials on this type of analysis. This would be on implemented actions and their relationship to performance measurement improvement.

Presentations of findings from the scan at appropriate regional and national meetings and conferences throughout the United States over the next year. Candidate venues include AASHTO annual, committee, subcommittee, task force, and regional organization meetings; Transportation Research Board annual and committee meetings; and meetings of organizations such as the Institute of Transportation Engineers, Association of Metropolitan Planning Organizations, and the International Bridge, Tunnel, and Turnpike Association.

Web-based distribution of materials. Investigate and select a logical Web-based home for the materials emanating from the scan, including reports, presentations, and implementation actions. Options include AASHTO, FHWA, or a university.

Follow-up review on the sustainability concept. New Zealand is attempting to incorporate sustainability into transportation decision- making. At the time of the scan, the national transport agency was six months away from proposing how it would incorporate sustainability goals into its performance measurement system. The team recommends that a Transit New Zealand official be invited to the United States after the agency implements the proposal to discuss the topic.