U.S. Department of Transportation
Federal Highway Administration
1200 New Jersey Avenue, SE
Washington, DC 20590
202-366-4000


Skip to content
Facebook iconYouTube iconTwitter iconFlickr iconLinkedInInstagram

Office of International Programs

FHWA Home / Office of International Programs

Chapter 4: Observations and Lessons Learned

GENERAL OBSERVATIONS

The scan team has identified the following observations on the use of performance measures that would be of interest to officials in the United States. Many of these observations are of broad interest to transportation professionals, while others are of interest in a specific context.

  1. The scan team found examples in which the processes of setting priorities and making planning, investment, and management decisions use performance measures to a much greater extent than is typical in the United States. In cases where performance measures were used for priority setting, the process of using such measures represented a new level of organizational behavior. The vertical integration of information flow represented in the use of performance measures for different levels and purposes of decision-making characterized the better examples of performance measurement. The scan team did note that where overall policy goals and important performance measures were set primarily in a political process, transportation agencies often struggled to determine how the desired performance could be achieved.
  2. Perhaps the most impressive application of performance measurement, in terms of showing how the process can influence governmental policy and budget determinations, was in the area of road safety. Impressive results in reducing fatalities and injuries have occurred in some of the sites the team visited through a comprehensive program of engineering, enforcement, and education. Especially in Australia, the team noted that performance measurement activities of VicRoads in relation to safety were impressive, and should be understood and considered carefully by U.S. officials.
  3. A common frameworkto performance measurement was apparent in all cases the team examined. Each effort was related to a broader set of goals and objectives defined either by a legislative body or through a public visioning process. These goals and objectives led to the identification of transportation system-specific performance measures, often tied to the target values to be achieved in a future year. Strategies and investment actions were then chosen on their effectiveness in achieving desired performance outcomes, as well as on political considerations.
  4. Transportation officials in the countries visited appeared to have a general understanding of the difference between the concepts of outcomes and outputs. Outcomes were viewed as the ultimate characteristic of transportation system performance, while outputs were the products and services of the organizations that led to these outcomes.
  5. In each case, transportation officials explained the initial movement toward performance measurement as a means of providing greater accountability and visibility to the publicof the agency's activities. Different targets for credibility were apparent in the approach taken to performance measurement. In some cases, the initiative to enhance agency credibility reflected a desire to show legislative bodies that the agency was managing the transportation program efficiently and effectively. In others, the performance management effort targeted the general public to increase perception of agency performance. In still others, the effort resulted in enhancing the credibility of agency activities in the eyes of other agencies that held an important place in transportation program development ( e.g., the environmental protection agency in Brisbane allowed transportation agencies to self-certify their environmental activities) .
  6. In addition to accountability and public visibility, many officials commented that performance measurement can be used to educate both elected decision makers and the general publicon the role transportation plays in society and/ or the need for additional investment.
  7. Performance measurement was applied at several different levels of planning and decision-making. For example, many measures were targeted at strategic investment decisions on long-term improvements to the transportation system. In other cases, performance measures were used to manage network operations ( e.g., the rail system in Queensland and the tram system in Melbourne) . Different levels of application usually meant the use of different performance measures.
  8. In Japan, the national ministry established a core set of 17 performance measuresas a guide for all regional offices and prefectures ( states) in the country. In addition, the prefectures could establish their own measuresappropriate to their circumstances. This was an example of a phenomenon common to all applications—the need and desire to determine a small set of measures that were really important to an agency.
  9. All of the sites visited used measures commonly used in the United States. Road network congestionwas a transportation system measure found in all of the performance measurement applications examined. Officials often viewed this as one of the important issues facing their region and agency. Other network measures included accessibility and mobility provided by the transportation system, road safety, travel time, and trip reliability.The latter measure was found in the performance measures in Australia, British Columbia, and New Zealand. Understanding trip and travel time reliability is an important underpinning for developing incident management programs and implementing other actions to deal with nonrecurring congestion.
  10. Environmental measureswere also present in all of the performance measure efforts the team examined. Of all the performance measurement categories examined in this scan, environmental measurement was the one creating the greatest challenge to transportation agencies. The team found few examples in which environmental outcomes had been identified and the relationship between outcomes and transportation system performance had been defined, although Transport Canada has developed an important framework for establishing this relationship. An interesting challenge is occurring in New Zealand, where government policy requires transport agencies to define how their programs will help achieve a sustainable land transport system for the country.
  11. Measures of customer satisfactionwere found in all of the sites visited. The measures most often related to the average scores obtained from public surveys ( target values in some cases were quite low, e.g., exceeding 50 percent public acceptability in Vancouver) . Also, New Zealand's approach to customer satisfaction measures focused on identifying issues for which the customers were dissatisfied, and targeting organizational action to deal with those issues. One concern expressed by scan members was that customer surveys seemed to be used by many as a placeholder or crutch when more quantifiable measures could not be defined.
  12. Measures relating to transportation system securitywere not found in any of the highway and port performance measurement examples. In many ways, this was viewed as an American issue, although transportation officials in Queensland and New South Wales did say that security was becoming a more important issue and that a security indicator would most likely be incorporated into their performance management regime in the future. Also, the scan team did not interview many transit officials; the transit industry has made greater strides in measuring security than the highway sector.
  13. Many performance measurement applications targeted rural transportation network performanceas a specific category for measurement. This was particularly true in the Australian states visited, where each state has one major urban area dominating the state's economy. This was viewed primarily as a regional equity issue.
  14. The level of integrated data collection strategiesas they related to performance measurement varied from one site to another. In Japan, ITS technologies were used to collect data to support operations planning and decision-making, and they fed into system performance measurement ( e.g., the use of probe vehicles) . In Queensland, a data collection strategy is related to each performance measurement category that defines the overall performance measurement program. In New South Wales, a traffic management center is the cornerstone of RTA's efforts to collect data for system performance measurement. In Victoria, VicRoads has an extensive data collection program that relies on ITS -ype technologies. Some of the more successful performance measurement programs occurred in data-rich environmentswith a history of strong data collection and analysis.
  15. Before-and-after studieswere important elements of the performance measurement systems in Japan and Australia. The performance measurement cases the team examined in those countries paid considerable attention to discovering what impacts adopted or implemented measures have had on selected performance measures. The results of these studies act as feedback to the decision-making process and to determine the likely results of similarly adopted actions in other parts of the region.
  16. Graphic ( and visual) presentation of performance resultswas viewed as a key component of the performance-management process, as was the identification of measures that the public can relate to. Most officials involved in managing performance measurement efforts stated that unless top decision makers understand the information presented, performance measurement efforts will be ineffective. Visualization of information is thus critical to successful performance measurement.
  17. Comparing performance against peer organizations and jurisdictions was used in Japan, Australia, and New Zealand to understand better the key factors distinguishing economic and transportation performance, compared to other states. This comparison effort was used simply as a screening tool to identify differences ( in both quantity per capita and relative ranking) , which then led to a more detailed examination of why those differences occurred.
  18. The key measure of success for performance measurement itself is the degree to which it influences decisions and budget allocation. This was a difficult linkage to pin down during the scan visits. The most advanced application appeared to be in Queensland and Victoria, where the performance measurement process was important in determining program priorities for safety actions.
  19. Few examples were found where performance measurement resulted in multimodal investment tradeoffs. In most cases, performance measurement was implemented in a modally focused agency, so performance measures were targeted at decisions relating to the performance of the respective modal network.
  20. Measures relating to freight movementwere found in many performance measurement efforts. Examples of measures included commercial trucking travel time between economic gateways ( British Columbia) ; mode split for goods movement ( Vancouver) ; increase in average annual truck payload capacity, percent of illegally overloaded trucks, and freight rail delays and travel time variability ( Queensland) ; and freight productivity, access to ports, freight rates, freight-related infrastructure condition, and congestion effects on freight movement ( Victoria) .
  21. Organizational productivity measures were found in all of the performance measure applications studied. Agency managers viewed these as a critical element of the performance measurement exercise because they indicated the degree to which their agency was delivering the products and services needed to meet other performance objectives.
  22. Monitoring project delivery through performance measures was also a common approach in most of the sites visited. An interesting aspect of this project delivery monitoring was the effort to assess project performance against project expectations.
  23. In cases where performance management was most successfully institutionalized in agency operations, top management leadership of and commitment to the processof development and continuous usewere essential to get performance measurement past its infancy. This was particularly important for agencies where the organizational culture and even the culture of the society itself were not conducive to open and responsive performance-based planning and decision-making.

LESSONS FOR THE UNITED STATES

Similar to scan team observations, a large number of "lessons learned" came out of this scan:

  1. Safety was viewed as a strategic use of performance measurementthat has resulted in a significant decline in fatalities. The team believes a great deal can be learned from this application of performance measurement, especially as it relates to identifying strategies and actions that need to be in place to achieve reductions in road fatalities.
  2. Meaningful performance measurement is a product of extensive outreach and discussion and collaboration with partners.Performance measures are readily available and easy to create, but without a comprehensive ( internal and external) outreach process, their value as a behavioral influencer is limited. Open, inclusive planning processes are fundamental to good measurement of performance.
  3. In the best examples of performance measurement, officials were still refining what measures to useand how to make sense of the political guidancethey received. This suggested that performance-based planning and decision-making are never-ending processes, and must be viewed as such by top agency officials. This also implied that an agency should not measure too many objectives, only what it needs to make business decisions. Too many measures can be a burden on staff.
  4. The key to success for measurement is accountability. This translates into the need for understandable measures, and a systematic follow-through to determine the level of failure or success in meeting the objectives defined by the measure. This issue is important across all transportation areas, but has particular short-term relevance and application in the safety area.
  5. Performance measurement is most relevant when linked to decision-making, especially resource allocation. Elements to consider include allocation of funding at the program and project level, multimodal tradeoffs, and distributional equity. A multimodal approach is best, although the team found few examples of such an approach. Performance measures position agencies well to engage in debate, but may not necessarily be the determining factor in a decision, especially in the legislative arena. Measures sharpen and focus the debate, and help clarify organizational direction.
  6. Quality systems have been put in placeand appear to be replacing externally defined quality criteria ( e.g., ISO 9000) . Many agencies are redefining quality and finding that the ISO process does not meet their needs.
  7. For safety, enforcement strategies were the key to changing driving behavior and affecting overall success. The best-case examples go beyond the mandatory use of seatbelts ( alcohol testing and speed enforcement) and are succeeding at reducing fatalities. In addition, the team found that engineering strategies were important elements of an overall safety program resulting in significant short-and long-term improvements.
  8. Analysis is an important underpinningfor setting targets and determining the effectiveness of actions to reach these targets. The scan team found a much greater use of benefit-cost analysis as a method to determine desirability of projects, and a desire to determine after the project was implemented whether the initial benefit-cost analysis was close to the resulting performance.
  9. Customer surveysare an important element of determining organizational performance, if done appropriately and in a valid manner. Many countries appeared to be most successful at getting useful data from customer surveys by asking more focused questions on specific aspects of transportation system performance. Many survey efforts focused on getting feedback from users on what should be changed and how it should be changed.
  10. A strong linkage exists between performance measurement and asset managementin Australia and New Zealand. Scan participants had a sense that, in both cases, transportation officials have a much better handle on the changing value of their asset base than most transportation agencies in the United States. The team believes U.S. agencies could learn from these examples of how to better link asset value to decision-making via performance measurement.
  11. Although all of the sites visited professed interest and concern for environmental quality, all found itdifficult to come up with area-wide environmental measures. This was the most disappointing aspect of the site visits for the scan team. New Zealand faces a significant challenge because recent legislation requires the national transportation agency to redefine its mission in the context of sustainability, or how its actions will contribute to a more sustainable land transport system for the country. The resulting actions over the next six months will be worth watching.
  12. Do not measure too many things. The most important measures are those needed to influence budget allocations and investment decision-making. In situations where large numbers of performance measures were considered, a lack of focus resulted in little influence on the decision-making process.
  13. Post evaluations/ assessments should be part of performance measurement. In many cases in the United States, little effort is made to determine the aftereffects of transportation investment. One of the surprising results of this scan was the widespread use of before-and-after studies as a means of determining the effectiveness of implemented actions.
  14. Performance measurement is a complex, evolving area of opportunity. The U.S. transportation industry is advanced in many areas of performance measurement, but still has much to learn. A coordinated, structured approach to sharing and advancing in the performance measurement area would serve the U.S. transportation industry well. This approach should emphasize the business model approach to assessing performance, which represents a shift to a corporate business model for management.
<< Previous Contents Next >>
Page last modified on November 7, 2014
Federal Highway Administration | 1200 New Jersey Avenue, SE | Washington, DC 20590 | 202-366-4000