U.S. Department of Transportation
Federal Highway Administration
1200 New Jersey Avenue, SE
Washington, DC 20590
Notceable throughout the scanning study were the large amounts of performance data the agencies produced. Their annual reports, service agreements with the central government, and midyear progress reports all were voluminous in the depth, scope, and scale of their performance reporting. It was apparent that legislative and budget officials could evaluate the agencies' performance across a wide range of activities. They could produce results showing their accomplishments on highway asset conditions, highway and transit performance, environmental impacts, operating efficiencies, and public satisfaction with their agencies. The performance information was highly detailed and tended to track results over time (figure 14). It was clear the agencies had fully embraced performance management as the framework for running their organizations.
Figure 14. Highway fatalities in Victoria have fallen significantly.
It was apparent that legislative and budget officials could evaluate the agencies' performance across a wide range of activities. They could produce results showing their accomplishments on highway asset conditions, highway and transit performance, environmental impacts, operating efficiencies, and public satisfaction with their agencies. The performance information was highly detailed and tended to track results over time (figure 14). It was clear the agencies had fully embraced performance management as the framework for running their organizations.
The use of professionally produced reports for public information was common. The agencies produced annual reports, strategic plans, strategy reports, and other documents printed on magazine-grade paper stock, with full-color photography and advertising-quality graphics. While such high-cost reports may be criticized in the United States, they were commonly accepted in the countries the scan team visited as an appropriate means for conveying complex and important information to the public. All of the agencies produced a significant number of such publications each year. One agency staff member said the agency uses the reports as a recruiting tool with young professional job candidates. The high quality of the publications, which depict the importance of the agency's work, illustrates the personal and professional rewards that can come from working for the organization, the staff member said.
Reviews of the agencies' performance were common. These reviews were often required through statute or regulation, such as the quarterly progress reports SRA provides to central government budget officials in Sweden. The Results and Service Plans in New South Wales and the Statements of Intent in New Zealand also were followed with regular progress reports to central officials throughout the year.
Agency officials said that the updates keep the agency focused on results, but that they also have other important benefits. They said regular reporting to central agencies tends to increase the central officials' understanding of the issues confronting the agency.
"We rely on dialogue, dialogue, dialogue," said an SRA official in explaining the strategy for communicating needs, results, and constraints to central budget officials. SRA officials meet with budget officials monthly. Formal quarterly reports are required on their progress toward the annual operational plan. However, the reporting sessions tend to involve more informal dialogue than formal review of detailed performance metrics, SRA officials said. They said they and their agency benefit from the continuous interaction with budget officials because it enhances the central government's understanding of the agency's needs and what it realistically can accomplish within the constraints it faces. Instead of penalties for failing to meet a specific target, the discussion usually focuses on factors that influenced actual performance.
A New Zealand transportation official made similar observations about the importance of regular reporting to ministry officials. "Many of these communications are formal and many are informal. In reality, organizations are people. Much of this is how people interact. Our interaction is heavily dependent upon the trust and dialogue between people. It takes trust for government to say, 'We'll leave these decisions up to you.' Letting agencies have large budgets and influence does require a high level of trust in making these decisions."
The agencies commonly hold regular internal update meetings in which agency managers report on progress on agency goals.
SRA uses a Balanced Scorecard tracking process at all levels of the organization. Each manager has his or her unit's own Balanced Scorecard report that feeds into the organizational scorecard. Regular updates on progress are held with the senior staff, often monthly. All major aspects of the agency's Balanced Scorecard reporting are also tracked on the agency intranet, which includes voluminous reports on agency performance. SRA also has invested in intense leadership training with small groups of managers to ensure they understand the performance management framework of the organization.
In all of the agencies visited, performance audits were common and were usually required by law. The countries or states had Treasury officials or auditors general charged with the performance audit function. The auditors published reports and recommendations, which were incorporated into the management priorities for subsequent years.
"Do it with people, not to them" was both a direct quote and a common sentiment heard from transportation officials during the scan. From Sweden to New Zealand, transportation officials advised that carrots versus sticks, incentives versus penalties, and dialogue versus dictates were preferred in the intergovernmental management of performance.
Experienced agency managers urged cooperation when the central government sets measures for transportation agencies to achieve or transportation agencies set targets for local agencies. They universally advised that collaborative processes to assess strategies and set goals were preferable to performance mandates. The scan team found few cases in which one level of government penalized another for poor performance. The performance measures found were benchmarks for continuous improvement and dialogue, rather than milestones for penalty.
As mentioned previously, the British performance management process has evolved from a prescriptive process to a cooperative and collaborative one. The national performance management process began in 1998 with a long list of performance measures that the central agencies, regional governments, and local governments were required to report on. Web sites and news reports revealed that regional and local governments had significant difficulty with the time and expense the reporting requirements involved. The number of required measures steadily declined, culminating in 2007 in a significant change in the government's approach. In that 2007 update, four key changes were made:
Department for Transport officials said that the decade of trial and error in performance management taught the government that performance measures are a key component of improving government services, but they are best used in a collaborative, cooperative process. Performance measures work best when departments and agencies being measured have a sense of ownership of the measures and believe they reflect a shared priority.
"We also work with them to set the target, because we want them to own it," said a Department for Transport official who works with local governments to reduce congestion in their communities. "We very much went through a process to try to get them to own it…. One of the benefits is they now say, 'We much better understand our network and we much better understand how we can manage it.'…. The big lesson is this has to be worked out over a number of years. There is a lot of learning on both sides. The fact that we have persuaded the local authorities to do this illustrates that we came up with evidence to overcome their skepticism…. The locals said the money was good but the real value was, 'we felt important, we got to interact with the DfT, which we don't normally get to do. You shined a light on us and we got to interact on strategies with the DfT to improve.'''
The British experience demonstrated a repeated pattern of officials setting several narrow measures as surrogates for achieving a larger social goal. Over time, they lessened reliance on many measures of output for fewer measures of outcome. "Be absolutely ruthless about what you measure, how you measure, and what you mean," advised one British veteran of the performance measure evolution.
The British also advised that performance measures combined with financial incentives and penalties are only a very small part of improving local or regional government performance. They said they use a broad range of other strategies to convince regional and local governments to want to improve their performance, rather than forcing them to improve their performance. These strategies include the following:
The Department for Transport does not use the measures and targets in a punitive fashion to demand performance from local government, officials said. "Targets are not the only lever. We've moved away from looking only at measures (of whether local government accomplished desired results). We've expanded our evaluation to look at overall, 'What did it do? What does it want for itself ? Is it achieving those goals?' The measures form the basis for the discussion, but are not the only focus of the discussion."
"You'd better be very clear about what you want to do and what you want to get. Also don't do it to people; do it with people."
Officials said the lesson from their experience is that improving performance of local governments involves constructive engagement in the form of workshops, benchmarking, newsletters, peer reviews, and consulting assistance. The semiannual review of budgets and performance is only one part of the process to set and achieve goals. "You cannot do it only in the context of a spending review," one said. "Understand your delivery method. This is absolutely key."
"Have indicators but not targets. Think hard at the beginning to have broad indicators and categories but not targets. You want to be focused without having specific targets."
Figure 15. Tracking trajectory against target is a British strategy to achieve performance.
The United States and British transportation governance structures have significant parallels. In both countries, the central government works with a large number of regional and local partners to carry out government transport policies. Although HA provides more direct management of central roads than does FHWA, the British agency still relies on regional and local authorities to manage 97 percent of the country's road-miles. HA manages 2 percent of the network, but that includes high-volume routes that carry 31 percent of all British motorway traffic. The remaining 69 percent is under local jurisdiction.
When the government sets goals to reduce crashes, reduce emissions from transportation, or get best value for the money from transportation investment, it must rely on the performance of the regional and local governments that control most of the highway network. The central government in England works with 34 counties, 238 regional district governments, 83 unified districts that include both counties and cities, 33 London boroughs, the seven largest cities (which have significant autonomy), and six integrated transport authorities. In short, central government officials must work through a wide array of broadly distributed and individualist local, regional, county, and city agencies to achieve national performance goals, such as congestion reduction.
"The delivery of this becomes quite a bit interesting because we don't have control over this," said a Department for Transport congestion official. "It is a matter of 'what are our levers to deal with this?'"
Despite the complexities of the governance structure, the Department for Transport has engaged in performance improvement in a broad number of highly complex, highly distributed areas, such as congestion reduction in the 10 largest urban areas. Because the Department for Transport and HA do not control the cities' routes, traffic signals, transit operators, or rail operators, their approach has been one of education, consensus, and incentives. One of the few programs the scan team found that actually included financial incentives or penalties was the English urban congestion program. This program sets goals for modest reductions in congestion levels in urban areas. Despite the modest level of congestion reduction sought, the result would be a significant improvement in the growth rate of congestion, which had been steadily increasing for a number of years.
The program provides GBP60 million (USD96 million) in supplementary funding over 4 years for projects, operational improvements, and other strategies in the 10 urban areas. Payments to the cities are based on their performance and are staggered to incentivize them to exceed their agreed-on targets. If an area fails to achieve its target, the Department for Transport makes a qualitative decision on how the area has been performing, with up to 25 percent of its original allocated funding available. If the Department for Transport decides that the local government is not making a sufficient effort to meets its congestion target, no reward funding is provided. To date, no area has lost all funding, but two areas were deemed as making insufficient progress. They were subject to increased assessment to improve their strategies and received 25 percent funding that year.
New Zealand also has a negotiated target-setting process in which it works with local governments to set transportation performance expectations. Although a small country of 4 million people, it works with 85 local councils to set targets.
The New Zealand structure has some parallels to the United States in that it has regional councils that function somewhat like metropolitan planning organizations (MPOs) in collaborating with the central government to select projects. In fact, NZTA officials said that the agency proposes few projects that do not come from regional and local officials. NZTA's own state highway projects must be included for funding consideration in regional programs. In project selection, much of the emphasis flows upward from local officials to the central government. In two other important ways, the emphasis flows downward from the central government to regional and local officials. Local officials are required to develop Long-Term Council Community Plans (LTCCP), asset management plans, and financial management plans, which NZTA must audit and approve. NZTA is interested in two components of the LTCCPs, the asset management plan and ensuring that the community was consulted on projects for which financial assistance is requested. The effect of these processes is local input on project selection and programmatic or statutory requirements for local officials to demonstrate basic competency in areas such as asset management, local planning, and long-term financial management.
Regional land transport strategies must take account of local priorities while also demonstrating that national objectives have been considered. In addition to land transport strategies, local governments must adopt a community plan and an asset management plan. The asset management plan is a derivative of national financial sustainability requirements that local governments must demonstrate. The long-term sustainability of NZTA-owned assets is considered one component of the long-term fiscal solvency of the local government. The asset management plan includes targets for different levels of service for different types of assets. The necessary levels of investments needed to achieve those targets also must be identified. The asset management plans and their accompanying fiscal plans are subject to review by the Office of the Auditor General.
When asset management plans were first required in 1996, many communities hired consultants to develop their plans. NZTA officials said some viewed the plans as stand-alone requirements necessary to satisfy the central government. Over time, continuing consultation among NZTA officials, the auditor general, and local officials led the local officials to understand that asset management is an ongoing approach of implementing good practices, not a stand-alone report or information system. Although NZTA sets performance expectations, the agency does not audit local governments or their LTCCPs. Asset management now is widely accepted by the local governments because the nation has developed a wide cohort of local officials with a strong understanding of asset management principles and strategies.
The central government through NZTA has supported the local asset management evolution by providing data through the national Road Asset Management System (RAMS.) RAMS includes not only an inventory of all pavements, but also a history of all their treatments and a forecast of their deterioration, section by section. "You can actually tell in front of every house in New Zealand how many potholes they have and how many cracks," said one official.
Local and regional governments are also required to develop their own targets for crash reduction, transit operations, and walking and cycling goals. NZTA officials called the safety goals "tremendously useful" in encouraging local governments to focus on crash-reduction efforts over the long term. Although there are no national targets for transit use, regional targets are set that include factors such as total overall patronage, percent of income from the fare box, passenger-kilometers traveled, and service provision to the elderly.
The Queensland Department of Transport and Main Roads took coordination with local governments to a new level by leading the formation of regional Roads Alliances. These alliances are formal, collaborative groups of local governments that form asset management and safety management plans for the entire network of roads in their regions. These alliances and their networks cut across local boundaries and include all local and state routes in a region. To date, 32,000 km (19,883 mi) of local and state routes have been brought under the jurisdiction of Local Roads of Regional Significance.
The department proposed Roads Alliances as a regional and collaborative approach to provide seamless decisionmaking across the state on local, primarily rural, roads. Because the public expects consistent conditions across the entire network, the alliances offered a way to pool resources, knowledge, and decisionmaking for more consistent roadway performance across the regions.
Through Roads Alliances, members develop an asset management and a safety management plan for the road network in their region. A Roads Alliance board oversees the overall process, while Regional Roads Groups select projects and make local decisions. Each Regional Roads Group is supported by a technical committee of experts from the Department of Main Roads and Transport and local engineering experts.
Queensland officials said the Roads Alliances have led to regional approaches to transport planning and road works delivery, increased local government influence on state decisionmaking, and increased the availability of technical expertise to local governments.
Money is available for alliances to improve their capabilities in five performance areas:
A Transport Infrastructure Development Scheme provides funding to local governments of about AUD63 million (USD46 million) annually. To access funds, alliances must demonstrate road stewardship capability and that most of the money will be managed through the Regional Roads Groups and their cooperative prioritization process.
The Roads Alliances are pursuing three new capacity-building products. One is called NetRisk, interactive software that allows the user to identify high-risk areas of the local road network and evaluate countermeasures. The alliances are also developing a statewide approach to asset management and road safety, including the use of the NetRisk tool.
The Swedish model illustrates that an institutional culture of consensus can be used to cascade broad national goals into local performance. Two hundred and ninety Swedish municipalities govern all cities, towns, and undeveloped areas between the towns. No counties or unincorporated areas exist in Sweden. Using outreach, promotion of best practices, benchmarking, small grants to encourage behavior, and continuous dialogue, SRA has extended national government objectives into the practices, processes, and projects of the local municipalities. SRA's ability to direct municipal agencies is nonexistent, yet the Ministry of Transport and Parliament expect SRA to use persuasion, education, and benchmarking between municipalities to instill central government policies into local governmental actions. SRA officials acknowledged that local performance is not uniform, but they reported continuous improvement among local governments, despite a lack of control over them.
"It is a very cooperative process," said a Swedish official in describing SRA interaction with local governments. "It doesn't have to be. We could use our power to be road builders, but we don't do that. We think we reach our goals much better if we cooperate."
SRA's example illustrates the success of a transport agency reaching beyond its normal stakeholders to create alliances in other sectors to address cross-cutting issues such as vehicular safety and emission reduction. It regularly collaborates with automobile manufacturers on vehicle emission reduction and safety improvement. It also works with trucking firms to help them identify ways to increase vehicle efficiency, reduce emissions, and earn accreditation as "green" businesses. Such accreditation can be an important marketing tool in environmentally conscious Sweden. It has worked with automotive manufacturers to adopt advanced skid-protection technology (electronic stability program) in cars, which is estimated to reduce the risk of fatalities by 15 percent. SRA campaigns on behalf of vehicular safety and regularly works with the media to promote vehicular safety as an important automotive feature. As a result, it is difficult to sell an automobile with less than a five-star safety rating to the informed public, SRA officials said.
It was common in almost every agency visited to find a cooperative forum through which the central government and transportation stakeholders collaborate to establish transportation goals and targets.
In England, local authorities and development agencies advise the Department for Transport on goals, targets, projects, and priorities. In New Zealand, regional councils play a similar role. New Zealand also has a National Road Safety Council and an Asset Management Steering Committee that influence government policies in those areas. In Australia, the Council of Australian Governments represents state and local governments in ongoing consultation with the central commonwealth government. In each state, regional councils work with the state governments. Sweden and other European nations work with the European Union and the Organisation for Economic Cooperation and Development to set common goals in areas such as automobile safety, emissions, and benchmarking.
The Swedish experience also demonstrates the influence of benchmarking with other transport agencies to identify best practices. SRA has regular benchmarking efforts with Poland, Lithuania, Latvia, and the Nordic Association, which includes Norway, Denmark, Iceland, Finland, and the Minnesota DOT.
One of the most continuous transportation benchmarking and peer review processes is that run by Austroads, the association of road authorities in Australia and New Zealand. Since 1993, it has coordinated the publication of National Performance Indicators for New Zealand, as well as the states and territories in Australia.
Current measures include several categories of crash data and costs, pavement smoothness, a general measure of the benefit-cost ratios of projects, travel speeds, congestion, and a user satisfaction index. Under development are measures for the following:
The publication of performance information enables the eight state and territorial transportation agencies in Australia and the national authority in New Zealand to benchmark themselves at both a national and international level. The voluntary effort helps the authorities improve their effectiveness and efficiency as part of the overall Austroads goal to identify and implement international best practice in the management of roads.
Austroads officials reported that the effort began in an era when the Australian and New Zealand governments were focused on improving corporate governance. The agencies involved decided they should adopt common performance metrics and benchmark their performance before the central government mandated it. They believe that the widespread availability of such robust data may also have played a role in the Australian central government not establishing its own set of performance measures for states and territories.
The transportation agencies the scan team visited all were impressive in the depth and breadth of technical performance data they presented. At the same time, they consolidated the highly technical data into categories of performance that were understandable and relevant to the public. Rather than extensive technical and financial data, public reporting focused more on outcomes of relevance and clarity to the typical user.
"We make the good journey possible" is the Swedish Road Administration's motto. Its entire budget is categorized in four broad areas: "Capture customer needs, improve transport conditions, offer travel opportunities, and provide support during the journey."
The British Highways Agency spoke of three overriding objectives: "Safe Roads, reliable journeys, and informed travelers."
The Victoria Department of Transport's annual report has broad goals for environmental sustainability with calculations of energy consumption and greenhouse gas emissions down to the number of reams of paper the department uses. It appeared that the departments made consistent efforts to translate their goals and strategies into meaningful examples relevant to the public.
The experience of the international agencies was that performance management is a very effective way to demonstrate results and accountability, but agency officials also warned that it can take many years. The conditions on large systems such as transportation networks do not change quickly, but only after years of sustained efforts. Gathering accurate data, achieving consensus on goals with other partners, and constructing projects that actually improve conditions can take a decade or more.
Many noted that developing successful performance management systems is a long-term, iterative process that requires a commitment of funding and staff resources. The measures and targets evolve over time to meet changing fiscal and policy needs. While short-term, quickly implemented dashboards are helpful tools, the ultimate benefit of performance management is the continuous improvement of transportation programs to reach long-term societal goals.
"Evolution is my favorite word," said a New South Wales official. "It's about biting off what you can chew."
The British case study illustrates that successful transportation performance management systems require time, consensus, and clarity, said one British official. "You can't invent this kind of scheme overnight. It has taken us 10 years." The British spent 4 years developing their reliability measure, and they already plan to revise it to be more understandable to the public.
Important outcomes that are difficult to measure in the United States were equally elusive in the agencies studied. Such measures as transportation's effect on the economy, travel time reliability, and transportation's effect on the environment were not easily captured by the international agencies. All of the agencies expressed a desire for continued evaluation of ways to measure transportation's effects in these areas, particularly on the economy.
"Our lips are not necessarily connected to our checkbooks," said one official, who meant that despite a desire to fund broad outcomes such as economic development they were still limited to traditional funding categories. "The money pots still influence decisions. We don't fund quality of life; we fund transport."
A Swedish official said they struggle for and have never settled on a good metric for measuring mobility. "For a long time it has been hard to set a target for mobility. What is it really? Reliability? Travel time? What is it?" he asked.
A British official who works with hundreds of local governments said the more they parse measures such as "sound maintenance" or "accessibility" or "air quality," the harder it is for them to define good performance. Therefore, they use performance metrics as one, but not the only, indicator of performance.
All of the agencies support performance management, but they also spoke of a tendency for elected officials to emphasize short-term accomplishments over long-term trends. Several agency officials cautioned that while frequent budget reporting of results achieved short-term transparency, they feared that emphasis on "bean counting" skewed performance toward easily measured, short-term accomplishments. They advised that a better system was one that tracked accomplishment on long-term goals, which may be more ambiguous to measure but more important overall. Important issues such as the public's satisfaction with the journey, transportation's support of economic development, and transportation's link to environmental sustainability may be vitally important but difficult to measure in monthly increments.
One senior transportation official had worked in the national auditor's office, where he was involved for many years in measuring agency performance. As an auditor he was an advocate for strict measurement, but now as an agency executive he sees how strict focus on achieving specific targets can divert an agency's focus from more important but difficult-to-measure objectives.
"One of the dangers of performance management… for people like me in my former role at the auditor's was that performance management was an accountability tool. What it has generated is a dislocation between what is reported and what organizations actually use to manage performance. Focusing on accountability as a way to think about performance management forces us to focus on the things that are easily measured and which can be measured over a shorter time period. That has forced us to measure on short-term things we can count. Also, it leads to risk aversion to the type of targets you set. Accountability is not as helpful as it could be to drive performance. Speaking from my old perspective, I have strong feeling for the need for accountability. In a parliamentary system it is about how the Parliament is a check on the executive branch. But ultimately the Parliament and public are concerned about getting the best value for money. If we get perverse results, that is not the best value of money. The issue is how do we get performance management and targets that actually are going to drive our performance? One of the issues is there is a need for a longer time period. We may need more than a 12-month cycle. We need to look long term rather than 'what did we do in the last 12 months just so I don't get beat up.'"
Two other officials expressed frustration over the large number of measures and the frequent audit processes they experience. "We feel we are submerged in the audit process. By evaluating short-term auditing and accountability measures we often get distracted by whether we've made incremental gains rather than the end game… It becomes paralysis by analysis. You spend more time reporting what you are doing that you spend time doing."
"You are better off to have a few (measures) that everyone understands than this broad suite. You start thinking you need a wide array, but that becomes almost impossible to measure."
"We've been running 8 years on performance reports, and we find that everyone is so exhausted from dreaming up the measure and populating the measure that the consequences of failing to meet the measure is zero."
Several agencies cited examples in which their performance reporting was used to criticize the agency, either in the media or in political debates. Several acknowledged that such criticism creates a desire to set easily achievable targets, hide or downplay problems, or play "metric games," which undermine the transparency and accountability of performance management.
All of the agencies had some form of candid, confidential reporting of results to central ministries. Great Britain used the Prime Minister's Delivery Unit (PMDU) delivery assessment, an uncompromising, truthful assessment conducted every 6 months as a confidential tool. It was conducted as an evidenced self-assessment and action-setting exercise by government departments that was challenged and moderated by the PMDU. It drove the agency's actions for the next 6 months. It allowed for candid discussions with central governments about agency performance, steps that may be needed to improve performance, improvement ideas, and best practices.
The Australian Results and Services Plans were confidential, as were the monthly Swedish reports to the central budget authorities. While it is unlikely that U.S. transportation agencies would be able to produce documents that are exempt from public record laws, all of the officials said that a forum for candid discussion with central agencies about the realities of performance and why it may be lagging is an essential component of their performance management systems. The candid forums allowed realistic discussions about what hinders performance or the reasonableness of targets.
|<< Previous||Contents||Next >>|