The debate over PPP in North America has frequently suffered from oversimplification. In both Canada and the US, critics of PPP frequently rely on simplistic comparisons of borrowing costs for PPP project companies and sub-sovereigns.

Ontario – one of the most active, and influential, jurisdictions anywhere for PPP – could present a large and reliable dataset for assessing whether PPP meets its promises. The province has brought 47 PPPs to completion, and another 20 are in construction.

The auditor looks over AFP

In December 2014 the Office of the Auditor General of Ontario issued its annual report for 2014. The report included a section on alternative financing and procurement (AFP), the name that the province uses for procuring infrastructure with PPPs.

The AFP section of the annual report ran to 24 pages, including two appendixes and space for Infrastructure Ontario (IO), the province’s procurement agency, to respond to the comments by the auditor, Bonnie Lysyk. It made a thorough attempt to try and compare two types of procurement with very different types of risk transfer.

The report praised the crown corporation’s record of bringing projects to completion on time and on budget. But it offered specific criticisms of the way it had occasionally priced design, permitting and residual asset risks. According to a news report shared with IJGlobal by IO, its chief executive Bert Clark was “stunned” at the report’s findings.

The report was almost immediately the subject of wilful misunderstanding. In this regard, the auditor’s own press release on the report was probably the most culpable.

The report said that there was a difference in costs between AFP and conventional procurement of C$8 billion, but almost immediately turned to IO’s explanation for the difference, most of it down to the way that IO prices the different risks associated with infrastructure procurement. The report then examined these explanations one by one.

The office’s gloss on its own report presented the cost difference without any context, and the auditor’s examination of the validity of IO’s explanations as “other findings.” The release, unlike the report, made no distinctions between design-build-finance-maintain and design-build-finance projects.

Unhelpful reactions

The Canadian Union of Public Employees (CUPE), a consistent opponent of PPP, seized, predictably enough, upon the headline figure, but also on the weaknesses that the auditor identified in IO’s value-for-money calculations. CUPE devoted particular attention to criticism of IO’s procedures for retaining conflict of interest declarations.

Supporters of AFP – including the Ontario Chamber of Commerce and TD Economics – argued that the C$8 billion number was an oversimplification, without noting that the auditor had been at pains to delve into the pricing of the risk transfer that explained the gap.

TD’s report included a chart that highlighted improvements in pricing on PPP debt in the last five years. This was designed to counter the argument that high spreads on PPP debt relative to government borrowing account for much of the nominal cost difference.

TD also noted that the high proportion of build-finance projects in Ontario’s PPP portfolio meant that the totals included some projects where cost totals included life-cycle risk assumption, and a number where it didn’t.

The Ontario Chamber was less measured, accusing the auditor of using “baffling logic,” and pointing to four directly procured projects that went significantly over budget. It overlooked the section of the auditor’s report that said “we acknowledge that there are examples of recent projects delivered by the public sector that have experienced cost overruns.”

The report, significantly, continued: “There is no empirical data supporting the key assumptions used by Infrastructure Ontario to assign costs to specific risks. Instead, the agency relies on the professional judgment and experience of external advisers to make these cost assignments, making them difficult to verify.”

Towards a useful procurement benchmark

The auditor has highlighted the fact that, nearly ten years after the first PPPs in Ontario, there is still no robust independent data to compare the potential for cost overruns on PPP projects with that on direct public procurement.

The Melbourne Engineering Research Institute carried out a benchmarking exercise in Australia in 2008 that covered only long-term concession-based PPPs. That report noted that most of the (primarily UK-focused) comparisons suffered from small sample sizes, and the methodology was open to criticism.

And so the auditor has asked that “Infrastructure Ontario should, in conjunction with the Ministry of Economic Development, Employment and Infrastructure, gather data on actual cost experience from recent public-sector infrastructure procurements and alternative financing and procurements (AFPs).”

The request coincides with a recommendation that the ministry use IO more frequently to advise on traditional procurement. The mandate is a huge opportunity, because it would expand IO’s mandate into almost all of the province’s infrastructure procurement, and give it access to more data.

In a response to IJGlobal’s enquiries, IO stressed that “there is still significant empirical evidence available on cost overruns in the traditional model, along with specialist professional firms who estimate and track project costs,” and says that it “continues to develop its data based on the best available information.” It did, however, direct IJGlobal to a report from Deloitte highlighting a lack of key data that would allow for comparison between the two procurement methods.

The auditor’s findings arrive as IO prepares to release a new value-for-money methodology. IO says that the new methodology will draw on “input from industry experts and key stakeholders,” and recognises the “impact of private-sector innovation and differences in approach to lifecycle management.”

Whether or not the new methodology quiets PPP’s critics in Canada, the report acknowledges that PPP and IO are now central to infrastructure procurement in the province.