Input/output economic impact studies only tell part of the story
Economic impact studies have become a staple of postsecondary institutional accountability but the basic ‘input-output’ approach, while relatively simple and inexpensive to undertake, is telling only part of the story – and not the part that’s of greatest interest to the public or government decision-makers, according to a new report from the Higher Education Quality Council of Ontario (HEQCO).
Project description
Measuring the Economic Impact of Postsecondary Institutions is a high-level introduction to economic impact analysis for the postsecondary education sector, geared to those wishing to measure the economic impacts of their institutions or understand the methods, findings and limitations in studies done elsewhere. The report, by KPMG LLP, is based on a review of literature on economic impact methodologies as well as a number of publically available Ontario, Canadian and international economic impact studies.
Findings
Many postsecondary economic impact studies tend to relate to internal operations and short-term flow-through impacts on the local economy, such as the so-called multiplier effect, which mainly measures how much money is redistributed into the local economy by virtue of the institution’s presence. While generally accepted by the academic community and government funders in the past, such metrics rarely discriminate among the successes of different institutions and are silent on impacts of most relevance to the public, policy-makers and government decision-makers today. In the worst case, an institution with lackluster educational outcomes and poor research results can generate a large multiplier effect simply by spending money.
What we really want to know are the outcomes or returns generated by the institution fulfilling its mandate by graduating highly qualified personnel who contribute to the economy, making research discoveries that translate into innovation and commercialization, and providing service that enhances lives to the community. These “downstream” value-added outcomes and impacts take place largely outside the institution itself, and may take years to manifest. “In general, institutions have not recognized their power in helping explain their achievements to funders and the general public,” says KPMG.
Because downstream impacts are typically much more difficult to define and measure, they are often not measured at all, which can have a chilling effect on postsecondary activities that foster such impacts. “The key thing to bear in mind is to try to measure impacts as far downstream as possible and consider as many impacts external to your institution as possible,” says KPMG. “These external impacts for industry and society are of most interest to the general public and policy-makers, and the reason postsecondary education institutions are provided with public funding.”
Acknowledging that there is no one-size-fits-all solution, the authors also suggest that institutions link metrics to each institution’s differentiated mission and goals. As a result, both metrics and methodologies will differ among institutions, and governments should not focus on benchmarking one institution against another. “Although this lack of consistency may initially appear to be a negative, it is actually far more useful than using very simple metrics which, while consistent, are mainly correlated with expenditures rather than success.”
Using externally focused, downstream metrics will help institutions identify important impacts as they arise and to track their success over time. “This requires a different approach by institutions and industry liaison offices to developing their performance measurement systems and to conducting follow-up studies,” says KPMG, “but over time will be far more revealing of their true successes and the routes leading to them. This in turn can help improve programming.”
These approaches align with HEQCO’s overall drive to move the examination of postsecondary education from a focus on inputs (money spent, faculty hired, students enrolled) to outcomes (skills learned, graduates with jobs, measurable research impacts, service to the economy) and with its recommendations to foster greater differentiation between institutions.