Harvey P. Weingarten — Measuring Academic Quality: International Perspectives

In my last blog, I argued that the quality of a postsecondary education could be measured by the degree to which it met academic requirements or needs. I also argued that one non-negotiable and essential requirement or need of a postsecondary education is that students receive an education that equips them with the knowledge and skills they need to lead successful personal and professional lives. We already know the overarching categories of skills that they need to secure such success (disciplinary knowledge as well as basic cognitive, higher order cognitive, and transferable skills).

For the moment, I will ignore the disciplinary knowledge that students require. This content is usually defined by those in the discipline (although there are calls for more involvement of others outside of the academy, especially employers, to shape curricula). It is what faculty typically teach and what they largely evaluate. We seem to be doing a pretty good job teaching and measuring content and disciplinary knowledge.

On the skills side, though, several significant questions remain. From the long list of skills or competencies that are discussed, which are essential or the most important? What are reliable and valid measures of these skills and competencies, and what strategies or approaches are the best to make these measurements? And how should these measurements be used, by whom and to what purpose?

Our fixation with measuring quality is motivated further by the fact that one of HEQCO’s legislated mandates is to evaluate the Ontario postsecondary system and to make the results of that evaluation public. To this end, we are developing a dashboard that would succinctly present the state of the Ontario postsecondary system. We have a relatively good handle on measures for access and financial sustainability (areas we think a dashboard should contain), but we are stumped by what the best indicators are to monitor and evaluate academic quality. We do not want to ignore this component because, after all, measuring academic quality should be central to any evaluation of a higher education system or institution, no matter how difficult the challenge.

Luckily, we are not the only people worrying about this. So, we are convening in May a two-day workshop in Toronto of a group of experts from around the world who are working on this problem.

Participants in the workshop, entitled Higher Education Performance Measurement: Moving beyond Enrolments, are coming from the United Kingdom, Ireland, the Netherlands, OECD, Australia, US and Canada. These people were not chosen at random. They reflect a group of people who are actively working on the problem and, most importantly, who encompass a range of approaches to the challenge. They champion different measures. Some support a bottom-up, faculty-driven approach. Others promote a top-down, institution-wide standardized measurement. Some advocate direct measurement of skills. Others advocate for the use of surrogate, indirect markers of quality. Some are biased to input measures; others to the output side. Some believe in a strong role for government; others adopt a more hands-off government approach. Some focus on indicators most relevant to students. Others emphasize indicators more relevant to society and governments.

We asked workshop participants to address three questions:

1. What aspects of postsecondary academic quality, student learning or the educational experience do you feel are most important to measure or that give the best insight into these issues?

2. Specifically, how do you currently (or plan to) measure these attributes? What specific indicators do you recommend and use, and why?

3. How are, or should, these measures be used by institutions and governments (or bodies associated with government such as quality assurance boards, accreditation bodies)? How are these measures reported and disseminated, and, from your experience, what impact have they had? On this latter issue, we are particularly keen to hear views and evidence about how these academic quality measures have influenced postsecondary curricula and programs, government policy, and/or resource allocation by governments and institutions.

We have deliberately kept the workshop small and by invitation because we intend to shamelessly cajole participants to get maximal value from them.

For completeness and clarity, our focus on measuring academic quality has nothing to do with ranking institutions. Rather, it is an exercise in the continuous improvement of postsecondary education. You can’t manage or improve things you don’t measure.

We are looking forward to this workshop and will be sharing what we learn from it with you. The participants will also be submitting chapters summarizing their work and advice that we will publish as a book through the Queen’s School of Policy Studies in 2018.

Thanks for reading.