In my last blog, I argued that the quality of a postsecondary education could be measured by the degree to which it met academic requirements or needs. I also argued that one non-negotiable and essential requirement or need of a postsecondary education is that students receive an education that equips them with the knowledge and skills they need to lead successful personal and professional lives. We already know the overarching categories of skills that they need to secure such success (disciplinary knowledge as well as basic cognitive, higher order cognitive, and transferable skills).
For the moment, I will ignore the disciplinary knowledge that students require. This content is usually defined by those in the discipline (although there are calls for more involvement of others outside of the academy, especially employers, to shape curricula). It is what faculty typically teach and what they largely evaluate. We seem to be doing a pretty good job teaching and measuring content and disciplinary knowledge.
On the skills side, though, several significant questions remain. From the long list of skills or competencies that are discussed, which are essential or the most important? What are reliable and valid measures of these skills and competencies, and what strategies or approaches are the best to make these measurements? And how should these measurements be used, by whom and to what purpose?
Our fixation with measuring quality is motivated further by the fact that one of HEQCO’s legislated mandates is to evaluate the Ontario postsecondary system and to make the results of that evaluation public. To this end, we are developing a dashboard that would succinctly present the state of the Ontario postsecondary system. We have a relatively good handle on measures for access and financial sustainability (areas we think a dashboard should contain), but we are stumped by what the best indicators are to monitor and evaluate academic quality. We do not want to ignore this component because, after all, measuring academic quality should be central to any evaluation of a higher education system or institution, no matter how difficult the challenge.
Luckily, we are not the only people worrying about this. So, we are convening in May a two-day workshop in Toronto of a group of experts from around the world who are working on this problem.
Participants in the workshop, entitled Higher Education Performance Measurement: Moving beyond Enrolments, are coming from the United Kingdom, Ireland, the Netherlands, OECD, Australia, US and Canada. These people were not chosen at random. They reflect a group of people who are actively working on the problem and, most importantly, who encompass a range of approaches to the challenge. They champion different measures. Some support a bottom-up, faculty-driven approach. Others promote a top-down, institution-wide standardized measurement. Some advocate direct measurement of skills. Others advocate for the use of surrogate, indirect markers of quality. Some are biased to input measures; others to the output side. Some believe in a strong role for government; others adopt a more hands-off government approach. Some focus on indicators most relevant to students. Others emphasize indicators more relevant to society and governments.
We asked workshop participants to address three questions:
1. What aspects of postsecondary academic quality, student learning or the educational experience do you feel are most important to measure or that give the best insight into these issues?
2. Specifically, how do you currently (or plan to) measure these attributes? What specific indicators do you recommend and use, and why?
3. How are, or should, these measures be used by institutions and governments (or bodies associated with government such as quality assurance boards, accreditation bodies)? How are these measures reported and disseminated, and, from your experience, what impact have they had? On this latter issue, we are particularly keen to hear views and evidence about how these academic quality measures have influenced postsecondary curricula and programs, government policy, and/or resource allocation by governments and institutions.
We have deliberately kept the workshop small and by invitation because we intend to shamelessly cajole participants to get maximal value from them.
For completeness and clarity, our focus on measuring academic quality has nothing to do with ranking institutions. Rather, it is an exercise in the continuous improvement of postsecondary education. You can’t manage or improve things you don’t measure.
We are looking forward to this workshop and will be sharing what we learn from it with you. The participants will also be submitting chapters summarizing their work and advice that we will publish as a book through the Queen’s School of Policy Studies in 2018.
Thanks for reading.
3 replies on “Harvey P. Weingarten — Measuring Academic Quality: International Perspectives”
I am glad to hear about this workshop as I believe that educators need to give both knowledge and skills to students. I look forward to hearing about the results of this workshop.
No one is absolutely required to know what they are going to do when they graduate. It’s great if they do, but it isn’t a requirement. Nor should programs of study always be unambiguously connected to the student’s career path. That said, unless one is pursuing a clear path to a specific job, within the context of a community college training program, or even pursuing an advanced professional program (e.g., medicine) it grows increasingly harder for young people to know what they want to “do” with their education and whatever portion of their lives they want to allocate to that path.
That’s not a criticism of post-secondary education; more a reflection of the contemporary nature of work and careers, and the ever-expanding nature of programs and curriculum to try and keep pace with that. That said, part of what “quality” consists of is in helping young people to figure out their direction, and how to leverage their higher education in a way that serves them, without any meandering odyssey being necessary.
So, a suggested 4th question, if time and interest permits: What role, if any, should student career-guidance and/or planning, provided by institutional programs, play? Should connecting students with the world of work and career possibilities (as we saw so exquisitely illustrated here) be considered an essential part of “quality”?
Is this meeting open to others in the sector?