In an excellent new paper, David Trick and Jinli Yang analyze Ontario’s existing surveys of college and university graduates, and look ahead to imagine the next generation of performance metrics for our public institutions. The surveys have been running for almost 20 years. Trick and Yang note that the data is of broad interest to higher education managers to measure outcomes at the system, institutional and (with some gaps due to ever smaller samples) program levels. It is also used by students, in public communications, and as a broad accountability metric. It is not perfect – Trick and Yang make important recommendations for improvements.
The story of the Ontario’s graduate outcomes surveys and other “key performance indicators” (KPIs) introduced at the same time (graduation rates, satisfaction surveys, and OSAP default rates) is about good things that happen when data is collected well and then made broadly available to all interested parties. Government might have mandated it, but institutions were heavily involved and have considerable ownership. Government may tout the annual “headline” metrics, but institutions have access to the underlying data and are using it within the academy for program and service level analyses. So are research bodies like HEQCO.
The KPIs illustrate how sharing data multiplies its application and effectiveness beyond what would have happened if it had been locked in a vault at government headquarters. Perhaps this is why Ontario’s KPIs are still flourishing two decades later, despite the fact that the costs of collection are considerable, and the headline numbers don’t generate many headlines any more. KPIs have become a broad management tool, particularly within the colleges (the university side was much slower in bringing up the response rate, and it still lags). From the institutions themselves: “These KPIs provide important insights into our strengths and help identify areas where improvements can be made”, and “The KPI surveys provide a unique opportunity for the college to receive feedback that reflects on student life from the moment an individual walks through the campus doors to the day that their experiences help them to land their dream job”.
Today, we are on the cusp of opening up another, powerful new data set in the same way, with revolutionary potential for our understanding and stewardship of the system, and for improving student outcomes: data linked with the Ontario Education Number (OEN). OEN is the warp to the weft of our other datasets, enabling us to assemble a tapestry of information about the many journeys into, through and beyond higher education.
Will we make OEN tagged data as broadly available as we did KPIs? The signs say yes. The OEN informs policy files at the centre of provincial priorities: mobility, equity of access, student success, and institutional differentiation. The data has been rigorously collected on a census basis; the sample is everyone and the risk of getting it wrong is extremely low. The pendulum on protection of privacy is swinging from 20 years of “play safe: don’t share anything” to a balanced approach that protects individuals while promoting evidence based policy and program design. The Ontario Ministry of Advanced Education and Skills Development is signalling a willingness to share OEN data, appropriately protecting privacy, with the broader community, and is taking steps to do so.
At HEQCO, we are looking forward to doing better research with OEN tagged data, just as we have with KPI data. Many of you are as well.
Martin Hicks is HEQCO’s executive director, data and statistics.