By Stefania Kuczynski & Jacqueline Whyte Appleby
The Ontario Council of University Libraries (OCUL) is a member-based consortium of academic libraries. In 2024, OCUL launched the OCUL Artificial Intelligence and Machine Learning (AIML) Program, which aims to promote responsible, ethical AIML use in the academic library environment while building related knowledge and skills across the OCUL membership and beyond. The program consists of five distinct projects. One of the projects, focused on enhancing access to government documents collections through AI-generated metadata, is part of HEQCO’s AI Consortium. Visit the OCUL website for more information on OCUL’s AIML program and projects.
AI is often spoken about as a tool to make our lives easier and our work more productive. This can be true, but getting these AI tools to function in a way that is consistently helpful requires significant human labour!
Figuring out how to ask the AI models for useful results is, as anyone who has worked through prompts for large language models (LLMs) knows, a huge amount of work. Prompt engineering is the process of creating and fine-tuning instructions for LLMs. Getting high-quality outputs requires significant prompt engineering, including writing, tweaking and re-writing sets of instructions. We have found that many of these LLMs require very, very specific directions, because they each have their own challenges and quirks. Some read instructions, no matter how specific, as mere suggestions. Some follow these instructions for the first five pages and then ‘forget’. Some can’t resist adding additional context and notes. In the end, the AI tools are only as good as our instructions.
Getting AI tools to function in a way that is consistently helpful requires significant human labour
As part of HEQCO’s Consortium on AI, our project team — consisting of librarians, developers, systems support specialists and co-op students — is exploring the application of generative AI and optical character recognition (OCR) technologies to improve the metadata quality and discoverability of libraries’ digital collections. This project will benefit the discoverability of 50,000 government documents digitized by the University of Toronto Libraries, which are openly available on the Internet Archive.
The first phase of our project involved testing multiple OCR tools to determine which one was most suitable for our project (you can read about it here). Now, we’ve begun work on the next phase: metadata extraction. Identifying details like author names, publication year and document type is essential for efficiently discovering, organizing and leveraging the valuable information found in these resources. We’ll start with a small number of documents and a series of prompts and then test, fine-tune and record results as we go, improving prompts iteratively. The modification of prompts will be based on our own findings, as well as feedback from the government information community and changes in the rapidly evolving AI landscape.
Identifying details like author names, publication year and document type is essential for efficiently discovering, organizing and leveraging the valuable information found in these resources.
We provide high-level instructions to the LLMs, including Llama4, Qwen and Gemma3. Here is an example of the specific language we use in leading prompts to get well-structured output from Gemma3:
You are an assistant designed to follow instructions carefully. Your primary role is to process scanned government documents. You should be thorough and methodical. You can only respond with JSON data, do not give additional context or comments before or after the question.
Note that JSON (JavaScript Object Notation) is a standard text format that is widely used to structure and share data.
From here, we move on to prompts targeting specific metadata fields. Authorship, for example, includes individual authors, editors and corporate bodies who may be authors (frequently the case with government documents). Our instructions must be detailed:
Please extract the following information from the document text: (1) authors — a list of name or entities responsible for the document, which may include individuals, committees, ministries, agencies, or groups related to a ministry, including any person who signed an introductory letter. If there is no clear author, use the responsible level of government as the author (e.g., “The Government of Canada”); and (2) editors — a list of names of individuals or entities credited with editing the document. Each list should contain only strings representing the respective names. Return the result in JSON format with only the fields “authors” and “editors”. Do not include any explanation, note(s), reasoning, or additional text. Only return the JSON object.
Here is an example document we used for testing, released by the Ministry of Municipal Affairs of Ontario in 1986:
When we examined the LLM output from the authorship prompts, we got different responses: only Gemma3 understood that the Ministry of Municipal Affairs was the author of this document; Llama4 and Qwen were unable to identify an author. As we move further in this testing, we’ll need to decide which LLMs are worth continuing to tweak, and which are simply not a good fit for the questions we need answers to.
In these early days of prompt-engineering testing, we are also comparing our outputs with human-produced catalogue records. While many of the documents have no metadata, there are some that have fulsome MARC (Machine-Readable Cataloging) records — an older standardized format used by libraries to store and share metadata — and this is a useful way to see where our records fall short. In the future, we may be able to identify where older records are lacking, and it will be exciting to see where our LLM outputs can possibly enrich or enhance existing records.
Government documents are full of quirks, and their histories and contexts are essential in understanding what makes an appropriate and useful output. We are lucky to have support from the Ontario government information experts who understand how the world of government information is structured and used. When the experts say a prompt is not performing as it should, new questions or parameters may be introduced to our prompts.
We’re frequently sold scenarios where AI produces astonishing results instantly with little human support, but this is not the case. Our project is labour-intensive, and the goal is not to replicate work being done by humans but to see where we can use AI tools to improve the discoverability of content that has long been undescribed. The tools exist and hold great promise, but they require investments of time and expertise to be properly configured and used.
Our goal is not to replicate work being done by humans but to see where we can use AI tools to improve the discoverability of content.
About OCUL and Scholars Portal
Scholars Portal, the digital service arm of the Ontario Council of University Libraries (OCUL), provides shared services and technical infrastructure in support of academic teaching, research and learning in Ontario’s universities. Together, Scholars Portal and OCUL respond to the needs of post-secondary institutions through innovative information services and ensuring access to and preservation of research and content.