The Team That Taught a Machine to Assess Children’s Reading

About 80% of ten-year-olds in Pakistan cannot read and understand a simple text by the age of ten, on the World Bank’s learning-poverty measure. The Pakistan Institute of Education, part of the Ministry of Federal  Education and Professional Training (MoFEPT), can only close a gap this size if it can measure it—and measure it quickly enough to act on what it finds. Assessing reading child by child on paper is slow, costly and hard to run across the country’s many dialects and accents. By the time the data reaches the ministry to be evaluated for targeted interventions, the situation on the ground has often shifted, with recurring climate disasters exacerbating the situation.

Most AI models that work with speech are designed to infer audio input and transcribe—to convert what they hear into text as accurately as possible. What this team needed was different: not just to hear what a child said, but to judge whether they read it correctly—which requires the model to understand what ‘correct’ sounds like in the first place, across multiple languages and accents, for a child’s voice. That’s a much harder problem, and why this pilot required a collaborative effort across three countries rather than an off-the-shelf model. Getting there wasn’t a single leap either: over the course of the project the team tested and set aside more than one modelling approach and tried different speech-model architectures. They finally landed on a model that agreed with human markers closely enough to trust, with a small enough number of parameters to make it feasible for scaled usage. 

The idea is simple to picture. A child sits with a tablet and a headset and reads letters and words aloud, one at a time. The tool records them and, in real time, marks whether each was read correctly. In Pakistan, where the cost and effort of assessing reading at scale is enormous, automating it could transform what data decision makers receive, and at what frequency, in order to tackle the learning poverty crisis. 

The reading assessment technology began life in Australia, was rebuilt to accommodate African minority languages for a pilot in South Africa, and was then rolled out by teams in Islamabad to collect data to train the AI model for a Pakistan context. Three groups did the work: Neurabuild built the app that captures a child’s voice offline and the bespoke system that marks it, Wyeside Consulting ran the data collection in Pakistan’s schools and used local language experts to mark the recordings, and then Western Sydney University trained the speech models based on that marked or ‘labelled’ data. The teams had to improvise, work fast and on a small budget to pull this off.  

Over six months and across six countries, six ministries of education joined EdTech Hub’s Ministry of Education AI Challenge. Each Ministry worked with embedded AI experts to build their first AI prototypes. Meet the AI Experts introduces the people behind those tools: who they are, how they think, and what it takes to build AI a ministry of education can actually use. Pakistan’s prototype was built by a team, so we’re featuring several partners in this profile.

The conversation below is drawn from the project interviews, lightly edited and condensed for clarity. 

Where did this tool actually come from?

Ben, Neurabuild: It started in Australia, as a way of automating an assessment where you show a child paper cards with words on them, one at a time. In the assessment, there are 300 words: a hundred normal words, a hundred irregular ones, a hundred made-up ones. Normally an assessor sits with the cards, records the session, then listens back and marks it, which is hugely laborious. A university proved you could do it with voice recognition. They came to us to put it online. We then connected with the Gates Foundation and the University of Cape Town, who were doing reading assessments in African classrooms, and asked: can we digitise this and make it work in minority African languages? So we rebuilt the whole thing from scratch to run offline on a tablet and sync to the cloud when connectivity is available. We built the first version of the app in 48 hours and were collecting data on the third day and started iterating based on observation and user feedback. Pakistan is the next chapter of that same tool.

You drew on experience in Australia and South Africa to make the case for Pakistan. What are the commonalities across these contexts?

Alberto Soriano Diaz, the Ministry’s point of contact: Reading and literacy assessments are expensive, technical and hard to sustain, which is exactly why a ministry struggles to do them at the scale that’s needed. I’d seen this kind of approach work in South Africa, and I thought: if South Africa can do it, Pakistan can too. In Punjab alone, foundational-learning programmes reach close to a million children, with a lot of resources going into assessing them at the start and end of the programmes. Make that cheaper and you free resources for teaching. The point was never the technology for its own sake. It was to give the Ministry an affordable way to evaluate where children can read, and where they need targeted support.

Why can’t you just use an existing speech recognition tool—why did this require building something new?

Saeed Afshar, Western Sydney University: People misunderstand how hard this task is. Every speech recogniser on the market assumes the speaker is saying everything correctly, so the only problem is working out what the words were. Here it’s the opposite. We’re fairly certain the child will make mistakes, and the whole point is to catch them. That is infinitely harder. Using a general speech model for this is like using a hammer to drive in a screw. It’s the wrong tool, built for a much simpler problem.

How did you actually go about building a model that could tell right from wrong – and how well does it work now?

Sergio Chevtchenko, Western Sydney University: We didn’t land on the final speech-model architecture straight away. What we needed was a model trained specifically to judge whether a word was read correctly or not, rather than a general-purpose speech-recognition model built to work out what you probably meant to say rather than flag when you got it wrong. Our target was to get a model that agreed with our three human markers at least 90% of the time.
So over the course of the project we fine-tuned and compared several speech-model architectures on the recordings Wyeside’s team had triple-checked in the field. We used a large portion of the labelled or marked data set for the training and kept a smaller independent set for testing.

Accuracy wasn’t the only thing we were optimising for, either. The whole point of this tool is to run offline on a tablet in a classroom with patchy connectivity, so we want something that can eventually run on the device itself rather than depend on a server in the cloud. A smaller model that performs nearly as well is often more useful to us than a slightly more accurate one that’s too heavy to deploy at the edge.

A large multilingual model, Wav2Vec2 XLSR-53, came out on top for English and held up well on our early Urdu results too. Data2Vec wasn’t far behind on accuracy but is a fraction of the size — given our preference for something that can eventually run at the edge, that trade-off matters as much as raw accuracy.

A very important pattern stood out to us: model accuracy closely matched how much our human markers agreed with each other on a given word. Where the markers themselves argued about what counted as correct — the accented sounds Sam mentioned earlier — the model struggled too. That’s reassuring in its own way: the AI isn’t failing randomly, it’s running into the same ambiguity a human listener would. 

Saeed Afshar, Western Sydney University: And when we compared the AI’s total scores on recordings it had not been trained on, the AI also matched the human marker’s scores closely across the full range of readers — from children getting almost everything wrong to those reading fluently. That’s the result that tells us this works and is worth pushing further, possibly toward a fully offline, phone-ready version.

Why does it matter enough to be worth all this effort?

Ben, Neurabuild: Reading is the foundation of learning, and the window to support children to learn to read is narrow. In South Africa only about one in five children can read properly by the age of nine. By the time you’re trying to help a child in grade four, it’s often too late. If you can measure reading early, cheaply and accurately, you can act in time.

Sam Wilson, Wyeside: There’s a hard-headed case too. A national reading assessment in Pakistan, done on paper for around twenty thousand children, costs over a million pounds to run. The savings from doing this with a tablet are huge, and the data is better quality as well as the AI model reduces the subjective and human errors. For a government, that combination of cheaper and potentially better is the thing worth emphasising.

Sam, you collected the data using a tool similar to the one that will be leveraged for this AI model into real schools. What was that actually like?

Sam Wilson, Wyeside: Messier and more random than you might expect. We went into 26 schools in and around Islamabad, collecting student responses in both English and Urdu. It was important that we spent time with students in government schools, in both cities and more rural areas, and a mix of boys and girls, so the AI was able to familiarise itself with a range of accents, voices and nuanced colloquialisms. Pakistan has a very rich variety of languages and dialects that lead to different accents and pronunciations, it was important that the model was trained to catch these differences as accents rather than mispronunciations.

The motivation to cast a wider net presented some challenges. For instance, I drove half a mile up a street to find it blocked by a sewage pipe, so we parked and carried the tablets across fields in washing baskets. Every night we’d come home, charge fifty tablets in our bedrooms and living rooms until the place looked like some sort of chaotic data centre, and upload the day’s recordings. The app was updated ten or fifteen times during the fieldwork. We’d report the bugs each evening and Graham would fix them overnight.

Graham Withey, Neurabuild: That nightly loop was important. The app had to work offline on the tablets, sync when it could, and not lose a child’s recording.

What did the data throw at you that you weren’t prepared for?

Amina Batool, Wyeside: Getting wrong answers. To learn, the AI needs to hear both a child reading a letter correctly and a child getting it wrong, but almost every child was getting the early letters and numbers right. We started with grade two, then went lower to grade one to find struggling readers, but those children were often too small to manage the headset and the tablet. The lesson for next time is to find older children with low reading ability instead. Then there’s the language itself. 

In a way marking was harder than collecting the data. We triple-marked every recording with three independent markers per recording. This meant that we tested 1700+ kids, produced 400,000 recordings and triple marked 200,000 of them, meaning 600,000 recordings were marked – that is a lot of marking! Where markers disagreed we brought in local language experts. One Urdu letter can make a ‘w’ sound when placed at the start of a word and make an ‘oo’ sound when it sits as a vowel, so two different readings can both be correct. The experts sat for hours debating exactly that, and what they settled became the gold standard the markers and the model were trained towards. During this process we discovered that even within the experts there was a lever of subjectivity when deciding whether a pronunciation is incorrect or merely accented. This was the point where we realised how huge a task it is to train an AI model to evaluate pronunciations when there are no recognised objective conventions that cater to all accents and pronunciations.

Saeed Afshar, Western Sydney University: That human layer never fully disappears, and people find that surprising. As you collect more data the amount of checking needed trails off, but it never hits zero. You always want a little quality assurance at the top. Think of a factory: you still want one person at the end of the line checking what comes off it.

With the clock running and the budget stretched, what helped move this work along?

Sam Wilson, Wyeside: Lots of things helped: being super organised and in rapid communication with all teams; knowing the environment on the ground; getting the government on board from the start; making sure we had the right access and permissions for data collection; coordinating with school leaders to make them comfortable; hiring the markers and setting the right incentives for them to crunch through a huge pile of markings at speed (they were paid per recording marked). Importantly, having a fab team that worked super well (and fast!) and learning from people who’d done it before. Neurabuild were great. They had run this in South Africa, and beyond the public case study they shared the detailed back-end reports: how many field offices and markers they used, how they approached schools, how they paid people. We copied their marker leaderboard and their pay-per-recording system outright.  They were fab throughout. 

Ben, Neurabuild: And we built only what we needed, when we needed it. The first version of the app did nothing but record a voice and upload it. The marking system, the three-marker accuracy check, the back end, all of that came later, as the problem demanded it. When you’re that resource-constrained, you take one challenge at a time and build a model for each, rather than chasing a single general model that might take three years and never arrive.

Where does the tool stand now?

Sam Wilson, Wyeside: The tool is still in its early stages—what exists now is a proof of concept, not a finished product. To complete the whole assessment we’d have needed about 1.4 million markings. We did 600,000, so we marked a targeted slice rather than the lot. Finishing it isn’t complicated, it just needs more marking time: sit a team down for a month and crunch through the rest.

If you had unlimited money and time, what would you want to see happen with it?

Amina Batool, Wyeside: Get it embedded where the work already is. There are big foundational-learning programmes running in Pakistan, every province now has a foundational learning policy to implement, donors like UNICEF supporting the next phase, a mandate in Punjab for every school to use AI. This tool could be a pillar of any of those. You’re on the road to building something genuinely valuable. But building a tool, no matter how good, is only the first step. Governments have to own it, to get behind it, to be supported to use it and fund it, and to be able to analyse and act on the results it produces. 

Saeed Afshar, Western Sydney University: For me it comes back to one thing: getting as close as we possibly can to the interaction the very best teacher has with a child learning to read. Close that gap, and you change lives. I’ve seen it with my own children. That’s what keeps me going.

What does the Ministry need to do next to turn these learnings into a tool it can actually use?

Dr. Zaigham Qadeer, PIE: The Prime Minister declared an Education Emergency because of two problems: children who are out of school altogether, and a learning crisis among the children who are in school. The second one is where this tool fits in. To act on a learning crisis you first have to measure it, which means assessing learning poverty properly, and that makes a strong foundational learning assessment absolutely critical. Right now our assessments run at the national, provincial, and regional levels, and even at that scale they’re expensive to carry out on paper. Learning poverty plays out differently district by district, sometimes school by school, so that’s the level we need this data at if it’s going to drive targeted interventions rather than broad, generic ones. The only realistic way to do that without the cost becoming prohibitive is to leverage technology and AI to move these assessments to a digital modality.

That’s the case I took to the Ministry and beyond to the Economic Affairs Division (EAD). It was well received — officials and partners agreed that moving from a paper-based to a digital assessment was the right direction. Particularly emphasis was put on the initial and recurring costs, and rightfully so, these costs will determine how feasibly we can scale these assessments to the districts. It was also agreed to link this to the National Open Data Portal (NODP) rather than build a separate structure for it, since the NODP will be pulling in data from all the provinces and this should sit inside the system the Ministry is already building rather than as another parallel system.

All of this is encouraging — from the EAD to the Ministry to our development partners, everyone is acknowledging the importance and the need for an AI-powered digital assessment tool like this.

Special thanks to Dr. Shahid Soroya, now former Director General of PIE, who championed the idea and facilitated all necessary support needed from the ministry and other ministerial bodies for the project.

< Back to all interviews