Using WhatsApp to Teach at the Right Level in Pakistan

In order to explore how technology can be used to deliver Teaching at the Right Level (TaRL) in low-resource environments, EdTech Hub partnered with Pakistan’s Sustainable Development Policy Institute (SDPI) on a study investigating the impact of learning outcomes through the use of EdTech during Covid-19 (Adil et al. 2021). This study has since been published as an open-access journal article (Adil et al., 2022). 

While the study showed us that TaRL can improve learning outcomes through the use of low-tech devices, it also helped us unearth valuable findings about implementing EdTech in Pakistan that shaped dynamic discussions at a dissemination event SDPI hosted on 25 August 2022. This blog explores how future EdTech programmes could be designed and implemented to best reach marginalised learners in Pakistan.

*Investigating the Impact of Learning Outcomes Through the Use of Ed-Tech During Covid-19 was developed by SDPI’s Fareeha Adil, Rabia Nazir, and Misbah Akhtar.

What did we do?

Knowing that three out of four school-going 10-year-olds in Pakistan cannot read a simple sentence (ASER, 2020), we wanted to explore methods of improving learning outcomes using scalable and cost-effective methods. The TaRL method has been proven to improve fundamental reading and maths skills by separating learners into groups by their learning levels rather than grade or age (Pratham, 2017). Traditionally, TaRL is delivered through face-to-face settings, but digital personalised learning (DPL) that supports TaRL has been proven to enhance learning outcomes at scale (Major et. al., 2021). With Covid-19 school closures limiting face-to-face interactions and with DPL technology being expensive to scale, we wondered how technology-enhanced TaRL could be delivered in a remote, scalable, and cost-effective way. Given that 90% of smartphone users in Pakistan access WhatsApp, we decided to use WhatsApp as a medium of delivery to reach students in Bahawalnagar, a socio-economically disadvantaged district in Pakistan’s south Punjab (Wilson et. al., 2022). Because government schools were less likely to deliver instruction online during Covid-19 school closures, our study assessed 12 randomly selected low-cost private schools for the three interventions, which included:

1. Digital training sessions for teachers
2. TaRL
3. Fortnightly formative assessments. 

Forty-three students participated in the TaRL-only intervention, and 35 received a combination of all three interventions.

Students were identified by learning levels in English, mathematics, and Urdu and clustered into different WhatsApp groups of 30–35 students accordingly. Instructors sent each of the groups the following via WhatsApp:

• Personalised learning material that matched the group’s existing learning level
• A learning problem (and a recorded video of the problem being solved)
• A short quiz. 

We found that TaRL had a significant and positive impact on the Urdu and English scores of students who were part of the TaRL-only treatment group but no significant impact on maths scores. In the process, the study revealed that low-tech devices could be used for effective education delivery in Pakistan — if certain implementation factors are considered.

Implementation lessons in Pakistan

Although the study focused on implementing TaRL to improve learning outcomes during Covid-19 in Pakistan, we learnt much about programme design that can provide valuable insights into delivering distance education. In dynamic discussions during SDPI’s dissemination event, stakeholders reflected on the study’s findings on programme design to explore how effective, equitable, and accessible solutions could be implemented in Pakistan. Some of these reflections are given below.

1. Actively involving school actors in programme design is important for the long-term buy-in of teachers and students 

The study found that school administrators were reluctant to run the interventions, fearing that it would place additional burdens on teachers and students and potentially distract students from routine homework and classwork. Specialist volunteers were hired to engage with students in the WhatsApp groups, but students tended to respond more frequently to their own teachers rather than the volunteers. As a result, we recommend that interventions should be integrated into existing teaching–learning processes at the school level to avoid overburdening both students and teachers.

With the finding that student engagement varied depending on the level of their teacher’s involvement, we see that teachers are at the very heart of EdTech implementation. At the dissemination event, stakeholders discussed ways to increase teacher involvement in programme design and whether there is potential for educationists across government and private schools to convene and exchange lessons learnt. This suggestion from stakeholders has the potential to not only deepen teacher buy-in but also increase the impact of interventions. A recently published Theory of Change by EdTech Hub and UNICEF Bangladesh finds that teacher engagement with EdTech content “would only result in high impact if the teacher was also designing, teaching, and reflecting upon activities for their learners and engaging in dialogue with others to understand the impact of these activities on their learners” (Clarke-Wilson et. al., 2022, p. 25). 

2. Consider how access and use of devices go beyond mere device ownership 

Even in households with mobile phone ownership, internet costs associated with accessing WhatsApp hindered learner engagement in the study. Even a small cost can limit poor learners from accessing interventions and benefiting from them. This highlights the importance of considering multimodal approaches to delivery.

The study’s focus on delivering TaRL through low-tech devices ignited a discussion on how educational content should be delivered to accommodate access challenges. Although much investment is being made in radio and TV content to facilitate distance learning, a government stakeholder argued that EdTech interventions in Pakistan should not be “reinventing the wheel” through investing in radio content. Instead, he suggested exploring multimodal approaches, including audiovisual lessons, interactive voice recognition (IVR), audio lessons over the phone, texts, WhatsApp messages, and educational exchange across social media platforms.

3. Community perceptions of technology must be factored into the planning of any technology-supported education programmes

Parental attitudes and perceptions must be considered or addressed before significant investments are made in digital education. For example, our study found that parents view technology as a medium of entertainment, and this could have an impact on its usage for educational purposes. Evidence globally suggests that parents and teachers are the “gatekeepers to girls’ access to technology” (Webb et. al., 2020). This implies that even in households with access to technology, other barriers may prevent their equitable usage.

The study’s recommendation to factor community perceptions of technology into EdTech programme design opened up a dialogue about how attitudes can form barriers to technology-enhanced learning. A stakeholder in academia emphasised that more studies should focus on the ways communities perceive EdTech. His reflections mirror findings from EdTech Hub and UNICEF Pakistan’s Digital Learning Landscape Analysis highlighting that many parents consider technology to be “a mechanism through which children would access entertainment, rather than educational content” (Zubairi et. al., 2022).

4. Parental and caregiver education levels are also a key factor when it comes to quality engagement of learners with technology

The children whose parents or caregivers had some level of education were found to be more engaged in using technology for learning than those whose parents or caregivers had no education at all. This means that accessing content is not the only challenge that can impact the effectiveness of EdTech interventions. Instead, parental and caregiver levels of education can impact the quality of student engagement itself. Reflecting on this finding, stakeholders identified various factors that impacted parental and caregiver  involvement in supporting learning continuity during Covid-19 school closures. Their involvement varied depending on socio-economic status, awareness, and level of education. Stakeholders emphasised that community perceptions of technology also impact the way content is delivered.

What did we learn?

TaRL has been proven to improve learning outcomes and improved student learning outcomes in English and Urdu in this study. However, with substantial investment being made in using technology to improve children’s learning outcomes  in Pakistan, stakeholders need to consider responding to access challenges, capacity limitations, and attitudinal barriers that may impact programmes’ effectiveness and scale. These considerations might include: 

1. Using devices that students and teachers already know how to use and have access to.
2. Effective learning must be enabled through a supportive environment of teachers, parents, and caregivers.
3. Community attitudes towards technology need to be addressed for parents and caregivers to champion technology-enhanced learning.
4. Programmes require teacher buy-in and should not add to the  burdens of school actors. This can be encouraged by including teachers in programme design.

One final challenge governments face in using rigorous research to inform their decision-making emerged during the stakeholder session. A government stakeholder expressed concerns that evidence is often not readily available to inform expensive and time-sensitive decisions about EdTech. His opinion on the challenges of producing and using evidence is linked to themes we have observed across countries (Mazari, 2022). To help bring the right evidence to decision-makers at the right time, EdTech Hub has partnered with Ministries of Education in Pakistan at the federal and provincial levels. Stay tuned for more updates on the invaluable insights these collaborations will no doubt generate.