Digital Personalised Learning in Kenya: Emerging Findings from a Randomised Controlled Trial Involving Pre-Primary Learners and EIDU

Researchers’ note: This blog was written after a provisional analysis of the midline data. Since the blog’s publication, endline data has been collected, the final sample confirmed, data analysed and results submitted for peer-review. The final publication and results will be linked here when the review is complete and journal article published.

This blog forms part of a wider research study on digital personalised learning to improve literacy and numeracy outcomes in Kenyan classrooms, and outlines emergent findings from a randomised controlled trial following the first two rounds of assessment (baseline and midline). The preliminary findings will be strengthened after further analyses and endline assessment.

Most research on digital personalised learning (DPL) has taken place in high-income countries, with less known about its effectiveness in low- and middle-income countries (LMICs) (see UNESCO’s 2023 GEM report). Existing studies have also mainly explored DPL as a ‘supplementary’ intervention, involving additional technology sessions separate from regular instruction, rather than aligned with the curriculum or integrated into classroom practice (Major & Francis, 2020). There is potential, therefore, to evaluate low-cost approaches that integrate DPL into classrooms aligned with teachers’ practices (UNICEF, 2022). 

Our in-process research in Kenya, conducted in close collaboration with Women Educational Researchers of Kenya (WERK), is evaluating the effectiveness of the EIDU DPL platform in improving learning in pre-primary classrooms. Developed and implemented in partnership with Kenyan county governments, the research combines DPL with the Tayari structured pedagogy (SP) programme. Over the final school term of 2022 and the full 2023 school year, a randomised controlled trial (RCT) is being conducted as part of a multi-year research study. The objective is to evaluate the impact of EIDU’s DPL platform and digitised Tayari SP resources (referred to in this blog as ‘DPL-SP’) on the development of emergent numeracy and literacy skills among pre-primary learners. The RCT complements other EdTech Hub research, including ongoing design-based research and an innovation sandbox, to explore the potential of DPL in Kenya.

Our initial analysis indicates that DPL-SP had a significant overall effect size of .425 in improving pre-primary learning outcomes in this context. 

These preliminary findings indicate that the EIDU DPL-SP model has the potential to positively impact emergent numeracy and literacy learning outcomes for pre-primary learners aged 4–5 in Kenya. This suggests promising implications for the use of DPL aligned with SP in LMICs.

The EIDU Model: Aligning DPL with Structured Pedagogy

EIDU provides a DPL mobile application deployed on low-cost Android devices in pre-primary and primary classrooms. Through the use of personalisation algorithms, learning can be tailored to the individual needs of each learner. During the RCT, two EIDU devices are used in classrooms. Aligned with the pre-primary competency-based curriculum in Kenya, EIDU supports DPL by offering high-quality content, including onecourse. Additionally, devices assist teachers by providing digitised Tayari SP resources, which have previously been shown to improve pre-primary learning (Piper et al., 2018). 

By integrating these two approaches into their adaptive software, EIDU offers a DPL model that is aligned with classroom practice, rather than serving as a supplementary tool. Learners benefit from DPL content that reflects teaching received in the classroom and is personalised to individual learning progress. Moreover, teachers receive ongoing support from early childhood development officers, who are employed by the county government and trained in the Tayari approach. The officers regularly visit teachers to provide feedback and guidance on delivering DPL and Tayari. 

This study is taking place in the context of EIDU’s ongoing rollout in Kenya. EIDU has agreements with 46 out of 47 Kenyan counties to provide the platform to all pre-primary schools. They aim to reach two million pre-primary learners in Kenya by 2026.

A Brief Overview of the RCT Research Design

An RCT enables the evaluation of educational interventions by randomly assigning participants to receive the intervention or be in a control group. This allows for the measurement of an intervention’s impact on learning. RCTs provide valuable insights into its effectiveness and can contribute to evidence-based decision-making.

The objective of this RCT is to investigate the impact of EIDU’s combined DPL and structured pedagogy model on emergent numeracy and literacy outcomes. The study includes:

• one treatment arm that receives EIDU’s DPL-SP model 
• one control arm with no EIDU platform or SP programme (although learners will receive both interventions following the RCT) 

Assessment points in the study are: baseline (October 2022, Term 3 of the 2022 academic year), midline (May 2023, the start of Term 2 in 2023), and endline (October 2023, the end of Term 3 in 2023). The RCT is taking place in 291 schools across four sub-counties of Murang’a county (Gatanga, Kandara, Maragua, and Mathioya). Randomisation was carried out at the school-level. Ten learners, aged approximately 4–5 years old, were randomly selected at each school for assessment. The International Development and Early Learning Assessment (IDELA) assessment tool, developed by Save the Children, was used. A comprehensive protocol providing detailed information on the research approach will be published in September 2023.

Emerging Findings From Baseline and Midline Assessment

Sample

A total of 2,885 learners were assessed at baseline, with an equal distribution between the control and treatment groups, as well as among learners’ genders. Out of these, 2160 learners were re-assessed at midline, resulting in an attrition (‘drop-out’) rate of 21.8%. 

Analysis

Preliminary analysis indicates that learners’ overall assessment scores increased from baseline to midline for treatment and control groups combined: a mean positive difference of 14.8%, which was statistically significant (p < .001). 

While no overall significant score differences existed between the treatment and control group at baseline (p = .153), the treatment group performed significantly better than the control group at midline, with a mean score of 49.7% compared to 43.5% (p < .001). The figure below visualises the overall difference in IDELA scores between the treatment and control groups at midline. 

The control group started with a baseline score of around 32.4% and increased by 11.1% at midline, while the treatment group increased by 17.3%. This means that students in the treatment group showed improved reading and numeracy skills compared to the baseline control group. The treatment had an effect size of .425, calculated by comparing the gain score difference between the treatment and control groups to their baseline standard deviation. 

Effect sizes for numeracy and literacy scores were .399 and .404 respectively. Additionally, emerging findings show only a small difference between male and female learners, with an overall effect size of .436 for boys and .416 for girls. 

The final round of endline assessment will indicate whether such between-group differences remain after learners and teachers have been using EIDU and Tayari for a full school year. The research will also continue to analyse if specific socio-economic factors are related to learning outcomes. 

Interpreting These Provisional Effect Sizes 

A key question is how to interpret reported effect sizes? It is important to exercise caution as this study is still ongoing. The findings presented are provisional and represent preliminary analysis between baseline and midline only. Nonetheless, results indicate significant learning effects.

In addition to demonstrating learning effects, effect sizes can also provide an estimate of equivalent additional learning time (based on previous research including Dowd et al., 2018). In our study, the overall effect size of .425 observed at midline can tentatively be interpreted as an additional 0.80 years of learning. Specifically, the effect size suggests an additional 0.89 years of learning in numeracy and 0.84 years of learning in literacy when using the DPL-SP intervention.

Assessing the practical importance of standardised effect size estimates for educational interventions lacks a universal guideline (Bakker et al., 2019). Consensus suggests that effect sizes should consider the nature of the intervention, the target population, and the outcome measure(s) used (Hill et al., 2008; Pigott & Polanin, 2020). As we continue our investigation, understanding how to interpret observed effect sizes in the specific context of this research remains a priority. Nevertheless, our provisional conclusion is that these results are positive and potentially educationally important. 

Ongoing Research Priorities 

Although preliminary findings of RCT data thus far are considered positive, the endline assessment scheduled for October 2023 will provide more detailed information and further confirmation. For instance, the endline will allow us to determine potential ‘ceiling effects’, or if there appears to be a higher rate of learning gains in the initial stages before the control group catches up. We will also further investigate who benefits from such an intervention and the cost-effectiveness of implementing DPL. 

Work is planned to evaluate potential threats to the validity of these findings, including in relation to observed attrition. Additionally, the implications of the study design, sample consisting of young learners, and setting of the RCT will be carefully considered before finalising conclusions and generalising to other contexts. Full RCT findings will be published following data collection and analysis in early 2024.