Enhancing Children’s Learning with Edutainment – Can Following Set TV Schedules Make a Difference?

Working with children poses a great and unique challenge in terms of ethical considerations, logistics, and overall implementation. This calls for a collaborative approach between parents, caregivers, the children, and the research team. While this is usually a pretty aligned team of individuals (all trying to understand how to make the world more livable for children), it is also often made up of three groups of individuals who live vastly different lives, on different schedules, and with different ways of interacting in the world. 

At Busara, we’re currently working with EdTech Hub to effectively measure the impact of the Nuzo and Namia show, a new show by Ubongo aimed at improving outcomes that affect children’s learning such as literacy, social-emotional learning and gender attitudes, for children aged six to nine, while teaching them about different cultures in East Africa. We’ve been studying this show in Kenya since its inception. Our study involves working with children, teachers and parents in 7 out of 8 provinces. That is Coast, North Eastern, Eastern, Central, Rift Valley, Western and Nyanza. We excluded Nairobi for multiple reasons including the risk of existing over-saturation of the intervention to be studied and increased risks of spillover due to a dense and mobile population. You can read about the pilot stage here, then find out what happened during randomized controlled trials.

We’re currently wrapping up the stage where we try to understand what would get parents and children to actively adhere to the viewing schedule. Around this stage of the study, the work goes through one of the most collaborative stages where everyone, in their own environment, needs to work together on a regular basis to test what works and what doesn’t. 

We selected half of our sample population to be encouraged in subtle ways to stick to the schedule while the other half continued with their normal television viewing behaviors. In doing this, we were hoping to see which forms of encouragement worked to increase adherence. 

Setting the stage for collaboration: Pitching parents and caregivers

A week before the show aired, we invited selected parents and caregivers for a session where we planned to set the stage for our collaboration. We grouped schools and invited parents to central locations to exhaustively reach all the target respondents. Where we couldn’t group schools, we leveraged the existing relationships we had with the area chiefs and village elders to allow us to make in-person visits to the parents and caregivers.

A day prior to the session, parents and caregivers received a text invite that contained the session venue and a one time passcode (OTP), unique to them. The OTP helped us ensure that only targeted respondents showed up and participated in the sessions. At this stage, we realized a few of our participants had unintentionally restricted short code messaging. We were able to guide them in the process of unblocking the shortcode, which was important as messages were part of the larger encouragement design. 

The session started off with a one-and-a-half-hour section in which we had at least two field officers pitching both Ubongo, the makers of the show, and the show itself to parents and caregivers. They also had a chance to watch an episode of a slightly similar show created by Ubongo. Additionally, parents and caregivers were directly encouraged to note airing times and watch the show with their children. Below is the encouragement they received: 

To really get the benefits from watching Nuzo and Namia we think that your children should watch twice per week.

Nuzo and Namia is broadcast on Akili Kids!: a free-to-air television channel available on any digital or smart TV. No subscription is necessary. We’ll explain later how you scan with your digital or smart TV and an aerial and look for Akili Kids! under PANG channels.

The show broadcasts at 18:30 pm on Wednesdays and 10:30 am on Saturdays. Each episode lasts just over 20 minutes.

Next was a section on planning where we detailed the activities children engage in before, during, and after the show. The planning activity was geared towards identifying potential barriers to engaging with the show and collaboratively working to fix them with the parents and caregivers. The planning tool was then handed to parents and caregivers to take home. They were also reminded that they would receive weekly text reminders 30 minutes before the shows and quizzes immediately after the show. The text acted as an additional nudge for parents to remind their children to watch the show, and quizzes would help us understand how engaged the children were while watching. 

Additionally, parents and caregivers were also asked to sign a commitment form promising to watch the show at the allotted times and a viewing tracker that was to be placed in clear sight within the household. Finally, we gave them a poster to give to their children that encourages kids to learn and solve new problems as they have fun with Nuzo and Namia. 

Behind the scenes: Designing the SMS system 

TV show engagement is less structured and more difficult to measure than most other things. In cash transfer studies, for instance, implementers only need to ensure that the money was received by respondents. For the context of our project, we defined engagement as children watching and understanding the concepts presented in each episode. 

While it would have been ideal to ask children directly if they watched the episode, it was not feasible in practice to survey children weekly. Four factors were at play here that had to strike a balance in the final solution: 

• The need to measure engagement accurately — accuracy often involves a high level of detail. Especially when trying to measure something as unstructured as engagement. 
• How easy it is to interact with the system — Given that we were working with a sample of respondents with varying levels of access to technology and education, it would be important that the system is relatively simple to interact with. 
• How accessible the respondents would be — A system like this would be self-reported. As such, we were heavily reliant on respondents for their participation. 
• The cost of the system — Our project budget. 

Striking the balance between these factors made it pretty apparent that an SMS system would work best in this scenario as most parents and caregivers had access to mobile phones.

Our final system involved weekly SMS reminders about the show and concepts covered that week, as well as an SMS quiz after each episode. The quiz consisted of one multiple-choice question about which country the characters visited in that week’s episode. The idea is that a correct response to the quiz can be interpreted as engagement with the show. To incentivize parents to respond to the quiz, the SMS system would respond whenever the parent or caregiver answered. They would score 100 points each time they answered correctly. Monthly assessments were sent to the parent’s phone, showing how their child performed on the quizzes relative to other children in the sample, which would theoretically motivate both parent and child to answer the question correctly.

The main event: Unraveling the mystery of parental engagement — the SMS dilemma

Picture this: we had a stable version of our SMS system that allowed us to receive high-frequency data on a weekly basis. Parents and caregivers had been primed to interact with this system. The show was set to launch. 

We knew things weren’t going too well when we received the first batch of answers, and the numbers were dismally low.  Upon further investigation, we found out that around 10–20% of messages sent weren’t even reaching the parent’s phones. Then, even when the messages did land, only 30% of parents responded each week. Our current incentive system, which rewarded correct answers with points and showcased performance to each parent, simply wasn’t cutting it. 

But wait, there’s more. 

We thought deciphering the responses from parents who did engage would be a breeze, but it turned out to be a riddle wrapped in mystery. Given our design, with 60–70% of respondents answering correctly, it would sound like we could confidently quote that as the percentage of respondents who engaged with the show. But could we? Here are a few reasons why we are still perplexed.

Comprehension: Our SMS quiz was designed to be simple, asking a singular question about which country the characters visited in the latest episode, with three answer options. But even the simplest question can hide complexity. We realized that to answer correctly, the children had to not only watch the show but also remember the country. What if they watched but simply missed that detail? 

Parent–Child Communication: We directed our questions at parents and caregivers, assuming they’d communicate with their children after the show. But life, as we know, isn’t always that straightforward. If parents and caregivers watched alongside their children, they could easily answer correctly without consulting the young viewer. Our data would show correctness, but it wouldn’t reflect the child’s engagement. 

Guessing: The age-old dilemma of multiple-choice quizzes. There’s always the chance of a random guess leading to a correct answer, masking true understanding. In our context, we had to accept that our quiz method, while valuable, was inherently imperfect due to the possibility of guessing.

To make sense of it all, we embarked on a daring journey of statistics. By weighing the overall percentage of correct responses against the probability of guessing correctly (33.3%), we could confidently estimate that only around 12–15% of our sample had truly watched the show.

Looking ahead

To be a fly on the wall is every researcher’s dream. A fly on the wall is not subject to logistics, scrutiny, suspicion, system failures — maybe just the random fly swatter. Navigating the amount of collaboration necessary to achieve as close to this effect when working with children is a tricky balance between design, methods, costs, and people. 

We have a lot of challenges that we are yet to solve in this study, and undoubtedly more to come. Currently, we are considering several changes to our encouragement design (enter the intriguing concept of community-based monitoring schemes, a beacon of hope on our horizon) as well as approaches to measuring engagement that seek to improve the issues discussed. We hope that, with these improvements, the study will successfully evaluate the impact of educational television and contribute to the wider literature on educational broadcast content. In the world of research, challenges are the norm, not the exception. Yet, the commitment to overcoming these challenges and enhancing processes remains steadfast. These hurdles may slow progress, but they do not deter the mission.

This blog was reviewed by: Laila Friese and Jillian Makungu