Review #2
AI tutoring evidence, and how do we remember things?
This week I will be discussing two things, first some evidence on using AI as a tutor, and some thoughts on memorisation, repetition and what this can mean for higher education. I’ll then consider the synthesis of these two things, and practical steps.
AI tutoring
Personalised AI-tutoring has been identified by multiple authors and AI companies as a potentially important use case for generative AI. With the release of the new version of ChatGPT and the introduction of OpenAi for education, as well as developments such as Khan Academy’s AI tutor, we are closer to this becoming the norm for educational settings. What would be nice is some evidence on the effectiveness of AI tutoring, and we are starting to see some of this now.
This paper by Henkel et al is randomised controlled trial of an education intervention using an AI tutor to help teach Maths to Ghanaian school children. This video captures elements of what the app is doing:
Important elements include:
personalised learning that is suited to different abilities.
hints that help the student if they get things wrong (this is where the LLM AI element starts to come in)
The AI element comes in again when having conversations within the app about the subject matter, getting more detail on why your answers may be right or wrong.
The study finds both a statistically significant positive effect on test scores, and a relatively large effect, equivalent to an extra year of schooling in fact! This is important as many interventions may have a statistically significant effect, but still do very little in practice, therefore being unlikely to justify their cost.
The main weakness of this study in my view is that the intervention is two 30-minute sessions with the app whilst at school, so we aren’t really looking at the use of this out of an educational setting that we are most interested in. Much of the sample (around a quarter) dropped out due to low school attendance and it is these students that we might be most concerned about and could benefit the most from an intervention of this sort.
As the authors discuss, there is also a technically limiting factor in that the study was not blinded, so there may have been a behaviour change by teachers towards students in different arms of the trial that then affected the results.
Like any evidence, we should be cautious, but this offers some support that AI-tutoring can have a positive impact. Much more evidence is needed before we are very sure, and experiments to be done, but this is likely to be something we all have to interact with in the future.
Scott Alexander on memory and learning
Scott Alexander (SA) recently wrote about education, and specifically the degree to which students actually remember anything they are taught. Survey evidence and introspection tells us that much (likely most) of what we spend many years learning at school, is fairly easily forgotten.
However we do in fact remember and know a great many things, not everything is forgotten, what explains this? SA uses the example of things we likely remember because we are reminded of them repeatedly, and then delves into some harder evidence (though the overall level of evidence about this seems not very robust) about how repetition of information affects memorisation. The repetition can be in a formal education setting, but also in a broader cultural context. An added confounder is that some people will have opportunities for repeated learning because of their own behaviour, they choose to read books or engage in other hobbies that give them the opportunity for repetition and learning.
Alexander concludes with a fairly negative assessment of education. Not much gets remembered, the repetition that would allow for deeper learning is not normally well planned, and is as likely to come from a wider social context as it is from formal education. There are plenty of people who know a lot of things, but this is more because they have habits of reading and other hobbies that repeatedly put facts in front of them that they will then take on long-term.
My own experience suggests to me that this story about memorisation and repetition is largely true, and the features of a university education do not help us get this right. In particular the siloing of learning into distinct modules taught by different staff and often with little programme level overview means there is little coordination of material to allow for well timed repetition that will reenforce important ideas in a way that will aid memorisation. I have been particularly worried about this with my own students recently as we have moved to a block model of teaching where students are only on one module at a time, but doing this one module in an intensive fashion. I teach first year introductory microeconomics in the autumn, then the same cohort come back for their intermediate microeconomics in the winter a year later, with little in the way of any other microeconomics in-between. The consequence being that they have retained very little in that period, we have to do a lot of catching up, leaving little time for new material.
What could improve the situation? Firstly a programme-level view of what they most important ideas are, and how to sequence them throughout the course. This would require coordination between multiple course leaders in a way that rarely happens at university (at least in my experience). It would also mean sacrificing breadth of knowledge for depth, but this is no bad thing. Too often I have heard academics worry about discussing a given number of ideas, with little consideration for whether students actually learn or retain those ideas.
The other argument for higher education that stems from SA’s analysis is that we can help inculcate a love of learning. Most learning by students is done independently so we need to provide a broad structure and let them get on with it. Maybe I am pessimistic, but this seems like something that for most people would have happened already. How many 18 year olds who don’t really enjoy learning, come to university and then discover the love of learning they never picked up during 14 years in school?
Am I being too miserable, can anyone give a more positive vision?
Conclusion and practical steps
I believe these two topics relate to each other. We have a model of learning that requires well timed repetitions, and the the development of AI-tutors that can directly help with this, especially in a higher education setting. What steps can we take?
Have a look at the video above and the link to Khan academy to get an idea of the technology that is now being developed. If your students aren’t using something like this now, they will be soon, and being able to integrate this into your own teaching will be vital.
Think about how students could use an app like this for your courses. What do they struggle with? What are the important ideas? How do you assess them? How does it complement existing resources?
Think about the element of repetition, when you introduce an idea, when do they have the chance to see it again, to implement it, to put it into practice? Is this being done in a way that will help them keep it in their long-term memory, to use it later in their degree or their career? If not then you need to talk to others who are teaching on your course and do better, perhaps with the aid of technology such as an AI-tutor.
(hey look, the image has some correct words now!)