Re-evaluating AI in Education: Balancing Enhancement with Autonomy
Discussion of 'Impact of AI assistance on student agency'
I have written before about the use of AI in teaching, and one specific area I think has great promise is for student feedback (see here and here for my tools).
A recent paper by Darshivi et al appears to challenges some of my optimism on these points. As per the principles of this blog I want to be driven by the empirical evidence, therefore in this post I will review the paper and review some of my earlier points, updating as necessary.
First a summary of what the paper argues (courtesy of GPT4):
Background and Purpose: The use of AI in education, particularly for automating and scaffolding learning activities, has grown. However, the impact of these technologies on student agency and self-regulated learning is less understood. The study aims to explore how AI assistance, particularly in the context of peer feedback, affects students' learning behaviours and their ability to work independently without such support.
Methodology: A randomized controlled experiment was conducted with 1625 students across 10 courses, examining the effect of AI assistance on student agency. The experiment had students initially receive AI-guided peer review assistance for four weeks, followed by a division into four groups with varying degrees of AI support and self-monitoring strategies to see how they adapted to providing peer feedback without AI help.
Findings: The study found that students tended to rely on AI assistance rather than learn from it to improve their feedback quality independently. When AI support was withdrawn, there was a noticeable decline in the quality of student feedback, indicating a dependency on AI for assistance. However, self-regulated strategies, such as self-monitoring checklists, helped bridge the gap somewhat but were not as effective as AI support. The research also noted that a hybrid approach combining AI assistance with self-regulation strategies did not significantly outperform the AI assistance alone.
Implications: These results suggest that while AI can enhance learning outcomes by providing personalized support, its use should be balanced with strategies that promote student agency and the development of self-regulation skills. The study cautions against over-reliance on AI, which might hinder students' ability to engage independently with the learning material.
Conclusions: The paper concludes with a discussion on the broader implications of using AI in education. It emphasizes the need for careful integration of AI tools that support student agency and highlights the potential challenges and benefits of relying on AI assistance in educational settings.
Why they did it
I first want to consider the question being answered by this research. It is focused on a student’s ‘agency’, which in this context is their ability to use AI feedback and other support to assist them to learn independently. As the authors put it:
that involves individuals making informed decisions and developing the capacity to set learning goals, evaluate progress, reflect and learn from feedback throughout their lives…..
…the capacity for students to actively shape their learning experiences, make responsible decisions, and control their educational journey.
This study has the authors providing students with AI assistance, and then taking it away or altering it to see how students respond. I am not convinced that this is the best way to approach the role of AI (or any other technology or learning technique) in education. The implication is that a technology is only truly helpful if it helps us learn so that we can do the task without it. But this is too strong an argument, my use of a calculator to do long sums has not helped me to learn to do these sums without a calculator. Instead it has meant that I do not need to learn to do long sums, because I will always have a calculator available to help me and I should dedicate my time to learning something else. The authors use an example of spellcheckers, citing research showing that we do not learn much from these, continuing to make similar spelling mistakes. Spelling is perhaps a little different because I may occasionally write by hand or in a format where I do not have an in-built spell checker, but for anything important (please feel free to embarrass me by pointing out any errors in this essay), I will always have that, so is this something that is now so important to learn?
I believe the better question to ask is, what things can we now rely more on AI to do, and what should humans be focusing on? Students will always have AI now, it will be everywhere for all of their lives, and in forms radically different from today, adapting to this, and quickly, will be vital.
Given this, the most interesting aspect of this paper to me, is the research question which looks at whether AI and human self-regulation strategies work as a complement, given the growing ubiquity of AI.
What they did
The authors consider a specific feedback tool where learners give feedback on learning resources. The specific AI-assistance on their reviewing platform includes:
Suggestion Detection. AI scans documents to see if there are suggestions already, if not the student reviewer is prompted to provide some.
Relatedness Score. Feedback is measured for relevance to the resource, if this is low, the student reviewer is prompted to consider their feedback more deeply.
Similarity Score. Comments by the reviewer are appraised for their similarity to existing comments in order to reduce repetition, and encourage originality.
In the authors words:
The primary objectives of these functions are to ensure that feedback is not only relevant but also original and constructive.
This seems like an excellent use of AI to help improve the quality of a task. Importantly for this research, the AI not only provides prompts to the reviewer to look again at their work, but explains why they have been asked to do this, therefore providing a potential learning opportunity.
As to the specifics of the study, it seems to be well-constructed, with a large sample size (1625 students) and suitable control methodology (students randomised into control and treatment groups). All students received the full set of AI help for the first four weeks of the study. For latter four weeks, the students were divided into four groups. The first group continued with the full set of AI assistance, the second group had all AI assistance removed, the third group had the AI prompts replaced with a self-monitoring checklist, and the fourth group had both the AI help and the self-monitoring checklist, thus testing the possibility of complementarity in the approaches.
Whilst the authors aim to develop a set of objective measures for assessing the reviewing process, this can never be perfect for something like a written review, and therefore some doubt will remain.
There is also an irony in that some of the measures used to assess the reviews, are themselves derived from AI models. The authors have embraced AI for specific purposes in their own work, but seem to not be entirely convinced that others should do the same.
A final comment on methodology is one that no-one researching AI can avoid, which is the speed of development. The models assessed here are by 2024 standards, now rather outdated. Speed of development in AI does not align well with the years normally required in the academic publishing process.
What they found
Comparing the control group (using AI throughout the entire time), to the group without AI in the latter part of the experiment, there is a statistically significant lower performance for the non-AI group.
Similar results are found for comparison between the control group and the group where AI is replaced with a self-monitoring checklist. Comparison of the control group to the group using both AI and the self-monitoring checklist finds no statistical difference.
There are two things missing from the analysis. Firstly there is no comparison to a group that never used AI. We see that the group that stops using AI does worse than the one that continues, but how would they compare to a group that never had it to begin with? Without this we can’t truly see if the use of AI helped the students learn to perform the task better (this limitation is acknowledged by the authors).
Related to this, we don’t get to fully understand the learning effects of using AI as the students get better at adapting to the prompts they are given. We could do this by comparing the performance of the control group in the first four weeks to the latter four weeks. If there are further learning effects after four weeks this affects the interpretation of the comparison to the treatment groups.
Implications
The authors conclude that:
Our study showed that the integration of AI in learning environments could impact students’ agency to take control of their own learning.
I think they are wrong to focus on this conclusion. The key takeaway from this research is that with AI assistance, the students did a significantly better job. Yes when you take it away their performance drops, but this tells us that you have designed a good AI tool, of course when you take that away the performance drops, why would you be happy to find your AI tool was not so useful?
The authors are correct to think about the correct use of AI as informed by pedagogical research, but I believe they are wrong in this case to interpret their findings in a somewhat negative manner. This does not make me strongly reconsider my practice in adopting AI to help with feedback, though it does make me think more carefully about the specific mechanisms that can be used to induce better feedback. Though this study focused on a somewhat different setting (AI prompts to help write feedback, rather than AI feedback to produce better presentations or other work) the general idea is similar, with the help of direction from the AI, you are able to produce better work (as I hopefully do with these essays).
Practical steps
If you aren’t already, start using AI feedback to help you with your own work, and encourage your students to do the same. This needs to be done carefully, but as demonstrated in this study, if done correctly, can have a significant effect on results. The AI prompts discussed in this study may not be relevant to the type of work you or your students are doing, but give a good idea about the directions you can head in.
