AI-Proofing Your Assessment (part 2)
Solutions that exist
In part 1 of this series, I made the (hardly original) point that many assessment types are now not fit for purpose in the age of AI. I will now move on to analyse what can work, beginning with assessments that are used already, and in the next section moving on to some more innovative assessments that work within the new paradigm of learning and assessing alongside AI.
Things that can work (if done the right way)
Of the main existing assessment structures, what can still work?
In-person tests and exams
‘Old-fashioned’ tests and exams with no computer are an obvious remedy to our AI-problems, and one that are still of major importance in many disciplines. Whilst there have been arguments to reduce the use of such assessments to focus on more authentic assessments that look like real-world work, I believe there is still a strong case for needing to know things, and that exams are a good way of testing this knowledge and how to apply it when designed intelligently. These assessments will still allow students to use AI in preparing for the test, and in-part this should be what AI is best used for in a learning context. Students will be pushed to use AI in the most effective way to actually learn something that they can then put into use when they don’t have the AI to help them.
Since the advent of search-engines, the argument has been made that knowing things is less relevant, rather being able to find information, analyse and synthesise it, are important. Certainly, it is true that what you do with the information is important, but I would argue that there is strong evidence that domain knowledge facilitates higher-order thinking and transfer; thus, we should retain assessments that test for this. Academics should recognise this more than most. We are not in our jobs simply because we are good at finding information or using that information but because we actually have more knowledge than our students, hence why they pay us to teach them.
Finally, there is the benefit that exams test the ability to work under pressure and in a time constrained manner, still likely to be important skills even in a labour market beginning to be transformed by AI advances.
I do not believe that an assessment structure that relies entirely on this type of assessment is optimal, however. There is a good reason for asking student to write essays and reports over longer periods of time. Writing helps us to think better spending long periods of concentrated time on a specific output is an important skill. Showing you can use electronic tools, including AI alongside other software and research tools to produce an output, is important. As the type of work that people do changes, we should want at least some assessments that reflect, and shape, what that work looks like as part of training our students for their careers.
So yes, let’s use well designed tests and exams in an academic programme, but we need other assessments to complement them.
Presentations
These have the benefit of fostering a real skill; the ability to present in-front of an audience (something that at least for some time may be even more important than currently due to its human features) and do so without the immediate assistance of AI.
My personal recent experience of the reality of presentations has unfortunately been underwhelming. LLMs are very good at creating a presentation structure, notes for discussion, organising the visual elements of slides and giving feedback for polish on a presentation. The result can be presentations of good quality, where I am not convinced the students really have learnt very much in the preparation of the work, making these little better than essays and reports where students have copy and pasted direct from an AI.
Can presentations be salvaged? Yes! With the major change being to emphasise the Q&A element of the presentation. In my own presentation rubrics this element is currently 10% of the mark, but in future this will be 30, 40 or even 50% of the mark. The presentation itself will therefore serve largely as an introduction to the part that matters, the ability to answer questions to reflect your genuine understanding of the issues you have discussed. We therefore combine elements of the test/exam response to AI, but in a spoken rather than written format. This is a very traditional1 assessment, and not without its downsides either. It may reward memory more than genuine learning in the same way as a test or exam, and those that speak more fluently and confidently may be unduly rewarded over those that know their stuff but lack some confidence.
Modified versions of a presentation may involve group presentations, some element of peer questioning where students are partly assessed on the questions they ask of others, and an early submission of slides so that the assessor can prepare their questions in advance of the presentation (with an opportunity to use AI to help).
I think presentations are important, and would aim to incorporate them into a programme-level assessment with this added emphasis on Q&A. It helps foster research skills and looks more like a real task that we are training students to do well in. This still does not fully overcome the issue of writing as an aid to thinking however, we still lose something without longer essays/reports.
Reflective work
One variety of assessment that I had not had much experience of until recently is reflective work where a student analyses a process such as the team dynamics within a groupwork assignment, or the process of producing a piece of work. This can take a more standard essay form, or the form of a diary, potentially across a spread of time or done in relation to multiple other assessments.
I believe this can be an assessment that works relatively better than other written assessments in that first it is designed to be personal, such that it would be hard (though not impossible) to produce an acceptable answer directly from an LLM.
Also, and especially when in the form of a diary, the reflective work moves us in the direction of an assessment method I will discuss in the next section where the focus is on the process of producing work more than on the finished product.
This type of assessment is likely to work better when certain elements have been included in the teaching of the course. You could discuss and analyse what reflection is and how it is valuable for example. If you were asking them to reflect on the tools you used then did you teach them about pros and cons of different tools, or introduce relevant reading? If you want them to reflect on teamwork, did you do exercises where they analysed group dynamics, considering what good teamwork looks like etc?
Part of a current assessment I use to complement a group presentation requires the following:
Reflect on your experience of working in a group. This should be around 500 words.
Include a discussion of what went well and what didn’t. What difficulties did you face and
how did you overcome them? How did you divide up tasks and share information? You
should include some mention of how you used AI or other technology. Finally
what have you learnt about working in groups that can help you in the future?
Earlier in the course we had workshop sessions analysing examples of problems that groups might face and talked about what makes a good team member. We have also extensively analysed when and where to use AI, so students should be well-prepared to answer this element.
Final thoughts
A common theme in the analysis of these assessment types is the tension between assessment as a way for students to signal their ability versus assessment as something that encourages learning, both whilst preparing and undertaking that assessment, and by learning from feedback to improve for the future.
Both are valid and important. If we were very focused only on the signalling aspect, then we might be happier to rely more heavily on traditional exams and tests. As I argue, these avoid any doubts that AI has directly helped students and therefore preserve the integrity of the signal better than many other assessments. On the other hand, there has long been an argument that these assessments often emerge as an ability of memory rather than higher order abilities. This memorisation that is used to do well in the exam may also be fleeting rather than genuine long-term learning, and this is also a legitimate concern about a heavily exam-oriented strategy, or one that relies on heavy questioning after a presentation.
Reflective work, or presentations with less reliance on questions, potentially have a greater educational benefit. In the process of preparing a presentation or reflecting on various aspects of the subject matter and the process of preparing the work, students may pick up more genuine learning and get more meaningful feedback, as they would have with a more traditional essay or report. But with the potential to use AI, these methods become less reliable as a signal as well as losing the learning aspect for those that take shortcuts with AI
In the next part of this series I will discuss innovative assessments, drawing from my own ideas, a variety of ideas I have seen elsewhere, and on the fundamentals of what makes good teaching, learning and what we are trying to do with assessment, reflecting on the importance of learning and signalling respectively.
