In Coversation with OxfordAQA: What’s in a Question?
After Dale Bassett’s fascinating talk on the transformation of assessment at Outstanding Schools Middle East 2020 and ahead of our Outstanding Schools Middle East one day teaching and learning event in March 2021 on Raising Attainment, we asked OxfordAQA to tell us 'What’s in a question?'
How Fair Assessment is now a key driver for leading international exam bodies
Creating exam questions used to be easy. Exam writers wrote the questions, candidates tried to answer them, marks were collated, and results were sent out. Done.
But not anymore.
For leading bodies, developing exams that are based on the latest research into learning and assessment, the process of question writing has become a science in its own right. Selecting, formulating, and interrogating each draft question before they gain entry into the published exam paper involves a complex process to ensure each and every one is fair, valid and reliable.
The development of International GSCEs and A-levels in recent years has added a whole new dimension as well. Discussions (and debates) at a typical validation meeting might include the following:
“Is the draft question for next year’s Science paper loaded with cultural bias?”
“Would candidates taking their A-level Maths spend more time thinking about the English used in this question, rather than focusing on demonstrating what they have learned numerically?”
“Does the illustration in this GCSE Geography paper assist or distract candidates? Perhaps it shouldn’t be there in the first place?”
Fair Assessment: some principles
In other words, leading examining boards and their question writers are well aware their questioning can either help candidates (making it precisely clear what knowledge is being tested), hinder (for example, vague or confusing commands), or even mislead. This is how the notion of Fair Assessment has come to life.
It’s a big area and not something we can try to summarise in a single blog. But here are three important factors mentioned above:
- Fair: Each year – and in each location around the world – the standard of assessment needs to be the same. For example, the GCSE Maths candidate in Karachi this year should be assured that her grade 5 is the same as that achieved by the candidate in Bangkok last year.
- Valid: Questions need to measure what they are supposed to measure: candidates are tasked with demonstrating subject knowledge rather than how well they can decipher the way questions are constructed.
- Reliable: Candidates should be given enough opportunity to demonstrate what they have learned and the right marking process needs to be in place to recognise this.
So, what’s in a question?
Digging into the detail, question writing at leading international assessment bodies is now a precise and in-depth process involving teams in conference meetings – and on the ground around the world – rather than individuals huddled away in ivory towers.
Based on extensive studies by organisations such as the Centre for Education Research and Practice (CERP), a clear understanding of what can help, hinder or mislead candidates has now started to emerge. Some examples include:
Do use:
- Reduced word counts
- Simple language
- Careful ramping of demand
- Contexts that are not country or region specific
Don’t use:
- Inaccessible language
- Superfluous or confusing contexts, pictures or diagrams
- Confusing command words
- Incomplete or unsuitable mark schemes
As Jamie Kirkaldy, Head of Teaching and Learning Support at OxfordAQA, put it:
‘Fairness in assessment is even more important in an international school setting, where a number of students won’t have English as a first language. It would be completely unacceptable for this to be a barrier in demonstrating their knowledge and skills in Maths, say, or Science, or any other subject. Making sure exams test what they are meant to test – and nothing else – should be the number one priority of any exam board, but especially those that operate internationally.’
Although firmly established, the concept and detail of Fair Assessment is still a major topic of discussion in educational circles. No doubt it will continue to evolve as more and more international schools join the debate.
Fairness first. Isn’t that the least your students deserve? Learn more about OxfordAQA’s approach to Fair Assessment.
