How NOT to design online assessment
Posted September 6, 2014on:
Recently I read an article on The Atlantic, The End of Paper-and-Pencil Exams?
The headline asked a speculative question, but did not deliver a clear answer. It hinted at mammoth change, but revealed that dinosaurs still rule.
Here is the short version.
This is what 13,000 4th grade students in the USA had to do in an online test that was part of the National Assessment of Educational Progress. They had to respond to test prompts to:
- Persuade: Write a letter to your principal, giving reasons and examples why a particular school mascot should be chosen.
- Explain: Write in a way that will help the reader understand what lunchtime is like during the school day.
- Convey: While you were asleep, you were somehow transported to a sidewalk underneath the Eiffel Tower. Write what happens when you wake up there.
This pilot online assessment was scored by human beings. The results were that 40% of students struggled to respond to question prompts as they were rated a 2 (marginal) or 1 (little or no skill) on a 6 point scale.
This was one critique of the online test:
One downside to the NCES pilot study: It doesn’t compare student answers with similar questions answered in a traditional written exam setting.
I disagree that this is necessary. Why should the benchmark be the paper test? Why is a comparison even necessary?
While the intention is to compare the questions, what a paper vs computer-based test might do is actually compare media. After all, the questions are essentially the same, or by some measure very similar.
Cornelia Orr, executive director of the National Assessment Governing Board, stated at a webinar on the results that:
When students are interested in what they’re writing about, they’re better able to sustain their level of effort, and they perform better.
So the quality and type of questions are the greater issues. The medium and strategy of choice (going online and using what is afforded there) also influence the design of questions.
Look at it another way: Imagine that the task was to create a YouTube video that could persuade, explain, or convey. It would not make sense to ask students to write about the video. They would have to design and create it.
If the argument is that the YouTube video’s technical, literacy, and thinking skills are not in the curriculum, I would ask why that curriculum has excluded these relevant and important skills.
The news article mentioned some desired outcomes:
The central goal of the Common Core is deeper knowledge, where students are able to draw conclusions and craft analysis, rather than simply memorize rote fact.
An online test should not be a copy of the paper version. It should have unGoogleable questions so that students can still Google, but they must be tested on their ability to “draw conclusions and craft analysis, rather than simply memorize rote fact”.
An online test should be about collaborating in real-time, responding to real-world issues, and creating what is real to the learners now and in their future.
An online test should not be mired in the past. It might save on paper-related costs and perhaps make some grading more efficient. But that focuses on what administrators and teachers want. It fails to provide what learners need.