Another dot in the blogosphere?

Posts Tagged ‘grading

A few weeks ago, I shared how I provided feedback and graded assignments for an intensive course I facilitate. Short version: An intense course has intense assignments and my process for review and critique are intense.

That course has ended this semester and I am left with another set of assignments to process. The end-of-course essay is even more challenging, so I provide more time and space to do this.
 

 
Given that we are effectively in a lockdown to counter the spread of COVID-19, I thought it prudent to just tackle one assignment a day instead of my usual two. Spreading this out gives me something mentally stimulating to do every day.

I could go faster, but that would compromise the integrity of the grading process because I might be tempted to complete the race instead. Just like staying physically apart now for physical health, I keep my grading efforts a day apart for mental health.

Added after the fact: I have completed the grading ahead of schedule and well before the administrative deadline. I only have administrative tasks ahead. Here is looking forward to jumping through hoops!

Let’s face it — it sucks to grade papers. Whether you do it well or not, it is tedious work that requires a balance of judgement and empathy. It takes energy and focus to plow through a stack of actual or e-papers.
 

 
But assessment and evaluation are where student learning is most evident and where the most learning happens. Only when students are challenged to use what they think they know do they really know if they know.

The teacher also learns more about his or her students during grading. Perhaps not just about their aptitude, but also about their attitude. How the students take care to organise, present, or persuade reveals as much about their heads as their hearts.

This is why I think that extracting principles from video game-based learning is important. Video games test players immediately and constantly. Players figure out strategies — alone or together — and try them out. The test comes before the content. The evidence of learning is obvious because of the test.

So while I plod through a heavy set of papers, I remind myself just how important assessment and evaluation are. They are evidence of learning. They are why I invested months of design and revisions before committing the questions to paper.

But even as I grade, I remind myself that the numbers or letters that result are often barriers to continued learning. Students tend to focus on the numbers or letters instead of the quality of their learning. The numbers or letters scream louder than the feedback that comes from a place of care.

This is why educators need to educate administrators and spreadsheet managers about the importance of formative feedback. The quality of the feedback matters more than the quantity of marks or the distribution on a curve.

Two weeks ago, I shared this announcement about Singapore’s ten-year plan for AI and focused on how it might affect schooling.

I left my reflection on AI for grading on slow burn for a while. I am enjoying a break, but I also enjoy wrestling with dubious change.

Yes, dubious. But first, two pretexts.

First, the vendors that the Ministry of Education, Singapore, works with are not going to be transparent with their technologies, so I cannot be absolutely certain of the AI development runway, timeline, and capabilities.

Next, the field of AI is not new and it is diverse. Parts of it evolve more quickly or slowly than we might expect. For example, handwriting recognition has been around since before Microsoft released its slate PCs. It was good enough to recognise some doctor scrawls even back then!

However, the Hollywood vision that AI will replace or even kill us off has not materialised. An expert might point out that AI is not good at making social predictions and ethical decisions. I simply point out that artificial intelligence is still no match for natural stupidity.

Back to the issue — we need to consume claims made by policymakers and edtech vendors critically. And more critically if they are reported by the mainstream media that thrives on sensationalism.

Do not take my word for it. Take this expert’s view that some claims are “snake oil”. In his slides, he put these claims into three categories: Genuine and rapid progress; imperfect but improving; and fundamentally dubious.

An expert’s view that some AI claims are “snake oil”. In his slides, he put the claims into three categories: Genuine and rapid progress; imperfect but improving; and fundamentally dubious. Slide #10 at https://www.cs.princeton.edu/~arvindn/talks/MIT-STS-AI-snakeoil.pdf
Source

I highlighted “automated essay grading” in the screenshot above because that coincides with our 2022 plan to “launch automated marking system for English in primary and secondary schools”.

The fundamental issue is AI’s ability to automate judgement. Some judgements are simple and objective, others are complex and subjective. Written language falls in the latter category particularly when the writers get older and are expected to write in more complex and subjective ways.

Anyone who has had to grade essays will know what rubrics and “standardisation” sessions are. Rubrics provide standards, guidelines, and point allocation. Standardisation meetings are when a group of assessors get a small and common set of essays, grade those essays, and compare the marks. Those same meetings set the standard for the definitions of subjectivity, disagreement, and frustration.

Might AI in three years be able to find the holy grail of objective and perfect grading of subjective and imperfect writing? Perhaps. If it does so, it might be less a result of rapid technological evolution and more one of social manipulation.

To facilitate AI processing of essays, students might be required to use proprietary tools and platforms. For example, they might have to use word processed forms instead of handwriting. They could be told to write in machine readable ways, e.g., only five paragraphs, structured paragraphs, model phrases, etc. In other words, force-fitting writers and writing.

This is already how some tuition and enrichment centres operate. They reduce essay writing to formulae and teach these strategies to kids. Students are not encouraged to make mistakes, learn from them, or develop creative and critical thought. They are taught to game the algorithms.

The algorithms are the teachers’ expectations and rubrics now. They could be the AI algorithms in future. But the same reductionist strategy applies because we foolishly prefer shortcuts.

The AI expert I highlighted earlier focused on how ill-equipped AI is to predict social outcomes. He concluded his talk with this slide.

Concluding slide (#21) from https://www.cs.princeton.edu/~arvindn/talks/MIT-STS-AI-snakeoil.pdf
Source

We might also apply his last two points to automated essay grading: Resist commercial-only interest aimed to hide what AI cannot do, and focus on what is accurate and transparent.

This is not my way of stifling innovation as enable by educational technology. I wear my badge of edtech evangelist proudly. But I keep that badge polished with critical thought and informed practice.

 
I am in the middle of an intense grading exercise. This happens every semester because I get at least one Friday class which has deadlines on Friday evenings. This means that I burn weekends to give timely feedback to my students.

This practice is not unusual because I was a teacher and professor. I am married to a teacher and my parents were teachers. Marking scripts on weekends is a sad family norm.

What I find irritating is that I need to have at least five (sometimes six) web browser tabs open to provide feedback and grade. These is a tab each for:

  1. The institutional LMS (to access the online scripts)
  2. The feedback tool (for commenting on student work)
  3. A timer (because the LMS page times out every 30 minutes)
  4. A Google Sheet (for recording my marks)
  5. A Google Doc (for backing up my comments)

When I am in a public place like a library, I have a sixth tab to play soothing music through my headphones. I do this to drown out inconsiderate users who talk in the study space.

Technically I should need only three tabs open: The LMS (to get the scripts), the feedback tool (to read and comment on them), and the Google Sheet (to record the marks).

However, the auto-timeout of the LMS page requires me to refresh it, so I need the timer to remind me. If I do not pay attention, the LMS page times out in the background and my last set of comments does not get saved. The feedback tool also just freezes.

I need the Google Doc tab to back up my comments. I have been burnt by a systems administrator who removed my LMS account and its data (all the assignment comments), so I need the Google Doc for continuity between semesters.

Then there are the people who talk in the library… sigh. Ambassador-at-large, Tommy Koh, recently described us as a first world country with third world citizens. It is the small things that add up. Small things like being considerate in a shared space.

Needing five (or six) tabs open to grade and give feedback is a first-world problem. But the problems that cause them have been around for as long as Man has soiled the Earth.

This rant has been brought to you by Misanthrope Plus.

The issue of “grading on the curve” raised its ugly head in the news. This time the headline was a bold declaration:

But there was more to the headline. The article highlighted a variety of curve equivalent and curve adjacent schemes. Then there was a university don’s claim:

The claim that was not substantiated seemed to be that grading on a curve was part of assessment and that this was useful feedback. Specifically:

  • How is grading on a curve part of assessment when the other entities in the same article also claim they have done away with such moderating?
  • How exactly does sorting students on a curve provide feedback on meeting course objectives?

I do not know if he did not elaborate, or if the journalist or her editor left this out. Either way we have a claim without explanation or backing. None of us should take unsubstantiated claims seriously. Thankfully none of us will be graded on a curve to be critical thinkers.

Far wiser and more articulate people have shared their thoughts on assessment, grading and feedback. So I reshare what they shared.

From these and the work of others, I distill some wisdoms into these image quotes.

Formative feedback

Quantitative grading ends learning. Quality feedback sustains learning.

By the time you read this, I might be done with four intense sessions of performative evaluations of adult learners over the last two days.

Long story short: My learners need to provide evidence that they can facilitate student-centric lessons in a university context. This is as challenging to do as it is to evaluate.

To complicate evaluative matters, I have two batches of learners. I will need to grade the final written assignments of the second cohort as I evaluate the performative skills of the first.
 

 

This got me thinking about how much grading can be like the Rotten Tomatoes (RT) rating system for movies. If you are not familiar with RT, the video below provides a concise description and criticism of its flaws.
 

Video source

People might refer to a movie’s RT score as representation of its performance and then decide whether to watch it or not. While a single number is quick and convenient, it may not be valid or reliable.

The same could be said about essay grading and performative evaluations. As much as we use guidelines, standards, or rubrics, subjectivity is still an important factor.

I have reflected before on why and how I embrace subjectivity. This is particularly important when we need to find a balance between maintaining standards and treating each learner as an individual.

I am heavily influenced by Todd Rose’s work, The End of Average. We need to learn to recognise the contexts where using an average is meaningless. If we insist on grading strictly on a curve or using an unrealistic standard, we do more harm than good.

This harm affects our learners because we do not treat them as individuals and start where they are. The harm taints the teaching profession because we practice blindly. The harm persists when we act without question.

Does it take a rotten tomato hitting us in the face before we change?

Yesterday I shared some thoughts on providing formative feedback on written assignments. Today I share some strategies I use to do so efficiently and effectively.
 

Get out of comfort zones
I try not to work at home. There are too many distractions there largely thanks to my fast Internet connection. I might get sidetracked by YouTube videos or social media alerts.

I make my way to a library or a coffee place. I do not have the comforts of home and I switch to my all-business frame of mind.

One benefit of being at a coffee place is that I cannot stay long. I cannot hog a table or I might have time-limited Internet access. The time limit is an incentive to work efficiently.
 

Block distractions
Being in a public place means it gets noisy much of the time. I counter this by donning a pair of noise-cancelling headphones.

I have different playlists to help drown out ambient noise or bad store music. Some music is calming if the work I am reviewing agitates me; some of the tunes are energetic if my human battery runs low.

I also remove my glasses as I do not need them for close up work. Doing this has the benefit of not being distracted by what appears in my peripheral vision.
 

Set a quota and stick to it
I use the number of scripts and the feedback due date to calculate how many assignments I need to process a day. Then I divide each day’s quota into a before-lunch and an after-lunch set.

The quota needs to be achievable. Some initial trial and error has built my experience up on this matter. Calculating the quota is the easy part.

The difficult part is keeping to the quota. By this I do not mean completing the AM or PM session’s amount. I do this consistently. I mean stopping when I seem to have gained momentum.

It is important to stop because I do not want to mentally exhaust myself. If I am tired, it will show in my feedback and this is not fair to my learners. I also need time to rest, play, or finish the now cold cup of coffee on the table.
 

Revisit earlier scripts
It is tempting to grade, provide feedback, and never want to see a piece of writing again. However, I find it useful to revisit earlier scripts to
to check for the consistency of my feedback.

I think that we operate like the tides when we process assignments. We seem to wash up consistently like waves crashing on a beach. But no two waves are identical and there is the overall ebb or flow of tides too.

This could be due to my energy, the time of day, the overall nature of the class, the behaviour and expectations of learners who submit at different times, and so much more.
 

Embrace subjectivity
I do not try to be a robot or strive to be absolutely objective. This is impossible.

I embrace subjectivity but do so professionally. By this I mean that I take into account how a group responded to one or more lessons, what strategies I used with them, what content we uncovered together, etc.

We might have a menu to follow, but each dish is a bit different from another. I acknowledge and respect that by seeing how it was prepared (focusing on the process), not what it looks like (focusing on the product).
 

Strive for timeliness
One thing that makes feedback effective is timeliness. Wait too long to provide it and all effort it lost because learners have forgotten what they wrote.

The assignments I grade and provide feedback on are submitted online. After the submission deadline, I process assignments in the order they were submitted. This means that the first person to submit gets feedback first.

I could resort to an administratively convenient alphabetical order, but this is not fair to students who might have Z surnames but A response times.

End note: I prepared much of this reflection after processing a day’s quota of assignments. My mind was still buzzing, so I put my energy into sharing some ideas.

Grading Papers All Day (9/365) by Ms. Tina, on Flickr
Creative Commons Attribution-Noncommercial-No Derivative Works 2.0 Generic License  by  Ms. Tina 

Walking about in a mall, I saw a promotion to Pay Less, Taste More. Our Ministry of Education advocates Teach Less, Learn More.

After reading these three articles and reflections, I think we could and should Grade Less, Learn More.

The first piece by Bud the Teacher started the ball rolling . In his own words:

For starters, I think teachers, in general, grade too many things. So one way to streamline would be to “grade” less. And that doesn’t mean that teachers shouldn’t ask students to write, and write often. But we don’t need to grade everything that comes to us. In fact, we should grade very little of it.

In the context of writing, Bud went on to say that teachers have built up the wrong expectations that they are their students’ only audience and that they alone evaluate what that writing is worth.

He argued that technology should not be the means of streamlining grading when the problem lay with the assumptions, design, and expectations of grading.

Put another way, the questions he might have asked could include: Why are students writing for just one person? Why not leverage on authentic and public writing spheres? Why not approach the task as a fellow reader and learner in order to critique instead of just grade?

Cathy Davidson wrote the second article and she was as thought-provoking as always.

She started by describing seven reasons why standardized testing, while great for grading, are not that great for evaluating learning. So she suggested a key alternative: Get learners to create by learning how to code.

I do not think that all learners need to code, but I do think that all learners need to be given the freedom and opportunity to express themselves and to create. Only then are they putting theory into practice. Only then are they making mistakes. Only then are they really trying.

As a result of reading these and other articles, I had to ask myself if I had reflected on these issues before. I had in the third piece.

Two years ago, I had concluded that modernizing a problem (e.g., creating grading software) was not the solution. We might create more intelligent systems, but the problem of testing irrelevant skills in irrelevant ways would still remain.

No, the solution lies in questioning the established but problematic model. It starts with accepting that grading more sometimes leads to learning less.

Grading takes time and students do not get timely feedback. Students do not pick up as much content or as many skills as teachers wish. Teachers do all they can to keep up with the grading load and do not have the bandwidth to learn new strategies.

A solution might lie in figuring out how to grade less so that everyone learns more. By grading less I do not mean asking for lazier teachers. We need teachers who question, learn, and act on what they realize is better for their learners.

Teachers could grade less by redesigning tests and focusing on key things that need grading. They could leverage on their learners’ need to communicate, create, and critique. Teachers could model these life skills and teach their students to peer assess in authentic environments.

As they do this, I bet they would learn more about what their learners are capable of and what new value they as teachers bring to the table.

“News” broke recently of software that helped to “grade” student essays in some schools in Singapore. Here is one source that you don’t have to pay for to read about the use of WriteToLearn and Criterion.

I call it “news” because the software has been at least 10 years in the making and because the headline (Hey, the computer gave me an A for my essay) is sensationalist.

It might be news to some, but technology that is ten-years-old cannot be described, as it was in the article, in its infancy.

The headline is designed to give the impression that computers are taking over. Why not, right? After all, Watson, IBM’s latest AI has beaten human champions over the last three days in the gameshow, Jeopardy.

While some might be wondering when we are going to bow to our technology overlords, at the moment technology like WriteToLearn and Criterion are still just tools. We use them, not the other way around.

Those writing tools are great at doing what people cannot do: Using tireless brute force to analyze an essay with rules and provide recommendations more quickly than a teacher can for the learner. They are not designed to replace the teacher. Not yet anyway.

Two unoriginal thoughts came to my mind:

  • Any teacher that can be replaced by a computer deserves to be. Depending on your search, that quote might be attributed to David Thornburg or Arthur C Clarke.
  • The essence of being human involves asking questions, not answering them. I think that quote came from John Seely Brown (at the end of this NYT article).

We may have a finite capacity to hold knowledge. But we have an infinite capacity to wonder and create.


http://edublogawards.com/files/2012/11/finalistlifetime-1lds82x.png
http://edublogawards.com/2010awards/best-elearning-corporate-education-edublog-2010/

Click to see all the nominees!

QR code


Get a mobile QR code app to figure out what this means!

My tweets

Archives

Usage policy

%d bloggers like this: