Another dot in the blogosphere?

Posts Tagged ‘grading

One of the easiest ways to spot a teacher on public transport or at an eatery is when she brings out a stack of papers to grade.

This is true of my wife now and my parents decades ago. I grew up seeing our dining table topped with books and papers, and helped them mark assignments and tests.

So it was no surprise to learn that the soon-to-be First Lady of the USA, Dr Jill Biden, was known to grade on the run when she was Second Lady.

When I read the tweet above that claimed that Michelle Obama said that Jill Biden was always grading papers, I had to find a source. I could not find the exact quote, only a similar one from this Washington Post article.

The enduring image former staff members repeatedly say they have of Jill Biden during the Obama administration is her carrying a stack of papers she had to grade on state trips to places like Israel, Japan or the Democratic Republic of Congo. One time, Jill asked to leave her office hours 10 minutes early, recalls Jim McClellan, NOVA’s liberal arts dean, because Joe was sitting in Air Force Two waiting for her to show up so they could do a three-country tour of Latin America. “I said, ‘Well, since the jet is revving up and using gas sitting on the runway, go ahead,’ ” McClellan says. She left with a stack of papers to grade and had them all done in time to be back in the office at 7 a.m. for her Tuesday classes.

A teacher’s dedication today might be a bit more obvious to stakeholders in the era of COVID-19 and Zoom. But if they paid more attention before this, they might have noticed teachers grading papers whenever they could. They do this not because they are poor managers of their time, but because they have so much to take care of.

It is a pity that stakeholders do not have insights on the planning, worrying, counselling, chaperoning, and parenting that teachers do every day. Sadly, not all teachers are that dedicated. But those that are make the grade.

If ever there was professional development that school teachers and university faculty need, it is on grading…

… and ungrading.

The latter tweet provides insights for the autodidact to learn how to ungrade.

I have been following the disaster that was the algorithmic grade prediction for the International Baccalaureate in the US and the GCEs in the UK.


Video source

The video above is one of several news reports on the issue. The NYT piece offers an excellent overview and critique.

I offer an oversimplification of the problem. Students could not take their exams due to the coronavirus pandemic. Administrators decided to rely on algorithms to predict their grades after using teacher-graded work and the school’s past performance as indicators.

Unfortunately, some students received lower than expected grades. Worse still, these students were disproportionately from poorer or otherwise disadvantaged schools.

A problem that administrators wanted to avoid was grade inflation, i.e., better than expected results compared to the previous year. This was an underhanded way of suggesting that teachers might grade their students too generously.

Another problem that they and other observers worried about was the impact of remote lessons and exam preparation on students. This was a shady way of saying that learning online was inferior and/or that teachers thrown into the deep end of emergency remote teaching did not do a good enough job.

However, what resulted was grades that were better than expected. They were so good that they might have been algorithmically adjusted downward.

So here is an unoriginal thought: Either the teachers had no integrity (they cheated by being lenient) or that online teaching and learning was better than expected. The en masse fraud among teachers is unthinkable. Some people do not seem to want to give credit to online teaching and learning. So what happened then?

A few weeks ago, I shared how I provided feedback and graded assignments for an intensive course I facilitate. Short version: An intense course has intense assignments and my process for review and critique are intense.

That course has ended this semester and I am left with another set of assignments to process. The end-of-course essay is even more challenging, so I provide more time and space to do this.
 

 
Given that we are effectively in a lockdown to counter the spread of COVID-19, I thought it prudent to just tackle one assignment a day instead of my usual two. Spreading this out gives me something mentally stimulating to do every day.

I could go faster, but that would compromise the integrity of the grading process because I might be tempted to complete the race instead. Just like staying physically apart now for physical health, I keep my grading efforts a day apart for mental health.

Added after the fact: I have completed the grading ahead of schedule and well before the administrative deadline. I only have administrative tasks ahead. Here is looking forward to jumping through hoops!

Let’s face it — it sucks to grade papers. Whether you do it well or not, it is tedious work that requires a balance of judgement and empathy. It takes energy and focus to plow through a stack of actual or e-papers.
 

 
But assessment and evaluation are where student learning is most evident and where the most learning happens. Only when students are challenged to use what they think they know do they really know if they know.

The teacher also learns more about his or her students during grading. Perhaps not just about their aptitude, but also about their attitude. How the students take care to organise, present, or persuade reveals as much about their heads as their hearts.

This is why I think that extracting principles from video game-based learning is important. Video games test players immediately and constantly. Players figure out strategies — alone or together — and try them out. The test comes before the content. The evidence of learning is obvious because of the test.

So while I plod through a heavy set of papers, I remind myself just how important assessment and evaluation are. They are evidence of learning. They are why I invested months of design and revisions before committing the questions to paper.

But even as I grade, I remind myself that the numbers or letters that result are often barriers to continued learning. Students tend to focus on the numbers or letters instead of the quality of their learning. The numbers or letters scream louder than the feedback that comes from a place of care.

This is why educators need to educate administrators and spreadsheet managers about the importance of formative feedback. The quality of the feedback matters more than the quantity of marks or the distribution on a curve.

Two weeks ago, I shared this announcement about Singapore’s ten-year plan for AI and focused on how it might affect schooling.

I left my reflection on AI for grading on slow burn for a while. I am enjoying a break, but I also enjoy wrestling with dubious change.

Yes, dubious. But first, two pretexts.

First, the vendors that the Ministry of Education, Singapore, works with are not going to be transparent with their technologies, so I cannot be absolutely certain of the AI development runway, timeline, and capabilities.

Next, the field of AI is not new and it is diverse. Parts of it evolve more quickly or slowly than we might expect. For example, handwriting recognition has been around since before Microsoft released its slate PCs. It was good enough to recognise some doctor scrawls even back then!

However, the Hollywood vision that AI will replace or even kill us off has not materialised. An expert might point out that AI is not good at making social predictions and ethical decisions. I simply point out that artificial intelligence is still no match for natural stupidity.

Back to the issue — we need to consume claims made by policymakers and edtech vendors critically. And more critically if they are reported by the mainstream media that thrives on sensationalism.

Do not take my word for it. Take this expert’s view that some claims are “snake oil”. In his slides, he put these claims into three categories: Genuine and rapid progress; imperfect but improving; and fundamentally dubious.

An expert’s view that some AI claims are “snake oil”. In his slides, he put the claims into three categories: Genuine and rapid progress; imperfect but improving; and fundamentally dubious. Slide #10 at https://www.cs.princeton.edu/~arvindn/talks/MIT-STS-AI-snakeoil.pdf
Source

I highlighted “automated essay grading” in the screenshot above because that coincides with our 2022 plan to “launch automated marking system for English in primary and secondary schools”.

The fundamental issue is AI’s ability to automate judgement. Some judgements are simple and objective, others are complex and subjective. Written language falls in the latter category particularly when the writers get older and are expected to write in more complex and subjective ways.

Anyone who has had to grade essays will know what rubrics and “standardisation” sessions are. Rubrics provide standards, guidelines, and point allocation. Standardisation meetings are when a group of assessors get a small and common set of essays, grade those essays, and compare the marks. Those same meetings set the standard for the definitions of subjectivity, disagreement, and frustration.

Might AI in three years be able to find the holy grail of objective and perfect grading of subjective and imperfect writing? Perhaps. If it does so, it might be less a result of rapid technological evolution and more one of social manipulation.

To facilitate AI processing of essays, students might be required to use proprietary tools and platforms. For example, they might have to use word processed forms instead of handwriting. They could be told to write in machine readable ways, e.g., only five paragraphs, structured paragraphs, model phrases, etc. In other words, force-fitting writers and writing.

This is already how some tuition and enrichment centres operate. They reduce essay writing to formulae and teach these strategies to kids. Students are not encouraged to make mistakes, learn from them, or develop creative and critical thought. They are taught to game the algorithms.

The algorithms are the teachers’ expectations and rubrics now. They could be the AI algorithms in future. But the same reductionist strategy applies because we foolishly prefer shortcuts.

The AI expert I highlighted earlier focused on how ill-equipped AI is to predict social outcomes. He concluded his talk with this slide.

Concluding slide (#21) from https://www.cs.princeton.edu/~arvindn/talks/MIT-STS-AI-snakeoil.pdf
Source

We might also apply his last two points to automated essay grading: Resist commercial-only interest aimed to hide what AI cannot do, and focus on what is accurate and transparent.

This is not my way of stifling innovation as enable by educational technology. I wear my badge of edtech evangelist proudly. But I keep that badge polished with critical thought and informed practice.

 
I am in the middle of an intense grading exercise. This happens every semester because I get at least one Friday class which has deadlines on Friday evenings. This means that I burn weekends to give timely feedback to my students.

This practice is not unusual because I was a teacher and professor. I am married to a teacher and my parents were teachers. Marking scripts on weekends is a sad family norm.

What I find irritating is that I need to have at least five (sometimes six) web browser tabs open to provide feedback and grade. These is a tab each for:

  1. The institutional LMS (to access the online scripts)
  2. The feedback tool (for commenting on student work)
  3. A timer (because the LMS page times out every 30 minutes)
  4. A Google Sheet (for recording my marks)
  5. A Google Doc (for backing up my comments)

When I am in a public place like a library, I have a sixth tab to play soothing music through my headphones. I do this to drown out inconsiderate users who talk in the study space.

Technically I should need only three tabs open: The LMS (to get the scripts), the feedback tool (to read and comment on them), and the Google Sheet (to record the marks).

However, the auto-timeout of the LMS page requires me to refresh it, so I need the timer to remind me. If I do not pay attention, the LMS page times out in the background and my last set of comments does not get saved. The feedback tool also just freezes.

I need the Google Doc tab to back up my comments. I have been burnt by a systems administrator who removed my LMS account and its data (all the assignment comments), so I need the Google Doc for continuity between semesters.

Then there are the people who talk in the library… sigh. Ambassador-at-large, Tommy Koh, recently described us as a first world country with third world citizens. It is the small things that add up. Small things like being considerate in a shared space.

Needing five (or six) tabs open to grade and give feedback is a first-world problem. But the problems that cause them have been around for as long as Man has soiled the Earth.

This rant has been brought to you by Misanthrope Plus.

The issue of “grading on the curve” raised its ugly head in the news. This time the headline was a bold declaration:

But there was more to the headline. The article highlighted a variety of curve equivalent and curve adjacent schemes. Then there was a university don’s claim:

The claim that was not substantiated seemed to be that grading on a curve was part of assessment and that this was useful feedback. Specifically:

  • How is grading on a curve part of assessment when the other entities in the same article also claim they have done away with such moderating?
  • How exactly does sorting students on a curve provide feedback on meeting course objectives?

I do not know if he did not elaborate, or if the journalist or her editor left this out. Either way we have a claim without explanation or backing. None of us should take unsubstantiated claims seriously. Thankfully none of us will be graded on a curve to be critical thinkers.

Far wiser and more articulate people have shared their thoughts on assessment, grading and feedback. So I reshare what they shared.

From these and the work of others, I distill some wisdoms into these image quotes.

Formative feedback

Quantitative grading ends learning. Quality feedback sustains learning.

By the time you read this, I might be done with four intense sessions of performative evaluations of adult learners over the last two days.

Long story short: My learners need to provide evidence that they can facilitate student-centric lessons in a university context. This is as challenging to do as it is to evaluate.

To complicate evaluative matters, I have two batches of learners. I will need to grade the final written assignments of the second cohort as I evaluate the performative skills of the first.
 

 

This got me thinking about how much grading can be like the Rotten Tomatoes (RT) rating system for movies. If you are not familiar with RT, the video below provides a concise description and criticism of its flaws.
 

Video source

People might refer to a movie’s RT score as representation of its performance and then decide whether to watch it or not. While a single number is quick and convenient, it may not be valid or reliable.

The same could be said about essay grading and performative evaluations. As much as we use guidelines, standards, or rubrics, subjectivity is still an important factor.

I have reflected before on why and how I embrace subjectivity. This is particularly important when we need to find a balance between maintaining standards and treating each learner as an individual.

I am heavily influenced by Todd Rose’s work, The End of Average. We need to learn to recognise the contexts where using an average is meaningless. If we insist on grading strictly on a curve or using an unrealistic standard, we do more harm than good.

This harm affects our learners because we do not treat them as individuals and start where they are. The harm taints the teaching profession because we practice blindly. The harm persists when we act without question.

Does it take a rotten tomato hitting us in the face before we change?


http://edublogawards.com/files/2012/11/finalistlifetime-1lds82x.png
http://edublogawards.com/2010awards/best-elearning-corporate-education-edublog-2010/

Click to see all the nominees!

QR code


Get a mobile QR code app to figure out what this means!

My tweets

Archives

Usage policy

%d bloggers like this: