Another dot in the blogosphere?

Posts Tagged ‘comparison

My RSS feed showed me this graphic on media literacy in European countries. No surprises — the Scandinavian countries lead the pack.

I wondered if there was anything similar for this part of the world. So I searched for NGOs that researched it starting with the one credited in the graphic.

I could not find anything from such organisations so I widened my search for “media literacy in southeast asia”. Except for a few old articles like this one in 2008, I could not find comparison or rank tables.

It is not that the rankings are important. I want to know WHAT we are doing compared to elsewhere. An initial comparison of HOW we are doing might have opened doors to the WHAT.

But since media literacy across curricula and goes beyond formal schooling, it must be difficult to collect and make sense of such data. So now I wonder how that credited organisation actually ranked those countries.

Methodology of

I downloaded the PDF of the report which had a one-page description of its methodology. It turns out that that the “measurement” was not about media literacy. It was about predicting media literacy with components like PISA scores. What? My question exactly.

So my tweet is not an endorsement of the graphic. It is an example of not taking data presentation at face value.

The Organisation for Economic Co-operation and Development (OECD) released the 2018 results of its Teaching and Learning International Survey (TALIS).

The Today article summarised some TALIS data.

Taken at face value, the sample of lower secondary school teachers in Singapore spent less time teaching but more time marking student work than their OECD counterparts.

So might the 2005 initiative to Teach Less, Learn More become Teach Less, Mark More now?

I jest. Here was another tweeted article on TALIS.

As I read both articles on survey findings, I wondered:

  1. How were “working hours” of the sample of lower secondary school teachers calculated?
  2. Did these hours include work done outside school, e.g., grading student work at home or at a Starbucks?

Why ask these questions? If the working hours were only based on official time tables and co-curricula commitments, then the figures did not capture the extent of work.

Even if we assume that there was a valid and reliable way of collecting outside school work across various OECD countries, how meaningful is an average for each country?

Consider how much more time a language teacher takes on grading and feedback than would a teacher who relies on online quizzes and scanned bubble sheets.

Consider how one teacher might be able to enjoy practically the whole of the June vacation while another has to chaperone students on an overseas trip, stay in touch with school leadership and parents, or prepare university admissions materials for graduating students during the same period.

I revisited my second question after reading both articles: Did these working hours include work done outside school?

The CNA article claimed that the working hours included “those spent working out of school” while the Today article did not. Both articles mentioned time spent marking and efforts to reduce administrative work. However, they did not stipulate whether this was marking and administration done in school or outside it.

Consider this anecdote from the Today article (I emphasised the key finding):

Four teachers teaching the lower secondary level, who spoke to TODAY on the condition of anonymity as they were not supposed to speak to the media, said that they did not feel their working hours have been reduced.

Most of their time is spent on administrative work, planning lessons as well as co-curricular activities and other school activities, they added.

Those interviewed said that they clock from 47 hours to more than 52 hours a week, taking into account the hours spent on some Saturdays due to co-curricular activities.

This did not clarify the hours of in-school work and outside school work. Both teachers and administrators probably did not see a need for such a distinction. But a data collector and analyser should. That is lesson number 1 on research.

Lesson 2 on research: When the quantitative and qualitative findings paint different pictures, we should give pause. Both might not wrong. Instead, both might be spotlights on a larger and more complex picture. We should embrace nuance instead of simplicity.

In Part 1 of my reflection on PSLE2021, I elaborated on why the move from norm-referenced testing (NRT) to criterion-referenced testing (CRT) was a fundamental shift. It could set the tone for the desired change from unhealthy competition and comparisons to a focus on individual achievement. To the latter end, PSLE2021 will have Achievement Levels (ALs) 1 to 8 instead of scores.

To claim that ALs will help students know where they stand is one thing. To say that ALs will help students focus learning is another.

TODAYonline claimed that the new system will “help children focus on learning instead of marks”. It mirrored the official message at MOE’s PSLE microsite.

The current PSLE does not help children focus on learning. The PSLE2021 with its ALs does not guarantee that either. This is because the:

  • PSLE is summative
  • Preparation for exams discourages it
  • Comparison and competition lead to stress
  • ALs are not actionable

Summative nature of PSLE
The ALs are part of summative assessment, which typically happens at the end of a course of study. Summative assessment is sometimes referred to as assessment of learning (AoL). The PSLE ALs should indicate what the student has learnt at the end of Primary school.

Summative assessments tend to focus on the products of learning, e.g., grades or ALs, instead of processes of learning.

The PSLE in its old or new form is like the quality control near the end of a production line. It rates the product of schooling (fail or pass; if pass, then how well) and determines where it goes next (to recycle or discard; to channel to which Secondary stream).

Exam preparation
With so much at stake, kids are drilled as the PSLE draws near. However, there is much to take in because an entire Primary schooling experience boils down to exam papers.

Students learn to game a system that is stacked against them. They learn shortcuts from their classrooms and tuition centres. Memorisation is key because it is the path of least resistance and learning is for the short term. The time-honoured strategy is GIGO — garbage in, garbage out.

Comparison, competition, stress
ALs will not stop comparison and competition. This is what creates the unnecessary PSLE stress.

At least two news agencies [1] [2] explored the effect of stress. One parent reportedly said that “students will not have to stress about having to outscore their peers under the new system”. I disagree.

Children are taught to compare by adults. They are conditioned in the classroom, home, and other social environments to do this. Students find out how their peers have done. Parents compare notes because they realise their children compete for limited places in their Secondary school of choice. As a result, students are told they can do better than themselves and someone else.

Enrichment tuition centres already take advantage the desire to compare and compete. The tuition industry will tweak its marketing messages and promise better ALs instead of better letter grades or scores.

ALs are not going to remove comparisons.

ALs are not going to stop enrichment tuition as the chase will be for AL1.

There is a simpler way to examine this issue. Some have remarked on the similarity of ALs to O-level grades [1] [2]. Is there less or no competition at the Singapore-Cambridge O-Level Examinations?

Reality bites and at least one parent put it plainly in this report:

Ms Deborah Giam, whose daughter is a student at Methodist Girls’ School, agreed it will be fairer to judge a student’s own performance but also felt the PSLE will “continue to be a source of stress”. “In the case of the T-score, a lot of it was out of people’s control … so all you can do is always push your kid to do the best that they can. With this new system, it’s still the same because … at end of the day, it’s still about how many points you score,” she said.

ALs are not actionable
The PSLE is taken at the end of a child’s Primary school experience. What difference can a student make if he or she has an AL4 in an academic subject? It is not as if there are any lessons of note after the examinations. The papers are not returned to students so that they can address what they have not mastered.

Short of retaking the PSLE, a student cannot try to get a better AL. The Secondary schools students are posted to are also not necessarily going to take note of the AL and remediate.

PSLE2021 Yoda.

The fictitious Master Yoda once said, “Fear is the path to the dark side. Fear leads to anger. Anger leads to hate. Hate leads to suffering.”

If there was an Edu Yoda, he might say, “Summative assessment is the path to the Dark Side of PSLE2021. ALs lead to comparisons. Comparisons lead to competition. Competition leads to stress.”

There is a lot of noise amidst the signals of PSLE2021. Strip away the noise and you might detect the important signals.

One signal is the shift towards on standards or criteria of learning. The focus is the learner and the mastery of learning. The other is the fact that the design and implementation of PSLE2021 is still summative. The result of this is still quality control and sorting, and with it the socio-cultural baggage of comparing, competing, and unnecessary stress.

The two signals do not harmonise. One hints at a new tune, the other repeats an old refrain.

If I can keep my figurative noise-cancelling headphones on long enough, I will pick on a few more PSLE2021 notes in more reflections.

Read Part 3: Differentiation vs granularity.


Usage policy

%d bloggers like this: