On phasing out feedback.

Thanks once again to Bruno Setola for putting me on to this very interesting take on feedback (and I can recommend his site for some interesting further work on TCs).

This is an invited lecture (the whole video is 88 minutes) from Royce Sadler of the Griffith Institute for Higher Education, Griffith University, Brisbane, given for the WriteNow Centre for Excellence in Learning and Teaching, and organised by Liverpool Hope University in May 2010.

At first sight Sadler directly contradicts Hattie's major finding from his meta-analysis--which is in itself interesting enough to make him worth listening to. But there is more to his approach than that, bearing in mind that he is talking about the assessment of complex learning among university students, rather than for example the development of simpler skills among children.

The abstract is here. I am not going to cover the entire lecture, but to give pointers to some of the most interesting parts of the argument below the video:

---

---

(The video starts with introductory remarks by Lin Norton and Bart McGettrick. The numbers are the time index from the very start of the video. Numbers in [square brackets] refer to my notes at the end.)

• 25:45   Sadler argues that feedback is over-rated, and outlines the basic argument. 
• 35:00   Writing feedback on students' assessment tasks is both demanding and time-consuming for the marker, so we are entitled to ask about the return on investment of this activity. Many students don't even pick up their work after marking, still less read the comments, still less learn from them...
• 37:30   The basic problem is that giving feedback is about telling students what the problems are and how they might improve; we no longer think that the "transmission" model is a good way of teaching content, so why should we think it's a good way of conveying feedback?
• 44:00   Students are rarely familiar enough with the process of marking/grading work to have any idea what actually constitutes quality--a good piece of work--so often what we tell them means nothing to them and they don't know what to do with it. [1]
• 52:00   Accomplished practitioners in all disciplines no longer rely on other people giving them feedback based only on externally determined criteria. They know for themselves;
• When and how to adjust their provisional plans, as they are going along.
• What issues they can ignore; what doesn't matter (and of course what does).
• They notice inconsistencies in their own work, and take steps to correct them as they are going along. [2]
• 58:00   But students do not yet have those skills. So the challenge is how to inculcate them. We have them, based on hours and years of practice in assessing students' work, but that does not mean that we can communicate them.
• 71:30  Sadler describes one approach to developing students' skills in self-monitoring and -assessment in tutorial:
• He gives out a sample piece of submitted work. He asks the students;
• Does it address the task as set?
• Is it any good? (He refuses to set out the criteria for them. [3])
• Why do you say that?
• What advice would you give about how to make it better?

I don't think there is a single mention of "reflection" in the entire lecture!

[1]  It's not only students. This is a little-mentioned feature of involving less-experienced colleagues, such as postgrads (and here), practice supervisors and mentors in the assessment process. They may be highly- experienced as practitioners, but not as assessors. A course on which I work matches each student with a work-place based mentor, who receives some training and undertakes some direct assessment of teaching, among other duties. But many mentors only work with one student, and often have fallow years with none at all. They will undertake just four formal observations with each student over two years--and those may well be the only occasions they use the observation protocol. The tutors do dozens or scores of observations every year, and develop a common approach to making judgements, but we can never be sure that mentors have had a chance to internalise those standards. No--you can't address the issue with more reams of spurious "guidance"...

[2]  Sadler goes on to suggest that waiting to give feedback until after a piece of work has been finally and formally submitted is pointless and counter-productive. The point at which to make suggestions for improvement is when the student still has a chance to implement them. A course on which I teach and which I helped to design many years ago was recently severely criticised in an internal review for permitting "dry runs", when students could submit partially completed assessments in advance for just such feedback. The basis of the criticism was that it conferred an unfair advantage on those students who made use of it. This seems to me to suggest that the reviewers did not have a clue... They had lost sight of the idea that being able to produce a good piece of work on a well-designed assessment task is what we are all working towards; effectively handicapping students by denying them feedback when they can use it is a grodd distortion of the process. (Yes, there is a cap on how many times it can be done...)

[3]  He is of course helping them to develop their own frame of reference, and capacity to notice factors germane to the quality of the piece. But I wonder how he learned this bit? I was asked a few years ago to contribute to a staff development event on "Assessment on Taught Master's Courses", a joint enterprise between two universities. I devised an exercise which was almost identical to Sadler's except that the participants were themselves experienced tutors and assessors at Master's level--and it ground to an embarrassing halt when they all claimed they could not judge how good the samples were without the rubrics and criteria to guide them. I felt very remiss for not having provided  them, and--even worse--unprofessional, because I know what I am looking for, and the qualities and trade-offs to notice. I suspect now that they felt them same but could not admit it in front of their peers. I feel a little vindicated that Sadler adopts the same approach.

There is something similar underpinning this piece by Gary Klein--in quite a different context!

On one reason not to grade...

It presupposes a degree of maturity on the part of students, but the course with which I am still associated, operates only on a pass/refer basis (one opportunity to re-submit referred work; fiture at that point is failure and requires a re-take of the whole module, at best). Many students don't like it--particularly those who are themselves graduates and have a long history of assessing themselves by their marks, often of course relative to their class-mates.

The situation Mary Beard describes in the linked post is one reason (and it is incredibly time-consuming to adopt her solution, particularly for big courses).But it goes a bit further than that.

Let's apply the Dale/Bruner “Cone of Experience” to the situation--slightly odd, but it fills the bill and such tools are never more than pragmatic devices. In terms of assessment;

• what the students actually know/can do etc. is at the enactive level. It's real life. It's incredibly rich and complex and inherently unassessable because of that, so
  
• we devise means of assessing which necessarily lose a lot of detail in order to be manageable. Sometimes the detail we actually lose is precisely what we wanted to assess in the first place, but we just get it wrong. This is at the iconic level. The students do the assessment, but in order to compare them against a standard, or even against each other, or previous performance...
  
• we have to assess the assessment, and that gets even more abstract and symbolic. And attaching a single number or letter is as abstract as one can get (apart of course from manipulating those number mathematically) --or in other words loses the most information.

Assessment is always a lossy process, but numerical marking exacerbates the situation. That magic number is a distant and distorted proxy for the blooming buzzing complexity (if not confusion) of the student's experience and conduct; which moreover obscures that enactive level. We have made grades etc. so central to a mass education system (no, I don't know the alternative either) that once one has been issued it is almost impossible to get through any more sophisticated message to the student;

"Yes, it did only get a C+, but that was solely down to those daft mistakes you made with the statistics; if you'd got them right it would probably have been an A-"

"Yeah, right..."

Such practices are actively inimical to effective feedback and assessment for learning, in the current jargon.

On explaining grades

The linked piece is a column (dated yesterday but I think I've seen it before) in the Chronicle of Higher Education by Gina Barreca of the University of Connecticut. What I found most remarkable was not the piece itself but part of the first comment:

"Nobody should explain a grade to a student."

As flat and dogmatic as that. On the contrary, nobody should ever give a student a grade without explanation.

The principles of formative assessment (or "assessment for learning") suggest strongly that students can learn an enormous amount from good feedback. Indeed, Hattie concludes that the single most effective way of improving learning is to give "dollops of feedback". And all assessment can be formative. The mark or grade is simply a summary or proxy for the feedback, but it is so abstracted as to give practically no information. On a course on which I currently teach, there are no grades, just "pass" and "refer" (which caan lead to an outright "fail"). But there are dollops of feedback.


Last week one of the participants--"student" is not quite the word for well-qualified professionals in their fields who happen to be learning to teach that field--did ask a colleague for a grade.He demurred, but he could see how uncomfortable it made her. She had been through school, an undergraduate degree and a Master's getting grades all the time; a piece of work had not been properly assessed unless it got a grade, as far as she was concerned even if as in this case, it carried credits if it passed.

It's interesting that the education system is so hooked on grades; in the rest of the working world we get plenaty of feedback, but only in teaching do we get graded (by Ofsted)*.

I suspect that the sub-text of the original comment is that no student should ever question the judgement of a faculty member, however arbitrary.

*Not entirely true, but substantially so.

On an ingenious approach to marking assignments

Steve Hill of Southampton Solent University has been experimenting with Camtasia, a screen recording package which also takes audio, as a means of recording his spoken comments on students' work, which he can then send to them.

Students submit assignments electronically using our Moodle-based VLE. I then get a student’s Word file on screen and edit it, whilst simultaneously giving a live commentary on the changes that I’m making. This is like giving a student their own personal tutorial.

Sounds like an excellent idea!

On feedback

Do read the linked column—and, as important, the comments.

The author (now a professor) has kept her former professor's annotation of an essay she wrote in her second year as an undergraduate. She reproduces them and her readers comment on them. It's fascinating, and also rather disconcerting to see how many ways the notes can be interpreted, as well as to consider whether that kind of feedback would be acceptable thirty years on.