On phasing out feedback.

Thanks once again to Bruno Setola for putting me on to this very interesting take on feedback (and I can recommend his site for some interesting further work on TCs).

This is an invited lecture (the whole video is 88 minutes) from Royce Sadler of the Griffith Institute for Higher Education, Griffith University, Brisbane, given for the WriteNow Centre for Excellence in Learning and Teaching, and organised by Liverpool Hope University in May 2010.

At first sight Sadler directly contradicts Hattie's major finding from his meta-analysis--which is in itself interesting enough to make him worth listening to. But there is more to his approach than that, bearing in mind that he is talking about the assessment of complex learning among university students, rather than for example the development of simpler skills among children.

The abstract is here. I am not going to cover the entire lecture, but to give pointers to some of the most interesting parts of the argument below the video:

---

---

(The video starts with introductory remarks by Lin Norton and Bart McGettrick. The numbers are the time index from the very start of the video. Numbers in [square brackets] refer to my notes at the end.)

• 25:45   Sadler argues that feedback is over-rated, and outlines the basic argument. 
• 35:00   Writing feedback on students' assessment tasks is both demanding and time-consuming for the marker, so we are entitled to ask about the return on investment of this activity. Many students don't even pick up their work after marking, still less read the comments, still less learn from them...
• 37:30   The basic problem is that giving feedback is about telling students what the problems are and how they might improve; we no longer think that the "transmission" model is a good way of teaching content, so why should we think it's a good way of conveying feedback?
• 44:00   Students are rarely familiar enough with the process of marking/grading work to have any idea what actually constitutes quality--a good piece of work--so often what we tell them means nothing to them and they don't know what to do with it. [1]
• 52:00   Accomplished practitioners in all disciplines no longer rely on other people giving them feedback based only on externally determined criteria. They know for themselves;
• When and how to adjust their provisional plans, as they are going along.
• What issues they can ignore; what doesn't matter (and of course what does).
• They notice inconsistencies in their own work, and take steps to correct them as they are going along. [2]
• 58:00   But students do not yet have those skills. So the challenge is how to inculcate them. We have them, based on hours and years of practice in assessing students' work, but that does not mean that we can communicate them.
• 71:30  Sadler describes one approach to developing students' skills in self-monitoring and -assessment in tutorial:
• He gives out a sample piece of submitted work. He asks the students;
• Does it address the task as set?
• Is it any good? (He refuses to set out the criteria for them. [3])
• Why do you say that?
• What advice would you give about how to make it better?

I don't think there is a single mention of "reflection" in the entire lecture!

[1]  It's not only students. This is a little-mentioned feature of involving less-experienced colleagues, such as postgrads (and here), practice supervisors and mentors in the assessment process. They may be highly- experienced as practitioners, but not as assessors. A course on which I work matches each student with a work-place based mentor, who receives some training and undertakes some direct assessment of teaching, among other duties. But many mentors only work with one student, and often have fallow years with none at all. They will undertake just four formal observations with each student over two years--and those may well be the only occasions they use the observation protocol. The tutors do dozens or scores of observations every year, and develop a common approach to making judgements, but we can never be sure that mentors have had a chance to internalise those standards. No--you can't address the issue with more reams of spurious "guidance"...

[2]  Sadler goes on to suggest that waiting to give feedback until after a piece of work has been finally and formally submitted is pointless and counter-productive. The point at which to make suggestions for improvement is when the student still has a chance to implement them. A course on which I teach and which I helped to design many years ago was recently severely criticised in an internal review for permitting "dry runs", when students could submit partially completed assessments in advance for just such feedback. The basis of the criticism was that it conferred an unfair advantage on those students who made use of it. This seems to me to suggest that the reviewers did not have a clue... They had lost sight of the idea that being able to produce a good piece of work on a well-designed assessment task is what we are all working towards; effectively handicapping students by denying them feedback when they can use it is a grodd distortion of the process. (Yes, there is a cap on how many times it can be done...)

[3]  He is of course helping them to develop their own frame of reference, and capacity to notice factors germane to the quality of the piece. But I wonder how he learned this bit? I was asked a few years ago to contribute to a staff development event on "Assessment on Taught Master's Courses", a joint enterprise between two universities. I devised an exercise which was almost identical to Sadler's except that the participants were themselves experienced tutors and assessors at Master's level--and it ground to an embarrassing halt when they all claimed they could not judge how good the samples were without the rubrics and criteria to guide them. I felt very remiss for not having provided  them, and--even worse--unprofessional, because I know what I am looking for, and the qualities and trade-offs to notice. I suspect now that they felt them same but could not admit it in front of their peers. I feel a little vindicated that Sadler adopts the same approach.

There is something similar underpinning this piece by Gary Klein--in quite a different context!

On one reason not to grade...

It presupposes a degree of maturity on the part of students, but the course with which I am still associated, operates only on a pass/refer basis (one opportunity to re-submit referred work; fiture at that point is failure and requires a re-take of the whole module, at best). Many students don't like it--particularly those who are themselves graduates and have a long history of assessing themselves by their marks, often of course relative to their class-mates.

The situation Mary Beard describes in the linked post is one reason (and it is incredibly time-consuming to adopt her solution, particularly for big courses).But it goes a bit further than that.

Let's apply the Dale/Bruner “Cone of Experience” to the situation--slightly odd, but it fills the bill and such tools are never more than pragmatic devices. In terms of assessment;

• what the students actually know/can do etc. is at the enactive level. It's real life. It's incredibly rich and complex and inherently unassessable because of that, so
  
• we devise means of assessing which necessarily lose a lot of detail in order to be manageable. Sometimes the detail we actually lose is precisely what we wanted to assess in the first place, but we just get it wrong. This is at the iconic level. The students do the assessment, but in order to compare them against a standard, or even against each other, or previous performance...
  
• we have to assess the assessment, and that gets even more abstract and symbolic. And attaching a single number or letter is as abstract as one can get (apart of course from manipulating those number mathematically) --or in other words loses the most information.

Assessment is always a lossy process, but numerical marking exacerbates the situation. That magic number is a distant and distorted proxy for the blooming buzzing complexity (if not confusion) of the student's experience and conduct; which moreover obscures that enactive level. We have made grades etc. so central to a mass education system (no, I don't know the alternative either) that once one has been issued it is almost impossible to get through any more sophisticated message to the student;

"Yes, it did only get a C+, but that was solely down to those daft mistakes you made with the statistics; if you'd got them right it would probably have been an A-"

"Yeah, right..."

Such practices are actively inimical to effective feedback and assessment for learning, in the current jargon.

On game-changing

I have my doubts about "reflection" as a panacea for professional development, despite the title of this blog. But in practical terms, my most effective learning from experience occurs when I find that themes from several different instances/areas/events/ideas come together to make a point.

But! I find myself warning some of my best and most curious students (I don't mean "freakiest", although the categories are not mutually exclusive...) against producing spurious syncretist theories of everything. I may be guilty of that, too.

What prompts this thought is a political debate (of course), editing a video (discussed more in the 10 May post), and a TV series on the history of science.

Politically, proportional representation as a form of electoral reform is a game-changer. It's irreversible short of a period of anarchy/revolution. I'm all for it, incidentally, as long as the system maintains an adequate link between representative and locality (which is a little more specific than "constituency"). But it is a BIG step.

It's an instance of second-order change (Watzlawick et al 1974)

• First-order change: Plus ca change, plus c'est la meme chose... The more things change, the more they stay the same (attributed to Karr, 1849)
• Second-order change: Things will never be the same again.

I note in the post on 10 May "On another threshold concept, rediscovered" that Kuhn's notion of "paradigm shifts" is an instance of second-order change in the history of science. Mosley's excellent BBC2 series (linked to above, and on again at 9pm today) has very effectively explored such instances, and in so doing demonstrated that--for later students--they constitute threshold concepts.

The video includes some plenary discussion of students' conceptions (and misconceptions) of threshold concepts, and watching it over and over again for editing purposes is a good way of at least forming some hypotheses about their understanding (getting feedback from them, as Hattie emphasises). I am struck by how "safe" are the TCs they chose to mention, in particular. As commentators Peter H and myself found ourselves challenged potentially to accept the proffered examples as they stood, even when they did not quite hit the target--thereby running the risk of diluting the idea to the status of "statements of the bleeding obvious", which a few students obviously thought we did, from their evaluations. Apart from the initial statement about the limitations of the applicability of TCs to practice with students with Special Education Needs, clearly students did not want to "show themselves up" in front of the assembled multitude by risking a contentious suggestion. And interestingly, few referred to their own learning of their discipline--which was what the exercise asked--but reverted to issues about teaching it.

What all this is reminding me of, apart from the general point about the much-neglected emotional agenda in teaching and learning, is just how disconcerting and disorienting is the liminality of TCs, and how much people may avoid it, because they have not got their bearings in the new space. Just as I hear politicians on the radio resiling for fear of what lies beyond the portal of PR, which is indeed scary until you think that it is what every other European democracy bar one uses. Just like scientists wishing no-one had ever discovered quantum physics.

On explaining grades

The linked piece is a column (dated yesterday but I think I've seen it before) in the Chronicle of Higher Education by Gina Barreca of the University of Connecticut. What I found most remarkable was not the piece itself but part of the first comment:

"Nobody should explain a grade to a student."

As flat and dogmatic as that. On the contrary, nobody should ever give a student a grade without explanation.

The principles of formative assessment (or "assessment for learning") suggest strongly that students can learn an enormous amount from good feedback. Indeed, Hattie concludes that the single most effective way of improving learning is to give "dollops of feedback". And all assessment can be formative. The mark or grade is simply a summary or proxy for the feedback, but it is so abstracted as to give practically no information. On a course on which I currently teach, there are no grades, just "pass" and "refer" (which caan lead to an outright "fail"). But there are dollops of feedback.


Last week one of the participants--"student" is not quite the word for well-qualified professionals in their fields who happen to be learning to teach that field--did ask a colleague for a grade.He demurred, but he could see how uncomfortable it made her. She had been through school, an undergraduate degree and a Master's getting grades all the time; a piece of work had not been properly assessed unless it got a grade, as far as she was concerned even if as in this case, it carried credits if it passed.

It's interesting that the education system is so hooked on grades; in the rest of the working world we get plenaty of feedback, but only in teaching do we get graded (by Ofsted)*.

I suspect that the sub-text of the original comment is that no student should ever question the judgement of a faculty member, however arbitrary.

*Not entirely true, but substantially so.

On an ingenious approach to marking assignments

Steve Hill of Southampton Solent University has been experimenting with Camtasia, a screen recording package which also takes audio, as a means of recording his spoken comments on students' work, which he can then send to them.

Students submit assignments electronically using our Moodle-based VLE. I then get a student’s Word file on screen and edit it, whilst simultaneously giving a live commentary on the changes that I’m making. This is like giving a student their own personal tutorial.

Sounds like an excellent idea!

On evaluating the one-off lecture

OK, it happened, and indirect feeedback via a colleague who could not attend but who spoke to people after the event was that it had been well-received, even remarked on as a "proper academic lecture" (not sure whether or not that is a compliment!)

There are of course standard procedures for evaluation, including the usual feedback sheets and the like. Such procedures are unsurprisingly not routine on this course, and it didn't seem appropriate to introduce them--after all, this was from the students' point of view just another session in a regular timetable. It was my decision to get all self-conscious about it.

So just a few remarks on practical aspects, and then a more general thought.

• time-keeping and pacing is always tricky the first time around; I seem to have got it pretty well right, though.
• The presentation? You can see it via the link on yesterday's post, but the version there has been annotated and expanded. I don't like too much content on the slides, and I tried to use them as headings to show the students where we were. That was OK with a heading like "fashions and fads", but not really satisfactory when I talked ever so briefly about a particular theorist; I was using slides cannibalised from another lecture including images which may not have been the very best choices. 
• Delivery? A guy at the back said he couldn't hear me in the first few seconds. I thanked him and talked louder, but I didn't continue to check--which I could have done very easily with the traffic-light cards all the audience had. Specifically everyone had a red, a yellow and a green A5 card, with which they could signal answers to questions, like "If you have never heard of Skinner, show a red card. If you've heard of him but don't know much about his ideas, show yellow. If you know quite a bit about him show green..." It's clickers-lite, but a lot easier to manage spontaneously.
  
• As to use of the cards; they were appreciated and commented on afterwards, and they are so easy to use. No downside, if you sort out the logistics of distribution. As ever, it is not merely the technical advantage they confer above a simple show of hands which makes the difference--it is what the very fact of issuing them and using them says about the kind of interaction the lecturer wants within the session. They send a message--"I want you to be able to communicate back to me".
• And as for that great bug-bear of the lecture, attention-span, that little bit of activity every few minutes seemed to get the students re-engaged. Visual clues suggested they were still with me at the end of the session.
• The two-minute buzz-group exercise came almost exactly half-way through; frankly it was as much about breaking the session up as it was about content, but some of the ideas were interesting enough to weave into the rest of the session. It also gave me a chance to nip out and buy an extortionately priced bottle of water--why no water-coolers?

More generally, though; did the session achieve its objective of providing an orientation to learning theories so students can engage with them on a fairly deep level? As ever, I think the argument was clearer to me than it was to them, and I perpetrated the sin I have been trying not to commit for years and years (and which I can avoid as a part of regular teaching); I elaborate arguments in the hope of making them clearer, but in practice I am obscuring and burying them in a mass of marginally relevant detail. Once I get into the swing of a lecture I can always think of more stuff to pile in. When it is something I do regularly, I can prune it and filter it so that what gets said is what needs to be said and no more, but I'm still not good at that on a one-off basis.

And is a lecture, at this stage in their programme, the appropriate means of communicating such a message? Probably not, but given the constraints it was a reasonable strategy to choose.

One comment my colleague heard was that I had rubbished some current fads, or even sacred cows. Indeed I had, and I now wonder whether or not I should have done. After all there was a team of about a dozen other colleagues attending, most of whom I don't know, and I may well have stepped clumsily on a few people's toes. I'm pretty sure the students enjoyed it, but that's not the point. Had I had more notice I could and should have circulated some of the proposed content in advance for comment; who knows, perhaps one of those colleagues will have to deliver a session next week advocating learning styles! An interesting and open question about the responsibilities one has to the rest of the team...

Still, I have to admit that I enjoyed myself thoroughly; not something to be aimed for, but a good spin-off if it happens.

On feedback

Do read the linked column—and, as important, the comments.

The author (now a professor) has kept her former professor's annotation of an essay she wrote in her second year as an undergraduate. She reproduces them and her readers comment on them. It's fascinating, and also rather disconcerting to see how many ways the notes can be interpreted, as well as to consider whether that kind of feedback would be acceptable thirty years on.