On phasing out feedback.

Thanks once again to Bruno Setola for putting me on to this very interesting take on feedback (and I can recommend his site for some interesting further work on TCs).

This is an invited lecture (the whole video is 88 minutes) from Royce Sadler of the Griffith Institute for Higher Education, Griffith University, Brisbane, given for the WriteNow Centre for Excellence in Learning and Teaching, and organised by Liverpool Hope University in May 2010.

At first sight Sadler directly contradicts Hattie's major finding from his meta-analysis--which is in itself interesting enough to make him worth listening to. But there is more to his approach than that, bearing in mind that he is talking about the assessment of complex learning among university students, rather than for example the development of simpler skills among children.

The abstract is here. I am not going to cover the entire lecture, but to give pointers to some of the most interesting parts of the argument below the video:

---

---

(The video starts with introductory remarks by Lin Norton and Bart McGettrick. The numbers are the time index from the very start of the video. Numbers in [square brackets] refer to my notes at the end.)

• 25:45   Sadler argues that feedback is over-rated, and outlines the basic argument. 
• 35:00   Writing feedback on students' assessment tasks is both demanding and time-consuming for the marker, so we are entitled to ask about the return on investment of this activity. Many students don't even pick up their work after marking, still less read the comments, still less learn from them...
• 37:30   The basic problem is that giving feedback is about telling students what the problems are and how they might improve; we no longer think that the "transmission" model is a good way of teaching content, so why should we think it's a good way of conveying feedback?
• 44:00   Students are rarely familiar enough with the process of marking/grading work to have any idea what actually constitutes quality--a good piece of work--so often what we tell them means nothing to them and they don't know what to do with it. [1]
• 52:00   Accomplished practitioners in all disciplines no longer rely on other people giving them feedback based only on externally determined criteria. They know for themselves;
• When and how to adjust their provisional plans, as they are going along.
• What issues they can ignore; what doesn't matter (and of course what does).
• They notice inconsistencies in their own work, and take steps to correct them as they are going along. [2]
• 58:00   But students do not yet have those skills. So the challenge is how to inculcate them. We have them, based on hours and years of practice in assessing students' work, but that does not mean that we can communicate them.
• 71:30  Sadler describes one approach to developing students' skills in self-monitoring and -assessment in tutorial:
• He gives out a sample piece of submitted work. He asks the students;
• Does it address the task as set?
• Is it any good? (He refuses to set out the criteria for them. [3])
• Why do you say that?
• What advice would you give about how to make it better?

I don't think there is a single mention of "reflection" in the entire lecture!

[1]  It's not only students. This is a little-mentioned feature of involving less-experienced colleagues, such as postgrads (and here), practice supervisors and mentors in the assessment process. They may be highly- experienced as practitioners, but not as assessors. A course on which I work matches each student with a work-place based mentor, who receives some training and undertakes some direct assessment of teaching, among other duties. But many mentors only work with one student, and often have fallow years with none at all. They will undertake just four formal observations with each student over two years--and those may well be the only occasions they use the observation protocol. The tutors do dozens or scores of observations every year, and develop a common approach to making judgements, but we can never be sure that mentors have had a chance to internalise those standards. No--you can't address the issue with more reams of spurious "guidance"...

[2]  Sadler goes on to suggest that waiting to give feedback until after a piece of work has been finally and formally submitted is pointless and counter-productive. The point at which to make suggestions for improvement is when the student still has a chance to implement them. A course on which I teach and which I helped to design many years ago was recently severely criticised in an internal review for permitting "dry runs", when students could submit partially completed assessments in advance for just such feedback. The basis of the criticism was that it conferred an unfair advantage on those students who made use of it. This seems to me to suggest that the reviewers did not have a clue... They had lost sight of the idea that being able to produce a good piece of work on a well-designed assessment task is what we are all working towards; effectively handicapping students by denying them feedback when they can use it is a grodd distortion of the process. (Yes, there is a cap on how many times it can be done...)

[3]  He is of course helping them to develop their own frame of reference, and capacity to notice factors germane to the quality of the piece. But I wonder how he learned this bit? I was asked a few years ago to contribute to a staff development event on "Assessment on Taught Master's Courses", a joint enterprise between two universities. I devised an exercise which was almost identical to Sadler's except that the participants were themselves experienced tutors and assessors at Master's level--and it ground to an embarrassing halt when they all claimed they could not judge how good the samples were without the rubrics and criteria to guide them. I felt very remiss for not having provided  them, and--even worse--unprofessional, because I know what I am looking for, and the qualities and trade-offs to notice. I suspect now that they felt them same but could not admit it in front of their peers. I feel a little vindicated that Sadler adopts the same approach.

There is something similar underpinning this piece by Gary Klein--in quite a different context!

On writing a dissertation

This is what I have been up to when not posting to this blog. Honest!

These notes originate from my efforts to help an old friend undertaking a taught Master's course, at a university which seemed to devise its regulations around the key principle that under no circumstances should, or could, a supervisor be helpful. So my reaction to his first almost complete draft was to say that it needed to be completely re-organised; it was great content, but did not follow the conventions of a dissertation. Quite reasonably he asked how he was supposed to know that? This is a game he would only play once, whereas supervisors and assessors and external examiners are familiar with it. He suggested,

"I should have found (useful) a model outline for the MA, like

A. Intro could include..........and should not......

B. Literary review could..... and should not..............

C. Further Chapters could.... and should not...........

D. Conclusion could ...and should not...... "

So this is my attempt to provide just that. I had just assumed that it already existed in practically every text on Research Methods, but on the basis of the sample I have consulted. it doesn't. I've already modified it in response to some very useful suggestions from people on lists I belong to, and plan to continue to do so—so please write to me (address in the footer) and together we can improve it further. Thanks.

On how to write a research paper

From Andrew Gelman (picked up on the Marginal Revolution blog). Very sound and clear advice.

The order is rather different from that which I suggest for writing a dissertation here. That is because Gelman's approach suits writing up a paper after the research has been completed, and is about communicating it effectively. Students doing dissertations are often writing concurrently with undertaking the work, and the structure of the article should provide some scaffolding of that process. Having said that, I acknowledge that "writing up" often does not work that way.

On less being more at Master's level

I have been on a validation panel today, grappling with the problem of how to specify modules at Master's level. It was very instructive.

The team developing the programme had understandably assumed that outcomes for a Master's level module would have to be more tightly specified than for a lower level course, in order to ensure that the learning and the assessment would be at greater “depth” than for an undergraduate module.

First, though, it is quite easy to specify “learning outcomes” for low-level courses. When people are learning the basics of any subject or skill, what any one person learns will have to be the same as any other learner. It's easy to assess, and “correct” knowledge or performance is clear-cut. It's not so easy at higher levels. Knowledge and “understanding” (not to mention, for those who care, the higher regions of Bloom, or Krathwohl and Anderson) may be contestable, and indeed one person's understanding or “take” on the subject may quite legitimately be different from their neightbour's. So when you get to Master's level it may be reasonable to specify that the outcomes will include so-and-so, but it may well be patronising and simply counter-productive to presume to set them out completely and exhaustively.

Even so, how do you incorporate the academic level requirements into the outcomes? It's traditional to use all those recommended “Bloom verbs” to produce “SMART” objectives. (What's the difference between an objective and an outcome, in this context? Strictly between ourselves, I no longer have to pretend that I know, and I don't care.) So the first-year students “list” or “describe”, the second-years “analyse” and the third-years “evaluate”... So Master's students? They “critically evaluate”, it seems. (That means in practice that they evaluate on the basis of one or more over-arching frameworks, showing that there are no simple answers.)

That is fair enough but it does risk becoming formulaic, and also implying that there is a correct procedure for doing it. The more specific the directions, the more restricted the outcomes, and the less the scope for the exercise of individual initiative and creativity (if that is desirable in your discipline, of course!) Master's students are experts, or at least nearly there. They need to be given their heads rather than constrained. (I am referring mainly to experienced practitioners of their discipline undertaking Master's study part-time, here; I am aware that full-time “second cycle” students who were undergraduates last year may not fit this picture.)

Personally I would rather just set out the aims of the module just so the students know what they are letting themselves in for, and recognise that the outcomes will be different every time it runs, and different for every participant. But that won't wash in the compliance climate where standardisation is all. So what can we get away with?

(Specify the level and assessment criteria at a scheme/programme level so you don't have to do it at a module level)

"On completion of this module, participants will (I prefer “participant” to “student” at this level);

• Come to their own informed conclusions about the significance of...
• Explore ... in the context of ...
• Use ... as the basis of original work on ...

Or have I got it wrong?