On phasing out feedback.

Thanks once again to Bruno Setola for putting me on to this very interesting take on feedback (and I can recommend his site for some interesting further work on TCs).

This is an invited lecture (the whole video is 88 minutes) from Royce Sadler of the Griffith Institute for Higher Education, Griffith University, Brisbane, given for the WriteNow Centre for Excellence in Learning and Teaching, and organised by Liverpool Hope University in May 2010.

At first sight Sadler directly contradicts Hattie's major finding from his meta-analysis--which is in itself interesting enough to make him worth listening to. But there is more to his approach than that, bearing in mind that he is talking about the assessment of complex learning among university students, rather than for example the development of simpler skills among children.

The abstract is here. I am not going to cover the entire lecture, but to give pointers to some of the most interesting parts of the argument below the video:

---

---

(The video starts with introductory remarks by Lin Norton and Bart McGettrick. The numbers are the time index from the very start of the video. Numbers in [square brackets] refer to my notes at the end.)

• 25:45   Sadler argues that feedback is over-rated, and outlines the basic argument. 
• 35:00   Writing feedback on students' assessment tasks is both demanding and time-consuming for the marker, so we are entitled to ask about the return on investment of this activity. Many students don't even pick up their work after marking, still less read the comments, still less learn from them...
• 37:30   The basic problem is that giving feedback is about telling students what the problems are and how they might improve; we no longer think that the "transmission" model is a good way of teaching content, so why should we think it's a good way of conveying feedback?
• 44:00   Students are rarely familiar enough with the process of marking/grading work to have any idea what actually constitutes quality--a good piece of work--so often what we tell them means nothing to them and they don't know what to do with it. [1]
• 52:00   Accomplished practitioners in all disciplines no longer rely on other people giving them feedback based only on externally determined criteria. They know for themselves;
• When and how to adjust their provisional plans, as they are going along.
• What issues they can ignore; what doesn't matter (and of course what does).
• They notice inconsistencies in their own work, and take steps to correct them as they are going along. [2]
• 58:00   But students do not yet have those skills. So the challenge is how to inculcate them. We have them, based on hours and years of practice in assessing students' work, but that does not mean that we can communicate them.
• 71:30  Sadler describes one approach to developing students' skills in self-monitoring and -assessment in tutorial:
• He gives out a sample piece of submitted work. He asks the students;
• Does it address the task as set?
• Is it any good? (He refuses to set out the criteria for them. [3])
• Why do you say that?
• What advice would you give about how to make it better?

I don't think there is a single mention of "reflection" in the entire lecture!

[1]  It's not only students. This is a little-mentioned feature of involving less-experienced colleagues, such as postgrads (and here), practice supervisors and mentors in the assessment process. They may be highly- experienced as practitioners, but not as assessors. A course on which I work matches each student with a work-place based mentor, who receives some training and undertakes some direct assessment of teaching, among other duties. But many mentors only work with one student, and often have fallow years with none at all. They will undertake just four formal observations with each student over two years--and those may well be the only occasions they use the observation protocol. The tutors do dozens or scores of observations every year, and develop a common approach to making judgements, but we can never be sure that mentors have had a chance to internalise those standards. No--you can't address the issue with more reams of spurious "guidance"...

[2]  Sadler goes on to suggest that waiting to give feedback until after a piece of work has been finally and formally submitted is pointless and counter-productive. The point at which to make suggestions for improvement is when the student still has a chance to implement them. A course on which I teach and which I helped to design many years ago was recently severely criticised in an internal review for permitting "dry runs", when students could submit partially completed assessments in advance for just such feedback. The basis of the criticism was that it conferred an unfair advantage on those students who made use of it. This seems to me to suggest that the reviewers did not have a clue... They had lost sight of the idea that being able to produce a good piece of work on a well-designed assessment task is what we are all working towards; effectively handicapping students by denying them feedback when they can use it is a grodd distortion of the process. (Yes, there is a cap on how many times it can be done...)

[3]  He is of course helping them to develop their own frame of reference, and capacity to notice factors germane to the quality of the piece. But I wonder how he learned this bit? I was asked a few years ago to contribute to a staff development event on "Assessment on Taught Master's Courses", a joint enterprise between two universities. I devised an exercise which was almost identical to Sadler's except that the participants were themselves experienced tutors and assessors at Master's level--and it ground to an embarrassing halt when they all claimed they could not judge how good the samples were without the rubrics and criteria to guide them. I felt very remiss for not having provided  them, and--even worse--unprofessional, because I know what I am looking for, and the qualities and trade-offs to notice. I suspect now that they felt them same but could not admit it in front of their peers. I feel a little vindicated that Sadler adopts the same approach.

There is something similar underpinning this piece by Gary Klein--in quite a different context!

On surrogates and assessment

I'm starting off with two observations about quite different things.

The first is this one from Ben Goldacre's brilliant "Bad Science" blog at the Guardian. His topic concerns a press release about a potential new treatment for Duchenne's Muscular Dystrophy.

"...this story is also a reminder that we should always be cautious with "surrogate" outcomes. The biological change measured was important, and good grounds for optimism, because it shows the treatment is doing what it should in the body. But things that work in theory do not always work in practice, and while a measurable biological indicator is a hint something is working, such outcomes can often be misleading."

And later...

"...improvements on surrogate biological outcomes that can be measured in the body are a strong hint that something works – and I hope this new DMD treatment does turn out to be effective – but even in the most well-established surrogate measures, and drugs, these endpoints can turn out to be misleading."

A fairly basic point, of course, but it did set me thinking about how the extent to which we have become obsessed with measurement in many fields, including teaching and learning, had led to increasing reliance on fairly dubious surrogates.

And then I came across this commentary on Standard and Poor's revision of the the US credit rating:

"[Ratings agencies] ..are human enterprises, fallible institutions—and like other institutions, they have procedures, interests, and histories. Their records deserve inspection. In the scientific spirit, in the spirit of show me, they deserve scrutiny."

A credit rating is a complex construct (I presume). Since it is supposed to have predictive value (other than merely being part of a self-fulfilling prophecy), it must be put together from a raft of surrogate measures, presumably of directly observable factors which co-vary with an institutions credit-worthiness. But it is only as good as the choice of those surrogates*.

Which led to some general thoughts in relation to education.

Today is the day when GCSE results are published (the exams taken at age 16 by practically all pupils in the UK). The press stories are predictable, suggesting grade inflation and the exams being dumbed-down. (Or of course if the pass-rates were not an improvement on last year, there would be jeremiads about further decline in educational standards...) The press discussion will not be sophisticated, but it will at least acknowledge what the politicians and the educational establishment will deny, namely that the examinations are not realistic proxies for educational achievement.

This is leaving aside the issue of the tail wagging the dog, of "teaching to the test" without ever asking whether the test is valid or reliable. Beyond that, the logistics and practicalities of mass assessment distort the process, and it has ever been thus. When Liam Hudson (1967) discussed convergent and divergent thinking styles, he noted that convergent thinking was privileged in school at least in part because its testing could be standardised.

But these artificial surrogate assessments are increasingly separating the formal educational system from the "real world", particularly that of communities of practice. This is not an original observation; while I'm on "golden oldies", I'll refer to Becker's wonderful 1972 paper School is a Lousy Place to Learn Anything In, which is based inter alia on a similar argument.

It is out of an awareness of the intrinsic limitations of such surrogacy that a course on teaching with which I have long been involved has attempted to develop a more authentic assessment strategy. Of course, teaching courses have always routinely involved direct observation of teaching, but not everything is amenable to direct observation. The traditional solution on most** other courses has been set assignments; our course moved away from that to negotiated submissions based on a learning contract. Learning outcomes are specified and students decide, in consultation with a tutor, of course, what evidence they will submit to demonstrate that the outcomes have been met. This is a step closer to reality, but of course only insofar as the specified learning outcomes correspond to the real world.

The course has just been internally reviewed for routine reasons, and it is apparent that the bureaucrats hate the assessment scheme. Work is not graded, for example. The scheme is not suited to anonymous submission, because the students are talking about their own practice and work setting (it is an in-service course). Not all work is suited to electronic submission via Turnitin.... the list of complaints goes on.

The real problem is that validity, reliability and fairness--the traditional requirements of an assessment scheme--are now subordinated to standardisation, administrative convenience, and security***.

These are considerations for the legitimation of surrogates and proxies--the same kind of consideration as applies to the regulation of second or third-order derived financial instruments which no longer bear any relation to buying and selling stuff which is any actual use.

---

* I am not relying entirely on a single blog-post here! See also Dan Gardner's excellent and accessible Future Babble: Why expert predictions fail and why we believe them anyway. (London; Virgin Books, 2011) It's a great corrective to all the doom and gloom surrounding us. Incidentally, he draws a lot on the work of Philip Tetlock, the subject of this interview by Jonah Lehrer in Wired.

** Most but not all, our approach owes much to work at the University of Huddersfield, particularly in the early '90s.

*** Security in the sense of not being vulnerable to plagiarism, although the emphasis on discussion of one's own practice and production of examples and resources means that the approach is fairly protected in any case.

Becker H (1972) “School is a Lousy Place to Learn Anything In” American Behavioral Scientist (1972):85-105, reproduced in R G Burgess (ed.) (1995) Howard Becker on Education Buckingham: OU Press

(Update later today: Many thanks to David Stone, who writes; "I was happy to discover that my institutional subscription gave me access to the original Becker article. Just in case others should be as lucky, here is the DOI link:
http://dx.doi.org/10.1177/000276427201600109 ")

Hudson L (1967) Contrary Imaginations; a psychological study of the English Schoolboy Harmondsworth: Penguin

On the acceptability of originality, criticality and context.

I came to university with a history--at least in my mind; probably no-one else ever noticed. I failed O level English Literature (now GCSE, and apparently almost impossible actually to fail). I passed on a re-sit. But as far as I can reconstruct, given that the whole assessment process was inaccessible to scrutiny in 1960, I rose to the challenge of some question about Wordsworth (whom I hated at that time with a passion worthy of himself) by referring to Aldous Huxley's speculations* about the neurological or even pharmaceutical roots of Wordsworth's vision;

There was a time when meadow, grove, and stream,
The earth, and every common sight,
To me did seem
Apparell'd in celestial light,

Ode on Intimations of Immortality from Recollections of Early Childhood ll.1-4

Fifty years on, I think the marker was probably right! Chris Woodhead (a marmite figure if ever there was one!) has periodically made the point in his columns that schools are not about encouraging originality--pupils do not know enough at that age to make useful original contributions to anything; and even Diana Laurillard  argues that in science in particular, school and undergraduate study can be no more than an initiation into what is already known. (I think she concedes that an undergraduate final dissertation may perhaps occasionally have a touch of originality.) But of course by the time one embarks on a doctorate, the rubics will require "an original contribution to knowledge".

I've just caught up with two of Jim Hamlyn's recent posts; here and here.** The pedantry of the marker is mind-boggling, of course, but for me Jim's experience also poses the question of why he or she should find Jim's independent-mindedness so objectionable (even perhaps threatening) as to put in the considerable effort
required to nit-pick so obsessively. After all, this is a post-graduate course, and Jim's own discipline (art) is one which values and encourages creativity and originality and (critically) critical thinking, as this post of his demonstrates, so he can be trusted to have a handle on those values and attributes.

So where does the energy come from to set out so comprehensively to miss the point and to inhibit learning? After all, the marker could just have let things ride, and the outcome would almost certainly have been better on all fronts. I suspect that it has something to do with "the notion that the less meaningful a belief, the greater the passion and dogmatism with which it is held. This was observed by Swift (the war between the Big-Endians and the Little-Endians in Gulliver's Travels, 1726, ch. IV) so it's not exactly original, and of course as in that story, it is religious disputes which spring to mind as the most obvious examples." I'm quoting myself here, from another post on 3 March 09.

That also fits with points I made at about the same time about referencing, and the other day about the "fetishization of citation". All these "technical" issues are no more than proxies for valuable knowledge which works--and you know you are in trouble when the proxies are elevated above the substance, or you stop seeing the wood for the trees.

Returning to Graham Gibbs' keynote; one of his opening slides was a quotation (I didn't make a note from whom) to the effect that "there are just a few factors which make a difference in teaching, and we know what they are." He explicitly denied that was the case, although of course such a contention is the foundation of teaching development programmes; it is an article of faith which tutors on such programmes must believe, otherwise they would not be able to carry on "delivering" them.

I admit that I am having something of a crisis of faith. Gibbs' account of the variability of student experience and the importance of context, and even Hattie's acceptance that "everything works" (in some measure) (2009: 1) accord much more closely with my experience than practically anything in the teacher education literature. Also at the conference, Ray Land recycled to great effect a lecture from a few years ago when he stood in for a keynote speaker who got stuck in France; he took us on a tour of the changing context of education and students' lives, and concluded that current models of the university were being superceded.

When teacher educators start behaving like Jim's (and Sean's) examples, and many others, we have to conclude that the implicit deficiency model is flawed, because we do not have the authority to which we pretend. So perhaps it is time to see the lecturer as the expert in teaching in his or her context as well as a subject expert, and the role of the teacher educator as no more than a consultant to the practitioner.

In practice I think that is what I have been doing for years... Of course, it screws up conventional approaches to assessment, but that's OK.

*  Huxley A The Doors of Perception and Heaven and Hell, Harper; various editions 1954-2004
** Of course I'm well aware that I'm accepting Jim's account as legitimate, and the whole story--but I have no reason not to, and this is the start of a meditation rather than an adjudication.

On reductionism in assessment

The link (valid until 23 August only, and only in the UK) is to a Radio 4 programme on the Advanced level examination in UK schools (except for those in Scotland) taken at the age of 18. It pays particular attention to A level English literature, concentrating on a question on Hamlet.

It considers how principally how the examination game is played and how students are trained unashamedly to get the maximum marks by engaging with the "Assessment Objectives" (AOs) of the subject, to the extent of a student readily saying that they can't start discussing the "texts" in class because that might take them away from the AOs, when they only have four weeks for each of their six texts.

The AOs for English Literature are;

1. Articulate creative, informed and relevant responses to literary texts, using appropriate terminology and
   concepts, and coherent, accurate written expression.
2. Demonstrate detailed critical understanding in analysing the ways in which structure, form and language
   shape meanings in literary texts.
3. Explore connections and comparisons between different literary texts, informed by interpretations of other readers.
4. Demonstrate understanding of the significance and influence of the contexts in which literary texts are written and received.

There's nothing much wrong with them as they stand. They do constitute a sub-set of the skills a critical reader ought to have, but...

• To see a work of literature as a text, a technical product, is very partial. Babies and bathwater spring to mind.
• This approach is even more stultifying that the French explication du texte.
• But most of all it is disabling; I now have a much clearer idea of what my humanities colleagues are complaining about when they bemoan the inability of first-year undergraduates in particular to work independently, to value their own responses to a work independently of what will feature in the assessment, and even to read the whole of a novel or play or poem as a work of art. 
• They have been drilled to fake appreciation.

On manuals and text-books

The other day I indulged myself by going to one of the world's great physical bookshops to browse and buy.

Convenient as the online bookshops are, there is to my mind no substitute for physical browsing. And browsing in a bookshop and in a library are quite different experiences, which inform and feed off each other. In a library I look up the UDC/Dewey reference to find a text on a shelf and may browse round it to find associated stuff on a similar topic, but usually I am pretty clear what I am looking for, and either I find it or I don't.

The bookshop is different. Certainly it is organised--much more broadly--by subject area,  but then by author, which leads to some interesting juxtapositions on the shelves which you can't find anywhere else, other than perhaps in a particular person's library.

Almost as a penance for the browsing and buying motivated purely by my interests (and some potential presents--why else would I clean out their stock of Frankfurt H (2005) On Bullshit Princeton N J; Princeton University Press? *), I had a look on the education shelves, in particular those on teaching in post-compulsory education, to note if there were more recent editions of any of the standard works on our reading list.

There were, so I flipped through them to see where the differences were and, being in that context in the bookshop, I was instantly bored stiff. Truth to tell, of course, I have never read any of these things. I've raided them for bits of information and to give a reference to something for students, and I do read the next level of collections of papers and research-based studies and the like, but I don't read the text-books.

Thinking about this driving home, I realised anew that the text-books, in a professional/vocational discipline, may well be inimical to effective learning how to practise. The text-books have been contaminated, or perhaps more generously constrained, by the pernicious micro-managed compliance culture of post-compulsory education promulgated by LLUK and the now-defunct Learning and Skills Council (it's not often I link to something just to show that it's not there!), but that is not the whole story.

They have managed to reduce all the knowledge to something to be learned about (See here for a very quick and dirty exposition) rather than something to know by direct experience. (Sorry; that is a very sloppy usage of one of the most venerable distinctions in philosophy). It is a slab of indigestible dough sitting out there which needs to be consumed**. It is inert.

Some might recognise that "inert knowledge" like that, is another of David Perkins' categories of "troublesome knowledge" (2006).

Many of our teacher "trainees", especially those from craft and practical disciplines, experience these 400-page tomes as serious obstacles to learning how to practise. And Howard Becker, in his excellent 1972 article, which pre-figures all the "situated learning" discussions of the '90s onwards, shows how the culture and conventions of the educational system militate against the effective learning of practical disciplines.

Now Sean has identified another instance of this in the case of a textbook all about management which is a liability to his students who need to learn how to be entrepreneurs and to manage a small business.

This is not of course to deny that there is value in these books and their counterparts which fuel further investigation into policies and practices; after all, this blog in its trivial way fits within that tradition. But just as a weed is a plant in the wrong place, much of the literature is a stumbling-block to new practitioners.

As I drove home, I was asking myself why we were putting our new students through that obstacle course. There are many reasons, of course; the academic level of the course, the inspection regime, even our desire to show off our academic credentials by announcing that we know all this stuff--and a corresponding desire to put initiates through a rite of passage, whose irrelevance and futility is of course part of its potency. But none of it is about getting them to teach better.

So perhaps we should forbid them to read anything like these textbooks for the first year? (Bearing in mind that for some of the graduates, that will work as a paradoxical injunction which will guarantee they devour the reading list...) Let them gain knowledge by acquaintance first, like people do in the real world, perhaps?

Strangely, something like this is happening by accident. The current structure of teacher training in post-compulsory education in England and Wales requires that all new staff undertake a short, survival-oriented course at the start of their job (PTLLS) and then go on to a full two-year part-time programme as soon as possible. In practice, that "as soon as possible" may well be the beginning of the following academic year. So they get a year's experience under their belts (with support, theoretically) and will only then get all these books thrown at them. That won't reduce the academic step some craft colleagues have to take, but it will mean that they may be ready for this more formal approach to learning when it comes.

And the reference to manuals? There are some authors out there who have little interest in covering the standard ground, but just presenting concrete advice to newcomers. They don't write textbooks; they write manuals. Their agenda is set by what their readers want to know. At different ends of the scale, Sue Cowley and Phil Race stand out, but they are not alone. They may not go far, but they make a much better start.

On the other hand, avoid like the plague anything which is sub-titled "meeting the LLUK standards" or "achieving QTLS" (it doesn't matter what it means). Those sub-titles are testimony to the authors having lost the plot; they are writing to "cover" the inert knowledge entombed in the standards and the syllabi, as a very poor proxy for improving practice.


* Actually, I wasn't terribly impressed. The whole thing is really summed up on p.47, "the essence of bullshit is not that it is false, but that it is phony." It's a conceit, that's all.
** Sean has a very interesting blog post about bread-making, incidentally, here.

Becker H S (1972) "A School is a Lousy Place to Learning Anything in" American Behavioral Scientist vol 16: 85-105 doi:10.1177/000276427201600109
Perkins D (2006) "Constructivism and troublesome knowledge" in J H F Meyer  and R Land (eds.) Overcoming Barriers to Student Understanding: Threshold Concepts and Troublesome Knowledge London; Routledge.

On one reason not to grade...

It presupposes a degree of maturity on the part of students, but the course with which I am still associated, operates only on a pass/refer basis (one opportunity to re-submit referred work; fiture at that point is failure and requires a re-take of the whole module, at best). Many students don't like it--particularly those who are themselves graduates and have a long history of assessing themselves by their marks, often of course relative to their class-mates.

The situation Mary Beard describes in the linked post is one reason (and it is incredibly time-consuming to adopt her solution, particularly for big courses).But it goes a bit further than that.

Let's apply the Dale/Bruner “Cone of Experience” to the situation--slightly odd, but it fills the bill and such tools are never more than pragmatic devices. In terms of assessment;

• what the students actually know/can do etc. is at the enactive level. It's real life. It's incredibly rich and complex and inherently unassessable because of that, so
  
• we devise means of assessing which necessarily lose a lot of detail in order to be manageable. Sometimes the detail we actually lose is precisely what we wanted to assess in the first place, but we just get it wrong. This is at the iconic level. The students do the assessment, but in order to compare them against a standard, or even against each other, or previous performance...
  
• we have to assess the assessment, and that gets even more abstract and symbolic. And attaching a single number or letter is as abstract as one can get (apart of course from manipulating those number mathematically) --or in other words loses the most information.

Assessment is always a lossy process, but numerical marking exacerbates the situation. That magic number is a distant and distorted proxy for the blooming buzzing complexity (if not confusion) of the student's experience and conduct; which moreover obscures that enactive level. We have made grades etc. so central to a mass education system (no, I don't know the alternative either) that once one has been issued it is almost impossible to get through any more sophisticated message to the student;

"Yes, it did only get a C+, but that was solely down to those daft mistakes you made with the statistics; if you'd got them right it would probably have been an A-"

"Yeah, right..."

Such practices are actively inimical to effective feedback and assessment for learning, in the current jargon.

On norm-referenced grading

A fascinating insight into how a US professor grades forty essays, on a norm-referenced basis. (If you're not familiar with that term, have a look here, with a more opinionated piece here.) Apart from her initial remakrs about approaching the drafts like a copy-editor, I have no idea what actual criteria she is using to make her judgements.

I referred here to another columnist's approach to grading, which seems to be based on the amount of work a student submits rather than on its quality;

"...you made the deadlines -- which is what I expect from everyone -- and you handed in the assignments -- which is also what I expect. That level of work secures students a "C" because that's what we call "meeting the minimum requirements" and in no case would that snag anybody an automatic A."

I may grumble about subject benchmarks and the generally dead hand of the Quality Assurance Agency on higher education in the UK--and of course the system conspicuously fails to deliver the consistency it claims--but at least the insistence on criterion-referenced assessment does mean that students and colleagues should know more clearly where they stand than in what appears from a distance to be a highly arbitrary world across the pond.

On explaining grades

The linked piece is a column (dated yesterday but I think I've seen it before) in the Chronicle of Higher Education by Gina Barreca of the University of Connecticut. What I found most remarkable was not the piece itself but part of the first comment:

"Nobody should explain a grade to a student."

As flat and dogmatic as that. On the contrary, nobody should ever give a student a grade without explanation.

The principles of formative assessment (or "assessment for learning") suggest strongly that students can learn an enormous amount from good feedback. Indeed, Hattie concludes that the single most effective way of improving learning is to give "dollops of feedback". And all assessment can be formative. The mark or grade is simply a summary or proxy for the feedback, but it is so abstracted as to give practically no information. On a course on which I currently teach, there are no grades, just "pass" and "refer" (which caan lead to an outright "fail"). But there are dollops of feedback.


Last week one of the participants--"student" is not quite the word for well-qualified professionals in their fields who happen to be learning to teach that field--did ask a colleague for a grade.He demurred, but he could see how uncomfortable it made her. She had been through school, an undergraduate degree and a Master's getting grades all the time; a piece of work had not been properly assessed unless it got a grade, as far as she was concerned even if as in this case, it carried credits if it passed.

It's interesting that the education system is so hooked on grades; in the rest of the working world we get plenaty of feedback, but only in teaching do we get graded (by Ofsted)*.

I suspect that the sub-text of the original comment is that no student should ever question the judgement of a faculty member, however arbitrary.

*Not entirely true, but substantially so.

On an ingenious approach to marking assignments

Steve Hill of Southampton Solent University has been experimenting with Camtasia, a screen recording package which also takes audio, as a means of recording his spoken comments on students' work, which he can then send to them.

Students submit assignments electronically using our Moodle-based VLE. I then get a student’s Word file on screen and edit it, whilst simultaneously giving a live commentary on the changes that I’m making. This is like giving a student their own personal tutorial.

Sounds like an excellent idea!

On feedback

Do read the linked column—and, as important, the comments.

The author (now a professor) has kept her former professor's annotation of an essay she wrote in her second year as an undergraduate. She reproduces them and her readers comment on them. It's fascinating, and also rather disconcerting to see how many ways the notes can be interpreted, as well as to consider whether that kind of feedback would be acceptable thirty years on.

On less being more at Master's level

I have been on a validation panel today, grappling with the problem of how to specify modules at Master's level. It was very instructive.

The team developing the programme had understandably assumed that outcomes for a Master's level module would have to be more tightly specified than for a lower level course, in order to ensure that the learning and the assessment would be at greater “depth” than for an undergraduate module.

First, though, it is quite easy to specify “learning outcomes” for low-level courses. When people are learning the basics of any subject or skill, what any one person learns will have to be the same as any other learner. It's easy to assess, and “correct” knowledge or performance is clear-cut. It's not so easy at higher levels. Knowledge and “understanding” (not to mention, for those who care, the higher regions of Bloom, or Krathwohl and Anderson) may be contestable, and indeed one person's understanding or “take” on the subject may quite legitimately be different from their neightbour's. So when you get to Master's level it may be reasonable to specify that the outcomes will include so-and-so, but it may well be patronising and simply counter-productive to presume to set them out completely and exhaustively.

Even so, how do you incorporate the academic level requirements into the outcomes? It's traditional to use all those recommended “Bloom verbs” to produce “SMART” objectives. (What's the difference between an objective and an outcome, in this context? Strictly between ourselves, I no longer have to pretend that I know, and I don't care.) So the first-year students “list” or “describe”, the second-years “analyse” and the third-years “evaluate”... So Master's students? They “critically evaluate”, it seems. (That means in practice that they evaluate on the basis of one or more over-arching frameworks, showing that there are no simple answers.)

That is fair enough but it does risk becoming formulaic, and also implying that there is a correct procedure for doing it. The more specific the directions, the more restricted the outcomes, and the less the scope for the exercise of individual initiative and creativity (if that is desirable in your discipline, of course!) Master's students are experts, or at least nearly there. They need to be given their heads rather than constrained. (I am referring mainly to experienced practitioners of their discipline undertaking Master's study part-time, here; I am aware that full-time “second cycle” students who were undergraduates last year may not fit this picture.)

Personally I would rather just set out the aims of the module just so the students know what they are letting themselves in for, and recognise that the outcomes will be different every time it runs, and different for every participant. But that won't wash in the compliance climate where standardisation is all. So what can we get away with?

(Specify the level and assessment criteria at a scheme/programme level so you don't have to do it at a module level)

"On completion of this module, participants will (I prefer “participant” to “student” at this level);

• Come to their own informed conclusions about the significance of...
• Explore ... in the context of ...
• Use ... as the basis of original work on ...

Or have I got it wrong?

On the other side of plagiarism

Or "confessions of a term paper ghost-writer". Interesting in its own right as a sub-cultural phenomenon, but with a sound point at the end;

I don't have the academic credentials of composition experts, but I doubt many experts spent most of a decade writing between one and five term papers a day on virtually every subject. I know something they don't know; I know why students don't understand thesis statements, argumentative writing, or proper citations.

It's because students have never read term papers.

It's an exercise re-discovered time and again, like the wheel; get students to mark/grade and comment on specimen papers written for your module, so they know what they are supposed to look like.