Assessing assessment

Bill Tomkins

For some time now people have been able to include the results of exams at grades 6, 7 and 8 in their tally of UCAS points. The amount of credit depends on both the level of pass – bare pass, merit or distinction – and the level of grade and whether it is for a theory exam or a practical one. A bare pass in grade 6 theory gives five points; distinction in a grade 8 practical gives 75.

With competition for places at leading institutions being extremely fierce and potential students keen to gain every possible advantage for their applications, it’s quite understandable that when the result of an exam is questioned there needs to be ready and rapid means of response by the exam board which conducted the assessment. This is particularly so when the exam takes place immediately prior to the confirmation of university and college places. It’s no exaggeration to say that a person’s future programme of study and subsequent career may be at stake because of slow remedying where there’s been a miscarriage of justice.

Rigour, comparability and validity

Assessment is an essential part of teaching and learning. Without reliable assessment pupils’ progress will be seriously impaired in a variety of ways. All teachers need to be taught how to assess. In any given learning situation most of the assessment should be done by the teacher and pupil(s) and it should be taking place continually. From the outset pupils should be encouraged to appraise themselves honestly. If they aren’t, then how can they use their time wisely when they practise?

Additionally, it can be very helpful for pupils to undergo some sort of occasional external assessment from parents, family members, friends, an adjudicator or examiner. Of these, assessments made by adjudicators and examiners are professional undertakings. We are right to demand high standards of competence from those who make such assessments and equally right to complain and seek redress when errors are made.

We should always demand that assessments are valid and that they measure what they purport to measure. To borrow a phrase from the world of DIY, valid assessments ‘do what it says on the tin’! A theory paper with questions written in adult language would be invalid if this were to cause a young candidate to gain fewer marks than if the language were simplified. An assessment of someone’s knowledge and understanding of content of music theory should differ significantly from an assessment of someone’s knowledge of English. It can be all too easy to overlook some aspect of assessment which is invalid, and especially so when working a system which is regarded as tried and tested. Unfortunately, the fact that something is the established way of doing things is no guarantee that it’s valid.

During the last 25 years there has been much effort expended to try to ensure rigour, comparability and validity in many public exam systems including GCSE, A level, graded exams, diplomas and degrees. The sheer fact that there continue to be complaints about unjust results shows that there is still some way to go.

Criteria

One of the most significant developments has been the introduction of criteria for the award of marks in particular categories. Search the websites of the various exam boards and you will quickly find which ones have put these criteria into the public domain, so that we can all know detail such as what it takes to obtain a pass mark for playing scales on the clarinet at grade 7, what more it takes to obtain a merit and what further is demanded for a distinction.

Criteria for repertoire may be subdivided into various headings such as notational accuracy, technical fluency and sense of performance. There may well be marks allocated to such headings, thereby allowing for even greater precision on the part of the examiner and even clearer understanding on the part of those reading the results as to how they were arrived at.

With the use of such explicit criteria there should be absolute comparability. A grade 3 pass awarded to a trumpeter in Hong Kong by examiner X should reward equivalent knowledge and skill to that of a recorder player in Huddersfield who is awarded a pass by examiner Q. If the examiners are working as generalists then the same outcomes should result if X were in Huddersfield and Q in Hong Kong. The same results should be obtained whatever time of day the assessment takes place, no matter which day of the week and no matter how many other candidates have been awarded the same result during the examination session. An organisation whose procedures embody rigour is almost bound to conduct assessments which are comparable. That said, considerable effort is needed to maintain a level of comparability which satisfies discerning teachers, parents and candidates.

If criteria are divided as outlined above and if each aspect has a stated proportion of the total marks available, there is little room for disagreement about the outcome. However, if assessments are made according to nothing more than what may be called a general impression based on an examiner’s lifetime’s experience, there is much greater room for disagreement, leading to much greater likelihood of injustice and complaint.

Examiners

Of course, examiners are just as fallible as the rest of us and there are bound to be occasions when something goes awry. One possibility is a mismatch of comment and mark such as, ‘This was a well controlled performance with clear tone, shapely phrases, neat articulation and at a suitable tempo’, coupled with a bare pass mark. The description suggests that the performance had considerable merit, though was not outstanding. The mark should match this. Ideally the written comments should guide those reading the report to understand exactly why the mark awarded is as it is, neither higher nor lower. Where there are published criteria those which apply can be inferred from the mark and the examiner’s written comment adds suitable glosses which are responses to the individual performance.

Examiners are obliged to assess what they hear, not what the candidate might hope they would hear. There are times when a candidate is unable to give of their best, due, perhaps to illness or injury. This is unfortunate but no allowance can be made for what might have happened under different circumstances. A professional performer who is unable to do the job properly will withdraw, even at short notice. It might be better for an exam candidate to do the same.

Recording Exams

The pros and cons of recording exams have been discussed over many years and there are interesting and important points to consider. First, why not go the whole hog, record all exams and have a panel of examiners who listen, discuss, agree and award the marks? Yes, why not? Speaking personally, I think that assessing on the spot is more realistic and less daunting. The campaign a few years ago to Keep Music Live! wasn’t about keeping concert halls open but about promoting live performances of all sorts. Graded exams are a special case of live performances. Special because usually the audience numbers only one person who has been paid to listen and appraise.

What about having back-up recordings which can be used to resolve disputes over the results? Yes, again, why not? In fact recordings are used in a variety of ways including gathering material to use for training and standardising examiners. But let’s consider for a moment. A result is contested on the grounds that the examiner has been excessively harsh by awarding a below pass mark. The report form has no discrepancies between comments and marks; there is no error in the arithmetic. The examiner is experienced and her records show that her marking is consistently in line, not only with the board’s overall standards but also with the year on year patterns at the various centres where she has worked over the last five years. A recording was made of the exam and when listened to by another examiner reveals that the candidate struggled with both technical work and repertoire. The original examiner’s decision may be upheld but will whoever appealed against the result necessarily be satisfied? By rejecting the appeal might not the board be seen to be covering up for someone?

But suppose whoever listens to the recording thinks the original examiner reached the wrong decision, are we to accept that a result based on the recording should override the live experience? If so, then why bother with live exams? Also, what of the comparability between panel members already discussed?

I have no dislike of recordings per se, but I do think we have to be very clear about the use to which they are put. The circumstances in which graded exams take place are very tightly regulated by the various boards and members of their examining panels are given thorough training and regular re-standardisation. In themselves these don’t guarantee absolute comparability and consistency; they are unachievable! However, just as every candidate is assumed to be about to pass the exam when he walks into the examination room, so every examiner has to be assumed to be intent on awarding a just and accurate result.

Consider the difference between, on the one hand, me sitting alone in my house in December writing this article followed by you sitting in your house reading it in February and, on the other, us sat together somewhere in conversation over the same subject matter. Both are acceptable ways of communication but they differ significantly in many ways. Something similar applies when comparing live and recorded assessments.

It may be helpful on occasions for two or more people to collaborate in the assessment process but I believe they can do so validly only by being present during the performance. A recording is a different matter from a live performance: neither can substitute for the other. If the requirements of rigour demand that more than one person make the assessment of a live performance – maybe in a diploma or degree examination– then I believe they must be together during that performance and make their shared decisions on the spot.

No particular aspect of music assessment can be treated exhaustively within a piece such as this.  I hope it can promote further discussion and look forward to a lively correspondence in the coming months.

Bill Tomkins has been involved in examinations since 1973: CSE, O and A level, GCSE, the Open University and, since 1993, on the panel of Trinity College London. He examines for graded and diploma practical exams and oversees one of the written diplomas. He has been a member of the ISM since 1966.

Add your comment

Your email address will not be published.

Add your comment

Name

Email

Location

Comment

Remember my personal information Notify me of follow-up comments?

Please enter the word you see in the image below:

Captcha