Dan at dy/dan challenged a few of us:

Choose any of your sections. Now, before you hand out your next objective assessment, write down a quick prediction of every student’s exam grade. Grade them and compare the results.

I did it. Three classes. On the technical side, Dan was working with A, B, C, etc. My predictions were numbers: 74, 82, 97, etc. I arbitrarily decided that within 9 points is correct, 10 to 19 points is one grade off, etc.

My results and discussion come below the fold —>

• Class 1, 10 correct, 1 underestimated by 1 grade, 6 over by 1 grade, 1 over by 2 grades.
• Class 2, 25 correct, 1 under by 2 grades, 1 under by 1 grade, 3 over by 1 grade, 2 over by 2 grades.
• Class 3, 12 correct, 2 over by 1 grade, 2 off the chart.

Totals: 47 of 66 were correct, I underestimated 3 (1 by 2 grades), I overestimated 16 (11 by 1 grade, 3 by 2 grades, and 2 far far away from disasters)

Some observations: The absolute (unsigned) errors by class were 9/student, 7/student, and 11/student (skewed by the two disasters. Removing them, 7/student). I was far more likely to overestimate than to underestimate the individual scores.

I disliked this game, and won’t play again. I am a fair grader, (imo). I take off points where they should come off. I even take off a few picky points that I swore when I started I never would (but need to break sloppy habits…) Since I am a stickler for accuracy, my hoping that the kids do well does not interfere with assessing them fairly when they don’t. So, me grading? I talk to the papers, yell at them for making silly mistakes, and root for them and encourage them. Sometimes I reduce these to written notes, sometimes with specific instructions on avoiding errors. More often I simply autoarticulate, looking to anyone who sees me like some crazy guy yelling at a stack of papers. But craziness aside, I root for the kids as I grade their papers, and this game tempered my rooting. I won’t play again.

1. April 8, 2007 am30 12:44 am 12:44 am

The term “game” (which, in fairness, was my description) diminishes the seriousness of the exercise. Everyone who “played” was far more inclined to overestimate his students’ comprehension than to under-. That fact makes me queasy, which is why I don’t “play” more than once a year.

2. April 9, 2007 am30 1:19 am 1:19 am

I still can’t help but wonder what the purpose of this exercise was. What does your guessing correctly, or overestimating or underestimating tell you?

3. April 9, 2007 am30 6:24 am 6:24 am

The overestimating tells me that I was (to some extent) refusing to play. I am, as I indicated, quite uncomfortable with this. Perhaps Dan could tell you more.

4. April 9, 2007 pm30 10:38 pm 10:38 pm

I think the point of playing the game is to test yourself on how well you know your students. If you overestimate, does that mean that you are assuming that they know knowledge that they don’t? If you underestimate, does that say that you could have challenged them a bit more or did the exam test easier concepts and ignore the more difficult ones? Hopefully, this reflection will help you better prepare your lessons and assessments next time.

You could look at it from a single student angle as well. If a student did worse than what you expected her to do, you could certainly ask yourself why you assumed she would do better.

5. April 10, 2007 pm30 7:12 pm 7:12 pm

So if you underestimate then you wrote too easy an exam. But if you overestimate then you assume they know things that they don’t. Maybe you wrote too hard of an exam? Maybe you tested for things that you shouldn’t have. Maybe the exam was too long and people didn’t have enough time to think. Maybe…. Maybe this game is meaningless.

6. April 11, 2007 am30 2:43 am 2:43 am

e, that’s just pointless provocation. Try to rise above it.

What this exercise presupposes is that the exam is fair, a requirement that isn’t too hard to satisfy. There’s a huge patch of area between too hard and too easy. Once there, the teacher predicts what each student will score.

Nothing about that task is inherently unpleasant or meaningless. This exercise is a lucid assessment of how well I assess. Doesn’t matter if I use this format or if I just ask myself regularly, “What does [student x] really know about [concept x]?” I’m sure we’d all agree that assessment is elemental but I’m wagering the number of teachers who assess their own assessment in any meaningful way is pretty low.

7. April 11, 2007 am30 8:38 am 8:38 am

Is that what you were after?

Assess my assessments (tests)? 1. did the students have adequate time. 2. When I read the test, do the students have opportunity to demonstrate knowledge of the topic(s). 3. Is there an adequate range of questions (iow, are grades based too heavily on a single skill or group of skills). 4. Is there sufficient opportunity to demonstrate basic, procedural knowledge/skill. 5. Is there some opportunity (perhaps bonus) to solve somthing related, but that looks different from what we’ve seen before. 6. Do the grades match, roughly, classroom perfomance. 7. Is there a single question that many students had trouble with. Is there a group of questions that no one had problems with (these may reflect on instruction more than on the assessment. 8. If I look at a grade of 80, does that really match my professional judgement for a kid being ready to move on, being ready for the next unit.

I think that any teacher who writes his own tests stops to look at them in at least some of the ways I’ve outlined above.

8. April 12, 2007 am30 1:11 am 1:11 am

Dan said :
e, that’s just pointless provocation.
I am sorry this is your perception. I still believe that there are too many variables involved in this game for one to make any meaningful conclusions.
Doesn’t matter if I use this format or if I just ask myself regularly, “What does [student x] really know about [concept x]?”
But isn’t this exactly what that assessment is supposed to tell you? Guessing (in)correctly student’s performance may tell you something about assessments you had administered earlier and on which you based your predictions, although I am arguing that even that is very unlikely.