CC Algebra – Fewer Strong Scores – Part III – How the Conversion Scale is Created
Set a pass score, set a mastery score, and perform a cubic regression. That’s the short hard-to-understand answer. Definitely needs explanation.
In Part I we saw how, when Common Core Algebra replaced Integrated Algebra, teachers tried to figure out how to adjust to the change. But even with adjustments, scores for top kids fell. I adjusted, as I had for exams in the past, but kids who seemed prepared for scores in the mid-80s saw scores in the mid- to high 70s instead. Similar reports came from across New York State.
In Part II we saw how kids no longer earn their scores directly. In the 1990s and before, score 85 points, and your score is 85. Since 1999 though, points and scores were untethered. The number of points was reduced to a number between 80 and 90. And for each exam administration a new “conversion chart” is created, that seemingly magically translates points into a score. Importantly, the conversion chart changes every time the exam is given. The number of points needed to passvaries each time the exam is given. The number of points needed to get an 85 varies each time the exam is given.
But how are these conversion charts created? The crucial element is something that New York State calls “performance levels” – and we have already created mystery for what should be an open process – but let’s go on. New York State numbers students as 1, 2, 3, or 4, and those numbers have names that go with them. In high school we don’t pay too much attention – those are mostly elementary and middle school numbers (I got a 2 on the ELA, but a 4 on the Math!). We think 3 means grade level.
For the first administration of an exam only, the cowardly agents of the State gather educators from around New York, and sit them in a room. They look over the questions, and guess what a level 1 kid would get right, a level 2 kid, level 3, level 4. They assign difficulty levels to each question. The State (or a contractor) examines the results, looks over the real tests, and combines them into a range of possible points that could equate to a 65 (passing) and an 85 (mastery). Then they bring those results to another room, where the cowardly agents of the State have gathered a mix of educators and administrators from around New York, and in that room they look at the percentage of kids who would fail under each scenario, and choose one.
(Cowardly? The data, as presented, leave very few choices. The process is predetermined. The educators in the room do not produce a result different from what the State produces. They are involved only so that the State has someone else to blame. Cowards.)
In any case, they walk out of the room knowing that 0 points = 0 score, X points = 65, Y points = 85, and all the points = 100.
For each subsequent test administration, a single room of educators determines X and Y (passing and mastery), consistent with the cuts that were set on previous exams.
So how do all the in-between scores get converted?
One method would be to ignore the committee work, and just use percentages. That’s what our gut tells us to do. So 72/87 = 84% (after rounding).
Another method would be to take the committee’s points, and use linear interpolation to fill in the values between. For example, in June 2013 it took 30 points to get a 65, and 68 points to earn an 85. (Ignore that 30 is 34% of 87, and 68 is 78% of 87 – we are no longer looking at percents). So 72 raw points would 4/19 of the way from 68 to 87, and would earn a score of 88.
But the method the State uses is different. They take the committee’s points, and they find a curve that fits the points. This process of curve fitting (they probably use a cubic regression) has absolutely no science, no math, no research to support it. It creates a conversion chart, so they use it. Without reason.
Anyway, here’s the June 2012 data. Notice the small differences between linear interpolation and cubic regression? They are completely meaningless and arbitrary – not really what you want to hear about when you are looking at grades going on kids’ transcripts.
Conclusion: Why did high scores drop when they shifted to Common Core? (posted here)