CC Algebra – Fewer Strong Scores – Part III – How the Conversion Scale is Created
Set a pass score, set a mastery score, and perform a cubic regression. That’s the short hard-to-understand answer. Definitely needs explanation.
In Part I we saw how, when Common Core Algebra replaced Integrated Algebra, teachers tried to figure out how to adjust to the change. But even with adjustments, scores for top kids fell. I adjusted, as I had for exams in the past, but kids who seemed prepared for scores in the mid-80s saw scores in the mid- to high 70s instead. Similar reports came from across New York State.
In Part II we saw how kids no longer earn their scores directly. In the 1990s and before, score 85 points, and your score is 85. Since 1999 though, points and scores were untethered. The number of points was reduced to a number between 80 and 90. And for each exam administration a new “conversion chart” is created, that seemingly magically translates points into a score. Importantly, the conversion chart changes every time the exam is given. The number of points needed to passvaries each time the exam is given. The number of points needed to get an 85 varies each time the exam is given.
But how are these conversion charts created? The crucial element is something that New York State calls “performance levels” – and we have already created mystery for what should be an open process – but let’s go on. New York State numbers students as 1, 2, 3, or 4, and those numbers have names that go with them. In high school we don’t pay too much attention – those are mostly elementary and middle school numbers (I got a 2 on the ELA, but a 4 on the Math!). We think 3 means grade level.
For the first administration of an exam only, the cowardly agents of the State gather educators from around New York, and sit them in a room. They look over the questions, and guess what a level 1 kid would get right, a level 2 kid, level 3, level 4. They assign difficulty levels to each question. The State (or a contractor) examines the results, looks over the real tests, and combines them into a range of possible points that could equate to a 65 (passing) and an 85 (mastery). Then they bring those results to another room, where the cowardly agents of the State have gathered a mix of educators and administrators from around New York, and in that room they look at the percentage of kids who would fail under each scenario, and choose one.
(Cowardly? The data, as presented, leave very few choices. The process is predetermined. The educators in the room do not produce a result different from what the State produces. They are involved only so that the State has someone else to blame. Cowards.)
In any case, they walk out of the room knowing that 0 points = 0 score, X points = 65, Y points = 85, and all the points = 100.
For each subsequent test administration, a single room of educators determines X and Y (passing and mastery), consistent with the cuts that were set on previous exams.
So how do all the in-between scores get converted?
One method would be to ignore the committee work, and just use percentages. That’s what our gut tells us to do. So 72/87 = 84% (after rounding).
Another method would be to take the committee’s points, and use linear interpolation to fill in the values between. For example, in June 2013 it took 30 points to get a 65, and 68 points to earn an 85. (Ignore that 30 is 34% of 87, and 68 is 78% of 87 – we are no longer looking at percents). So 72 raw points would 4/19 of the way from 68 to 87, and would earn a score of 88.
But the method the State uses is different. They take the committee’s points, and they find a curve that fits the points. This process of curve fitting (they probably use a cubic regression) has absolutely no science, no math, no research to support it. It creates a conversion chart, so they use it. Without reason.
My students have studied this, and are appalled. I use it to do some statistics, and later to talk about abuse of mathematics.
Anyway, here’s the June 2012 data. Notice the small differences between linear interpolation and cubic regression? They are completely meaningless and arbitrary – not really what you want to hear about when you are looking at grades going on kids’ transcripts.
Conclusion: Why did high scores drop when they shifted to Common Core? (posted here)
JD2718 
To start I have not looked at nor have a graded a Common Core Math exam at any grade level. I have graded NYS Algebra, Geometry, Algebra II/Trig, IM Course I, II, III and Math A and Math B exams. The early exams, i.e., pre-Math A and Math B have always been looked at as fairly straight forward. 30 Part I questions two points each and 4 Part II/Part III questions worth 10 points each. Total 100 points. Earn 84 points get an 84. A 65 was a 65%.
However, that’s not really what was going on.
There were 35 Part I questions from which students had to chose 30. Most students would skip items that they didn’t know and not be penalized. In addition, there were 7 Part II questions from which students could select 4 (Some of the exams had a Part II and Part III, but there were still choices.
So to pass an exam a student needed 65 out of a possible 140 points (35 x 2 + 7 x 10 = 140) a score of 46%.
Further, the grading of the exams changed too. A 10 point Part II question would penalize a student 1 point, (10%) for an arithmetic error. the same error on a 2 point question would be 1 point or 50% of the problem’s worth. On a four point question it would be 1 out of 4 or 25%. So with no choice to eliminate questions and small errors costing a larger percentage of available points passing scores dropped into the 30s.
Now I am not making excuses for the passing cut score nor the cut score for mastery. I simply wanted to point out that a direct comparison is not as straight forward as we might like. But before 1999 a 65% was never really a 65%. And with Math A and B the penalties were greater.
I would further note that when I was in the classroom I didn’t take away points. The whole notion that students start with a perfect score and that they lose points for mistakes is really counter intuitive to what is really going on. I gave my students points for things that they did right which when you really think about it that’s what we should be doing. The only exception to my grading procedure was on the NYS Regent’s exams.
All valid points, thank you.
I rambled into the old exams, while I was trying to discuss the new ones. I should have restricted myself to mentioning Course I, II, and III only for the idea of “untethering” points from scores. Which I still find very weird.
The real comparison I am making through this series (it comes in the next and last installment) is a dramatic, unannounced or barely announced change in scoring from Integrated Algebra to Common Core Algebra.
Across the state, kids who “felt like” 85-90 to their teachers scored 75-80. And it’s not by accident.
I have chatted with at least one person who has knowledge about how the conversion charts are created. From what I understand, the benchmarks for performance level (1 to 5) are set deliberately and then the scores are interpolated from those points. Why those values I am not sure, especially considering that schools, teachers, and students use the reported score as the standards.
If you look at the Algebra conversion curve, it definitely has changed with the Common Core flavor of exams. The negative curve at the top is just mean. Can you imagine if teachers implemented such a curve for in-class exams? What baffles me is why the conversion curve dropped from June 2014 to June 2015, but only at the upper level. So what was considered an 85 last year is now an 83. No reason is given as to why that happened. And, to me, the drop and the lack of transparency about it is a concern.
It’s not interpolation, but (until common core) cubic regression. Which is of course, silly.
They intentionally changed the shape of curve with common core (see the next and last post in the series)
By the way, we met, not sure if you remember, a few years ago, at Estelle’s retirement at Tosca…
Jonathan