Low proficiency rates, strange “bowl” curves, compact cut scores, huge TVAAS assumptions, and a disconnect from ACT outcomes among biggest problems
Teachers believe in assessments, we were the ones who invented them.
Yet TEA and others have growing concerns about TNReady as the testing data is provided to schools, teachers and parents from last year.
In a House Government Operations committee hearing December 15, Chairman Rep. Jeremy Faison had Education Commissioner Candice McQueen and her staff, along with a testing vendor official, testify on issues of TNReady. Prior to McQueen’s testimony, TEA lobbyist Jim Wrye made a presentation on the problems the association sees.
“TNReady issues go well beyond the mis-scored tests, glitches, and lateness in data. There are fundamental problems in the test data, and what it says about our students, teachers, and schools,” said Wrye. “The more we look into it, the bigger the problems become.”
Faison was blunt with the commissioner.
“We’ve put an immense amount of pressure on my educators, and when I share with you what I think you’d get a ‘one’ on, I’m speaking…from what I’m hearing from 99.9 percent of my educators, my principals and my school superintendents,” Faison said.
Government Operations does not normally look at education issues, but when problems surfaced for TNReady last spring, House Speaker Beth Harwell requested the oversight committee question the department and the testing vendor. Faison and Harwell have asked TEA about the testing system, and the chair provided an opportunity to present information the department doesn’t share.
TEA sees three very big problems with TNReady: the validity of TVAAS data used in evaluations; the stark contrast between high achievement in ACT and graduation rates, and the very low proficiency scores the state reported for all Tennessee students; and the lack of transparency in the new testing system.
Wrye explained TVAAS is a statistical estimate that uses standard deviations to determine scores between one through five. There are low confidence intervals at each point in the scale, and the scores are not fixed points but are recalculated in later years as more data is added to the statistical model.
“Imagine I’m a pollster and you are a candidate,” Wrye told lawmakers. “I tell you my poll says you are up 10 points, but then I tell you ‘plus or minus 32 percent.’ That is what most TVAAS scores mean.”
Wrye then noted the statistical model is supposedly based on three years of test data as a means to reduce volatility, and showed a long line of apples of past TCAP tests. Then there was an orange for TNReady.
“The tests are so radically different, it really is apples and oranges. How well a model can take in all the variables of students adapting to this new test is questionable,” Wrye said.
The problems are compounded by the failure of TNReady in 2015-16. For most teachers and students, there is not three years of data to generate an estimate. “Apple, then nothing, then orange is a real problem. The model doesn’t know what student growth was in 2016, but they spit out TVAAS numbers just the same.”
Wrye then used Williamson County to show the disconnect between TNReady and actual student outcomes, indicating real problems in the structure of the test. In 2017, Williamson County had 83 percent of all students go on to college, with an average 25.3 ACT score and 75 percent qualifying for the Hope Scholarship.
TNReady indicated that 54 percent of all Williamson students were below grade level in Math. Wrye countered that 80 percent of all system students scored a 22 or better in the Math portion of the ACT, the benchmark for college readiness.
“You can’t square the radically different outcomes, either the ACT or TNReady is wrong,” Wrye said, noting the ACT has 80 years of experience measuring college readiness, and TNReady has two.
And if TNReady is wrong about Williamson County, then it is wrong about every system in the state.
McQueen and her panel walked through the misscored tests and how TNReady now correlates with NAEP, the national test used to rank states. Her panel did not go into detail on issues with cut scores and other data problems outlined by the TEA presentation.
Rep. Antonio Parkinson was unimpressed. “What you’re telling me is that I run a race, and then you’ll decide after what a winning time is. That makes no sense to me.”
Faison summed up the testing situation in Tennessee and next steps.
“What we’re doing is driving the teachers crazy. They’re scared to death to teach anything other than get prepared for this test. They’re not even enjoying life right now. They’re not even enjoying teaching because we’ve put so much emphasis on this evaluation,” said Faison. “So I think you’re going to see movement in the legislature this year to detach the evaluation portion of the TNReady test from the teachers and the students.”