Jefferson Review

"Your Liberty is Our Interest"

Home /  Archives / Links / Quotes / Book Reviews / Advertise /Contact us / Subscribe / Calendar

CATS in Decline: Federal Yardstick Reveals Kentucky’s Education Testing Program Continues to Deteriorate

Executive summary

Ever since the implementation of Kentucky’s Commonwealth Accountability Testing System, known as the CATS, researchers have wondered if students were really doing better, or if the grading standards were simply getting easier.

Certainly, the new CATS results for 2007 show serious signs of inflation. There are extremely large differences in 2007 proficiency rates reported by the CATS’ individual academic subjects compared to proficiency rates reported by the National Assessment of Educational Progress – a highly regarded federal testing program. ¹

However, the Kentucky Department of Education publicly admits the 2007 version of CATS has so many changes that it must be considered “new.” Could the current differences all be the result of those 2007 changes? That raises the question: Were CATS scores artificially inflated even before 2007?

To answer that question, we developed a new technique to measure the rigor in the “old” CATS using a “yardstick” based on the national assessment. We found that virtually since CATS was implemented, standards have continuously declined. As a consequence, since its inception in 1999 the CATS program has offered an inflated evaluation of educational progress in Kentucky.

The consequences of such distortions are significant. There is now considerable misunderstanding throughout the commonwealth about the real rate of progress and current condition of our education system. Due to bias in the CATS scores that increasingly has favored elementary schools, scarce resources have probably been misdirected.

As an unreliable gauge of progress, the CATS assessment needs to be replaced by more credible tests. In hindsight, it was a mistake to charge the Kentucky Department of Education to both assist school systems in making improvements and be the sole administrator of the assessment system to determine if that effort was successful. A separate agency should be created to manage the assessment of educational progress in Kentucky’s schools.

Introduction

Since Kentucky implemented its Commonwealth Accountability Testing System, known as the CATS, controversy has existed about whether the results reveal true academic improvement or simply reflect lower grading standards.

Certainly, the CATS results for 2007 currently show serious signs of inflation compared to the federal assessment program. For example, the 2007 CATS reports 72.7 percent of our elementary school students read at or above the proficient level. However, the federal National Assessment of Educational Progress (NAEP) shows only 33 percent of Kentucky’s elementary kids are proficient readers.

But, what about earlier CATS scores up to 2006? Were they also constantly inflating? Using a new “yardstick” based on the national assessment, we developed an easy-to-understand measure that shows the answer is definitely “Yes!” Comparing pre-2007 CATS scores reported by state education officials with the relatively flat performance portrayed by the national assessment confirms the skepticism held by objective researchers about the rigor of CATS.

Measuring scoring stagnation, inflation

An example in Figure 1 will be used to explain a “yardstick” based on the federal assessment that can measure the rigor in state assessments. This tool will help reveal the decline in the rigor of CATS over time.

The green arrow shows that only 23 percent of Kentucky’s students scored “Proficient” or better on the 2005 national assessment in eighth-grade math.^{2 3} In contrast, when plotted on the same scale, the blue arrow shows that the state’s CATS assessment – known as the Kentucky Core Content Test (KCCT) – reported a much higher proficiency rate of 36 percent.

In fact, Figure 1 shows the percentage of proficiency reported for the 2005 CATS lies about a third of the way between the percentage of students that the federal test calls “Proficient” and the percentage of students who scored at the national assessment’s next-lowest scoring level “At or Above Basic” (indicated by the yellow arrow).

Clearly, because the federal standards have not changed over time, the location of the CATS proficiency rate arrow between the national assessment’s scoring arrows indicating “Proficient or Better” and “Basic or Better” offers a useful and accurate measurement of the rigor of the state CATS assessment. If the CATS proficiency arrow plots closer to the national assessment “At or Above Basic” rate, it shows Kentucky’s standards are weak. If the CATS arrow plots closer to the national assessment’s “At or Above Proficient” arrow, it indicates Kentucky’s standards approach those of the high, but carefully established national assessment norms.

To simplify our measurement, we converted the information contained in the federal yardstick to a percentage model (shown by the “NAEP Ruler” scale on the right side of Figure 1). As the “ruler” on the right side of Figure 1 (which runs from zero – using the “At or Above Basic” line as the beginning reference point – to 100) shows, the state’s proficiency standard appears only about 68 percent as difficult as the national test’s standard. We call the numbers measured by our ruler the “State-to-NAEP Rigor Ratio.”

State-to-NAEP Rigor Ratio exposes CATS inflation

The State-to-NAEP Rigor Ratio enables us to measure the academic strength of Kentucky’s proficiency standards and to track changes over time. The results are shown in Figure 2.

As with the model in Figure 1, scores close to zero indicate that what Kentucky calls “Proficient” work really is what the national assessment considers only a “Basic” performance. Proficiency rates close to 100 indicate that the state tracks more closely to the national assessment. Numbers greater than 100 indicate the state’s standards are more rigorous than the national assessment. Conversely, ratios less than zero indicate the state employs severely watered-down standards.

Figure 2 records the results of our analysis.

We include some results from Kentucky’s older assessment program to show that even from its outset in 1999, CATS was inflating results. It should be noted that pre-1999 federal testing didn’t allow special accommodations for students with learning disabilities; however, the impact of that policy on the comparisons in Figure 2 amounts to a few points, at best.

The ratios in Figure 2 offer clues that the oft-stated claims of substantial improvements over time among Kentucky students may in fact come primarily from constantly inflating scores. In every case in Figure 2, the State-to-NAEP Rigor Ratio declined since CATS made its debut in 1999.

Figure 2 indicates the worst decline in Kentucky’s standards occurred in the CATS elementary school reading assessment, ironically coinciding with a time period when some claim notable improvements in reading took place.

In fact, according to our State-to-NAEP Rigor Ratio, the commonwealth’s elementary school reading proficiency standard actually was significantly more demanding than the national assessment before CATS began. Kentucky’s standards have rapidly deteriorated since CATS was implemented. What the state now calls “Proficient” elementary school reading actually represents less of an accomplishment than even the national assessment’s “Basic” level of performance, which denotes only partial mastery of material. Evidence that the CATS severely misrepresents the performance of Kentucky’s elementary school readers – and that the misrepresentation has gotten much worse over time – cannot be much clearer.

Deterioration of CATS standards is not confined to reading. Figure 2 highlights that in every case shown – regardless of school level or subject – the CATS’ standards for proficiency declined over time when measured by the national assessment yardstick. Furthermore, this deterioration of standards began well before this year’s highly inflationary resetting of CATS standards made things even worse.

But Figure 2 shows still more. Comparing subject graphs horizontally across Figure 2 reveals that elementary school grading in CATS is easier than in middle schools; the largest discrepancy occurs in reading. The national-assessment yardstick shows that grading of CATS middle school reading in 2007 was 28 points more difficult than in elementary schools. This undermines often-heard claims based on the CATS that Kentucky’s education reform is working best in our elementary schools. The federal test shows there are scant differences in the true performance at these two school levels.⁴

Conclusion

While inflation of CATS proficiency rates markedly worsened in 2007, using the NAEP as a yardstick indicates Kentucky’s state test scores have been inflating on a continual basis ever since CATS replaced the old KIRIS assessment in 1999.

Everyone from policymakers to educators, parents, students and the general public has been misled about the real performance of Kentucky’s education system. As a result, few recognize the excessively slow rate of education progress in Kentucky and the inadequacy of the effort to achieve meaningful and effective education reform.

The results of the State-to-NAEP Rigor Ratio make it clear: It’s time to make major changes to the state’s assessment.

• CATS has become an inaccurate and bloated measurement. It needs to be replaced by tests of more widely accepted relevance and accuracy with scoring scales that are not controlled by parties with vested interests.

• Clearly, officials held accountable for the state’s education system should not also manage the system’s performance measurement. History shows the temptation to reduce standards becomes too great. Instead, an independent agency should take control of the state’s education assessment while the Kentucky Department of Education retains its sole and primary duty to improve education. Separating those functions should yield a more reliable yardstick to measure the progress of education in Kentucky and offer students the rigorous academic program they need and deserve.

Richard G. Innes is an education analyst with the Bluegrass Institute for Public Policy Solutions, Kentucky’s free-market think tank.

Endnotes

¹ A forthcoming Bluegrass Institute report examines inflation in the 2007 CATS in detail.

² See any recent NAEP report card for a full description of the achievement level scoring system which forms the basis for our NAEP yardstick, online at: http://nces.ed.gov/nationsreportcard/. NAEP scoring data used in this report was assembled in October 2007 from the NAEP Data Explorer, online at: http://nces.ed.gov/nationsreportcard/nde/ .

³ CATS KCCT proficiency rates used to assemble this report were calculated by the Bluegrass Institute from reported percentages of students in the “Proficient” and “Distinguished” scoring levels in various Briefing Packet, State And Regional Release, Commonwealth Accountability Testing System (CATS), from the Kentucky Department of Education, online at: http://www.education.ky.gov/KDE/Administrative+Resources/Testing+and+Reporting+/Reports/CATS+Briefing+Packets/. Older KIRIS data came from hardcopy in Table 4, Kentucky School and District Accountability Results, Accountability Cycle 3 (1994-95 to 1997-98), State and Regional Scores by Subject, Kentucky Department of Education, Frankfort, KY, 1998.

⁴ The National Assessment of Educational Progress is a sampled test, so the published proficiency rates include sampling error, reported as “Standard Error.” We looked at the impact of those errors on our 2005 NAEP eighth-grade math “yardstick” example. A very conservative estimate shows that the ratio should not vary more than plus or minus 8 points. Similar margins of error also will apply to the other ratios. The effect is too small to significantly impact the findings in Figure 2. NAEP also cautions that exclusion and accommodation rates on the test must be considered in doing year-to-year analysis. The combined rates of exclusion plus accommodation have generally increased over time on Kentucky’s NAEP assessments. The impact would actually make the situation even worse than that shown in Figure 2.
 

Download PDF

 

The Bluegrass Institute is an independent research and educational institution offering free-market solutions to Kentucky's most pressing problems.

Weather (Louisville) / Mapquest /  White Pages / Business Search / CNN / Dictionary / E-card / MSN