Jefferson Review

"Your Liberty is Our Interest"

Home /  Archives / Links / Quotes / Book Reviews / Advertise /Contact us / Subscribe / Calendar

Planning for failure:

Is the road to 2014 leading toward proficiency for all Kentucky students?

By Richard G. Innes

Executive Summary

Since the early years of the Kentucky Education Reform Act of 1990 (KERA), education leaders have steadfastly proclaimed all of its students would attain academic proficiency by year end 2014. Are they on course to meet this No Child Left Behind Act (NCLB) standard? Or, will Kentucky’s parents and students be forced to settle for less?

This report reveals that KERA’s assessment program, the Commonwealth Accountability Testing System (CATS), will fall far short of proficiency for too many students. With CATS as the standard, many students will fail to reach proficiency while the poorly performing schools they attend could be rewarded.

Certain schools (see Table 1) appear likely to score high enough to achieve immunity from all CATS sanctions in 2014 despite having unacceptable proficiency rates in certain subjects. In each of these schools, the rate of proficiency averaged across all the CATS-tested subjects is less than 50 percent. Still, Kentucky’s testing policies will not sanction any of these schools in 2014 as long as they maintain their anemic academic performance with absolutely no improvement at all.

Kentucky parents should not stand for maintaining the status quo concerning the lackluster performance of their children’s schools. Doing so fails to fulfill the intentions of NCLB and KERA, the high standards of which were established on the premise that all Kentucky children can learn at the highest level.

Sadly, the current CATS program guarantees that Kentucky’s education odyssey will end in mediocrity in 2014. Fortunately, changes could be made to improve the proficiency of Kentucky’s students. This report concludes by offering suggestions for change with the hope that Kentucky legislators and education officials will fortify the proficiency standards that ensure a quality education for all students in the commonwealth.

Introduction

Kentucky’s education leaders claim they support the goal of President Bush’s No Child Left Behind (NCLB) law requiring all students to become proficient in key academic areas by 2014. However, Kentucky’s testing policies indicate the state’s education bureaucracy intends to settle for far less.

In fact, the current situation will leave many students behind in 2014 with below-proficient performances while the state’s assessment apparatus – known as the Commonwealth Accountability Testing System (CATS) – not only fails to sanction poorly performing schools, but could actually reward them!

This clearly is not what Kentuckians want or what the legislature intended when it implemented the Kentucky Education Reform Act (KERA) in 1990. However, absent significant changes, Kentucky’s CATS program only guarantees mediocrity in 2014 by requiring little improvement beyond current performance in most of the state’s public schools.

The School Accountability Index (SAI)

Understanding the problem requires knowing how the CATS program rates the performance of Kentucky’s schools. To do that, this report relies extensively on the Kentucky Department of Education’s (KDE) “2002 Kentucky Core Content Test (KCCT) Technical Report¹, which we will refer to as the “Technical Report.”

Each year, the CATS creates a single number for each school known as the School Accountability Index (SAI). The SAI consists of an average of scores for a number of academic and non-academic elements.

Academic measures include the KCCT assessments of reading, mathematics, science, social studies, writing (including portfolios developed throughout the year and on-demand samples administered under test conditions) – plus arts and humanities and practical living/vocational studies. There also is a nationally normed achievement test for reading, language arts and mathematics².

A small portion of the SAI includes scores from non-academic elements, such as “transition to adult life” (the success rate of high-school graduates in going on to college or into the workforce) along with attendance, retention (the proportion of students not promoted at the end of the school year) and dropout rates (for middle and high schools).

Using a weighting system, CATS score components from each of these individual elements are averaged together to determine a school’s annual SAI.

Grading the school

Since these annual SAIs tend to fluctuate substantially from year to year, the KDE creates a “Biennium SAI” by averaging each school’s SAI scores for two years. Once a school’s Biennium SAI is determined, it is plotted on a grading graph, called a CATS Growth Chart to determine that school’s CATS accountability classification, which will be either “Meeting Goal,” Progressing” or “Assistance.”

Figure 1 shows a typical CATS Growth Chart. The left vertical axis indicates SAI scores and the horizontal axis shows the final year for each SAI biennium. The numbers on the right vertical axis show score levels required for earning special rewards³.

There are two important lines on the growth chart in Figure 1, which contains the CATS growth chart for Oran P. Lawler Elementary School in Grayson County. The first is the “Goal Line,” which separates the green and yellow regions. The beginning point of this line on the left is fixed at a number equal to the school’s baseline SAI from the year 2000 minus a small amount that compensates for the measurement error in test scores. Such compensation is a normal procedure found in credible testing programs.

Lawler’s baseline score in 2000 is 72.1 points which, when adjusted for a testing measurement error of 1.1 points, fixes the school’s 2000 SAI Goal Line at 71.0 points. The measurement error compensation of 1.1 points also explains why Lawler’s Goal Line ends at 98.9 in 2014, which is 1.1 points less than the officially proclaimed CATS accountability index goal of 100.

The growth chart’s other important line is the “Assistance Line,” which begins in the same place as the Lawler’s Goal Line – at the baseline score of 71.0 points – but increases only to a projected score of 78.9 (80.0 points minus the 1.1 error compensation in our Lawler example) in 2014.

The Technical Report provides absolutely no explanation why the Assistance Line terminates at a score near 80. While this resulting 20-point “gift” will be further examined later in this report, suffice it to say that such a variance has profound impacts on the end goal of KERA for the year 2014.

The two small dots on the Goal Line indicate Lawler’s Biennium SAI scores for 2002 and 2004. Lawler’s SAI was 73.9 in 2002, placing the school in the “Progressing” category while the score of 80.0 in 2004 plots in the “Meeting Goal” grading area.

Implications of ‘Meeting Goal,’ ‘Progressing’ classifications

Schools with SAI scores falling within the “Meeting Goal” classification face no sanctions. In fact, these schools can even qualify for monetary awards if they lower the number of students scoring in the CATS “Novice” category and reduce the dropout rates for middle-school and high-school students⁵.

While not currently funded, these theoretical rewards nevertheless remain a future possibility as enabling legislation approved by the Kentucky General Assembly remains in force. Also, the Kentucky Performance Report for each school continues to indicate whether a school is qualified for rewards even though there are no checks in the mail.

The critical issue concerns how the CATS program treats schools in the “Progressing” category. The “Technical Report” makes it clear that these schools “ … are held harmless in the accountability system.”⁶ In other words, as long as a school is “progressing” toward – instead of actually reaching – its goals when the year 2014 arrives, it remains immune from sanctions. The KDE has made it abundantly clear in its testing policies that schools need not have an SAI of 100 to avoid sanctions in 2014.

What is more, “Progressing” schools get little, if any, aid and attention to make further improvements. Even schools with flat or declining SAIs are immune from any real consequences provided their indexes fall within the “Progressing” zone.

It should also be noted that “Progressing” schools may actually qualify for rewards in pretty much the same manner as “Meeting Goal” schools. The Kentucky Performance Report for “Progressing” schools even notes reward eligibility with an announcement very much like that included in performance reports for “Meeting Goal” schools.

There is virtually no significant difference between schools fully meeting their CATS goals and those simply scoring in the “Progressing” category. Due to immunity from sanctions and the possibility of receiving awards, “Progressing” has essentially replaced “Meeting Goal” as the true practical aim of Kentucky’s school assessment program.

The impact of being in ‘Assistance’

At least some measure of genuine accountability – along with special supports to improve – remains in the CATS for schools that score in the “Assistance” zone. Schools with “Assistance” SAIs face measures ranging from audits by outside educators to other, far more aggressive actions such as possible takeovers.

It is important to note that “Assistance” schools qualify for extra financial and academic assistance as well – aid which generally isn’t made available to “Progressing” schools.

Overall, however, schools have strong motivation to avoid the “Assistance” category. That is very different from the motivation to avoid the “Progressing” zone. Thus, the real goal line of the CATS is not the nominally named “Meeting Goal” line. Instead, the real goal of the CATS is achieving an SAI above the “Assistance Line.” That makes the construction of the “Progressing Line” very important.

Dividing line between ‘Progressing,’ ‘Assistance’ improperly established

We have serious questions about the way the KDE draws the “Assistance Line” on the CATS Growth Charts. The positioning of the right end of this line seems highly arbitrary. The result is that a large number of Kentucky schools will probably be excused from sanctions in 2014 even though their actual performance will be clearly unacceptable.

We estimate that by 2014, the SAI for Kentucky’s elementary schools will be about 106 and that around half of the state’s 749 elementary schools will have an SAI below 100.

As mentioned earlier, the 2014 termination points of the “Goal” and “Assistance” lines are both adjusted downward slightly – 1.1 points for Lawler Elementary – to allow for test-measurement errors. The actual margin of this testing error varies by school and ranges from 0.5 to 3 points, which seems like a reasonable correction⁷.

However, the KDE’s technical documents offer no justification for the additional, very large, 20-point reduction that is used to anchor the right end of each school’s “Assistance” line. The testing-measurement error has already been separately considered, so this additional, extremely large adjustment is mystifying. For example, in the case of Lawler Elementary School, we understand why it would be reasonable to locate the right end of the Assistance Line at a value of 98.9. We do not, however, understand what justifies the line’s actual termination at a much lower value of just 78.9.

This unexplained 20-point factor in the creation of the “Assistance Line” greatly increases schools’ ability to avoid sanctions in 2014 despite low proficiency rates. It also eliminates real incentives for these mediocre schools to make substantial improvements between now and 2014. According to the KDE’s evaluation method, “Progressing” schools already perform well enough to avoid sanctions despite clearly unacceptable proficiency rates. Let’s look at some actual examples.

Exploring ‘Progressing’ schools’ proficiency rates in 2014

Table 1 provides examples of some schools whose current SAI scores would fall in the “Progressing” zone in 2014. Basically, these schools can simply maintain their current performance, making no further significant improvements, and still avoid all CATS sanctions in 2014.

(The goal here is not to criticize these schools, but rather question the seeming inability of the CATS program to bring about positive change. While these schools certainly need to make further progress and could do so through their own initiative, our point here is that the CATS system offers no real inducement for further improvement.)

The bottom line in Table 1 shows Lawler Elementary’s 2004 CATS Biennium SAI was 80.0⁹, which ranked 310 out of the 749 elementary schools identified in the KDE’s 2004 school report-card database¹⁰. With its current accountability index, Lawler will avoid all sanctions by placing in the “Progressing” zone in 2014.

Table 1 shows the KCCT proficiency rates by subject for the 2003-2004 CATS Biennium. These rates are based on scoring scales developed by CATS scoring committees populated largely by Kentucky teachers in various academic disciplines.

Notice in Table 1 that Lawler has some very unimpressive proficiency rates in individual KCCT subjects. More than one out of four students tested below the level Kentucky defines as being proficient in reading. The situation was much worse in math, where only 39 percent of Lawler’s students scored proficient or better. In both science and social studies, more than one out of three students was not proficient. Zero percent of Lawler’s students who took the on-demand writing assessment scored proficient or better.

Do Kentucky parents and taxpayers deem such performance an acceptable end goal? Undoubtedly not! Yet this performance by Lawler appears to be good enough for the KDE as the school will likely escape sanctions on the CATS judgment day in 2014.

The abysmal writing performance of the other schools in Table 1 shows they, too, don’t deserve to escape CATS accountability without making substantial improvement before 2014. In addition, the table reveals that fewer than 50 percent of the students in all three of these other schools are proficient or better in science. Also, more than one out of three students cannot read proficiently, and nearly every other child is being left behind in math. Is that really going to be an acceptable performance in 2014?

Overall, the average proficiency rate across all subjects is less than 50 percent in all four schools listed in Table 1. Yet each of these schools will be immune from CATS sanctions in 2014 if they merely maintain their current level of performance. The tragic news is, there are many more schools demonstrating similar results, and the CATS will likely excuse all of them from accountability in 2014, as well.

Summary

A testing policy that excuses the kinds of unacceptable proficiency rates highlighted in this report can in no way be deemed compatible with the goals expressed in KERA. Such an assessment policy also is way out of line with the requirements of federal legislation that insists no child will be left behind.

The problems highlighted in this report make it clear why some schools fare well on the CATS but are identified for sanctions and assistance under NCLB. Obviously, the standards in these two programs are very incompatible.

Conclusion: Replace or revamp the CATS?

Kentucky needs an accountability system that strongly encourages rigorous proficiency for all students in each subject and accurately targets real assistance – and consequences, if necessary – to schools that need them. From questions to scoring schemes, the entire CATS program must either be replaced or completely revamped to ensure minimum acceptable proficiency rates, particularly for core subjects like reading and math.

One fairly obvious improvement would be to change the right anchor point for the Assistance Line on CATS Growth Charts. The situation would improve considerably if the right side of each school’s progressing zone was re-anchored to match its Goal Line at a point equal to 100 minus the small amount that represents school’s known testing errors.

After all, the goal is supposed to be reached in 2014. At that time, a “Progressing” school really isn’t meeting the proclaimed standard. In essence, there should be no “Progressing” zone by 2014. A school either reaches – or falls short of – the goal. Changing the right anchor point would make that happen.

However, simply fixing the Assistance Line won’t be enough. While it is probably unreasonable to expect fully 100 percent of all children – including those with severe learning disabilities – to reach a truly proficient level of performance, it also is an unacceptable notion that only 39 percent of our students need to attain proficiency in math.

Acceptable proficiency rates might need to be moved to higher levels in key areas like reading and mathematics than in some other subjects in the KCCT program. The KDE could accomplish this by establishing minimum-proficiency rates for each subject that schools must attain in order to score in the “Meeting Goal” category in 2014.

Establishing such thresholds would help the state’s testing program better mesh with the philosophy of the subject-proficiency approach of NCLB; it would also offer a much more informed approach toward academic scoring.

One thing is very certain. Unchanged, the current CATS program does not guarantee anything close to universal proficiency in Kentucky’s schools when 2014 arrives. Our students deserve better.

– Richard G. Innes is an education analyst for the Bluegrass Institute, Kentucky’s free-market think tank.

Endnotes

1. “2002 Kentucky Core Content Test (KCCT) Technical Report,” Kentucky Department of Education, March 2004

2. Technical Report, Page 1-3.

3. See the “Tech Report,” Chapter 13 for more details on the growth chart.

4. Growth Chart from “Spring 2004 Kentucky Performance Report, Oran P. Lawler Elementary School, Grayson County,” (KPR) Kentucky Department of Education, Frankfort, KY, Page 3.

5. “Tech Report,” Page 13-6.

6. “Tech Report,” Page 13-3.

7. “Tech Report,” Page 13-3.

8. Data sources for Table 1 are the individual schools’ Kentucky Performance Reports for 2004. Proficiency rates are calculated by the Bluegrass Institute by summing the published percentages of students scoring “Proficient” and “Distinguished.” All KPR are on-line at the link in endnote 4.

9. Growth Chart from “Spring 2004 Kentucky Performance Report, Oran P Lawler Elementary School, Grayson County,” (KPR) Kentucky Department of Education, Frankfort, KY, Page 3.

10. Based on Bluegrass Institute Analysis of the KDE’s School Report Card Database for 2004.

The Bluegrass Institute is an independent research and educational institution offering free-market solutions to Kentucky's most pressing problems.

Weather (Louisville) / Mapquest /  White Pages / Business Search / CNN / Dictionary / E-card / MSN