Grading Those Test Scores by Hugh Kemper

(As visitors to this site know, Mr. Hugh Kemper is a man who is willing to test the conventional wisdom.  His exhaustive report on educational funding in Vermont is permanently posted on this site and an invaluable resource for anyone who wants to understand the extravagant per-pupil cost of sending a Vermont child to school.  Kemper looked at the recent reports on student performance in the state and decided to dig a little deeper.  His report makes for sobering reading.)

Vermont’s Public Education Report Card 2007:
Grade ‘C’

Only average? That couldn’t be true. Haven’t we all read recent news articles extolling the stellar performance of Vermont’s public schools? That is true. These articles, however, were based on a very casual reading of the results contained in the 2007 National Assessment of Educational Progress (NAEP) reports.  A more critical assessment revises Vermont’s apparent national rankings as follows:

4th Grade Reading: to 19th from 4th
4th Grade Math: to 23rd from 6th
8th Grade Reading: to 11th from 1st
8th Grade Math:  to 16th from 4th

Demographics account for these significant downward revisions of Vermont’s performance. Vermont’s students are 94% ‘White’.  Nationally, ‘White’ students, comprising 55%-58% of those tested, significantly out perform ‘Black’ (17%) and ‘Hispanic’ (18%-21%) students. NAEP highlights the importance of demographics in assessing comparative performance. Vermont’s revised rankings reflect how our ‘White’ students performed vs. other states’ ‘White’ students.

While the NAEP reports are billed as ‘The Nation’s Report Card’ they don’t award grades. To rectify this shortcoming, standard deviation methodology was applied to the test performances of white students in all 50 states with the grade of ‘C’ accorded its traditional role, i.e. indicative of average performance. Massachusetts performed the best with an aggregate grade of ‘A’ (two A’s and two B+’s) while West Virginia had an ‘F’ on all four tests. Vermont’s aggregate grade was ‘C’ (‘C’ on both 4th grade tests and ‘C+’ on both 8th grade tests). 

Vermont ‘White’ Scores vs. National ‘White’ Average Scores:
4th Grade Reading: Vermont 229 vs. USA 230
4th Grade Math: Vermont 247 vs. USA 248
8th Grade Reading: Vermont 273 vs. USA 270
8th Grade Math:  Vermont 292 vs. USA 290

Apologists for the exorbitant spending on Vermont public education often point to its ‘excellent performance’. That’s simply not true. And the solution to Vermont’s average performance is not more resources. As a recent report detailed there are significant opportunities to streamline Vermont spending on public education without compromising performance. Improving performance requires a complete rethink of what and how we’re teaching and administration/teacher accountability for measurable results. Are we up to the challenge? We need to be. We owe it to our kids.

Author’s Note:
The supporting data/analysis is available via this link . It contains five tabs- one for each test and one to a summary of aggregate performance. The two NAEP reports are available here and here .   

   

Comments

[George Cross]

Kemper continues to be hung-up on race as the factor which differentiates testing data. For years, many agreed with this differentiation. However, more recent stratification of testing populations has concentrated more on socio-economic conditions of the family. I suggest that Kemper look at how Vermont does when one segregates those on free and reduced lunch from the total cohort and then makes his comparisons for both groups. Commissioner Cate is on target with his continuing quest to reduce the gap between these two groups. However, most writers, both educational and sociological, suggest that this is a difficult road that is as much influenced by housing patterns and employment as with schools. Which means we need to be looking at all the factors related to children in order to effect change.

The comment about Commissioner Cate and his quest to reduce the gap between the two socioeconomic groups sounds awful. Does that mean we should strive to pull the poorly scoring group up and either let the others that are doing well slip or remain the same? Education is not one size fits all, we need to enrich everybody. It seems to me that the funds are already lopsided. How much do we spend on special ed vs. honors courses?

[George Cross]

So, we should simply argue that the gap is the natural order of things and forget about trying to narrow it? There was nothing in my comment about money. Narrowing the gap may not mean spending more money on education, but it may mean improving the housing and employment situations for some families. Does that take more money or just more creativity and less negativity probably depends on how one views the world? I will post the differences between racial differentiation and economic differentiation as soon as I can figure out how to transport a chart from one place to another. It is an important point related to comparisons across states.

[George Cross]

Kemper’s insistence that Vermont students should be compared to the nation only via how whites do on the NAEP ignores the issues related to socio-economic differences which play a much larger role in Vermont schools than race. NAEP provides data that identifies two groups of students along socio-economic lines; those who are eligible for free and reduced lunch and those who are not eligible. The chart below gives the reader a better view of how Vermont ranks: (Let’s be clear, the difference between one point on the ranking scale and another several places higher or lower might be the correct response to just one or maybe a couple of items on the test. These rankings are not like batting averages which are earned over a long baseball season.)

Test Total White Elig. Not Elig.

4 Read 4th 19th 9th 5th

4 Math 6th 23rd 8th 7th

8 Read 1st 11th 2nd 2nd

8 Math 4th 16th 2nd 6th

Obviously, if one wants to paint the worst case scenario for Vermont one will use the race card. However, that case fails to recognize the one characteristic which actually makes the greatest difference in Vermont. Kemper would suggest that Vermont should be compared in the white column above. I suggest that Vermont should be compared in three columns; Total, Eligible and Not Eligible. The great difference in size between the two cohorts, white and black, make those characteristics much less important than the economic ones in our state.

[Art Woolf]

The issue that's being debated here is how Vermont's NAEP scores compare to the U.S. when adjusted for all sorts of differences. Hugh Kemper focuses on race and provides scores and ranks. George Cross focuses on income (proxied by whether the students are eligible for free or reduced lunch) and provides ranks only.

How about controlling for both and looking at scores only? I did that for 4th grade reading.

The first number is the scale score for those students who are eligible for free or reduced lunch (poor students) and the second number for those not eligible.
Eligible--Non Eligible

US All 205--232
VT All 212--235
VT White 213--235
US White 215--235

Vermont 4th grade students eligible for free and reduced lunch do better than their peers nationally, as do those who are not eligible. But our low income white students actually perform slightly lower than their white peers nationally.

Yes, low income students perform lower than high income students. But despite Vermont's high spending, neither high nor low income students outperform their peers, adjusted for income and race.

This exercise can easily be done for the other NAEP tests. The results will probably be very similar.

[George Cross]

There are three problems with this continuing discussion:

1. NAEP results are at best estimates as it is only a small random sample of students who take the tests. Different schools are sampled by different subjects in different grades from year to year. Here is what NAEP says about its results:
“The average scores and percentages presented on this website are estimates because they are based on representative samples of students rather than on the entire population of students. Moreover, the collection of subject-area questions used at each grade level is but a sample of the many questions that could have been asked. As such, NAEP results are subject to a measure of uncertainty, reflected in the standard error of the estimates. The standard errors for the estimated scale scores and percentages in the figures and tables presented on this website are available through the NAEP Data Explorer.”
2. The percentages of students falling into the different categories being discussed varies widely and thus the comparisons begin to take on an “apples and oranges” environment. The chart below demonstrates these major differences: (The first percentage is the national one and the second is Vermont’s. White refers to the percentage of white students in the tested sample. Eligible and Not Eligible refer to the free and reduced priced lunch program.)
TEST WHITE ELIGIBLE NOT ELIG.
4Read 56–94 45–31 54–69
4Math 55–94 46–31 53–69
8Read 58–94 40–26 58–74
8Math 58–95 41–27 58-73

As you can see, Kemper and Woolf tend to concentrate on comparing 94% of Vermont students with less than 60% of the nation's students. Such a comparison is suspect at best. I must admit that the comparisons on socio-economic conditions are only slightly better as there is still a difference in the percentage of the total cohort that is included in the sub-cohort. However, it is certainly an improvement over the white only comparison.
3. To some degree this whole exercise, particularly as set forth here and in previous writings by Kemper and Woolf, tends to remind me of Governor Douglas’ fundamental business recruitment tool which is a constant barrage of rantings about Vermont being the highest taxed state in the nation (I suspect that is not the recommended recruitment technique taught in Marketing 101 at any Vermont institution of higher education). and the Democratic leadership’s focus on trying to understand the specifics about the various taxing mechanisms and how they relate to other jurisdictions. We all use the data to promote a position of some type. The notion that simply making a chart or reciting data is non-partisan and citing fact is nonsense. Reality is far more complicated at all levels and across all domains.