Richard Innes

1/ I become more and more dismayed by the huge number of supposed education experts who regularly use data from the NAEP improperly. In this thread I plan to outline some of the issues so the rest of us will have a better understanding of how people mislead with NAEP.

@CreditFlex Good point.

@Innes434 I don't understand why they munged together G3-4 scores with G8 scores. My district, for example, is 4 years into using an SoR approach. The expectation would be that the 3rd grade scores should rise. But that would be lost in this data.

A new Education Scorecard ranks states and some school districts for academic progress from 2022 to 2025 educationscorecard.org/?utm_medium=em…. It's an interesting report and adds to the discussion, but I have questions. The Scorecard adjusts test results from different state assessments -- which are not directly comparable -- onto a common scale using results from the 2019 NAEP. I am wondering why a more current NAEP, either 2022 or 2024, wasn't used and if there are potential problems with this approach. The rankings are for overall state scores only. I suspect that things could look very different if the results were broken out and compared in a more apples to apples manner, such as looking at white student and Black student scores separately. Page 32 in the NAEP 2009 Science Report Card talks about why such breakouts are important. A similar logic should apply here. There can be a notable difference between making progress and current standing. Consider information in Figure 11 in the national Education Scorecard educationscorecard.org/wp-content/upl…. It shows Tennessee ranked 4th best for reading growth, well ahead of Massachusetts. But check below to see how Tennessee ranked for NAEP Grade 8 Reading in 2024 (most recent available) compared to Massachusetts. Tennessee ranks statistically significantly behind Massachusetts for white student scores. Why didn't the scorecard include current standings along with the improvement information?

@MikeJoh14725926 That earns you a block. You clearly have no clue about conducting respectful discourse.

@Innes434 You are very impressed with yourself for having no experience in public schools.

I am still seeing some making claims about MS' reading reform not showing any improvement on the ACT. Well, it's simply too soon to see that. The post below has more details.

@MikeJoh14725926 I doubt you've read much of my research. Unfortunately, closed-minded attitudes in public education make it really hard get improvements that work for students. Still, those improvements are badly needed.

@Innes434 28 years as a public school teacher. I trust my experience over your research any day of the week.

@MikeJoh14725926 I've been researching Kentucky's public school education reform effort for more than 3 decades. I've learned a lot. What is your experience with education?

@Innes434 What experience do you have in public schools?

@MikeJoh14725926 Your personal experiences are limited. This is a big country; a lot is happening you cannot personally experience, but it is happening. I'll let others reading this discussion decide who knows what they are talking about.

@Innes434 I've never seen it prosecuted. I'm not guessing. I'm speaking from experience while you are doing internet searches. You don't know what you are talking about.

@MikeJoh14725926 Mike, "No one ever notices"? Here's just one article with the truth: usatoday.com/story/news/edu…. Truth involves determining all the facts, not guessing. Schools notice, and they prosecute, too.

@Innes434 Truth involves admitting when you are wrong, and you just double down on this nonsense. I've seen parents do it, and no one has been prosecuted. As long as your child isn't causing problems, no one even notices.

@MikeJoh14725926 What's true here is parents can get prosecuted for illegally entering their child in a public school other than their zip code. It's called residency fraud. And, in general neighborhoods with good schools are more expensive to live in.

Again, not true. Another way people do it is by moving into neighborhoods with good schools to begin with, and thus not having to move or transfer. I've seen poor people do it by using grandma's address to get into a different school. I've seen others use a friend's address. Your smokescreen doesn't hold up.

@MikeJoh14725926 In general, only parents with money have those public school choices.

Actually they can pull their students out. They do it all of the time. Vouchers are one way to do it. Some districts allow inter-district transfers. Some people move. What you are doing is posting a smoke screen to cover up that you want to hold public schools accountable, and other schools using public funds not so much.

Mike, that's really funny. Where NAEP has issues, I point them out. When I did it in 1999, it upset NAGB so much that they were still writing about it years later (Start at Pg. 7 in this: tinyurl.com/yjd3fue3). NAEP, properly analyzed, is helping by showing that more needs to happen and education is problematic in the US. It is also pointing to states which are starting to make better things happen for students. As far as vouchers go, there definitely is accountability, but it works differently. Parents can pull their child out of a private school that fails them. Parents generally can't do that with a public school, so public schools need different accountability systems.

You may have drunk the NAEP Kool-Aid, but I haven't. I do not see how this test has helped students prepare for life. It does allow politicians to pretend they know what they are talking about, but many of them have never set foot in a public school. It is a lot easier to create a chart and convince everyone that education is in crisis. I love it when people say that public education is paid for by the public and has expectations about outcomes. However, I have noticed that no such expectations exist with the school voucher money being handed out to private schools and homeschools.

I think you completely misunderstood me. NAEP shows many students are not getting the education they need for life: storage.ghost.io/c/53/d3/53d3d9…. It's a useful test. Most state tests, by comparison, set too low a standard for proficiency. That makes education in the state look better, but it isn't getting the job done for students. I see plenty in education misuse NAEP to try to claim all is basically well, disregarding things like NAEP's sampling errors in the process. They get the message wrong with poor analyses.

@Innes434 So we should just ignore the fact that American public school students are doing abysmally in math and reading? Or are you saying NAEP isn’t the determiner or any test isn’t and the kids are alright?

I continue to see educators misusing the NAEP, often by ignoring the sampling errors in the scores. This thread will help you to avoid being misled.

First, I have seen plenty of examples of educators jumping on fads that didn't work. Kentucky fell for many of them over the years. Just one example: eddatafrominnes.com/pdfs/KERAUP32.…. Next, if we are talking about reporting if students are getting ready-for-life proficiency, NAEP scoring is probably more valid than the vast majority of state tests. Check out why I say that here: storage.ghost.io/c/53/d3/53d3d9…. Finally, schools can't use NAEP because federal law does not allow NAEP results to be reported for either schools or individual students. Regarding politics, I can't think of any state testing that isn't political. Can you? Public education in general is political as the public pays for it and has expectations about outcomes.

I don't believe it should be disregarded, but I've seen it too many times where education has jumped on a bandwagon only to find out later it didn't really work. NAEP is a politician's test. No schools use it. It has problems like setting the proficiency level higher than the state's do with their own test. Many believe this is to artificially create a crisis to allow politicians to push through their radical agendas.

@h2sdrummer It's important for people who understand the data to comment on misuse. The NAEP is very useful, but it has limits and it can be misused when those limits are not honored.

@Innes434 You can see how tempting it is for people who want to sell an agenda, or product, to use the data and graphs to sell a story. It’s almost all you see on Twitter.

I see this differently. Many naysayers write that there are no miracles so MS should be disregarded with no indication that the situation might change. Their confirmation bias leads them to continue saying there are no indications in MS' NAEP Grade 8 Reading when the 2024 results show the state made progress compared to other states for both white and Black students. They cannot come to terms with the evidence (not just from MS) that schools can do better despite poverty and other issues. They want poverty to be an excuse.

@Innes434 Time will tell. I think that is what Diane Ravitch and others are pointing out. Nobody wants to kill anything, but before everyone starts copying this model, we need evidence that it works over time.

Well, I think the reform isn't a miracle, and many from MS who are involved seem to agree. That's not the real issue. The MS naysayers seem to want to kill what's happening because it contradicts their world view that schools should be left off the hook because of poverty, etc. If MS succeeds, they are wrong, and they can't live with that.

@Innes434 But I think that is the point. If it hasn't been around long enough to make an impact on ACT scores, then we shouldn't be declaring it a reading miracle.

@WSJopinion @shadwhite Well, the Mississippi gains in 4th Grade NAEP Reading are starting to appear in Grade 8, as well.

The “Mississippi Miracle” will fizzle if the state fails to extend gains to later grades with school choice, writes @shadwhite on.wsj.com/3NbwmJG

@plthomasEdD apparently doesn't understand that NAEP is a sampled assessment. All the scores have statistical sampling errors that make small score differences meaningless. The first table below is one he has used repeatedly, including in the linked article, which claims in Grade 8 NAEP Reading results that Black students in CA, GA, LA and MA outscored those in Mississippi (in 2024) (Forget the Dept of Defense schools, they are not like a state). However, after the score sampling errors are considered, the NAEP cannot tell us that the small score differences for Grade 8 NAEP for CA, FL, GA, and LA are actually different from those for MS. The second table set, generated using the NAEP Data Explorer, gives you the correct picture. Notice that Mississippi isn't at the bottom of the listing, either.

Grade 8 NAEP: A More Valid Reading Story (Although Still Incomplete) [Michigan, Missouri Updates] open.substack.com/pub/paulthomas…

3) @plthomasEdD and others are also fussing that MS has shown no progress on the ACT. That's really poor analysis. The MS 2013 legislation targetted K to Grade 3. There hasn't been enough time for retraining teachers and getting the program up to speed for results to work up into Grade 11 or 12. Again, @plthomasEdD seems to want to kill SoR before it has a full chance to work.

How to Manufacture Crisis with Line Charts: NAEP Reading Edition open.substack.com/pub/paulthomas…