Can Big Macs Affect Your VAM Score?

imageAccording to a recent study conducted by the Department of Human Sciences at The Ohio State University, this kid should have to wear a T-shirt to school declaring, “I’m a VAM Killer” to alert any potential future teachers. These educators will want to do everything they can to avoid having him assigned to their classes.

Why? Because the research appears to reveal that the amount of fast food children eat may be linked to how well they do in school. In short, eating a lot of fast food as an elementary school student can do much more than just make a kid fatter and less healthy–it can also literally make them DUMBER!

Lest you believe this is one of those satirical articles from The Onion, rest assured this is a real university study. However, The Onion did post a humorous response to the article HERE.

The researchers in this study found that the more frequently children reported eating fast food in fifth grade, the lower their growth in reading, math, and science test scores by the time they reached eighth grade. How much lower? Well, get ready for this: the study showed that students who ate the most fast food had test score gains that were up to about 20 percent lower than those who didn’t eat any fast food.

This study included about 11,740 students. They were tested in reading/literacy, mathematics and science in both fifth and eighth grades. They also completed a food consumption questionnaire in fifth grade.

If you are a skeptic like me, you undoubtedly are thinking, “aren’t there other factors like poverty, family structure, student considerations or quality of teachers and schools that may have affected this?” I mean, there has to be a way to blame this on rotten teachers, doesn’t there?

Well, like most good university researchers, the authors of the study thought about these things too. Using a regression analysis common in statistical studies to isolate the contribution of different variables, the researchers took into account a wide variety of other factors that may have explained why those with high fast-food consumption might have lower test scores, including how much they exercised, how much television they watched, what other food they ate, their family’s socioeconomic status and characteristics of their neighborhood and school. Even considering these other factors, the researchers found moderate correlations between fast food consumption and student performance on standardized tests.

What this means is that while this study cannot PROVE that fast-food consumption caused the lower academic growth observed in this study, the authors are confident fast food is explaining some of the difference in achievement gains over time.

It does seem to make some sense. If eating nutritious food has been shown to affect physical growth factors in children, it is reasonable to expect that eating poor quality foods could negatively affect a child’s cognitive development. That being said, I am certainly not convinced. Maybe highly intelligent people tend to eat more nutritious foods while less intelligent people tend to eat less nutritious options. But, just for fun, let’s assume that this conclusion is accurate.

If these are valid and reliable conclusions, it seems like we need to do some additional study of our current Oklahoma value-added models (VAM), particularly since we are implementing these quantitative measures into our state teacher and leader (TLE) evaluation system this year.

Based on this new research, I am wondering if our current VAM model will be able to factor out a child’s fast food consumption from the calculations. I think we can all agree that math and reading teachers have very little influence over how often parents of their students pass through the McDonald’s drive thru. As a result, all of the hard work we do to teach Johnny how to divide fractions or identify a subordinate clause during the day could be wiped out by one Big Mac value meal that evening!

image

Will Hunting is not available to help me fully decipher the VAM formula above, yet even with my own rudimentary skills in statistics I am unable to see ANY factors for fast food consumption—let’s call it the Big Mac Factor (BMF)—in this VAM formula.

How can this be? Since most reliable studies related to teacher effects on student achievement place that value somewhere between 8 and 14 percent, with another 10 percent or so being school-related factors (leadership, resources, and peer influences), you would think our state would certainly need to isolate the BMF (with its potential 20% impact) from our calculations.

Is it fair to hold me and my teachers accountable for the state assessment scores of our eighth grade students without addressing the potential likelihood that their consumption of fast food has obliterated many of their brain cells, clearly outside of our control?

The answer: Hell NO! This is simply NOT fair!

I am not being completely satirical here. I share this example to once again illustrate the absurdity of relying on these value added models as reliable and valid measures of teacher effectiveness. As I have written about in several posts before, VAMs may have some value in the teacher development process, but should not be used to make high stakes decisions about teachers or leaders.

To fully grasp the ramifications of value added, we need to realize that value-added metrics are designed to further lock public school teachers into the world of pseudo accountability, a world where they are held responsible for phenomena completely beyond their control.

Dr. Randy Hoover from the Ohio Teacher Advocate Project has published several studies on the difference between authentic and pseudo accountability in education. In THIS article, Hoover argues for the need to be vigilant in ensuring we have a system of authentic accountability as opposed to a system of pseudo accountability. What does he mean?

Hoover provides this simple example to highlight his point (emphasis mine):

“If the state or federal government were to hold local news weather reporters accountable for each day’s weather, we would think that to be unfair and uninformed accountability. The weather reporters have no control over the meteorological conditions they encounter because they can only control the their forecasts. Authentic accountability would focus on the accuracy of the forecasts, and pseudo accountability would focus on the weather itself. Therefore, to be fair and to generate valid and credible accountability reports to stakeholders, it is only right and proper to hold the practitioner responsible for only those things within their decision latitude—for only those things over which practitioners clearly have professional control and the power to affect.”

Hoover concluded by arguing that: “Value added as a mathematical procedure is neither bad or good in and of itself. However, the assumptions about teaching and learning upon which VAMs are based are entirely wrong. Value added is an example of pseudo accountability.”

This leads us to the critical flaw of VAM. Value added is based on the demonstrably false assumption that there is a one-to-one correspondence between the teacher and student achievement. In other words, the formulas assume that teachers are the direct cause of student scores—and that a specific teacher’s effects are revealed directly and fully by standardized test scores. Value added assumes that teacher effects are significant, specific, and precisely quantifiable when they are not.

It has become conventional in educational policy discussion to assert that “research shows” that “teachers are the most important influence on student achievement.” In fact, up until last spring, our own state department printed the following comment on almost everything related to TLE on their website:

“Educators and researchers agree that teacher effectiveness
is the single most important factor in student academic achievement.”

There is, in fact, no serious research that shows any such thing. The assertion results from a careless glide from “teachers being the most important in-school influence” to teachers being the most important influence overall. To their credit, the OSDE has gone back to their materials to add the words, “school-based” in front of “factor” in the above quote. This is more accurate, but still misleading. This is because school effects on average levels of achievement are smaller than the effects of families and communities, even if teachers were the largest school effect, they would not be a very big portion of the overall effect.

Again, here is what Hoover has to say about these relationships based on his studies of VAM.

“To accept that teachers are 100% responsible for student achievement test scores is to reject scientific reality and, worse, a monumental moral outrage. Teachers are not the most important influence on student achievement. No legitimate educational research has ever found that they are. Teachers do have an impact on achievement, but all forces and factors in a child’s life considered, the impact is far, far less than we are led to believe. Teachers have no control over the backgrounds of students they encounter—no control over the living conditions, the poverty or wealth, the advantages or disadvantages, the intellectual gifts or deficits, or the developmental readiness each child brings to school. The time has come to stop holding teachers responsible for these critical life variables!”

The bottom line is that a child with an average teacher who comes from a literate, economically secure, and stable family environment will, on average, have better achievement than a child with a superior teacher but with none of these contextual advantages. This is true regardless of how many Big Macs the former student consumes.

The lesson we should learn from this is that there are many factors that affect student achievement, to include, possibly, excessive consumption of fast food during their cognitive development years. TOO MANY. Even the best value added models are unable to account for all of these factors, which is precisely why they are they are considered unreliable for evaluation purposes (ASA Study HERE).

Given this reality, it is neither ethical nor legally defensible to base teacher evaluations on factors outside of their control. But the so-called reformers determined to paint public schools and public school teachers as failures need these models to drive their misguided narrative.

For this reason I will continue to sound the refrain “VAM Must DIE” until we are successful in getting them removed from our state evaluation system. Like the boy snarfing down burgers and fries at the start of this post, I too wish to have a t-shirt that proudly proclaims, “I’m a VAM Killer.”

Other than that, I think I will eat a salad tonight. I’m losing enough brain cells with aging as it is!

This Post Has Been Viewed 1,580 Times

14 thoughts on “Can Big Macs Affect Your VAM Score?

  1. And THANK YOU, Rob Miller . . . for likely causing me to forever link Big Macs and VAM. This negative association will be helpful in my quest for healthier living in the new year. 🙂

  2. What am I missing? Why is this so hard? Why is our government making it so difficult for us as teachers and administrators to do our jobs? Maybe I’m just a silly woman with my head in the clouds….just a simple soul with outdated ideas and only a rudimentary idea of advanced statistics and educational theory, in spite of the advanced degrees and years of experience. We love the children, and they love us. We pour our very lives into our jobs, and the students are learning. So when did we get to the point where society started pointing their fingers at the schools and shouting, “It’s their fault!! The economy is tanking, and we have no jobs and no money and no future and the police are shooting everyone and Lindsay Lohan is in jail again…..BLAME THE TEACHERS!”

    All of the studies, just like this one, show that the biggest indicators of student success come from outside the classroom. It seems very simple to me…..schools can’t fix all of societies ills, but a free public education is the last great hope for true social justice. By drowning our schools with VAM, TLE, SOO, SLO and the rest of the toxic alphabet soup that is being poured over our heads, they are sabotaging the only lifeline that some of these kids will ever have.

    For the love of God….quit choking the crap out of the loyal goose, expecting golden eggs to come out.

    • There is an ancient episode of The Electric Company in which Morgan Freeman dons SCUBA gear and dives into a giant cup of alphabet soup. I could so see a remake happening in which he fishes out our current educational acronyms . . . before narrowly escaping the steaming, toxic broth. 🙂

        • I’ve got it on DVD at school . . . but, for a change, I thought I’d stay home today! 😉

          The best I can tell from looking around online is that it’s #379 or maybe Season 3, Episode 119.

  3. I recently read an article in The Oklahoman similar to this one:
    http://newsok.com/despite-some-progress-oklahoma-drops-to-no.-46-in-national-health-ranking/article/5374632

    Are we going to be satisfied with #46? We need REAL health reform in Oklahoma . . . reform that serves us patients rather than the medical bureaucracy bent on maintaining the status quo. What we obviously need is for our physicians and medical care facilities to submit themselves to measures of accountability. Physicians should be evaluated on such items as the yearly changes in the weight, blood pressure readings, and blood glucose levels of myself and fellow patients. (Note: They may earn bonus points if patients show up for all their check-ups.) Physicians and health care facilities who cannot demonstrate consistent growth by these measures will be target or priority groups. These designations are not intended to punish them, rather to bring them to light through this fair and transparent process for the good of all Oklahoma health care consumers.

    Physicians and facilities designated as target or priority will be required to perform increasingly frequent weight checks, blood pressure screenings, and finger pricks. (There will also be periodic pre-screenings to ensure patients are adequately prepared for their high-stakes screenings.) Of course, the standards for what is considered “healthy” are subject to change. For example, just because 120/80 was considered “normal” blood pressure in 2010, that does not necessarily translate to normal today. The State Regents for Higher Health may need to certify our standards as Mountain and Marathon Ready. That said, the epidemic of Oklahomans moving though our health care system without emerging Mountain and Marathon Ready is pathetic. If physicians and health care facilities are unable to add value to indicators of health on a yearly basis, their personal and organizational evaluations will reflect this and patients will be given the option to transfer to higher-performing facilities. This commitment to transparency, accountability, and rigor will produce real health reform and ensure we do not lose another generation of Oklahoma patients to medical mediocrity.

    • TC, I enjoyed your metaphor! Though imperfect, these type of examples help illustrate the ridiculous logic of many of reformers who dismiss out of school factors when discussing student achievement. You might enjoy this post I wrote last September along the same lines. I may steal your “mountain and marathon” ready quote!

      • Ah, I missed that! It was posted way before I became aware of A View From The Edge and made it a daily stop on my Internet walk. Funny how we thought so much along the same lines . . . guess the analogy is almost too obvious . . . but you did a much better job with the alphabet soup!

        Feel free to use Mountain and Marathon Ready as you wish. It kind of goes along with the whole Race to the Top notion. 🙂

    • I love your metaphor, and the Mountain and Marathon Ready phrase. I would be the bane of some doctor’s existence, I am sure, with all the time I spend in front of the computer, instead of exercising. Oh! But that doctor must not be very good if he can’t inspire me to lay off the chocolate and jog a mile or two…

    • The health comparison is apt. I was surprised to learn recently that research suggests that the health care establishment (doctors, hospitals, etc.) accounts for only about 10% of an individual’s health outcomes; the rest depends on other factors such as environment, genetics and lifestyle.

      When it comes to fundamentals like health and education (or a healthy democracy), we have a misplaced consumerist mindset. That is, we think we can neglect our own responsibilities, then run to the professionals to buy the results we need, as though achieving wellness or an educated society were as simple as buying new underwear.

      • This is excellent insight, Doug! The second paragraph is all too true, sad to say. Thanks for the great comment!

  4. Very interesting Rob. Thanks for taking the time to share this information. Hopefully other leaders in education will be willing to consider this study and its results.

Leave a Reply

Your email address will not be published. Required fields are marked *