Want to Be a Bad Scientist?

Tonight, I would like to share something important with all of my readers. It’s simply not fair to keep this secret to myself any longer.

Are you ready? Here goes: I have discovered the key to maintaining a healthy height/weight ratio and performing at peak physical performance, even after one’s fiftieth birthday.

I suggest you stop reading and grab a pen. You will want to write this incredible information down to be able to share with others. Go ahead, I’ll wait for you here…

The clear reason I have been able to keep a relatively low BMI index and continue to run marathons at my age can be directly and unmistakably attributed to my rigid adherence to my “if you see a donut, EAT a donut policy.”

It’s really very simple. If you come up to me and ask, “Hey Rob, would you like a donut?,” my answer will be, “You’re damn straight!” Likewise, if I see a box of donuts setting in the mail room, I am getting mine before they are all gone. I won’t give it a second thought.

To summarize, when it comes down to choosing whether or not to eat a donut, I submit the best approach is: “Just say YES!”

I have eaten donuts my whole life and rarely have denied myself the pleasure of savoring this sweet confection. It started when I was younger when my grandma would visit and bring a dozen donuts. My brother and I would eat them all in about ten minutes. Maple and coconut are my favorites but honestly I am not that picky about it. Bismarks, glazed, blueberry, chocolate, baked, caked, sprinkled–I’ll eat um all. Sometimes I will cut one in half to give the impression that I am watching what I eat. I’m watching, all right! As soon as others turn their back, that other half will be gone too! I know you do this as well!

Last year, before running my best time ever in the Route 66 Half-Marathon, I consumed an entire QT maple bar a half hour before the race. My weight is also the same as it was when I left the Marine Corps 22 years ago and my dress blue uniform still fits well. Of course, I owe it all to my faithful and consistent consumption of donuts, especially as I grow older. My parents also eat a lot of donuts and are generally thin and healthy in their late 70’s. What more evidence do you need?

Okay, I’ll be serious now. What I have just presented to you is bad science. I do like donuts (more than I should quite honestly), but they are not the primary reason I am relatively fit. It’s more about genetics. I also run a lot and generally eat sensibly. Therefore, I have made an absurd claim about donuts and health based on flimsy evidence lacking clear reasoning and rationale. This is a classic example of what statisticians call a type I error, or false positive. I have not adequately considered the potential effect of other variables (exercise, diet, genetics, etc.) which likely have an even larger impact on my general health and wellness. I am making a correlation between variables (donuts and healthy body weight) that could not stand up to any real scrutiny.

Type I errors occur when people make correlations between variables when they really do not exist. The other type of common errors, type II or false negatives, happen when people fail to make correlations when they really do exist.

A common analogy from Larry Gonick’s “Cartoon Guide to Statistics” describes the difference between type one and type two errors like this:

Type I: Alarm with no fire.

Type II: Fire with no alarm.

Think of “no fire” as “no correlation between your variables” (or null hypothesis is accepted). Conversely, think of “fire” as the opposite, true correlation, and you want to reject the null hypothesis, because there really is something going on.

“Alarm” is evidence of correlation. So you WANT to have an alarm when the house is on fire…because you WANT to have evidence of correlation when correlation really exists.

Let me share one more example. National reformers commonly use the “poor” results of American students on international tests to make the correlation that public schools–and by extension, teachers and school leaders–are failing. In essence, they create an alarm when there is no fire. But they don’t stop there. They then proceed to label educators as arsonists for starting a fire that doesn’t really exist, or, at a minimum, is much smaller and more manageable than the reformers want the public to think.

All that to bring me to my current dilemma.

I provided the background above to explain the scenario I am going to face when I stand in front of my staff tomorrow to provide training on the state’s new Student Learning Outcomes (SLO) and Student Objective Outcomes (SOO).

All certified staff members who do not receive a value added score are required to develop a SLO/SOO which will count as 35% of their evaluation next year. After the OSDE provided training to district representatives in late October, sites were given until mid January to train staff and have SLO/SOOs developed.

So, tomorrow, I will tell my faculty that donuts will keep them healthy, that the Lochness monster exists (I’ve seen pictures), and that writing an SLO will make them a more effective teacher. All three are false claims.

In short, I will ask them to become bad scientists.

With limited previous training in statistics, quantitative analysis, or even basic scientific protocols, my teachers of social studies, art, music, PE, drama, computers, Vo-Ag, band, and so on—as well as my counselors, librarian, and nurse—will be asked to develop and conduct an quasi-experimental study to determine if a selected intervention has a desired impact on their study group composed of widely varied, unpredictable, and highly complex adolescents! What could possibly go wrong?

And the outcome of their study will count for 35 percent of their annual evaluation!

After I set the stage with some statistical mumbo-jumbo, I will proceed to explain to my teachers how they should identify an area of focus based on essential knowledge or skills they want their students to attain by the end of the year. Once they do this, they will need to evaluate which of their students’ characteristics might affect the SLO/SOO and collect baseline data of their students’ current skill levels. Are you with me so far?

Once the teachers have accumulated baseline information from each student using whatever “measure” they have developed, the next step is to create a growth target for each one, based on their best guess on where they believe students will be at the end of the SLO/SOO period. Since most of the teachers have zero historical data related to their growth measure and their students’ anticipated growth towards this target, they will simply be making numbers up.

In accordance with SDE guidance, the teacher will have to explain why the growth target is appropriate for each student or group of students, as determined from student characteristics and baseline or trend data. Since I assume the phrase, “I pulled the numbers out of my a@#” will likely not be acceptable for the SLO/SOO form (despite its accuracy), the teachers will have to put some convincing words together so their administrator can approve the targets.

What the teachers DO KNOW (because the OSDE has already established the required success rate for the growth targets) is that they need to set their goals “realistically low” to ensure they earn a certain score.

For example, if a teacher wants to earn a 5.0 on this portion of the evaluation, they know that 90% of their students must meet the growth target they set. For a 4.0, they need 80%; for a 3.0, 70%; and for a 2.0, 60%.

Thus, any teacher who makes the mistake of setting their growth targets too high risks being labeled ineffective or unsatisfactory IF less than 60% of their students fail to meet the goal. Again, the growth targets that each teacher sets are mostly subjective and derived from estimates and rosy predictions. The targets also fail to take into account all other variables that might affect the student’s achievement of their growth goal!

Finally, at the end of the SLO/SOO period, the teacher will assess how many of their students met the unscientifically derived growth targets that they arbitrarily established to earn a high score. This score will be sent to the SDE and combined with the results of the teacher’s qualitative evaluations and their Other Academic Measure (OAM) to give them one score: a score that can then be used to make high stakes decisions about that teacher.

The entire SLOBS process (I’m guessing you can figure that acronym out:-) is rife with the potential for numerous type one and two errors.

If a teacher or group of teachers do well in making their growth targets achievable, their students might score well. The teacher (and their evaluator) might incorrectly attribute the students’ success to the teacher’s intervention, when in fact other factors caused the improvement or the teacher simply set a low target. Regardless, the teacher(s) will earn a high score and be deemed highly effective or superior. It may or may not be true.

If a teacher or group of teachers set their growth targets too high and too many students fail to show “adequate growth,” the conclusion will be that the teacher(s) did not apply the intervention effectively. As a result they will be rated ineffective or in need of improvement. Again, it may or may not be true.

Two effective teachers in different subjects may develop a similar goal or area of focus. However, each teacher’s score will be derived from the setting of their individual growth targets. They also have different students so the setting of targets is completely inaccurate and unreliable. What works for one group of selected students may not work with a different group of students in another class or in another year. In short, they are not comparable lacking an analysis of other factors affecting student achievement, such as poverty, disabilities, or learning English. Some comparisons may be apples to apples, but most will be apples to oranges, or even apples to waffles. Data obtained in one classroom will not be transferable to another classroom in the same building, let alone another school with completely different variables.

I have written before on how value added models (VAM) are junk science. The logic behind SLOs and SOOs is worse. They represent voodoo science and are just about tinkering and wishing for the best. Yet we are “teaching” this crap to our educators to masquerade the fact that the state has no accurate way to conduct VAMs for teachers of untested subjects. So, to provide the illusion of fairness, the state makes up something completely and totally devoid of scientific value to waste our time. Thousands and thousands of hours of time that should be spent writing lesson plans or eating donuts.

Teachers SHOULD BE trained to use student data to guide instruction in accurate and reliable ways. This is important to promote transfer of lessons learned, both good and bad. Teachers should NOT be evaluated based on how good they are at playing the SLOBS game, but sadly, this is exactly what will happen.

I know it, my teachers know it, and people at the state department know it! 

It might be okay if they at least provided free donuts!

PS: I know this is already long but thought you might enjoy a quick rewrite of the first part of Foghat’s iconic hit, “Slow Ride.” If you have not heard or somehow forgotten the song, click HERE for the link.

<iframe width=”420″ height=”315″ src=”//www.youtube.com/embed/mIjZE4kcg_Q” frameborder=”0″ allowfullscreen>

SLO write, not so easy- SLO write, makes me queasy,
SLO write, not so easy- SLO write, makes me queasy,

Not in the mood, the timing’s not right
Half of the year’s gone, so this really bites!
Oooh, oooh, SLO write – oooh, oooh …

SLO write, not so easy- SLO write, makes me queasy,
SLO down, back down, why the rush to push
Hold it, scull it, SLO writin’ teachers are too tired
Not in the mood, the timing’s not right
I think this is stupid, and I’m ready to fight, yea.
Oooh, oooh …

This Post Has Been Viewed 951 Times

4 thoughts on “Want to Be a Bad Scientist?

  1. If I ever teach stat again, I am using that Type I/ Type ll analogy. And probably that SLOBS acronym as well 🙂
    You are right; this is a colossal waste of time, money, and energy.

  2. Rob-
    What happens if parents or students refuse to allow this slo data to be collected on the student? What if parents refuse to allow their kids to take the tests…does that impact the teacher’s VAM?

  3. Rob, I can’t imagine how hard it must be to stand up in front of your teachers and sell them this “bad science” because you are required to do so. I retired from teaching third grade last June and remember hearing about SLO and SOO. How fair is it that teachers in tested grades do not get to choose which outcome he/she wants to be assessed on for the year. I guess they are just SOL!!

Leave a Reply

Your email address will not be published. Required fields are marked *