Much educational research has a largely unrecognized limitation. You can read study after study and miss it because it is so familiar. Here is what I’m talking about. Scan research on student motivation, persistence in college, factors leading to college success, best strategies for helping students with ADHD find success in school, teacher effectiveness, strategies leading to student success in science and math, and a myriad of other topics. What do they all have in common? Almost all these studies are tied to a largely traditional school context. Consider the one about helping students with ADHD find success in schools. Many studies are looking at how to help these students fit within the confines of a traditional school context. Studies on how students with ADHD fare in adulthood with or without medication are interesting, as the samples in these studies have a bias toward students who would sit traditional schooling settings during their formative years. Change the context, and it may or may not be a different result, but much research doesn’t look at that. Could it be, for example, that an intervention of a different context has drastic changes on the well-being of such people later in life? There are countless studied to show the impact of a nurturing versus toxic environment on people’s development. In other words, the studies treat the overall school structure as a constant. As such, it would be helpful to have more studies that look at how a given demographic of student fares in different contexts.
The same is true about student motivation, performance on tests, or the vast majority of research about improved student learning. How many of those studies measure student learning using readily available measures like standardized tests and other similarly easily quantifiable sources? In other words, much modern educational research has a classroom or traditional school limitation. Most do not look at what would happen if you put students in a completely different learning context, or when they do, there are many studies that are measuring success or performance using traditional schooling assessments.
This is no small problem today, because it taints and shapes the modern conversation about education reform. Consider how quickly conversations about the Common Core State Standards turned into a debate about testing. Why? That is because standardized tests are the way we can easily measure student progress as it relates to the Common Core. The easiest measures are not the best ones. Yet, the use of tests is also a problem with many modern debates about the performance or under-performance of various magnet and charter schools. These schools are sometimes designed to nurture different competencies, but people are too often judging the school with traditional tests that measure or emphasize something else. We too often don’t use robust and multi-faceted measures that provide a full picture of what a given school tends to do both well and poorly.
This calls for a more nuanced understanding of what much educational research tells us. It calls for getting beneath the surface because the design of every study, like any technology, has affordances and limitations. Ironically, discovering these limitations calls for a type of thinking that is rarely measured on many tests today.