The Death of Testing and the Rise of Learning Analytics

I know that it is sad news for some, but more than a few of us have assessed the situation, and the prognosis is not good for our friend (or perhaps the arch enemy to others of us), the test. We might be witnessing the death of testing. Tests are not going away tomorrow or even next year, but their value will fade over the upcoming years until, finally, tests are, once and for all, a thing of the past. At least that is one possible future.

Tests are largely a 20th century educational technology that had no small impact on learning organizations around the world, not to mention teachers and students. They’ve increased anxiety, kept people up all night (often with the assistance of caffeine), and consumed large chunks of people’s formative years.

They’ve also made people lots of money. There are the companies that help create and administer high-stakes tests. There are the-the companies that created those bubble tests and the machines that grade them. There are the test proctoring companies along with the many others that have created high-tech ways to prevent and/or detect cheating on tests. There are the test preparation companies. There are even researchers who’ve done well as consultants, helping people to design robust, valid and reliable tests. Testing is a multi-billion dollar industry.

death of testingGiven this fact, why am I pointing to the death of the test? It is because of the explosion of big data, learning analytics, adaptive learning technology, developments around integrated assessments in games and simulations and much more. These technologies are making and will continue to make it possible to constantly monitor learner progress. Assessment will be embedded in the learning experiences. When you know how a student is making progress and exactly where that student is in terms of reaching a given goal, why do you need a test at the end? The student doesn’t even need to know that it is happening, and the data can be incredibly rich, giving insights and details often not afforded by traditional tests.

Such embedded assessment is the exception today, but not for long. That is why many testing companies and services are moving quickly into the broader assessment space. They realize that their survival depends upon their capacity to integrate in seamless ways with content, learning activities and experiences, simulations and learning environments. This is also why I have been urging educational publishing companies to start investing in feedback and assessment technologies. This is going to critical for their long-term success.

At the same time, I’m not convinced that all testing will die. Some learning communities will continue to use them even if they are technically unnecessary. Tests still play a cultural role in some learning contexts. My son is in martial arts and the “testing day” is an important and valued benchmark in community. Yes, there are plenty of other ways to assess, but the test is part of the experience in this community. The same is true in other learning contexts. Testing is not always used because it is the best way to measure learning. In these situations, testing will likely remain a valued part of the community. In some ways, however, this helps to make my point. Traditional testing is most certainly not the best or most effective means of measuring learning today. As the alternatives expand and the tools and resources for these alternatives become more readily available, tests will start the slow but certain journey to the educational technology cemetery, finding a lot alongside the slide rule and the overhead projector.

Education in a World Where Every Action is a Data Point

Remember the classic scene from Dead Poet’s Society where the new teacher, John Keating, instructs the students to rip pages out of the poetry text? He offers the students a different view of poetry, one that invites people to feel instead of analyze. There is something to be said for appreciating the forest and not just chopping it down into discrete pieces that are more easily quantified. At the same time, we are in an increasingly data-driven world, and I don’t expect to see it disappear from the field of education anytime soon. So, what are we to do?

people as data vennIn the Walking Dead of Higher Education, a software vendor wrote an article about the need for more satisfactory ways of approaching assessment in higher education. He contends that higher education administrators are having trouble proving the value of a college degree because they don’t have effective ways to measure what students are learning. This is not a new notion. In fact, it is a mantra in our increasingly data-driven world. We see the drive for more data-driven decision-making in everything from political campaigns to marketing plans, church growth strategies to public health initiatives, crime prevention efforts to assessment of organizational effectiveness. This is the expected progression of living in an increasingly digital and connected world. As I’ve mentioned elsewhere, the first wave of the digital era was about increased access to information through the Internet. The second wave was the discovery that it is the connections, communities and social interactions that have some of the most transformational possibilities in the digital age. Now, in this third wave, we see the convergence of the first two waves (These are not entirely distinct. There are social and analytic elements already in the first wave). Now we have increased access to more information than we ever imagined, but every one of our actions (social and otherwise) is becoming part of that information, data points to mine and analyze for meaning, and used to achieve business, educational or other goals.

When I first understood that Google was mainly an advertising company, I had a great illustration of this new age. I took people to Google, conducted a random search and then we looked at the results. What do you see? Most of us now get the general idea. The first results to show up are paid ads. What is Google selling? They are selling you, based upon data about you, data suggesting that there is a possible match between who you are, what you want or might want, and what others are selling. The consumer becomes the product being sold via a business-to-business interaction between Google and another company. This is very old news, but this concept, evident in many other online business strategies since the 1990s, illustrates the idea that our behavior becomes valuable data.

Now schools and education vendors are increasingly involved in similar practices. Education companies are coming to Universities and P-12 schools, offering ways to collect, track, analyze, and extract meaningful insights from student data, not just student performance on tests and key assignments, but thousands of other potential data points. Some are coming to schools that already have a massive collection of dust-covered student data, and companies are showing them the stories that these data tell, and how these stories are important for the work of the school. Others are starting at the beginning, offering new ways to collect data that have never been collected before, let alone analyzed.

I’ve written about adaptive software before, even providing a simple visual to illustrate the feedback loops involved with such software. Adaptive software is an example of how these data are being used on the micro level to create learning experiences where student performance and progress is persistently tracked and the software adapts accordingly (or, apart from  adaptive software, where the teachers or students are prompted to respond). The testing culture in American K-12 education illustrates some of these data on the macro level. There is no question that big data sets will be used in ways that will amaze, trouble and baffle many of us over the upcoming decade. This is certain to happen in the education sector as much as anywhere else.

Yet, it is important for us to remember that the technologies of modern statistics and data analytics have values. The more we seek to use them, the more they drive us to look for that which they are best at measuring. This means that standardized tests are likely to be used before portfolio assessment practices, personalized adaptive learning software before messier student-centered self-directed learning practices. Until we gain the ability to collect and analyze broader types and ranges of learner activity, these technologies have a way of telling us what to value as much as measuring that which we value. For example, suppose I wanted to understand how much my children love me. Would I be satisfied with a dashboard that indicates the number of times they spoke those words in the last month, the number of acts of kindness toward me, or something else that is easily quantifiable? I’m reminded of something I referenced by G.K. Chesterton back in a 2012 article about letter grades:

G.K. Chesterton, in The Everlasting Man, is talking about a completely different subject, but he discusses how many people seek to quantify things that are not inclined toward quantification. They are pictures, not diagrams. Consider your favorite piece of art, song, or sunset. Would you agree that a careful quantitative analysis and report on any of those would give an accurate picture of why it is your favorite? Would you be satisfied with a quantitative analysis in place of a picture or the real thing? Suppose you went to an art museum and discovered that every painting and sculpture was replaced with a chart or diagram that represents the same concepts or ideas that were previously illustrated in the work of art. I contend that letter grades too often do a similar thing.

This is an important caution as we move forward with learning analytics, big data, and strategies to identify quantitative measures for student learning. The measures are often (usually…almost always…maybe always) just approximations. The data will tell a story, but there are other important stories of learning to be told that are not read in the numbers, not in any set of easy measures that we develop. I am not arguing that we resist the move toward analytics in education, only that we better understand it, what it is, what it is not, how it can influence and shape our mission, vision, values and goals. The data are not just measures of our stand alone goals. Key Performance Indicators, for example, are hardly ever just measures of things we value. Over time, they influence what we value, sometimes in subtle and hardly noticeable ways, but other times (and over time) the influence can be substantial. Is it time to share one of my favorite Postman concepts again? With any technology (including big data, statistics, and learning analytics) it is not just about learning how to use the technology, but taking time to understand how it uses us.

The author of the article that I mentioned before (Walking Dead of Higher Education), for example, seems to propose that we provide quantitative answers to questions I’m not sure can be answered by quantitative measures alone, not unless we change the questions to make them more easily quantifiable. If we really want to get into what students are learning and how the higher education experience is impacting students, it will be a longitudinal mixed methods study on every learner. However, that is too messy, time-consuming, expensive and unrealistic…at least for some (but this might change). What usually happens is that we reduce the academic standards to something less (even as we do it in the name of something like academic rigor), but something less that is easier to measure, communicate and understand by parents, consumers, the public and politicians. The technology of modern statistics is not neutral. It values that which its current tools can measure and tends to minimize the rest. Without thoughtful deliberation, the charge to become more data-driven in education is a proposal to change the core values of schools to fit a new set of values embraced by a specific set of stakeholders. Only time will tell how this push for elevating the quantifiable will play out in higher education and P-12 schools.

As much as this article may seem to indicate otherwise, I am an advocate (or rather a Luddvocate) for learning analytics and big data in education. I am not, however, a champion for doing it with our eyes closed. It is time for us to count the cost as best as we are able, and to allow our institutional mission, vision, values and goals to maintain a strong voice in this growing world of data.

By the way, if I put my futurist hat on for a moment, I caution some of the loudest advocates for data-driven education and the dominance of quantitative measures in schools. Big data are here to stay, and it will push itself into many other areas, not just education. Politicians are using data to drive campaign decisions. Wait until data start to flip, when citizens start to use data to analyze the behaviors of politicians. Just as information was democratized with the first phase of the digital revolution, wait for some excitement when analytics becomes equally democratized (look for a full article on this topic in the near future).

Educational Publishers & Content Providers: The Future is About Analytics, Feedback & Assessment

What is the future of educational publishers and content providers? As more content becomes freely distributed online and there are more creative (and sometimes free) products and services that help aggregate, curate, chunk, edit and beautify this content; there are questions about the role of educational publishers and content providers. While there is something to be said for a one-stop-shop for content, that might not be enough to secure a solid spot in the marketplace of the future, especially given that content is not the only thing for which people are shopping.

Some fear or simply predict the demise of such groups, but I expect a long and vibrant future. In fact, over the past decade or two, we’ve already witnessed publishing companies rebrand themselves as education companies with a broader portfolio of offerings than ever before. They’ve done so by adding experts in everything from educational psychology and brain research to instructional design, software development to game design, educational assessment to statistics, analytics, and testing. These are exactly the types of moves that will help them establish, maintain, and extend their role in the field of education. This is a shift from a time when many educational publishers and content providers would suggest that it is best to leave the “teaching” up to the professional educators. Now, more realize that there is not (nor has there really ever been) a clear distinction between the design of educational products and services and the use of them for teaching. Each influences the other, and understanding of educational research is critical for those who design and develop the products and services that inform what and how educators teach students.

According to this article, the preK-12 testing and assessment market is almost a 2.5 billion dollar market, “making them the single largest category of education sales” in 2012-2013! A good amount of this is the result of efforts to nationalize and standardize curriculum across geographic regions (like with the Common Core), allowing education companies to design a single product that aligns with the needs of a larger client base. However, even apart from such moves for standardization, more people are becoming aware of the possibilities and impact of using feedback loops and rich data to inform educational decisions.

This is just the beginning. If you are in educational publishing or a startup in the education sector, this is not only a trend to watch, but one to embrace. Start thinking about the next version of your products and services and how learning analytics and feedback loops fit with them. If you look at the K-12 Horizon Report’s 5-year predictions, you see learning analytics, the Internet of everything, and wearable technology. What do all three of these have in common? They are an extension of the Internet’s revolution of increased access to information, but this time it is increasing a new type of information and making it possible to analyze and make important decisions based on the data. Now we have a full circle. Data is experienced by learners. The actions and changes of the learner become new data points, which give feedback directly to the learner, to a teacher, or the product that provided the initial data. There is a new action taken by the learner, teacher and/or interactive product and the cycle continues (see the following image for three sample scenarios).

Screen Shot 2015-02-16 at 2.36.14 PM

Some (although an increasingly small number) still think of the Internet and digital revolution in terms of widespread access to rich content. Those are people who think that digitizing content is adequate. Since the 2000s, we’ve experience the social web, one that is read and write. Now we live in a time where those two are merged, and each action individually and collectively becomes a new data point that can be mined and analyzed for important insights.

While there are hundreds of analytics, data warehousing and mining, adaptive learning, and analytic dashboard providers; there is a powerful opportunity for educational content providers who find ways to animate their content with feedback, reporting features, assessment tools, dashboards, early alert features, and adaptive learning pathways. Education’s future is largely one of blended learning, and a growing number of education providers (from K-12 schools to corporate trainers) are learning to design experiences that are constantly adjusting and adapting.

The concept that we are just making products for the true experts, teachers, is noble and respectable, but the 21st century teacher will be looking for new content and learning experiences that interact with them (and their students), tools that give them rich and important data (often real-time or nearly-now) about what is working, what is not, who is learning, who is not, and why. They will be looking for ways to track and monitor learning progress. If a content provider does not do such things, it will be in jeopardy, with the exception of extremely scarce or high-demand content that can’t be easily accessed elsewhere.

As such, content still matters. It always will. However, the thriving educational content providers and publishers of the 21st century understand that the most high-demand features will involve analytics, feedback (to the learner, teacher, or back to the content for real-time or nearly now adjustments), assessment, and tracking.

5 Predictions About Educational Credentialing in 2024

I am doing a bit of consulting later in the week, and one of my tasks is to make a few predictions about education in 2024. My part of the day is focused upon alternate and micro-credentialing. With that in mind, here are five predictions. I don’t necessarily like all these outcomes, but based upon the trends, I see many of them as highly likely, especially as they relate to adult and continuing education; and education for trades and regulated professions. What do you think? As you read this short list, you may be surprised about how much does not seem to be directly tied to credentialing. That is because, at least in much of American higher education, credentials and assessments tend to shape and direct much education practice.

I’ve always seen assessment as a bit boring until I started to recognize how it has become the most powerful aspect of many education environments. Change or add a given assessment or evaluation practice and you can quickly see a transformation in an entire system. Look at the conversations about Common Core in K-12 education. It was when the use of assessments started to take root that the debates become most intense.

Do you have any predictions of your own?

1. Unbundled Education – Education will become increasingly unbundled and aggregated across networks and contexts. This will give way to increased grass-roots educational initiatives, the capacity for learners to self-blend learning experiences from multiple sources and organizations, and cross-organizational credentials. Highly regulated sectors and those with strong centralized professional organizations and standards will be most insulated from some of this. It will lead to significant turmoil and disruption in many higher education institutions.

2. Networked Learning will become a fundamental life and work skill. While the most regulated industries will be more insulated, there will be significant conflict between democratizing and authoritarian models of education and training. Regardless, a fundamental aspect of lifelong learning will be the development, maintenance and ongoing expansion of a personal learning network. Related to this, we will see massive formal learning networks within geographic areas, specific fields and professions, and other distinct physical or virtual communities.

3. For many professions and trades, competency-based education and assessment will largely replace assessment of readiness through traditional letter grade systems, GPAs and similar measures. Systems like traditional letter grades will be phased out with the emergence of more accurate and granular measures of learner progress and competence. This will impact both initial training and continuing education.

4. Depending upon the context, alternate and micro-credentialing systems will replace or supplement letter grades, course, credits, and degrees (but the most regulated industries will be more insulated from this disruption). These emerging credentialing systems will have features like expiration dates and detailed information about the criteria met to earn the credential.

5. Educational experiences will provide significant learner control and/or learner-specific adjustments of time, place, pace and learning pathway. As part of this, adaptive learning and robust learning progression designs will replace many industrial or one-size-fits all models of education and training. For better or worse, with the maturity of adaptive learning tools, there will be a renewed and invigorated battle between the  “science of teaching and learning” and the “art of teaching and learning.” Learning analytics and big data will drive the design of high-impact, competency-based individualized learning experiences.