Measuring Value of Investment for #EdTech in Schools

Is a 1:1 iPad program worth the time and cost? How about adding an iPad cart to a class, adding interactive whiteboards in every classroom, student response pads, switching to electronic books, or purchasing an adaptive learning math software package?  Or, how about the cost of a web platform to improve parent-teacher communication? With the price tag associated with such technology investments, it is common (and even important) to ask about whether the benefits are wort the investment. The importance of such a task is amplified by Larry Cuban’s 2001 Oversold and Underused, and countless articles like this highlight of the  2013 National Assessment on Educational Progress study, which seemed to indicate some less than promising practices.

While some policy makers like to talk about education in terms of Return on Investment (ROI) by looking at the economic benefits to society, and others talk about ROI for school marketing and recruitment plans, this does not necessarily match with goals of a social endeavor like education. For this reason, many educational technology leaders focus upon value of investment (VOI). What is the value that comes from a given educational technology investment. Does it help support the mission, vision, values and goals of the school? For such a measurement, that requires articulating goals.

In Here’s the 5-Step Tech: Here’s the 5-Step Tech Investment Plan Districts Should Be Using, Keith Krueger lists potential target areas.

Increasing student achievement
Increasing student engagement
Improving attendance and behavior
Attracting and retaining staff
Developing 21st century skills for students
Decreasing dropout rates for at-risk students
Engaging parents and communities

Other possibilities include targets like:

  • increase family access to high-impact learning experiences,
  • document student learning beyond the school building,
  • increase student access to global perspectives on current events,
  • equip students with what Tony Wagner refers to as “collaborating across networks”,
  • increase parent-teacher communication,
  • improve student and teacher digital literacy,
  • equip learners with the ability to flourish as digital citizens,
  • assist students in building personal learning networks to increase self-directed learning competency,
  • or create strategies that adapt learning experiences to the distinct needs of each learner.

Note that some of these might relate to academic goals or standards, while others get at related but distinct topics like engagement, increased access and opportunity, addressing equity issues, or improving communication among stakeholders. Once more general targets like these are identified, it can be helpful to articulate them as more measurable goals.  From there, we brainstorm possible strategies to achieve the goal(s), including learning about potential technology-enhanced solutions. Each strategy will have affordances and limitations, which will be an important part of the exploration.

If we make it this far, it is a matter of selecting the most promising option (hopefully based on some research, including getting into the current literature), and building a plan (which will have a heavy focus upon professional development if we want to avoid the mistakes of countless 1990s educational technology projects in schools).

Screen Shot 2014-02-27 at 7.17.29 PMAll of this sets us up to ask the question, How will be measure the value of our investment? This includes plans for collecting formal or informal data about the project: observations, teacher and student journaling, focus groups, interviews, surveys, analysis of student products and performances, support logs, etc. This might not provide us with the certainty of empirical research, but it can help us develop a better understanding of the value afforded by a given educational technology investment.

What about you? How do you determine the value of an educational technology investments in your learning organization?

How Digital Badges Can Bolster the Aging System of Grades & Transcripts

Suppose you have one or more degrees and I ask for evidence of what you learned from those degrees. You have a couple of options. You can show your transcripts, which have abbreviated names of courses, the credits associated with each course, and the grade that you earned. Or, you can share some of the projects and papers that you wrote or created. Of course, the best evidence would be to show me what you know now, but we’ll bracket that option for the sake of this article. For the first option, there are some limitations. By looking at the transcript, I don’t necessarily know any specifics about what you learned in a class. I also don’t necessarily know what it means to earn an “A” or “B” in a particular course. Does that include participation points? Was it graded on a curve or was it based on mastery of the concepts? For the second option, you can pull out your old tests, papers and homework; but who is really going to look through all that information?

These are limitations of the current system. Most Universities do not offer an easy way to communicate the discrete knowledge gained and skills developed as a result of a degree program. Perhaps we’ve come to a time in educational technology innovativion where we have some alternatives or supplements to the current system. One that is gaining increased attention is the concept of open digital badges. Toward that end, consider the following six affordances of badges that are not represented in most current school systems for documenting and displaying student learning.

Document Learning Beyond the Classroom – What about all the learning that takes place beyond the school or classroom but relates directly to one or more standards or objectives in a formal curriculum? Currently, evidence of such learning and accomplishments remains absent from a formal academic credential. What if open badges could bridge this gap, allowing learners to earn digital badges for learning outside of schools and inside school, displaying the joint collection in a digital backpack that one can share with prospective employers?

Documents Discrete Knowledge and Skills – As I stated at the beginning of the article, current transcripts usually do not give details about what knowledge was gained and what skilled were developed. Digital badges give an opportunity for micro-credentialing of each discrete bit of knowledge and skill. For each badge, there is a description and a list of criteria that one has to meet to earn the badge. If a person could display a collection of badges as part of one’s learning, this would allow for communicating much more meaningful information about what a student did or did not learn as part of a course or entire program.

A Badge is Badge – Since criteria are always included with a badge, this allows any viewer of the badge to understand the standard set for earning the badge.  This is a significant enhancement to the dominant letter grade system, where viewing a transcript does not communicate the standard used to decide the grade.

Comparing Evidence Across Different Organizations – Does earning an “A” in English 101 at Harvard represent the same thing as earning an “A” in English 101 at a local community college? Most might assume that the answer is “no”, but we don’t really know. To know that, we would need more detail about what was expected of learners each class, and a transcript with a letter grade does not communicate that. Using digital badges to document learning in classes has the potential of allowing us to actually compare knowledge and skill sets from different sources.

Differentiate Academic and Non-Academic Performance – As noted in #4, we don’t know the standards used to earn a grade in a particular class by looking at a transcript. This is interesting given that a number of non-academic factors sometimes influence a grade. For example, suppose a student masters all the concepts in a class, but is persistently late on assignments. The letter grade and transcript will reflect what, at first glance, might be misinterpreted as a lower level of content mastery, when what it really indicates is a person’s inability (at least during that class) to meet deadlines. By documenting learning with micro-credentialing, a badge is earned when the criteria are met. This removes the need to speculate about the meaning of the badge.

Allow for Designing More Effective Learning Progressions – We sometimes use performance in one class as evidence of preparation for another class or activity. Sometimes one class is a pre-requisite for another. Other times, an upper level class might require a “B” or higher in an earlier class. However, most of us have seen examples where this standard didn’t work. This is partly because of the limitations alluded to in #1-5. What if we could set up more discrete benchmarks for advanced learning activities using digital badges? Instead of the broad pre-requisite of a certain course or a grade in a certain course, we could drill down to the specific knowledge and skills required to progress to a next learning activity.

I have little expectation that letter grades and transcripts will disappear in the near future. They have a trusted role in education, and there is widespread acceptance of the current system. At the same time, there are clear limitations that become even more clear as new possibilities like digital badges emerge. As a result, I expect to see a growing number of learning organizations start to blend these two systems, keeping part of the traditional credentialing model, but bolstering it with something like digital badges or another technology with comparable affordances.

Democratizing Academic Credentials with Open Badges

In a recent post about my trip to the Open Badge Summit in Silicon Valley, I wrote the following as one of 10 takeaways from the event:

Learners Own the Evidence of Their Learning – “We own our learning data rather than institutions owning it” – Connie Yowell – This represents the potential of digital badges based upon the Mozilla Open Badge standard. It means that wherever you learn something, when it is documented with a badge, you can take that evidence with you. It isn’t owned by any single institution. You can pull it into your digital learning backpack. The credential may come from different groups, people or organizations; but the potentially powerful shift is that the credential is your’s to keep, without having to ask for a transcript or official action from any institution.

As I begin to introduce the idea of digital badges in formal education (especially to people unfamiliar with the concept of open badges), many seem to initially think of badging as a strategy for increasing learner motivation…as digitized star stickers and “Good Job!” comments at the top of a paper.  As the conversation progresses, one of the first “aha moments” comes when I show how breaking courses and programs into discrete competencies attached to badges allows for more accurate documentation of student learning. It becomes easy for educators to see the many benefits, providing far more detailed data about student learning than an overall course grade. However, it is when I start talking about democratizing academic credentials that I start to see disinterested or confused looks. 

How do open badges democratize academic credentials? What does this mean? And what are the potential benefits of it? To consider answers to these questions, I like to start with the current situation. If I apply for a job and people want to verify the degrees I claim on my resume, they can have me request a college transcript. To provide that, I must request it from the University where I graduated.  This ongoing verification of credentials is controlled by the institution from which I graduated. Registrar’s offices in Universities decide how it is presented, to whom it is displayed, and when it is disseminated; even withholding “official” transcripts until the balance on one’s account is cleared (As I understand it, FERPA regulations in the US give students the right to view or quest access unofficial transcripts when they want.). In this current model, my transcripts are separated by the institutions where I studied. I, for example, have degrees from four different institutions and have taken courses from many more. So, each transcript is separate. In addition, when one views my transcript, there is little information about what I did or did not learn. It just lists courses, credits, and grades. It doesn’t even tell me what was supposed to be learned in a given course…simply an often-abbreviated title like, “The Written Word” or “Hist of Westn Civ.” What does it even mean to get an “A-” in “Hist of Wetn Civ”?

Now consider the idea of college degrees consisting of a series of digital badges, each representing a discrete body of knowledge or skill acquired to meet graduation requirements. Each badge includes a description of the badge, and detailed criteria that one must meet to earn the badge. It includes information about the issuer so anyone viewing the badge can verifying the authenticity of it. The moment a learner provides the necessary evidence of meeting the badge criteria, the badge is issued to the learner (while a full degree, which could be represented in a mega-badge, might not be issued until all other badges are earned). This need not replace the current transcript and diploma system. Badges can be part of a course and traditional transcripts can be created as well, with badges serving as a supplement, but an integrated part of the course (I’m working on distinct but related version of this that I hope to soon reveal.).

From that point on, the leaner maintains control of who sees the badge, where it is displayed, and how it is presented. The learner can immediately place it somewhere like a Mozilla Backpack and from there share the badge in an electronic portfolio, a LinkedIn profile or a personal website to use as evidence of certain knowledge or skill. It is a stand alone credential (hence the concept of badges as micro-credentialing), allowing one to leverage it right away. One can bolster a résumé progressively, even as working through a degree program. It is also quite easy to rearrange credentials, highlighting the display of those credentials most relevant for a given purpose (like aligning with a specific job description). This also allows one to pull micro-credentials from multiple institutions. One might have several badges on computer programming from one University, others from a few MOOCs, more badges from respected training and credentialing programs, and perhaps badges provided by an employer the recognize accomplishment of certain programming projects on the job. The learner can pull these together and present them as a collective case for one’s competency for a given job or simply the public display of one’s knowledge and skill.

This is what I mean by democratizing academic credentials. Notice how it shifts ownership (or at least control) of the credential from the institution to the learner. It also potentially increases the value and immediate usefulness of discrete knowledge and skills as they are acquired.  I have no certainty that Universities will embrace open badges in this way, and there is much work to be done if we were to make such a shift. Nonetheless, I find the possibilities intriguing and potential affordances significant enough to call for further discussion.

By the way, I’m working on another post that will focus on why digital badges are superior to current mainstream methods for documenting student learning.

Moving Beyond the Creation/Consumption Debate in #1:1 Programs? Not Quite

As more schools continue to move to 1:1 programs, there is the ongoing question about which device. Should we choose iPads, Chromebooks, laptops, Android devices, or maybe the Microsoft Surface? If you ask school leaders how they made the choice, you find a fascinating and sometimes eye-opening variety of answers.

  • There was a sale on _______.
  • I like this brand best.
  • We had someone willing to buy this product for us.
  • This one had the best support plan.
  • A vendor made a compelling case.
  • A special grant or corporate partnership.

There are many others reasons as well. However, I’ve yet to find a school that described the following process.

  1. We decided what we wanted learners to be able to do.
  2. We tested the usefulness of each device in doing these things.
  3. We took into account the unavoidable realities of things like cost.
  4. We made our choices based upon steps 1-3, recognizing that we don’t always need to make a single decision. We might choose one device for each student, but we also make sure to resource learning spaces with enough other options to address the different usage scenarios we defined in step 1.

I’m sure there must be plenty who chose to do this. I just don’t hear about it. Yet, the more I think about it, the more sense it makes to me. I own all the devices listed above and I have a pretty good sense of which device is most helpful in accomplishing different tasks. If I am working on a complex project with multiple browser windows open, a spreadsheet (or two), and a document where I am recording my findings; I use a laptop connected to a docking station and two nice-sized monitors. When I’m traveling and working on the same project, I settle for my MacBook Air, sometimes using an iPad next to it as a second monitor. However, the second option takes me longer, sometimes longer. In fact, it slows the process so much that I often opt to work on projects with a less complex workflow when I am traveling and using my laptop. In other words, the device available to me influences the type of work that I choose to do.

I decided to test this out. I was working on a project where I had to analyzing content from multiple sources and organizing it in a Google spreadsheet. It required the use of three short web sites, a 5-page PDF document, and a 2000-word Google Doc. This was a task that is very similar to what I would expect to see of students in a project-based learning environment. To make things easier, I opted for a simple but common task of collecting quotes from the different sources and organizing them by source in a Google spreadsheet. Then I repeated the same steps using three other devices. Below are the results. Note that I should have gotten better at the task each time that I repeated it. In other words, the last devices should have had an advantage over the earlier ones.

dual screen setup with full-sized keyboard and bluetooth mouse – 5 minutes

I copied 22 quotes from three web sites, a PDF, and another Google Doc, placing them in the right categories in a Google spreadsheet, including references (just a simple name for the source).

single laptop (Macbook Air) – 5 minutes

I attempted to repeat the same steps. I finished 17.

iPad – 5 minutes

I attempted to repeat the same steps. I finished 12.

iPhone – 5 minutes

I attempted to repeat the same steps. I finished 7.

Of course, I didn’t take into account all the possible variables. Perhaps I could have set up a couple more similar tests and then averaged the results. In addition, there are skills to consider. Part of the challenge might be that I don’t type as quickly as others on an iPad or iPhone.

My point is not that we should give all students a dual monitor setup. Money, space and any number of other factors might rule that out. I’m simply suggesting that we test things out a bit more. Come up with the types of tasks that we want learners to be able to do and then involve learners in testing out which device is most helpful for different tasks. How might such a process change our device and learning space design decisions?

By the way, if you’ve done something like this at your school, I would love to hear from you.