Like every father, I think my daughter is pretty clever. She is inquisitive; creative and loves to discuss new discoveries with me. Nothing new here, I hear you say? Yes, probably quite typical of a healthy 9-year-old girl. I agree. You can probably give me a whole list of things that are stereotypical for most 9 year olds: they hate eating liver, love cookies and cartoons, and they hate math. Right? Well, yes on most counts, except one thing: last year she fell in love with math!
My daughter discovered her love for math through Khan Academy. Khan Academy provides gamified online learning through micro lectures in the form of YouTube videos and a whole suite of practice exercises and tools for educators. In 2010, the Khan Foundation (that owns the www.khanacademy.org eLearning site) decided to add Gamification elements by introducing experience points and badges to motivate its online students. Currently Khan Academy has more than 15 million subscribed users (although it is unclear how many are users that come back on a daily or weekly basis).
My daughter is a lucky girl! All good, right? Wrong… I discovered a few months ago that she stopped visiting Khan Academy altogether. Her last website visit – 3 months ago! How could this be? How could she go from daily excited visits to absolutely no visits at all? Let’s have a look at Khan Academy and Gamification through the lenses of the Octalysis Framework and find out what happened!
Gamification in Khan Academy
The Discovery Phase
Khan Academy is targeted at younger users (from around 6 years up) and particularly those that want to learn math. They use that knowledge wisely in the Discovery Phase of the experience. Their content and SEO ensures that they are near the top of the Google search result page when you search “Learn Math”. Inclusion of the word “free” in the page description also lowers the barrier for people to click on the link to their website. So far so good!
Once you land on the website front page, you directly get confirmation on who the target group is and what the desired action is:
The small hands on the keyboard provide “Social Proof” to children that they have landed in the right spot: “This is for me! Other children must also be using this site” (Core Drive 5: Social Influence and Relatedness). The statement “You can learn anything” pulls you in further as it appeals to our Core Drive 2: Development & Accomplishment. It also gives a hint about the progress you can make in Khan Academy, which furthers enhances Core Drive 2 and tickles your sense of curiosity about what can be found inside (which is Octalysis Core Drive 7: Unpredictability and Curiosity).
Khan Academy finally leaves no doubt about what the Desired Action is here: “Start Learning Now.” Young people only please! Notice how the buttons for parents and teachers are minimalized. We call this a Desert Oasis (Octalysis Core Drive 2): the Desired Action is the only part of the page that is visually “colorful,” and your eyes automatically are guided to it.
You could say that even more black hat motivational design (the kind that motivates you urgently but may leave a bad taste afterwards due to your lack of control), could be even more effective in pushing people to enter the site. You could point out what children would miss out on by not joining (Core Drive 8: Loss and Avoidance). Or you could even create a scarcity push by only allowing certain number of users in at a given time of the day (Core Drive 6: Scarcity and Impatience). Then again, the objective of Khan Academy is not to force people in but to soft sell children. As they have 15 million subscribers, it looks like they did a good job with its current push.
The Onboarding Phase
OK, so Khan Academy Gamification elements got 15 million people into the experience. Well done. But what happens then? This is where Onboarding starts. After you have been made enthusiastic about discovering a product, the designers then need to make you feel competent and engaged to actually want to use it (for more onboarding knowledge see also here).
After you have registered with your Facebook, Google or email accounts and have clicked the confirmation link your email, Khan Academy directly asks you to select an Avatar:
This is a clever way to immediately create a feeling of ownership of your identity in the experience (Core Drive 4: Ownership and Possession). The fact that you are asked to input mental energy by choosing from a range of avatars increases this feeling of ownership even more.
In the Onboarding Phase we also want to dangle future cool possibilities in the experience and showing cute and cool avatars creates a nice Core Drive 6: Scarcity and Impatience effect. Because we cannot have what we want now, but have to wait for it, we want it even more!
Now the user is ready for the next step: choose a subject and start to learn. Here Khan Academy tries to use Gamification best practices by including a tiny step-by-step-tutorial.
Your avatar welcomes you (more push to increase ownership) and tells you what to do next: choose a subject. A nice addition here is the text “Don’t worry, you can always change your mind later.” This takes away some of the anxiety of picking the wrong subject (Core Drive 8: Loss and Avoidance).
Not bad at all. BUT, the amount of subjects offered (150+) is still way too large. Even when I know I can always change my mind, I still don’t know where to start and the chance of choosing things I don’t like is still very high. User dropout is normally very high when users hit pages that they don’t like, and I expect user dropout to be quite significant here. In the Onboarding Phase we want users to feel satisfied with their choices and give them easy win-states. Allowing a user to choose between 150 subjects is a risky tactic: high value in allowing many autonomous choices, but high risk in terms of losing user engagement! Khan Academy could have incorporated more Social Proof here, by showing what other children within your age group are studying now or what subjects are “popular” for example. This Core Drive 5 design, would leave children more motivated to follow a certain subject.
Once you have chosen a subject, you get this:
The tutorial tells you what to do, which is good and leads to feelings of accomplishment. The announcement “Now for the fun part where you learn things!” creates extra Core Drive 7: Unpredictability and Curiosity push.
The challenge for Khan Academy here was to find out what you know and what you don’t know yet so that they can fine-tune the learning content they give you. So they do this:
To make sure that they give you the right content, Khan Academy lets you do some practice questions to tailor their follow up content to you. This is a start to what we call the “Alfred Effect,” a Core Drive 4: Ownership and Possession design. The Alfred Effect occurs when an experience is personalized in such a way to the user’s preferences and skills that they cannot imagine using another service. In a eLearning world with many online education providers, getting your content in tailored to users’ preferences is key.
The questions in the warm-up section are quite easy and even if you get one wrong you still get a lot of positive feedback. In the Onboarding Phase (remember, we are still trying out the core functionality of the site) it is important to give the user plenty of easy win-states so they feel awesome and accomplished (Core Drive 2). Khan Academy Gamification does that here through strengthening Core Drive 2 motivation (by earning experience points; earning your first badge, completing a progress bar and by giving the verbal feedback “Awesome”). In addition you are reminded that you can customize your avatar. Doing so would mean that you put even more mental energy in your identity on the site thereby increasing your feeling of ownership and belonging even more.
The Scaffolding Phase
OK, so far Khan Academy seems to do a reasonably good job in creating a rewarding experience for its users. Can you see why they are so successful in attracting people and drawing them into the experience? Let’s find out then, why people like my daughter dropped out at later stage? Somewhere in the experience, there must be sections where engagement is faltering and where users drop out.
Khan Academy Gamification continues where it left off in the Onboarding Phase: give users relatively easy content, with easy win-states.
Notice how you only need to get 1 right, so that you do not need to sit through exercises that are too easy. This is clever Gamification design as it avoids users becoming bored when challenges get too easy. In the site background Khan Academy run an algorithm that adjusts the level of questions you get until you reach your sweet spot between challenges and skills. Users get into Flow:
Right after you get in Flow, and you get quite comfortable with the exercise level, this happens:
Just as I am sitting in the middle of the Flow Channel, Khan Academy makes things much harder for me. Not only is the content more difficult, I also need to have 5 correct in a row! The Alfred Effect is kicking in. The site knows that I am good at Early Math and in order to still let me learn, they increase the challenges on content and flexibility. Core Drive 6: Scarcity and Impatience also kicks in, because something is less (easily) available, I want it more and will commit to the exercise more. Clever, clever!
Note that there is still the option of getting a hint or to see a video. The Desired Action for Khan Academy Gamification is “learning math,” not “getting 5 answers correct in a row”.
The Mystery is getting solved
Mm, we still haven’t found out why my daughter dropped out. So far everything looks just fine… But our search continues. I am sure we will find the culprit soon!
Let’s return to the overview of subjects:
We already noticed the huge amount of choices here and the damaging effect on user engagement that this page has right? Now let’s look at what I see if I return to Khan Academy one day later:
Hey! What happened to my guidance? Where is my avatar that greeted me on the first day and what should I do next and why? The Early Math box looks kind of interesting-ish, but these boxes below also look slightly appealing. What to do? All of a sudden I don’t feel so much in control anymore, and my Core Drive 2 motivation is lagging.
I finally decide to click on “Early Math” as the white color appeals more than the black boxes (coincidence?). Maybe that will give me more guidance:
Mm, nope. Now I am even more lost. What do I have to do here?? Continue with my “early math mission”? Or click on any of the other 8 options (some options are in the bottom half of the screen). Let’s try “Counting”:
Wow, another 22 options to choose from. You know what? Forget it. And in my search to feel in control I decide to go back to what I started already, the Early Math mission. Surely, I will get more guidance there right?
Ah, finally some ready made content here that I can interact with! BUT, I still need to choose from 62 different skills that need to be practiced 4 times. So I need to do all of these exercises 248 times?? Wow, my motivation just went down a lot. Core Drive 6 Scarcity is a powerful motivator, but if users see the challenge as being too difficult to achieve it will become Core Drive 8: Loss & Avoidance and leads to drop out. “Really?? I need to do these exercises 248 times just to master early math?? See you!” says the 9-year-old.
At that moment I checked out the Arithmetic module, the one following Early Math, hoping it would be not so bad. Wrong, another 115 skills that needed to be practiced 4 times for a total of 460 times! My motivation dropped even more, and no badges or experience points could counter it. Spending days of my time surely wasn’t worth earning a badge or even more experience points? Surely nobody else was stupid enough to spend so much effort to reach a goal that I wasn’t even sure what its value was??
I asked my daughter, at what point exactly she dropped out. “When I saw how many exercises I still had to do after Early Math, I lost interest Dad.” She actually completed Early Math, but more so to impress me (Core Drive 5: Social Influence & Relatedness, which is an Intrinsic Motivation Drive). Without me she would have quit much earlier she said.
Bingo! We found the spot in the experience where engagement cratered and users drop out. Here is also the proof that badges and experience points do not guarantee long-term engagement. It is the pitfall of using too much Left Brain (rational motivation) and Black Hat (motivational, but does not make you feel in charge) Gamification.
In Khan Academy Gamification, users easily lose touch with their journey “Is there a special reason why I am doing this?” (Core Drive 1: Epic Meaning and Calling). They are also unsure whether other users are going through it too (Social Proof from Core Drive 5) and do not experience any unpredictability push as they can already see exactly what is ahead of them in the future (low Core Drive 7: Unpredictability and Curiosity).
So in a nutshell, the Gamification used in Khan Academy is quite unbalanced. It is very much geared towards Left Brain, Extrinsic Motivation and there is not much Intrinsic Motivational Design. Extrinsic Motivation works well in getting people to try out your product in the Discovery Phase. It also works well in creating easy win-states in Onboarding. But during Scaffolding, we need to start designing for Intrinsic Motivation. And Khan Academy does not pass this test, unfortunately.
The fact that you can choose whatever exercise you want to do is intrinsic of course (via Core Drive 3: Empowerment of Creativity and Feedback), but this is almost completely counteracted by too many available choices (Anti Core Drive 6); no suspense about what will happen next (weak Core Drive 7); and, very importantly, absolutely no clear social proof nor social connections to keep you motivated to come back long term (Core Drive 5). Oh, and what happened to my higher goal in learning math? Mm, yes, not much Core Drive 1 either.
Conclusion about Khan Academy Gamification
Designing eLearning packages is not easy. Khan Academy did a great job in compiling a massive amount of learning modules on their site. The power of the Gamification approach used is good at attracting new users and getting them to “play” for a while. But the lack of balance in Extrinsic and Intrinsic Motivation results in many users dropping out during the Scaffolding Phase. This is sad as Khan Academy’s content is great and balancing the experience would lead to many more users having a long lasting and engaged learning experience.
We haven’t focused on the Endgame of the Experience, as we wanted to find out why my daughter dropped out. As we have seen the culprit was found during Scaffolding already. The Endgame is all about how you retain your veteran users and motivate them in becoming multiyear-long users. We may do more research during an Octalysis Consultancy for them one day on the final phase of the Experience. Who knows?
For more information on how to craft awesome Gamification for eLearning, contact: