Tuesday, April 3, 2012

REVIEW: Lumosity - What Everyone Should Know about the "Science" behind these Brain Games

Format:Online Flash games

Price: 3-days free, monthly subscription $14.95/mo

Release Date: 2007

Target Age Group: 18-89

Developer: Lumos Labs, Inc.

Genre: puzzle, mini-games, brain-training

What people are learning: 35+ mini-games designed to improve general brain function in the areas of memory, speed, attention, flexibility, and problem-solving, although not all games are of equal caliber or likeliness to improve any cognitive function.


So-called "brain games" have become big business since the break-out success of the Brain Age series first released on the Nintendo DS in 2006. The idea that 20 minutes a day with a casual game could have wide-ranging effects, from small benefits like making a person better at remembering names, to huge consequences, like helping to stave off dementia in the elderly, is promise that we would all like to see realized.  And it's not an unrealistic promise.  We know that games possess the potential to exploit the brain's natural plasticity and to both alter and improve cognitive functioning in some very modest sense.  Board games like chess, go, and mancala, which all have histories dating back hundreds if not thousands of years, have been scientifically scrutinized in the modern age and proven to have some positive effects on skilled players' memory, problem-solving, and resistance to age-related mental decline.  And many video games, even of the non-educational variety like the fantasy role-playing behemoth World of Warcraft, have been shown through scientific studies to provide cognitive benefits

Hence the very idea of games having positive cognitive effects is already a matter of science fact, not just science fiction. But for all of the many video game developers who have jumped on the this recent brain-game bandwagon, none have been able to show that their particular games offer real, scientifically validated cognitive effects. Lumosity, however, wants to be different. Lumosity has entered the fray with considerably more neuroscience under its belt than most of its competitors.  One of the company’s three founders is neuroscientist Michael Scanlon, a Stanford Ph.D., and Lumosity prominently proclaims partnership with such reputable and prestigious research institutes as Harvard, Stanford, UCSF, and Columbia University on their website. Moreover, Lumosity claims that its games are "clinically proven exercises" that can improve players' memory and attention, and they offer "challenges and tasks shown in experiments to produce significant improvements in cognitive performance." But is Lumosity really different from any other casual brain training games, or are they simply better at mobilizing vague scientific promises in their marketing campaign?   It is clear that Lumosity seems poised to become one of the next leading purveyors of "brain games," and they've invested heavily in marketing themselves as such.  So with the self-hype working in overdrive, I decided to put Lumosity to the test and see if these games could really deliver.

The Training Experience: Lifelong Health or Lifelong Research Subject?

Pick a training program to suit your needs, but not if you're under 18.  
Before getting into the details of the games themselves, it's worth noting that Lumosity is selling more than just a few casual web-games.  Lumosity.com is a website that offers something more akin to a whole fitness/training program:  multi-week training courses, performance-tracking, social-networking, the ability to compare results with other users.  First time users are asked what areas of cognition they want to improve and Lumosity chooses a training program that best suits those needs. Of course, few of the choices seem likely to be left unselected.   Who doesn't want to be better at "keeping track of several ideas at the same time" or "concentrating while learning something new?" But it is certainly possible for members to want to emphasize improving memory over calculating figures or being faster at decision-making.

As users progress in their training programs and play more games on the site, various measurement rubrics are applied to the results of their performance.  Some of these are as simple as tracking your high scores in each game, which allows players to easily see when their performance improves at a particular game.  Other measures of performance are more obscure, however.  For example, Lumosity has invented a measure called a Brain Performance Index, or BPI.  This is supposedly "a clinically designed measure" of cognitive performance that uses and undisclosed proprietary algorithm to translate game scores into some indication of cognitive function. There are also newly added "Assessment" tests, which assign scores based on performance, but gives absolutely no interpretation of those scores.  Another was of measuring brain-power is the "brain Grade" test, which seemed like a thinly veiled means of collecting player data about personal lifestyle habits and other demographic information.

All of the various test and measurements can be overwhelming, especially as some seem decidedly more arbitrary than others, but the idea of regularly measuring your fitness fits in with the overall image that's promoted by the Lumosity site: these exercises are contributing to a healthy lifestyle.  Nearly every page of the site contains health "tips" that encourage users to train their brains and train them often. "Did you know?" the site asks rhetorically before each tip.  "Did you know?  The ACTIVE study, funded by the NIH and involving 2832 adults, found that some benefits of cognitive training can last over five years after the initial training." The implication here is clear: train with Lumosity for life-long health benefits.

Books and barbells: Lumosity is so serious about fitness!
But who really has the most to benefit from regularly training with Lumosity?  The truth isn't as obvious as it may seem. At first blush, it seems almost self-evident that subscribers themselves have the most to gain from regular training. After all, every time a member plays one of these games he or she is purportedly building brain fitness.  And just like a regular gym membership, regular users ostensibly get the most health benefits at the same time that they are getting the most value for their dollar--a member pays just as much to use the site once as does the person who might use the site 100 times.  So frugal members and fitness junkies might see it as to their advantage to play a lot and play often, but they may just want to reconsider that plan.

What may not be abundantly obvious to Lumosity subscribers is the fact that Lumosity itself has quite a bit to gain from subscribers who log into the site and play the games frequently.  Besides their obvious interest in seeing users become habituated to playing these games and renewing their subscriptions, the company gets a second, much less visible benefit from regular site-users: more data to analyze and sell.    Players' user and performance data is rigorously tracked by Lumosity.  This data is then utilized for the company's own aggressive targeted advertising, as well as sold to various undisclosed third parties.  It's right there in the Privacy Policy.

Lumosity does at least disclose their data collection practices in the Privacy Policy, but it's because Lumosity markets itself as an educational and scientific enterprise that I found their collection and redistribution of player data to be rather unsettling.  As discomfiting as it is to know that many purely commercial entities like Facebook and chain stores like Target have little compunction when it comes to collecting data about our shopping habits and personal preferences, Lumosity is commercializing what essentially amounts to human behavioral research, and this move strikes me as taking the invasion of consumer privacy to a new level.   They've turned a long-standing model of research on its head: instead of paying people to be their human lab rats, they're getting the rats pay them.  And the data Lumosity gathers is hardly done in the interest of advancing pure science or public health.  The company can use the collected player data however it likes, whether that's to improve it's own games, develop training programs for the military, or sell it to third parties and outside researchers.

Even if the majority of adult players may not be overly concerned about the use of their game data, those who are considering the Lumosity suite of games for their children should know that because of these data collection practices, it is against Lumosity's Terms of Service for children under 18 to use Lumosity's games.   Thanks to the Children's Online Privacy Protect Act, websites are strictly limited in what kinds of data they can collect from children, and the FTC has become more involved recently in fining children's app developers for violating these privacy laws.  Because of this, Lumosity specifically notes that "the Site and the Software are not designed for or directed at children; the subject matter of the Site is not designed for or directed at children; and the content, including any video or audio, on the Site is not designed for or directed at children." But there is a fair amount of doublespeak involved on this point.  At the same time that the Privacy Policy makes explicit the fact that children should not use the site, the site has "Scholar" training programs that are designed for use by "students."  And in the sparse scientific data presented in their "Science Behind Lumosity" the Lumosity shows to substantiate it's claims to efficacy, middle-school aged children were the demographic that their studies tested.  So the unwary parent should take note that despite any appearances to the contrary, Lumosity's all-encompassing data collection practices that make this educational gaming site off-limits to the under-18 crowd.

The Games

Lumosity boasts having 35+ games available on their site, a full list of which appears at the end of this review.  Some of the games are quite similar, however. Games like Addition Storm, Subtraction Storm, Multiplication Storm, Division Storm, and Raindrops are all identical in gameplay, and games like Color Match, Memory Lane, Memory Match, Memory Match Overload, Speed Match, Spatial Speed Match, Rhyme Workout also have nearly identical gameplay. Overall, I would consider the number of truly unique games to be somewhere in the high teens or low 20's. I did play each game at least once, however, the review below only highlights a sampling of game from each of the five categories of mental fitness that Lumosity utilizes: Memory, Attention, Speed, Flexibility, and Problem Solving. Some of these games are actually available to play for free if you register for a 3-day trial, so you can test them out for yourself, but I have also made an effort to include many of the games that are restricted to paying members.
Lumosity lets you customize your brain training across five different areas and tracks your results in each category.

I also enrolled in three training programs, the 40-day Basic Training course, 30-day Lumosity PTSD course, and the 30-day Lumosity Scholar course, simply to see how the offering varied from program to program.  These training programs are helpful for ensuring that players don't simply stick to their favorite games, but beyond that, it was hard to see any clear logic behind what games were chosen for each program.  It was not uncommon to play the some of the same games in each of the programs.  For example, Day 1 of the PTDS program and the Lumosity Scholar games both included rounds of the flexibility game Word Bubbles and the memory game Monster Garden.  Your mileage may vary with the training programs.

Alas, Japanaese 101 this game is not.
Speed Match (Speed)
Lumosity says: used for thinking faster, faster reaction time, speeding up cognitive processes, and also exercises working memory.

Speed Match shows players various symbols (colored shapes, multicolored bullseyes, or japanese kanji) and then players have to determine whether or not the symbol they are currently viewing matches the symbol presented to them just before. Players are then scored based on their speed and accuracy.

There are only three possible symbols that appear, and the symbols are easily distinguishable from one another, making this a game of reflexes more than anything else. Although it bears a resemblance to an n-Back test, which is known to help improve memory and fluid intelligence, this is the watered down version of that. The goal here isn't to remember the symbols, but to respond quickly.  It's not an especially entertaining game to play, but a round is quite short, so there's not much of a time investment involved in trying to improve on your previous scores.  At the same time, it's also not really clear that playing this game can really "speed up cognitive processes." There are plenty of reflex-based twitch games, like the side-scrolling space shooters that used to dominate the arcade scene, many of which are more exciting and have more replay value than Speed Match, and they seem just as capable of facilitating thinking faster and having faster reaction times.

Actually running on ice may be less difficult than this maze.
Penguin Pursuit (Speed)
Lumosity says: used for sense of direction, visualization, reading maps, and for exercising spatial recall.

What starts out as a simple navigate-the-maze game quickly turns into an experience in disorientation and, initially at least, in frustration. The premise of Penguin Pursuit is simple: you are a penguin that been placed in a maze. You have to race against a rival penguin and be the first to reach the delicious fish waiting for you at the end of the maze. The faster you outpace your rival, the higher your score for that round. It sounds easy, but just like an M. Night Shyamalan movie, there's a twist. As you progress through the maze, the maze will suddenly and unexpectedly rotate. When it does, the control scheme also changes as well.  The right arrow may move your penguin left, or maybe up.  Likewise, pressing the up arrow might suddenly make your penguin move down, or to the right.  The trick to success in this game is being able to adapt to the constantly changing control scheme as rapidly as possible.

The first time I played Penguin Pursuit, I was embarrassingly unable to adapt to the change in controls. My score for the first playthrough was a whopping 178 points.  However, the very next time I played, I managed to get a score of 19160 points.  That's not a typo, I really did do several orders of magnitude better the second time I played. And the third time, my score was also drastically improved: 33280. I can't easily account for my dramatic improvement except to say that it's simply a matter of intuiting the mechanics of the game. It clearly wasn't a result of my brain training or of suddenly improved reaction speeds, since I played the first two matches consecutively.  Rather, it seems that Penguin Pursuit relies on your brain unconsciously adapting to the new control scheme.  In other words, it's like a gestalt switch: your brain either "gets" the new control scheme, or it doesn't.  It's not clear that you can really think your way through it. You just have to play until your brain adapts and gets it right.

But does that adaptation constitute improved brain function? Certainly, an unconscious leap in thinking took place, which is undoubtedly qualifies as some form of learning.  But is that kind of learning something that can be applied to anything beyond the playing of Penguin Pursuit?  I'm not sure.  Learning how to play the game isn't the same as learning skills that have applicability outside of the game's context.  Penguin Pursuit is certainly challenging, and it's satisfying to overcome that challenge and move the your penguin toward his reward.  But what would make it even more rewarding would be to know that studies have verified that mastery over this difficult game is truly indicative of improved visualization, spatial recall, and the like.  My drastic improvement between matches makes me skeptical that the trick of this game is anything more than simply learning the game, but there's no denying that some kind of mental flexibility is required to excel at this game.

Clip-art from a diversity-in-the-workplace seminar lives on.
Familiar Faces (Memory)
Lumosity says: used for learning people's names for the first time; remembering the name of someone you've met, and for exercising working memory.

This game is a prosopagnosiac's nightmare.  You play the waiter/waitress at a restaurant.  Various characters come into your establishment and you must learn their names and remember their food and drink orders.  At first, this is a fairly simple activity.  You mostly have to be concerned with remembering the character's names, as the food orders are pretty much impossible to mess up.  As you advance to higher levels (which you can generally only do when you complete a perfect round at your current level), the game gets considerably more challenging, as the cast of customers continues to grow, more orders are placed at a time, and red-herring orders will appear in your list of options.

I've always been downright awful at memorizing things, which made the whole idea behind this game initially daunting to me.  However, after playing it a few times, I was surprised to find that it was one of the Lumosity games that I enjoyed the most in the end.   The difficulty level of the game ramps up slowly, giving you a chance to really learn the current set of customers before more get thrown at you. And even if the customers are just pretend, I found myself being surprisingly satisfied when I got their names right.  That alone made the game rewarding enough to me that I wanted to continue playing.

While Familiar Faces was a reasonably entertaining game, I do have my doubts about whether playing this game can actually improve a player's day-to-day ability to learn new names and faces. All the things that make it an entertaining game are also what make it significantly different from what one would experience in real life trying to remember people's names.  For one thing, letting a few days pass between plays is a deadly mistake.  The greatest advantage you have as a player is the potential for unlimited repetition.  Characters' names don't change between game sessions, so you can keep playing previous levels over and over again until their faces and outfits are seared into your short-term memory. Also unlike real life, while playing the game, you have the luxury of making up zany mnemonics for each customer and saying them aloud. Of course, while repetition, mnemonics, and saying things aloud are good memorization techniques to use in your everyday life, you have to be much more discreet in real life than you can be playing a game. For example, I suspect saying my "Looney Linda with a feather in her hair" mnemonic aloud wouldn't go over well if I were meeting this character in person.  So enjoy the entertainment value which Familiar Face's role-playing aspects bring to the game, but don't expect to be a real-life name-remembering whiz by playing this.

Hey, haven't I seen you before?  Like, about 2 seconds ago?
Face Memory Workout (Memory)
Lumosity says: used for learning people's names for the first time, remembering the name of someone you've met, and for exercising working memory.

Of the five gaming categories offered on the Lumosity site, the memory category has the greatest number of games.  Although spatial and working memory games are well-represented, the site places a lot of emphasis on games featuring face-name recall, which, practical as that may be, ultimately comes off as a bit gimmicky, as in the above reviewed Familiar Faces. However, while the focus on human faces in Face Memory Workout is little more than a catchy hook, the mechanics behind the game do rest on a proven tool for boosting working memory, the n-Back test.

In n-Back tests, the challenge is not simply matter of recalling the previous symbol just shown, but the previous nth symbol just shown.  That is, does the current symbol match what you saw two symbols ago? Three?  Four?  In Face Memory Workout, players' memories and reflexes are tested as various different faces are shown one after another on the screen, and players have to remember if the currently highlighted face matches a face they had previously seen.  On the earliest level, players only have to match the current face to the one previously viewed, making the game identical to the Speed Match games. However, as the "workout" progresses, players are challenged not simply to identify whether the current face matches the one they've just seen, but instead they have to remember if it matches the face they saw two or even three faces previously.  Players are then ranked based on how quick they are in making their determination, and based on how accurate their answers are.

The n-Back test features heavily in the memory games of the site, including Memory Lane, Memory Match, Memory Match Overload, and Rhyme Workout, and there's good reason for this. These tests, which are believed to exercise working memory and attention, have been shown to have positive affects in fluid intelligence.  That is, success here can translate to better ability to perform unrelated cognitive tasks, or to "to reason and to solve new problems independently of previously acquired knowledge." For more on these findings, you can see the original 2008 research paper here.

Yet, even while the memory games listed above represent Lumosity's strongest claims for scientifically valid brain "exercise," almost all of them they still fall shy of the complexity of the kinds of n-Back tests that have shown the most considerable results.  Dual n-Back tests, for example, require players to remember not just a single visual image, but an image and a sound.  Only one Lumosity game, Memory Lane, actually uses this technique.  I suspect this is because n-Back tests, and Dual n-Back tests in particular, are truly difficult.  Playing one doesn't feel like a pleasant pastime; it feels like a workout.  And while Lumosity is supposed to be offering just this workout, in many ways, their emphasis on accessibility is giving their customers the short end of the stick.  The most cognitive gains are to be made with the more challenging tests, and Lumosity's offerings in this respect are no better than what is already freely available online. The researchers who originally showed the value of n-back and dual n-back tests offer their program as a free download from their site at the University of Bern.  There are also many n-back tests freely available these days, including an open-source downloadable version here, and a very nice, simple online version here.

Blandly dressed up psychological tests aren't very fun.
Lost in Migration (Attention)
Lumosity says: used for avoiding distraction, increasing work productivity, concentration, and for exercising response inhibition.

Lost in Migration is another reflexed based twitch-game, along the same lines as Speed Match, but this time presented with a rather bland bird theme.  A flock of birds appears on the screen, and players must use the arrow key corresponding to the direction in which the center bird is facing. The trick is to not get distracted by the direction of the other birds in the flock.  Only the center bird's direction matters.  The faster and more accurate your answers are, the greater your score will increase.

This "game" is actually a variant of the Eriksen flanker test, which Lumosity describes as "a widely used and rigorously tested psychometric assessment." This it is, but it is not considered a psychological training device.  The test is used to primarily to identify impaired brain function in a clinical context, but there is no evidence that this test has any usefulness as a training tool or that improving scores are generalizable to some other kind of brain fitness or life skill.  While it certainly can't hurt to try and sharpen your reflexes by playing this game, whether this has measurable effects beyond improving one's score is doubtful.

It would also seem doubtful to expect much success in trying to improve over time, no matter how much you practice this game. Success here depends largely on sharp reflexes, and you can expect to plateau on this game very quickly.  Despite Lumosity's advertised emphasis on "adaptive learning" in their games, Lost In Migration doesn't ever change as you become more practiced at it, and it never offers more challenge.  Interestingly, the little research that exists on practiced use of flanker test suggests that the best way to improve performance at this game isn't to keep training your brain, but rather to get your body moving. Studies have shown that participating in moderate physical activity can temporarily boost flanker test reaction time. So if you're only interest in playing is to Lost In Migration is to achieve the best score, get up out of your computer chair and jog in place while you play.  However, I'd recommend you skip this flanker test altogether and just go for a jog.  Your brain will probably be better for it.

Wearing nametags apparently goes against fish-etiquette.
Playing Koi (Attention)

Lumosity says: used for avoiding distraction, increasing work productivity, concentration, and for exercising response inhibition.

Many of the attention games in Lumosity's arsenal I found to be quite easy, to the extent that they were pretty mindless and not at all fun to play. As an avid gamer, it's entirely possibly that I'm more attuned to tacking visual changes than non-gamers, so my impressions may differ from a majority of Lumosity users.  But games like Lost in Migration (above) and Eagle Eye were painfully simplistic to me. They hardly felt like games, and were neither stimulating nor rewarding.  Unless the purpose of these attention games was to make player perform a mindlessly repetitious task time and time again, then maybe they are successful in that respect.  But the contrast in the weakness of most of the attention games is painfully noticeable when they're compared to the much more enjoyable and inventive (not to mention cleverly titled) game, Playing Koi.

In the game, players are tasked with feeding all the koi in a pond once a day.  The fish are all identical in appearance, and players must click on each fish once and only once in order to feed it for the day.  With each new round, the number of fish in the pond increases.  The first level is fairly simple: three fish become seven by the level's end, and while tracking the fish did require concentration, it didn't strike me as overly difficulty.  But as players master one level, harder levels are unlocked (the game has five levels in total).  More distractions appear: certain types of fish should not be fed and lilypads block your line of site.  The number of koi in need of feeding grows, the fish swim more erratically, and the time you have to wait between each feeding is extended.  All these factors combined makes it that much easier to lose track and feed one you've already fed.

The pleasingly soothing artwork belies the genuine tension involved in playing the game: Playing Koi absolutely require focus, and a lot of it. Even slightest waiver in focus spells doom, as it only takes a split second to lose track of that last fish that needs to be fed, and I found myself scrunching my face into all kinds of comic contortions as I desperately tried to track ten increasingly wriggly koi fish around the pond. There is little doubt here that success in this game hinges on your ability to stay on task, and each near-perfect round lends plenty of motivation to try again. I think the strength of this particular game is that it seems to always remain just challenging enough to make that perfect game elusive, but not so challenging that success seems impossible. 

However, as with Lost In Migration, I do worry that there is a plateau effect at work here as well.  No matter how many times I played, I didn't see myself doing that much better than my initial playthrough. I can complete all five levels, but by the fourth, I invariably start to lose track of a fish or two before the end of the round.  I still end up with a decent score, but no amount of practicing could make me consistently able to score a perfect game on the most challenging levels.  Attention is a something that can be improved over time, and I would like to believe that Playing Koi is a game where repeated practice could really pay off, but the feeling that I had reached my own personal limit made me doubt that the game could transform me into some kind of heroic, super-human master of attention like a Jason Bourne or Shawn Spencer.  Even so, I readily concede feeding fish has never been so fun.

Word Bubbles (Flexibility)
Lumosity says: used for language, tip-of-the-tongue word finding, thinking outside of the box, and for exercising information processing.

Described as Lumosity's most popular game, Word Bubbles gives players a three letter prompt and they are tasked with creating as may words as possible starting with those three letters.  Word length counts, and players only receive points for the first three words of the same length that they create.  Know at least a dozen six-letter words that begin with the letters d-r-e?  Well, you're out of luck because only three will count! While the limits on the number of words of any length can be inhibiting, it also introduces an element of strategy into the game.  Instead of just typing blindly every word you can think of, you get better scores for being able to "max out" each length category.   It's a small strategic element, but it certainly helps keep the game from feeling like a stale vocabulary lesson.

Word Bubbles challenges players to probe the recesses of their memories and find words, both common and uncommon, to fill out their list.  In this respect, the game is an exercise in both memory and vocabulary.  For example, who knew that I had the word "vertiginously" locked away in my little mind?  I didn't, but when v-e-r was my Word Bubbles prompt, out it sprang like a fully formed Athena.  Who knows, maybe I'll even get to use it conversationally at some point, now that it's been moved up into my conscious memory.  But aside from the occasional serendipitous discovery of a word you though you had long lost, Word Bubbles doesn't obviously contribute to expanding your vocabulary. If a prefix stumps you, there's little in the game that will make you likely to succeed the next time the prefix comes around.  User CoolCat put it best in the player-provided tip section, "Active reading, such as texts, novels and newspapers, builds greater word vocabulary and better scores." In other words, it's what you do outside of the game that's most likely to help you succeed at Word Bubbles.

Brain Shift and Brain Shift Overdrive (Flexibility)
Lumosity says: used for multitasking, shifting your focus of attention, cognitive control, and for exercising information processing.

Brain Shift is a game that is supposed to help your brain shift gears from one task to another.  In the basic version of the game, players are shown both a letter and a number.  If this letter/number combination appear in the top box, players must answer yes or no as to whether the number is an even number.  If the letter/number combination appears in the bottom box, player must decide, yes or no, if the letter is a vowel.  In the Overdrive version of the game, the basic gameplay is the same, only there are four boxes, and players must respond yes or no to the following questions, depending on which box the letter/number combination appears in: is the number even, is the number odd, is the letter a vowel, is the letter a consonant?

The point of the game seems to be to train your brain to respond quickly to shifting evaluative criteria.  In order to be successful, player must remember what question each box asks of the letter/number combination and then correctly evaluate whether that letter/number combination meets the criteria.  And while the criteria are simple, it is in fact quite challenging to shift mental gears and provide a quick an accurate answer.  Practice certainly helps, and the best scores seem to come from those moments when your focus and rhythm are in perfect sync.

While it is satisfying to feel that sense of "flow" while playing the Brain Shift games, it's not clear if there are lasting or meaningful cognitive benefits to be derived from the game.  Multi-tasking is one of the defining buzzwords of the twenty-first century, but most research suggests that our brains are not especially designed for this function.  When we try to multitask, or constantly switch our attention back and forth between two tasks, we are in fact not really paying attention to either activity. We're not actually multitasking at all, we're simply distracted.  And it's not clear that our brains can be "trained" to be truly effective multitaskers.  So while the Brain Shift games are certainly challenging, it's not actually clear if meeting the challenge of these games is going to make a player better able to "adapt to changing circumstances" or "respond to the spontaneity of daily life," as Lumosity claims it will.

Ho-hum.  It's like being in grade school all over again.
Raindrops (Problem Solving)
Lumosity says: used for mental calculations, increased aptitude with numbers, making estimates, and for exercising quantitative reasoning.

As was the case with almost all of the games in the "Problem Solving" category, Raindrops is basically a timed math test.  Equations are encapsulated in rain drops which fall slowly to the ground.  Players must solve the equation and enter the correct answer before the raindrop reaches the ground.  The better player do, the more raindrops fall and the more challenging the equations become ( 5+1 versus 34-16).  In Raindrops, all math functions are fair game (addition, subtraction, multiplication, and division), which makes it somewhat more challenging that the other math games on the site. Scores are calculated base on the number of equations per minute solved, the total number of problems solved, accuracy overall, and accuracy under pressure.

There's little doubt that playing this game repeatedly should boost your mental math skills a bit.  When your multiplication tables are rusty, practice helps refreshen the memory.  And the speed component will definitely push finger-counters to brush up on their addition and subtraction skills. But these games won't make anyone a math whiz, and the claim that these exercises are even an exercise in the cognitive function of "problem solving" seems a bit of a misnomer.  Strictly speaking, player are solving math problems, but I tend to think of "problem solving" as a category of mental functioning to be something a bit broader than simply mathematical ability.  As simple math-trainers, these games will do the task, but there are plenty of other math-trainers available freely online.  Raindrops is neither overly fun nor overly exceptional in the realm of math games.

By the Rules (Problem Solving)
Lumosity says: used for pattern recognition, dissecting complex arguments, problem solving, and for exercising task switching.

By The Rules, and it's clone game, Word Sort, are games that seem to be pulled straight out of an SAT test.  In By the Rules, players are presented with a card that has a colored shape on it.  They then have to decide with this shape follows the hidden sorting rule for the round.  For the first card of each round, the player is simply guessing. But based on whether or not that guess is correct, the player can make an educated guess about how the second card should be sorted. For example, if the first card is a green circle, I might guess that it follows the rule.  If I am correct, then when the next card comes up as a red circle, I know that if this card follows the rule, then the rule is "circles."  If, however, that second card doesn't follow the rule, then the rule is "the color green." By process of elimination, players should quickly be able to figure out the rule for each round. The player moves on to the next rule after he or she sorts six consecutive cards correctly.

The game is essentially a logic puzzle, but it's a fairly simplistic one.  In fact, despite what I said above, I feel fairly certain that the SAT Reasoning Test offers much more challenging logic puzzles, for which the meager pattern recognition exercise employed in By The Rules wouldn't really prepare you.  As I've said with some of the other games reviewed above, Lumosity has clearly worked hard to ensure that their games are accessible to all levels of players.  There's nothing wrong with this--at least not a a starting point--but the problem is that many of these games never really push players to grow and excel.

Despite the fact that "adaptive learning" is supposed to be one of the innovations of the Lumosity site, it is quite disappointing to see that many of these games do not in fact adjust in difficulty as response to a player's performance.  Virtually none of them respond "on a moment-to-moment basis" as the site claims, and even those with a difficulty structure based on a level-up system don't necessarily become that much more difficult.  By The Rules, for example, simply adds more types of cards to the mix, like shapes colored in with a gradient, or some have no fill color at all but are just a shape outline.  This means that it takes longer to deduce the rules because there are more possibilities than before, but the cognitive processes used to uncover the rules hardly change at all.

This kind of soft-handed approach to increasing difficulty really undermines the entire purpose of the site.  The brain, just like our muscles, grows in response to challenge.  Lifting a 2 pound weigh 100 times isn't going to do as much for your physical strength as lifting a 20 pound weight ten times.  Likewise, we may be more likely to play a game repeatedly because cognitively, it's relatively easy for us, but the fact that it has become easy tells us that we're not learning from it anymore, and it is almost certain that we're not gaining the kind of mental fitness that will transfer into other mental activities.  In the case of By the Rules, I would be downright shocked if there were actually clinical proof that this game was in any way improving my ability to dissect complex argument, even though Lumosity claims just such a mental benefit. I suspect a virtual chess game would serve players better as a tool for improving pattern recognition and problem solving, although it wouldn't be as slickly marketable.


Even for those of us who already believe that games posses the ability to be powerful learning tools, I'm always wary of products that seem to invoke the power of "science" in their advertising, as though just speaking the very word can serve as a magical incantation, giving legitimacy and validity to all of one's to one's pie-in-the-sky claims.  My experience with Lumosity did nothing to quell those suspicions.  In fact, Lumosity seems to be to "brain fitness" as the Nintendo Wii is to physical fitness. If you're serious about fitness, you are probably better off running on a treadmill instead of hula hooping on WiiFit, and if you're serious about keeping your mind sharp, you'll benefit just as much, if not more, from reading a book, learning or practicing a foreign language, doing a few math problems each day, or playing your favorite casual puzzle game, be it a crossword puzzle or Tetris.

There is no doubt that people can and do learn from games and that some games may be particularly well-suited to improving certain categories of cognitive functioning like attention, memory, spatial orientation, perceptual speed, etc.  But saying that Lumosity's games are any more sophisticated or better at increasing "mental fitness" or "intelligence" than the hundreds of other games that one might play strikes me as all too reminiscent of those diet pills that you see advertised on late night tv: "Clinically tested! Eat all you want and never gain weight!"  If you feel like you don't get enough mental stimulation from your daily life, you like puzzle games, and you need the idea of a structured "training program" and incessant emails reminding you to "work out" your brain, then by all means, a subscription to Lumosity might be good for you.  But if you think Lumosity games are going to change your life or make you the brainiac you always fantasized about being, then expect to be disappointed.

Lumosity is sleek, polished, and provides a reasonably entertaining diversion for someone who already enjoys puzzle games. I personally felt that there wasn't really enough diversity in the games to justify the $15/month price tag, but I'm also not a huge puzzle-game player, so that might be a matter of personal preference. My biggest reservation about Lumosity is not a complaint about the quality of the games as casual puzzle games, it's that Lumosity markets them as so much more than that.  If Lumosity were content with being a casual, if vaguely educational, puzzle game site, it would be easy to endorse their product as some harmless fun.  I could readily concede that their games might be a bit more of a cognitive workout than the average round of Bejewelled or Plants vs. Zombies.  However, it concerns me that Lumosity takes their claims about proven cognitive benefits too far, and their attempts to market themselves as working in partnership with major medical and academic researcher makes their their aggressive data collection practices seem outright predatory.

The error page you see when you seek to learn more about Lumosity's "proven results." 

After doing some independent research, including searches of major medical and neuroscience research journal databases, I was only able to locate one peer-reviewed study that specifically tested the cognitive benefits of Lumosity games.  You can read the published article for yourself here.  In a nutshell, this particular study focused on cognitive rehabilitation of children suffering from post-chemotherapy cognitive impairment, also referred to colloquially as "chemo-fog." As such, the study was designed less as a general test of the effectiveness of Lumosity as a brain-strengthening regime, and more about its capacity to serve as suitable rehabilitative therapy for children fighting cancer.  The findings did indicate that some Lumosity games seemed capable of rehabilitating  specific cognitive skills back to their pre-cancer-treatment levels, which is obviously a promising finding for children undergoing chemotherapy, but it is a considerably more modest finding than the claims to generalized brain-boosting power which are plastered all over the Lumosity website.  Also notable is the study's finding that in several categories, training with Lumosity games had no effect on the children's cognitive skills.

While consumers can easily avoid Lumosity's shady shenanigans and look elsewhere for their mental stimulation, educational game developers should take note of the effect that aggressive marketers like Lumosity may be doing to our industry. In the short run, Lumosity may be successful at building up a user base and even recruiting new players through their ads.  However, I cannot help but think that a backlash will be inevitable. There is a strong case to be made for the educational value of games, but in the long run, overstating the efficacy of our games is certain to increase the cynicism of the very people we want to play them.  In light of the lack of genuinely rigorous clinical research, Lumosity appears to playing fast and loose with the claims of being "clinically proven" and that their program represents "science that works." These claims may not be entirely spurious, but they're certainly exaggerated.  And while there will always be some people who will fall for pure hype, all present and future educational game-makers will be hurt if over-the-top marketing claims destroy consumer trust.  If more "brain game" companies like Lumosity continue to use the mantra of "science" as a means of distracting consumers from their overtly commercial interests, both researchers and game developers may find that they no longer have an audience for their work, no matter how innovative or well-intentioned it may be.

Pros: A varied selection of polished puzzle games and brain-teaser style games. Training programs make it easy to "work-out" every day. Tracks your high scores and lets players see their progress over time.

Cons: The number of unique games is significantly less than 35, as many games are very similar if not identical in gameplay. Games are marketed with vague claims about "stronger brains" and "mental fitness" that are misleading and pseudoscientific. Enrolled users become unwitting guinea pigs for targeted marketing and third-party research. 

Score: C-

Lumosity Game List

Penguin Pursuit, spatial orientation (Speed)
Rotation Matrix, spatial orientation (Speed)
Speed Match, information processing (Speed)
Spatial Speed Match, information processing (Speed)

Birdwatching, visual field (Attention)
Eagle Eye, visual field (Attention)
Lost in Migration, focus (Attention)
Observation Tower, visual field (Attention)
Playing Koi, focus (Attention)
Space Junk, visual field (Attention)
Top Chimp, visual field (Attention)

Addition Storm, arithmetic (Problem Solving)
By the Rules, logical reasoning (Problem Solving)
Word Sort, logical reasoning (Problem Solving)
Chalkboard Challenge, quantitative reasoning (Problem Solving)
Division Storm, quantitative reasoning (Problem Solving)
Multiplication Storm, arithmetic (Problem Solving)
Raindrops, arithmetic (Problem Solving)
Subtraction Storm, arithmetic (Problem Solving)

Face Memory Workout, face-name recall (Memory)
Familiar Faces, face-name recall (Memory)
Memory Lane, working memory (Memory)
Memory Match, working memory (Memory)
Memory Match Overload, working memory (Memory)
Memory Matrix, spatial recall (Memory)
Moneycomb, spatial recall (Memory)
Monster Garden, working memory (Memory)
Name Tag, face-name recall (Memory)
Rhyme Workout, working memory (Memory)

Brain Shift, task switching (Flexibility)
Brain Shift Overdrive, task switching (Flexibility)
Color Match, response inhibition (Flexibility)
Disconnection, task switching (Flexibility)
Disillusion, task switching (Flexibility)
Route to Sprout, planning (Flexibility)
Word Bubbles, verbal fluency (Flexibility)
Word Bubbles Rising, verbal fluency (Flexibility)


  1. AnonymousJuly 17, 2012

    Wow! Thanks so much - i've been looking for some sort of unbiased review of Lumosity for a while now. Most reviews seem designed to promote Lumosity and carefully skirt around providing actual evidence of its utility, other than user reviews. While i enjoyed the games during a 3-day trial, their marketing strategies were rather annoying. I was especially suspicious when i got no clear answer to my query to them on studies proving the benefits of these exercises. Rather, i got a somewhat manipulative email suggesting i wasn't using the games to their full benefit outside of the Lumosity "program".

    Your detailed review of the Lumosity games, questioning of the "science" behind these games and highlighting of their unethical usage of player data are well worth reading. I am now happier than ever than i didn't fall for their sales gimmick. Thank you again, for your objective and in-depth review - it has helped me clarify my stand on this product.

    1. I'm very happy that you found my review helpful, and I appreciate you taking the time to comment here!

      I am rather saddened that they tried to deflect your questions about the underlying scientific research by accusing you of using the product incorrectly. That kind of response strikes me as entirely antithetical to the true spirit of scientific inquiry. Hopefully Lumosity will one day be pressured into being more open with their customers about all their scientific claims.

  2. AnonymousJuly 30, 2012

    After reading your insightful review of the Lumosity service, I was struck by one component of the service that was underdiscussed: competitive comparison.

    One of the driving aspects of Lumosity's curriculum is found in comparing your daily results with other profiles in your age group (generally condensed by decades). For students and people concerned with how their brain-"power" translates into a social arena, the drive to score higher than the average is very alluring. From my own observations, I have seen some Lumosity subscribers go directly to the "How you compare" link after completing a daily regimine for the sole sake of confirming that they were above the average. By my understanding, there have been no significant statistics gathered from the bottom end of these studies, nor have I witnessed people (outside of mischevious adolescents) blatantly attempting to produce poorer scores -- which suggests, while being somewhat counter-intuitive for a "cognitive improvement program", that there is not a great deal of subscriber understanding for how the foundation of the Lumosity curriculum will change for an ever-improving subscriber score base.

    For example, unlike the World of Warcraft (which offered more challenging activities and, arguably, "scores" [in the form of achievements] for higher level players), Lumosity continues to base it's score system on an inflexible SAT-esque score range, setting the high point at 1600 and modifying subscriber play time (through their mysterious algorhythm) to somehow reflect higher scores of BPI over a period of months, despite when one may score a 1500+ in their first game. Aside from the profits driven by their marketing of subscriber data to third-party affiliates, it seems to me that the greatest (underspoken) motivation for Lumosity's continuing subscriber base isn't improving (or even maintaining) a certain level of "cognitive performance", but in drawing attention to how one's scores reflect in comparision to others. Those who reach the 90th percentile may be riding high (and may be prone to dropping the service from a lack of cognitive challenge) for a little while, but -- if my suspicions are correct -- when the majority of users maximize their own scores through dedicated play (i.e. repetition), the scores will inevitably level out, thus reducing the percentages of those at the top to the appropriate average -- or, if the scores do NOT reflect the averages of the specified age groups, remaining unchanged -- because, seriously, who wants to be at 89% when they were at 95% simply because other people have improved their scores?

    This pardigm presents a number of potential issues, though the one that stands out in my mind is the question of how Lumosity will adapt it's curriculum once its user-base reaches a point where the games' score threshold cannot be exceeded. When committed subscribers who have maximized their cognitive output (reaching scores of 1500+ through competitive play) are faced with declining percentages from an ever-increasing average of the majority of subscribers, how will they -- and how will Lumosity -- react?

    1. Personally, I didn't find the social/competitive aspects of Lumosity to be well enough integrated into the site experience to really affect how I used the site. In fact, I believe they got rid of the "friends" component of the site sometime after I wrote my review because it wasn't contributing much to user experience. But I think you're right that it remains to be seen whether their ranking system will, in the long-term, actually encourage or discourage continued play among their customers.

  3. Thank you so much for your review. I have to say: I had no idea that the site was technically not accessible or at least not approved for use by children and adolescents. Many of the studies that they claim back their game design are focused on children. As a clinical researcher focused on improving emotion regulation among children, I was impressed by Lumosity's affiliation with Stanford. However, that collaboration/partnership/what have you seems unethical given that people are paying to have data collected on them: how does Stanford rationalize that, given the strict guidelines put forth in each university's Institutional Review Board and national policies regarding protection of human subjects? In addition, they are preying on vulnerable individuals by marketing a program specific to those diagnosed with PTSD. The fact that they say this was guided by a researcher at UCSD is very misleading given that the games are no different from those in the basic course (I'm guessing their reply would be that the UCSD-based scientist (?) directed them on what games might be especially "helpful" to those with PTSD in improving their cognitive abilities - a very sketchy proposition). As someone familiar with the literature on cognitive functioning and cognitive remediation, some of their games are very much designed to improve working memory but few studies have found that these increased memory skills transfer to real-world activities. A compelling claim that many researchers make is that several of these games (or the tasks that many Lumosity games are based on) increase fluid intelligence. However, what the Lumosity literature doesn't make clear is that studies that find this "effect" are measuring fluid intelligence using only one particular test (a matrix task called Raven's) which is far from ecologically valid. Finally, I am interested in what clinical psychologists specializing in psychological assessment think about the wide availability of games that are ridiculously similar to the tests that psychologists pay a gazillion dollars to use on their patients (Wechsler tests of intelligence, executive functioning batteries like the D-KEFS or NEPSY). As psychologists we are prohibited (politically anyway) from throwing these tests in the trash for fear that they will be collected by someone and distributed, thereby tainting the utility of the tests. Isn't this what Lumosity is doing? Thanks again for your thoughtful review.

    1. I'm glad my review could be of help and I thank you for your insightful comments.

      I, too, find it disconcerting that so many researchers have been willing to extend their credentials in the service of Lumosity's unabashedly commercial interests, but I'm sure every researcher and research institute has their own motivations for supporting the site.

      Some very interesting points you raise about the whether tests these researchers use to measure things like "fluid intelligence" actually identify the kind of real-world effects that they imply they do, and whether people playing these games are somehow going to "taint" the results of any future psychological testing that they participate in. It would be nice to know how Lumosity would respond to those concerns.

  4. well written. kudos.

  5. An interesting article. I believe that the 'science' behind cognitive training is too young to definitively make claims on whether computer based training can potentially benefit us. I know that using games to practice things (mathematics, typing, word games) makes boring, repetitive tasks a lot more enjoyable, and you can improve your abilities in those tasks (mathematics, typing, word-recall). However, whether that makes you smarter... who knows.

    One things that makes me curious about Lumosity (and if I get the time, I'll contact them directly), is that the BPI calculation seems really bizarre. If I just do the core 5 assigned games per day, my BPI increases markedly. If I play around with multiple games (to make sure I've covered the different skills within a training type), my BPI flattens out. It therefore seems that to get a constantly good BPI, simply play less games! This goes against my thoughts that playing different games (for example, 'attention' games) should give you a more accurate BPI score. After all, if I was working out at the gym, using 2 or more exercises for each muscle group makes sense... wouldn't the same be correct for accurately rating an area of cognitive performance? Doing less, and rating higher proves littles, except that the BPI score seems useless.

    As far as cognitive training go, it would be interesting to follow research over the coming decades. Look at the work of Samuel Renshaw (tachistoscopic training) and the application of his work during World War 2. The APA website has an article entitled ‘Spotting the enemy’ – http://www.apa.org/monitor/2010/03/enemy.aspx . Some of the Lumosity training reminds me of this earlier concept (attention and shape recognition at speed). Also worth checking out is Harry Kahne. You can find his bizarrely named “Multiple Mentality” course online. There is discussion on various forums about people who have tried to apply Kahne's training with varying results.

    It stands to reason that practicing maths skills will make you better at maths, but not necessarily make you 'more intelligent'. It is the same as people who do bodybuilding to become better at fighting - it doesn't make someone a better fighter (althought the conditioning may give them some additional benefits). As far as transference of skills goes... it seems highly inconclusive from the current research.

  6. AnonymousMay 31, 2013

    I did find your comments interesting. My sister bought me a two-month subscription and I cannot develop a daily habit, but am playing as much as I can to not waste the gift. In response to some of your comments: The Eagle Eye game gets a little harder to score on as it speeds up and changes backgrounds. On familiar faces, I happen to be naturally good at remembering men's names/faces... weaker with women, but I manage. Familiar Faces excercise is a complete chore and a bore. I have no desire to keep playing it. The Penguin race game I am a hopeless failure at...yet I am the most directionally literate and skilled map-reader I have managed to meet. Schematics, maps, plans... no problem. Never need a GPS. Terrible with the Penguins. Don't know how it could possibly help since it's your fingers you re-teach how to operate the game, nothing having to do with using maps or turning them in your head. I can flip designs in my head, but I can't work the penguins. I have no plans to renew or keep using the site. I don't care how I compare to other players. I don't feel better when my scores go up - it does feel like work when I play games I don't like. And since I don't care about my score, I am happy to take zeroes on the games I lose interest in.

  7. I agree with many parts of the article. As someone who uses Lumosity almost exclusively to play speed match and who believes there to be some merit to the game I have a different take on that game.
    First, the comparison to a n-back test seems to miss the intent and even type of game it is. Therefore saying it is a "watered-down" version would seem pretty unfair.

    Frankly, I find speed match very challenging and I'm extremely good at it, relatively--best scores are in 11000s (giving me from 1620-1670 BPI on the game) and personal and unreached goal is 12000. Most people find improving in the game to be a struggle after familiarization despite hard work at improvement. It is odd you compare it to a relaxed old-fashioned arcade game when speed match is incredibly strenuous and frustrating. You also laud n-back later as feeling like a workout, yet you would not say Speed Match is? I guess it depends what your goal is. There is no final level or readily reached max-score plateau on the game, so speed match can almost always be improved. Everyone I know finds it to be work out, as do I. No matter how good at it I have gotten there is still a lot to be improved, too.

    It is a very simple game, as a game that would be focused on reaction time without the use of other skills would have to be.

    I believe this simplicity is the central strength of the game because it only has one part of your mind to use, and hopefully therefore allows you to improve that area. Most games allow you to improve by familiarization, learning tricks, or developing alternatives instead of improving the area intended to be exercised by the test. However, speed match is so simple the only way to improve is to improve something very fundamental. Moreover, when you try and "cheat" by predicting to aid in speed (even subconsciously) it penalizes you and forces you to perform honestly.

    That is the major reason that the best and simplest explanation I can conceive for improvements in Speed Match scores is actually due to some form of mental development, not the learning of the game.

    As far as more boisterous claims like the ones used in Lumosity, it is not worth considering Lumosity at their word. And as far as I know, there are little useful intelligence and brain measuring tools of validated meaning. While we can see Lumosity makes a painful mockery out of serious attempts at actual brain improvement most of the time (as you articulated at length) it does not mean all of their games are meaningless (I also hesitate to call speed match a Lumosity game; there are many analogs developed by others, some being almost exactly the same).

    As it is, rather than only share a personal anecdote on my individual opinion on Speed Match, I would compare it to how we see other activities. Things like playing an instrument, studying for exams, staying mentally active; i.e., things which in general stimulate the brain, and are difficult. If speed match, a game which allows you to perform isolated exercise for reaction responses, cannot lead to improvements in some mental faculty involving reaction time, it would be a depressing conclusion for people hoping for positive neuroplasticity changes based on mental activities. In my opinion it does allow for interesting training and positive neuroplasticity and I enjoy the straight forward challenge of the game.

    1. Hi JC,

      I think we're in agreement that not all Lumosity games are meaningless, but I don't think there is much evidence at all to support the conclusion that the majority, or even any of Lumosity's games are actually comparable to activities like learning a language or playing an instrument for improving neuroplasticity, and that's why I caution against them.

      Speed Match may challenge your reflexes, but so too will a game of Call of Duty, and Call of Duty will also improve your decision-making reaction time and skills at probabilistic inference. And how do we know this? Because there's peer-reviewed research to back that claim up--something that Lumosity still cannot truthfully say for almost any of its games.

      Lumosity's games may be better for you, cognitively speaking, than doing nothing at all. But I still say that's a pretty low benchmark for people who care about their cognitive health.