I’m not a serious polyglot, but I’ve tackled a handful of languages in just about every way a language can be learned: classroom, tutor, textbook, audio recordings, flash cards, software, and more. Learning languages was always a chore—until Duolingo. I looked forward to my lessons. And I was learning Italian! I walked through my home confidently talking about hiding a knife in my boot, when my master’s thesis was due, and how important it was to pay attention to the will of the people. Okay, so Duolingo’s sentences covered some strange ground. But surely, I figured, that would work to my advantage when I was faced with more mundane language demands as a tourist.

A week before we were to leave for Rome, my wife, Laurie, put me to the test. You’re at the airport outside Rome, she said, and you want to get downtown; how would you ask? I gaped like a fish. Words and phrases swam through my mind, but they didn’t add up to anything useful. Laurie switched to a restaurant scenario: “Do you have a table for four?” “I’d like two glasses of red wine.” I knew I had seen all the pieces in Duolingo’s sentences. But I was utterly unable to recall them and pull them together.

Panicking, I fired up Duolingo and almost instantly saw the problem. The app had made me a master of multiple-choice Italian. Given a bunch of words to choose from, I could correctly assemble impressive communiqués. But without a prompt, I was as speechless in even the most basic situations as any boorish American tourist. And this in spite of 70-plus hours of study.

But I still had a week. I got my hands on a self-study book, a travel phrase book, and a pocket dictionary, and started cramming. A funny thing happened: I started easily picking up what I hadn’t been able to get from Duolingo—grammar, vocabulary, and, most important, an ability to engage in simple conversations in typical situations. It seemed I had been getting something useful from my hours with Duolingo. The app had exposed me to a considerable vocabulary; I needed only minimal drilling with books to remember the words. Learning the verb conjugations was a breeze, too.

In the end, I did pretty well in Rome, engaging in simple, fractured semi-conversation in most of my encounters. Was that how the app was supposed to work?

I recently got in touch with Luis von Ahn, a co-founder and the CEO of Duolingo, to ask whether my experience was typical. I expected some defensiveness from him about my need to use books to get the conversational skills I had hoped to get from Duolingo. But instead he laughed and told me the app had done exactly what it was built to do. “The biggest problem that people trying to learn a language by themselves face is the motivation to stay with it,” he told me. “That’s why we spend a lot of our energy just trying to keep people hooked.”