Five years later, I’ve spoken with more than 150 authors for “By Heart” (and compiled Light the Dark, a collection based on the series). The conversations have frequently—by total chance, but with spooky accuracy—highlighted my own creative ups and downs. I’ve also learned that these solitary, patient creatures, whose books can take the better part of a decade to complete, tend to have something in common.

More than knockout sentences, more than their grasp of human character, more than anything that might broadly be termed “craft,” novelists are masters of one skill primarily. Their genius lies in an ability to suspend their skepticism over the long haul, to persist in the belief that—no matter how hard things get—the work is meaningful, and worthwhile, and will one day pan out.

* * *

As my interviews got underway, I discovered something surprising: The artistic process never seems to get easier, not even for the most successful, famous authors. They, too, wasted months of time chasing down material that ended up being no good. They, too, were sometimes wracked by self-doubt. They, too, also sometimes felt a sudden, sweeping urge, as bold as lust, to give the whole thing up. A few glowing reviews in the Times won’t change any of that.

“The job of writing is pretty uphill most of the time,” said Mark Haddon, whose best-selling novel The Curious Incident of the Dog in the Night-Time went on to become a critically acclaimed Broadway play, when we spoke:

It’s like climbing a mountain—you get some fantastic views when you pause or when you get to the top, but the actual process can be tough. … I wish I could enjoy the process more, but I think I’ve come to accept that for it to work, I have to be uncomfortable.

What makes the process so difficult? I think it’s the nagging feeling that the words aren’t enough, the painful recognition that your language still falls far short of the beauty and complexity you’d wanted to spill across the page.

That’s true even for someone like Khaled Hosseini, the author of The Kite Runner, whose books have sold tens of millions of copies. For him, disappointment is baked into the experience of authorship, and even the finished product rarely measures up to the initial gleam of inspiration. “You write because you have an idea in your mind that feels so genuine, so important, so true,” he said:

And yet, by the time this idea passes through the different filters of your mind, and into your hand, and onto the page or computer screen—it becomes distorted, and it’s been diminished. The writing you end up with is an approximation, if you’re lucky, of whatever it was you really wanted to say.

That’s the killer: that gap between intention and output. You don’t have to be an artist to understand this. Most people wake up in the morning fully intending to be their ideal selves. To finally get themselves to the gym. To be a better student, a better parent, a better citizen, a better friend. That’s why it’s so painful to fail, as inevitably happens: It hurts to feel the distance growing between who you are and what you wish to be.