Kevin Hart is one of the most successful comedians in America and one of the highest paid celebrities in the country too. He has headlined at major venues like New York City's Madison Square Garden, where nearly 19,000 people flocked to watch him perform in September.

But Hart didn't always inspire laughs and roaring applause. In fact, he remembers one pivotal moment in his career, when a power player in the comedy world suggested he quit the profession altogether.

"Did I have my moments where there was a question of whether is this it? Is this the career? Of course," Hart said in October at the WSJ Tech D. Live conference.

Hart recalls feedback he received early in his career from Lucien Hold, a comedy club talent booker and manager known for discovering Eddie Murphy, Chris Rock and Jerry Seinfeld, among other comedic icons.

"If this guy said you were funny, then you were funny," Hart says. "If this guy said you had it, you had it. And I worked so hard to get to this guy, and I got to him."

But the result wasn't what he had hoped.

"I performed for him, and he told me to my face that I should really think about finding another profession," Hart says. "He told me to my face that he didn't feel that comedy was for me.

"And I'm sitting in the chair, I'm in his office, and I'm waiting for the joke. I'm waiting for the punchline, and there wasn't one."

Hart remembers that it was a blow to his ego, and he had to grapple with what to do with that feedback, and what next steps to take.

"I had put so much pride and wear and tear on myself to get that information. And then when I got it, it wasn't what I expected it to be," Hart says. "And that could have been a damaging moment, but those are the moments I choose to let go of."

He learned a valuable lesson too: "Everyone will always have an opinion, everyone will always have something that they can say, but your action is based on your will and want," Hart says.

"So as long as you have the get up and go to just do, and never stop, then you'll be fine. When you allow others to dictate your movements, then you'll always be affected."

Don't miss: How Kevin Hart went from sleeping in the hallway of a one-bedroom apartment to making millions

Like this story? Subscribe to CNBC Make It on YouTube!