With the services of three All-Stars and command of the biggest payroll in NBA history, Tyronn Lue has been positioned for success since becoming the Cleveland Cavaliers' head coach in January 2016.

Even so, the setup comes with challenges that no other coach in the league has to face, according to Lue.

"It's the hardest job, by far," Lue said on ESPN's NBA Lockdown podcast, which came out this week. "It's the hardest job. But I've been through a lot of tough things in my life anyway, and I just try not to listen to the outside noise -- to the media and what they have to say. As long as I have the support of [Cavs general manager] David Griffin, [principal owner] Dan Gilbert, my players, the city of Cleveland, the state of Ohio, as long as I have that support, that's all that matters."

The difficulty, Lue said, comes from not trying to win games, but from all the scrutiny that comes with the process.

"When I was a player, probably about seven to eight years ago, I'm trying to attack every reporter that said something," Lue said. "But now I've come to the realization that it's not worth it. People are going to say what they have to say, and they got to sell books or they got to sell stories. They have to do it. That's their job.

"But it's just, I don't like it when they make stuff up. ... If I didn't do a good job or I didn't do something [correctly], then I understand that. That's your job. You got to write it. But when you make stuff up, that's the part that I don't get that kind of makes me mad."

Lue notably took exception to how his benching of Kyrie Irving and Kevin Love during the Cavs' 26-point comeback in Game 3 of their first-round series against the Indiana Pacers was portrayed.

The Cavs coach said he is generally averse to being in the spotlight.

Tyronn Lue says the extra scrutiny that comes with being Cavaliers coach makes it the toughest job in the NBA. Andrew D. Bernstein/NBAE/Getty Images

"I don't like the media attention. That's why this job is just so tough because you're out in front all the time," Lue said. "It's tough because whatever you say, if you say one wrong thing -- especially leading this team -- it goes viral. It's a big deal. That's why in the huddles, a lot of times when we're miked up, I never say anything. When the camera comes over there and they're all like, you got to talk to the team like, 'You gotta [this and that]!' Our team? They don't want to hear that anyway."

Perhaps Lue's most controversial decision this season has been resting his stars -- LeBron James along with Irving and Love -- at the same time on several occasions. Then again, Lue also was criticized for playing the 32-year-old James too much.

"That's why I know that the media has to do what they have to do, because when you rest him, they cry, 'Oh, he should play! [Michael] Jordan would never sit!' And then you play him 42 minutes: 'He's playing too much!'" Lue said. "Like, what? So if you don't play, you mad. If you play too much, you mad. ... And I've come to the realization that it is what it is. That's why I just try to tune the outside noise out."

Lue said his relationship with associate head coach Larry Drew, who was previously a head coach in Atlanta and Milwaukee, has helped him handle the pressure that comes with the gig.

"Last year and probably the year before when Coach [David] Blatt was here, I just got so caught up in trying to be perfect and do the right thing every single time that you never take time for yourself or you never have time to enjoy it, and I think this year, I think you can see I'm kind of different," Lue said. "I'm enjoying it more. I'm trying to enjoy it more and not let the outside noise get to me. ... [Drew] taught me how to just take a step back and you got to live and take time for you and enjoy it, and if you don't, it will kill you."

Lue also said his quick ascension has contributed to the challenges he faces roaming the sideline in Cleveland.

"I just got thrown right in the fire, not knowing how I wanted to coach," he said. "I had a thought of what I wanted to do as a coach, but when you're coaching LeBron James and Kyrie Irving and Kevin Love and the pressures of winning a championship, it's different.

"Like, I didn't have a chance to start lower and work my way up to the top. I just jumped right up to the top, right into the fire, and that was tough. That was the hardest part for me because I didn't get a chance to grow. I didn't get a chance to even like have time to prepare. It was just a tough situation, and it was all God. I think it was all God to get through that and win a championship and be down 3-1 the way we did it. I mean it was just, it had to be heaven sent."

Lue revealed that the Cavaliers reached out to Kevin Garnett -- who was working as a consultant with the Clippers and Bucks after retiring last season -- about coming to play backup center after Andrew Bogut suffered a season-ending leg injury.

"I was like, 'Man, you should come back and play for me.' He was like, 'Man, you all have a lot going on over there,'" Lue said. "That was before we hit our stride like we're playing well now. He was like, 'If you and [James] Posey were still playing, I would come.' But he said, 'But y'all are coaching and y'all are going through what you're going through.' He said, 'Ah, I'm going to sit this one out.' I said, 'OK. We'll call you next year.'"