LAS VEGAS – Damian Lillard overcame growing up in a tough East Oakland neighborhood. He was largely overlooked during college recruiting before playing for mid-major Weber State and eventually becoming an NBA star. No matter the odds, the two-time All-Star always seems to figure out a way to succeed.

And now with All-Star LaMarcus Aldridge and three other starters gone from the Portland Trail Blazers, Lillard faces another challenge: leading the franchise during its rebuilding process.

View photos Damian Lillard signed a five-year, $120 million extension with the Trail Blazers. (NBAE/Getty Images) More

"Everything has worked in my favor. I had all these great experiences and accomplished so much in three seasons and now your team takes a step back," Lillard told Yahoo Sports at the NBA's Las Vegas summer league. "And now it kind of knocks you back to square one. Now people are saying, 'His team is terrible.'

"Everybody is saying things like that. It puts me back to my natural habitat where I got to prove that we're better than that. We can accomplish more than people might think."

The Trail Blazers won 51 games and the Northwest Division last season, but lost in the first round of the playoffs. Throughout the postseason, there was plenty of speculation that Aldridge could leave during free agency. He did just that signing a four-year, $80 million deal with the Spurs to join a championship-caliber team led by Tim Duncan, Tony Parker, Kawhi Leonard and coach Gregg Popovich. Aldridge turned down an extra contract year to depart from the Blazers.

Aldridge told Lillard he was leaving Portland before his final decision to sign with San Antonio became public.

"We basically exchanged texts about how much admiration we have for each other," said Lillard, who signed a five-year, $120 million extension with the Blazers this week. "That change wasn't about me. I did express that I wanted him to be back. I told him I respected his decision. I respected that he told me before the news broke and I saw it on TV."

Was there anything more that Lillard or the Blazers could have done to keep Aldridge?

"I'm not sure there is," Lillard said. "I think he needed a change and wanted to go some place where he felt he had a better chance to win."

Aldridge is one of four Blazers starters to leave this offseason. Portland traded small forward Nicolas Batum to the Charlotte Hornets and lost center Robin Lopez to the New York Knicks and guard Wesley Matthews to the Dallas Mavericks in free agency. Reserve guard Steve Blake was traded to the Brooklyn Nets and guard Arron Afflalo signed with the Knicks. Lillard is the only returning player among the Blazers' top six scorers last season.

View photos Blazers general manager Neil Olshey (left) is rebuilding the team's roster. (NBAE/Getty Images) More

Story continues