ATLANTA -- Josh Pastner isn't promising a quick fix as Georgia Tech's basketball coach.

Hired away from Memphis on Friday, Pastner knows the Yellow Jackets face a couple of years of rebuilding before they can expect to compete at a high level in the Atlantic Coast Conference.

"I mean, look, it's not going to happen overnight," Pastner said. "We're going to have to take a step back to take two steps forward. That doesn't mean we're not going to try to win every game next year and compete, but it is what it is."

Pastner's deal with the Yellow Jackets is worth $11 million over six years, a source told ESPN's Andy Katz. He will owe Memphis $500,000 for leaving the job.

Pastner was 167-73 with four NCAA tournament bids in seven years at Memphis. He did not finish his stay with the Tigers on a high note. Memphis was a combined 37-29 the past two seasons, failing to land an NCAA invitation each year. It finished 19-15 this season.

Josh Pastner went 167-73 at Memphis, but the Tigers were 37-29 over the past two seasons without making the NCAA tournament. AP Photo/Sue Ogrocki

Pastner is seeking to dispute that he's difficult to play for. As of last summer, 10 players had transferred.

"I don't know why anyone would think it's an issue of Memphis, because college basketball has become where it's just such a high transfer rate," he said. "Last year over 700, and this year it could get to 900. I just think it's the culture of men's college basketball. There's no other sport like it."

Despite the downturn of the Memphis program the past two seasons, which included a drop in attendance, Pastner comes to Georgia Tech with a reputation as a strong recruiter.

He already has recruiting contacts in Atlanta. Two of his heralded recruits at Memphis were signed from Atlanta: 2015-16 senior Shaq Goodwin and Jelan Kendrick, although Kendrick later was dismissed from the team by Pastner before playing at Mississippi and UNLV.

Expectations at Memphis were high this season after Pastner landed a highly ranked recruiting class. Instead, a disappointing year left Pastner's status at Memphis in doubt.

He would have been owed $10.6 million if fired by Memphis, and that helped to save his job.

Memphis president David Rudd and athletic director Tom Bowen said March 18 that the school was sticking with Pastner but would "make the necessary investments and changes" so it could "compete at the highest level."

After Pastner's departure, Bowen wished Pastner well in a statement.

"Coach Pastner guided our program with integrity, high academic standards, and a commitment to the Memphis community," Bowen said.

Brian Gregory, who was fired March 25, was 76-86 in five seasons, including 27-61 in the ACC. This year's team finished 21-15, including two wins in the NIT. Gregory needed to make the NCAA field to possibly save his job.

The Yellow Jackets haven't received a bid to the NCAA tournament since 2010, and Pastner will inherit a young roster. Leaning heavily on transfers, this year's Georgia Tech roster included five seniors, including top scorers Marcus Georges-Hunt and Adam Smith.

Georgia Tech will owe Gregory more than $1.3 million for the final two years of his contract. The school also must pay Gregory's predecessor, Paul Hewitt, $2.7 million over the next three years, the remainder of a $7.2 million buyout he received after being fired in 2011.

The financial strain was impacted by dwindling attendance at McCamish Pavilion. This season's average home attendance was 5,993, down from last season's average of 6,201.

Athletic director Mike Bobinski said he and Pastner first met in person last weekend at the Final Four in Houston. Bobinski wouldn't say how many other candidates he has seriously discussed the Georgia Tech job with.

"The coaches I was able to speak to throughout this period believed completely that we have a tremendous opportunity to build a championship program here at Georgia Tech," Bobinski said. "That doesn't mean it's going to be easy or that it'll be immediate. But we have the elements in place to do that at a very high level."

Pastner received a raise to $2.65 million in 2013, when Memphis finished 31-5 with a win in the NCAA tournament.

Pastner played at Arizona from 1996 to 2000 before working at his alma mater as a recruiting coordinator, graduate assistant and assistant coach from 2000 to '08. He was a Memphis assistant before being named coach on April 7, 2009.

Pastner enjoyed a strong start as coach. He is one of only three coaches to lead Memphis to four straight NCAA tournaments (2011-14), following Dana Kirk and John Calipari.

After winning two straight Conference USA championships with Pastner, Memphis fell to third, fifth and seventh in the American Athletic Conference the past three years. This year's team was seventh in the conference at 8-10.

The Associated Press contributed to this report.