Former Indianapolis Colts head coach Jim Caldwell plans to join the offensive coaching staff of the Baltimore Ravens, a league source told ESPN NFL Insider Adam Schefter.

The Ravens announced last week that Cam Cameron would return as offensive coordinator, but they still need a quarterbacks coach for Joe Flacco. Caldwell now appears to be the choice.

Caldwell had visited Pittsburgh this past weekend. The Steelers are looking to replace Bruce Arians, who retired as offensive coordinator in Pittsburgh before taking the same position with the Colts.

Caldwell was fired after a 2-14 season in Indianapolis during which the team was without quarterback Peyton Manning, who missed the entire year while recovering from neck surgery.

The Colts fired Caldwell after three seasons. The team went to the Super Bowl during Caldwell's first year, but this year locked up the No. 1 overall draft pick with a horrid performance that also cost team vice chairman Bill Polian and his son, general manager Chris, their jobs.

Caldwell now joins a Ravens team that narrowly missed a trip to the Super Bowl after a loss to the New England Patriots in the AFC Championship Game.

After its season-ending loss, the Ravens watched as first-year defensive coordinator Chuck Pagano was hired to replace Caldwell as head coach in Indianapolis.

Information from ESPN NFL Insider Adam Schefter and The Associated Press was used in this report.