Tom Barrack, former Deputy Interior Undersecretary in the Reagan administration, delivers a speech on the fourth day of the Republican National Convention on July 21, 2016 at the Quicken Loans Arena in Cleveland, Ohio.

An activist investment firm is looking to oust Tom Barrack from his role as CEO of real estate and investment management company Colony Capital, saying his ties to President Donald Trump represent "significant personal distractions that are a blemish on the reputation of the company."

Blackwells Capital, which owns 1.85% in Colony, said Tuesday it plans to nominate five candidates to the board. It said the move comes after private efforts to engage with Colony "regarding badly needed business improvement initiatives and corporate governance reforms over the course of the last year."

Barrack had already agreed to hand over the role to a successor, Marc Ganzi, in 2021. But Blackwells isn't satisfied with that time frame or choice, noting that Ganzi is Barrack's friend.

Los Angeles-based Colony Capital is an international investment firm with more than 500 employees operating in 17 offices around the world, according to its website. Its market capitalization is $2.4 billion.

Blackwells sent a letter detailing its gripes to Colony's board Aug. 29, and disclosed it Tuesday. Read the letter here.

The move puts another spotlight on Barrack's decades-long relationship with Trump. Barrack helped chair Trump's Presidential Inaugural Committee, which has been investigated by federal authorities.

Colony issued a press release in response to Blackwells' letter, which said the firm "is already sharpening its focus on the digital infrastructure and digital real estate sector."

Colony said it has already added "three qualified and experienced independent directors" after entering into a cooperation agreement with Blackwells.

Deputy inaugural chairman Rick Gates pleaded guilty to federal charges stemming from special counsel Robert Mueller's probe. Among its concerns, Blackwells cites the addition of Gates to Colony's payroll without a clear role. "It is unclear to us whether Mr. Gates delivered any legitimate services to Colony during this period during which he was paid $20,000 per month," Blackwells' letter said.

Gates testified against other top Trump aides such as former campaign chief Paul Manafort and advisor Roger Stone – both of whom have been convicted in federal cases.

"It appears Mr. Barrack has used friendship and politics as criteria for business deals and for allocating shareholder capital," Blackwells wrote, citing Barrack's ties to Gates. A lawyer for Gates did not immediately respond to a request for comment on the letter.

Barrack previously agreed to hand over documents to the House Judiciary Committee's probe into Trump's dealings.

"Whatever their merits, these investigations certainly have been a distraction for Mr. Barrack and may blemish the reputation of the Company, if Mr. Barrack remains Colony's CEO," Blackwells wrote.

A June 2018 report said the company had raised more than $7 billion in investments since Trump was nominated for president, nearly a quarter of which reportedly came from the Persian Gulf.

Those investments, though, haven't buffered the company's share prices, Blackwells said, also noting a three-way merger between Colony Capital, NorthStar Realty Finance and NorthStar Asset Management — which closed January 2017. Shares of Colony have fallen roughly 18% over last year, according to the FactSet.

"Colony's Board has given Tom Barrack too much deference and too much latitude for too long. His continued, imperial reign over the Company damages its credibility and business prospects and creates a tremendous overhang on the stock," Blackwells wrote. "It is time for Mr. Barrack to go and for the Board to be reconstituted with truly independent and experienced business leaders, who can provide effective and constructive oversight."

Barrack did not immediately respond to requests for comment.

— CNBC's Kevin Breuninger contributed to this report.