Soon there will be only three black CEOs leading America's largest companies.

Kenneth Chenault, the longtime chief executive of American Express, is retiring early next year. Unless another company hires a black CEO, that will leave Merck, TIAA and JCPenney as the only businesses in the Fortune 500 with black leaders.

And all three of those leaders are men. There hasn't been a black woman CEO in the Fortune 500 since Ursula Burns left Xerox last year.

The number of black executives leading America's largest companies is always low -- though three is particularly staggering. In 2007, there were seven black CEOs heading Fortune 500 companies. The number was six in 2013, and five in 2015.

Chenault announced Wednesday that he will step down from the top job at Amex after 37 years, including 16 as CEO.

Leaders of pro-diversity groups were saddened by news of Chenault's departure.

Executive Leadership Council president and CEO Ron Parker said in a statement that "Ken has always been a true model of courageous leadership, operating with the utmost integrity," adding, "while we are excited to learn about Ken's next chapter in life, we are disappointed that his decision now leaves us with only three black CEOs running Fortune 500 companies.

Cid Wilson, President and CEO of Hispanic Association on Corporate Responsibility, noted that people of color "looked to [Chenault] as a beacon of what we as a community can do."

"His success...helped to set the example on Wall Street, on Main Street and throughout our education system that you too can succeed and one day become the next Ken Chenault," Wilson said.

The remaining three black CEOs in the Fortune 500 are Merck's (MRK) Ken Frazier, TIAA's Roger Ferguson and JCPenney's (JCP) Marvin Ellison. Frazier is the only black CEO at a company in the Dow 30.

Linda Akutagawa, President and CEO of Leadership Education for Asian Pacifics, wondered, "what's that going to do in terms of the next generation of leaders of color?"

"My challenge to companies, and in particular to American Express," she said, is to look "at their pipeline to make sure that there are people of color," who could take on greater roles in the future.

With so few black leaders, each exit is noteworthy.

Burns left her post as head of Xerox (XRX) last year when the company split in two, and Rodney O'Neal retired from his role as CEO and president of Delphi (DLPH) in 2015. Don Thompson stepped down as CEO of McDonald's (MCD) in 2015 as well.

The lack of diversity in corporate America extends to the boardroom: Just 5.6% of board seats at Fortune 500 companies are held by black men, and just 2.2% held by black women, according to the Executive Leadership Council, which promotes diversity in corporate America.

Wilson and Akutagawa both pointed to the low numbers of minorities holding board seats as part of the problem. "That's really where people are bring plucked from for [executive] jobs," Akutagawa said. "It's all connected."

"Corporate America has a lot of work to do to improve the diversity of their corporate boards," Wilson said.

Wilson says that corporations should think of diversifying as a business strategy. "If corporate America wants to be relevant today and in the future, they have to make diversity and inclusion not just a check box," he said. "It must be a business imperative, and that includes having people of color...rise up the ranks."