Even as Virginia Democrats on Sunday began considering impeachment proceedings for Lt. Gov. Justin Fairfax over sexual assault allegations, a group of Richmond activists rallied to keep the heat on Gov. Ralph Northam.

“Do not ask me about Justin Fairfax’s resignation before you give me Ralph Northam’s resignation,” Chelsea Higgs Wise, a social worker and host of the “Race Capitol” podcast who campaigned for Northam in 2017, told The Daily Beast at a demonstration in downtown Richmond. The rally was organized by Race Capitol and RVA Dirt, a grassroots website run by three women who primarily cover municipal politics, often live-tweeting Richmond City Council meetings.

Wise, an organizer of the event, had gathered with about 40 other people—both black and white—at the site of Lumpkin’s Jail, where enslaved people were sent before being auctioned off in one of the country’s largest slave-trading markets in the 19th century.

The site of the rally on Sunday took on added significance after Northam, already under fire for yearbook photos showing a man in blackface and another in a KKK hood, drew more fire for referring to the first Africans brought to Virginia in 1619 as “indentured servants” in an interview that will air Monday on CBS This Morning.

“I feel duped. He’s never been able to talk about blackness,” Wise said.

The Richmond rally came as much of the attention in Virginia’s leadership scandal had shifted to Fairfax, with a Democratic legislator announcing he will file impeachment articles against him Monday. A second woman came forward over the weekend to accuse Fairfax of raping her when they were students at Duke University in 2000.

Fairfax says all contact with both women was consensual, has called on the FBI to investigate, and has vowed not to step down.

At the rally, one of the speakers said getting rid of Northam should be the No. 1 priority.

“The momentum is to get Northam to resign,” said Rebecca Keel, a policy advocate for a youth-focused racial justice initiative in Richmond. “We need to stand by survivors,” she added, but “Northam needs to go first.”

In the CBS interview, Northam said he was “not going anywhere” and portrayed the past week of scandal among Virginia leaders as a learning experience. He said he believed that he and Attorney General Mark Herring, who admits he wore blackface in the 1980s, “have all grown” since they were students.

For activists in Richmond, the growth is too little, and too late—and many have already begun looking for new leaders.

Speaker Ravi Perry, an associate professor of political science at Virginia Commonwealth University, said the line of succession is complicated but assured demonstrators that it’s extremely unlikely that House of Delegates Speaker Kirk Cox, a Republican, would ever take office as governor, even if Northam, Fairfax and Herring were to resign. With staggered resignations, he added, the governor could appoint a new lieutenant governor who could then assume the governorship.

Perry said he would be upset if the only person to resign were Fairfax, who is black, with no consequences for the white politicians, including Republican state Sen. Thomas Norment, who has been caught up in his own racist yearbook scandal.

“Blackface in any form is disqualifying,” Perry said. “It’s all inappropriate. It’s all racist. It’s all deliberate.”