“Zero Percent of Blacks for Romney”—Oh, that headline hurts. And the WSJ/NBC poll can’t just be dismissed out of hand as the work of partisan hacks.

Of course, this doesn’t mean that no African-Americans will vote for the Romney-Ryan ticket, just that it will be within the margin of error and along the lines last time, when McCain-Palin somehow managed to score 4 percent of the black vote.

But ’twas not always thus for the GOP. Dust off your history books and you will see Republicans once had a virtual lock on the minority vote—and minority elected officials. The legacy of Lincoln was alive and well until not so long ago. Which makes the retreat of recent decades both unfortunate and ill-timed.

Consider that the first popularly elected African-American senator was a Republican, Ed Brooke from Massachusetts, in 1966. Likewise the first Asian-American senator, Hawaii’s Hiram Fong, who was first elected in the Eisenhower era. The first Native-American senator, Charles Curtis—who went on to be Herbert Hoover’s vice president. The first Hispanic senator, Octaviano Larrazolo, also was a Republican. Ditto the first woman popularly elected to the Senate, Maine’s Margaret Chase Smith.

“The Republican Party was the party that gave hope and inspiration to minorities—and there was a coalition at first,” says Ed Brooke, now 92 and living with his wife, Anne, in Miami. “My father was a Republican. My mother was a Republican. They wouldn't dare be a Democrat. The Democrats were a party opposed to civil rights. The South was all Democratic conservatives. And the African-American community considered them the enemy.”

That’s why every single one of the 23 African-American members of Congress before 1900 was a Republican. They wouldn’t have dreamed of being anything other than members of the Party of Lincoln—Democrats were the party of the Confederate South. Frederick Douglass summed up the sentiment when he said, “I am a Republican, a black, dyed-in-the-wool Republican, and I never intend to belong to any other party than the party of freedom.” This legacy echoed for generations.

“When I first went to the Senate there was one woman there, Margaret Chase Smith, who was a Republican,” remembers Brooke. “Of course, I was the only African-American. But there were a couple of Jewish senators—Jacob Javits [a Republican from New York] and Abe Ribicoff [a Democrat] from Connecticut. We had some diversity—racial diversity and [gender] diversity—but it was very small, of course. But we also had a degree of diversity as far as political ideology. We had a group of moderate senators who met for lunch once a week and we had a block of eight that usually voted together on these issues.”

The decline of centrist Republicans was one important reason for the decline in the GOP’s diversity over recent decades, according to Brooke. The shift of the party’s political base to the states of the former Confederacy coincided with the rise of social conservatism and states’ rights in what had been the progressive party in the era of Lincoln. The historic irony of a Southern Democrat, Lyndon Johnson, signing civil rights and voting rights bills into law (which his 1964 opponent Barry Goldwater opposed) solidified the shift of African-Americans into the Democrats’ camp, capped by the election of the first African-American president a half-century later.

“When you say Republican Party to the average citizen now, what comes up in their minds?” asks Brooke. “Not that it was the first party to elect an African-American to the United States Senate or the first woman. And then Hiram Fong and [S.I] Hayakawa, representing Hawaii and California. There were all these instances where Republicans were first. But they never really were able to capitalize on that. They always looked like they wanted to keep that hidden. If the Democrats had done that, we would have never stopped hearing about it.”

Demographics are destiny, and looking like the party of old white men is not a recipe for Republican success in the future. That’s why this forgotten legacy of diversity should be respected and celebrated, even as the Party of Lincoln has turned into the Party of Reagan. Because these forgotten figures deserve to be remembered by Republicans and all Americans as the pioneers they were.

• Sen. Charles Curtis of Kansas served in the Capitol for most of the first three decades of the 20thCentury. Native-American on his mother’s side, Curtis spent much of his childhood growing up on the Kaw reservation with his grandparents, descendants of Chief White Plume. His presence on the 1928 ticket alongside Herbert Hoover made him the first non-white on a major-party presidential ticket and the first to serve as vice president.

• Octaviano Larrazolo served as first governor of, and then senator from, New Mexico. Born in Mexico, he was a lawyer by trade, and his gubernatorial term overlapped with Woodrow Wilson’s administration. He won a special election to the U.S. Senate in 1928, but declined to run for reelection due to illness, and died in 1930.

• Margaret Chase Smith was elected to the Senate in 1948, after serving four terms in the House of Representatives. She achieved national renown after becoming the first Republican to stand up to Joe McCarthy on the floor of the Senate, delivering a now-classic speech, “Declaration of Conscience.” She ran for president against Barry Goldwater in 1964 and her Senate career ended in 1972 after three decades of principled public service.

• Hiram Fong was the first senator elected from Hawaii after statehood was granted in 1958. A Chinese-American, Harvard-trained lawyer, Fong served in World War II as a major in the Air Force. He was speaker of the House in Hawaii and his service in the U.S. Senate spanned two decades. He ran for president in 1964 and 1968. He died in 2004.

• Ed Brooke became the first African-American popularly elected to the U.S. Senate in 1966, and the first to serve since Reconstruction. He previously served as attorney general of Massachusetts, a state that was less than 3 percent black at the time. He served two terms and was mentioned as a possible replacement for Spiro Agnew as vice president, which might have made him the first African-American president back in 1974. He was awarded the Presidential Medal of Freedom in 2004.

• S.I. Hayakawa was elected to the Senate from California in 1976 at the age of 70. The first Japanese-American to serve in the Senate, Hayakawa was a Canadian-born former professor of semantics who came to prominence standing up to protesters as acting president of San Francisco State University in 1968. A committed iconoclast, jazz-lover and friend of Ronald Reagan, Hayakawa defeated Democratic incumbent John V. Tunney. Saying that “bilingualism for an individual is fine, but not for a country,” Hayakaya sponsored a Constitutional amendment making English the official language of the country.

This forgotten legacy of diversity within the GOP is a source of strength the Republican Party can draw on going forward.

Republicans deserve much more credit than they receive for recently fielding successful minority candidates for governorships—notably Louisiana’s Bobby Jindal, Nevada’s Brian Sandoval, New Mexico’s Susana Martinez, and South Carolina’s Nikki Haley.

But in contrast to the past, Republican ranks in the Senate have been monochrome since the early 1980s, a dynamic improved with the election of Marco Rubio and the likely victory by Texas Republican Ted Cruz.

The ranks of African-American congressmen have grown since 2010, to include South Carolina’s Tim Scott and Florida’s Allen West. But they double the number of Black Republican congressmen who have served since the civil rights era (Gary Franks and J.C. Watts being the other two). In that same period, Democrats have elected more than 90 African-American congressmen and women—and, of course, the first African-American president.

As Republicans gather in Tampa—Florida is a must-win swing state with a sizable minority population—the opportunity and obligation for the party to reach out beyond their white base should be clear.

But simply calling itself the Party of Lincoln and the Party of Freedom is not enough to heal the breach, because that ignores the reason this legacy of diversity has been lost for so long. Mitt Romney—whose father, George, was an advocate for a more inclusive Republican Party when he was governor of Michigan—might be in a strong position to address these hard truths and make amends. But that will require the political courage to admit his party’s failings on this front, recognizing the lost legacy of diversity is a rational reaction to specific policies the GOP embraced in the not-so-distant past.