The firebrand conservative senator Rand Paul evoked the reported youthful drug use of George W Bush and Barack Obama in a pitch to young black students on Wednesday.

In part of a speech delivered at Howard University, a predominantly black college near Washington DC, Paul argued for a repeal of mandatory minimum sentencing laws, arguing that being jailed for some non-violent drug crimes ruined lives. In order to make his point, he teed up a story about two young American men – one of them white, rich and privileged and one of them mixed-race and from a single parent household. "Both of them were said to have used illegal drugs," he said.

Paul went on: "Now you might think I'm about to tell you a story about racism in America, where the rich kid gets off and the black kid goes to jail. It could well be, and often is, but that is not this story. In this story both young men were extraordinarily lucky. Both young men were not caught. They weren't imprisoned. Instead they both went on to become presidents of the United States. Barack Obama and George Bush were lucky."

In his autobiography, Dreams of My Father, Obama confessed that as a young man he smoked "weed" and had taken "maybe a little blow" – common slang for cocaine. Journalists and authors have repeatedly raised allegations that Bush used cocaine when he was a young man known for a hard-partying lifestyle. Bush has skirted the question. Asked about the rumours in 1994, in Texas, he replied: "What I did as a kid? I don't think it's relevant."

The libertarian-leaning Paul used both men as an example of how tough laws on drug issues can blight lives – a particular problem among some poor black American communities, where incarceration rates are high. "We should stand and loudly proclaim enough is enough. We should not have laws that ruin the lives of young men and women who have committed no violence," he said, pointing out that he had submitted a law to repeal federal mandatory minimum sentences.

Paul's speech at Howard comes as the Republican party is seeking to broaden its appeal to minority voters after Mitt Romney's decisive defeat by Obama in 2012. That loss has provoked a fierce internal debate between those who say that the GOP has to shift its policies to broaden its appeal, and those who say that it just has to sell its core conservative message more effectively.

The Kentucky senator acknowledged the party's challenge in persuading black Americans to vote for it. "Some have said that I'm either brave or crazy to be here today," he said. "My hope is that you will hear me out, that you will see me for who I am, not the caricature sometimes presented by political opponents."

The Tea Party favourite, who has inherited the libertarian followers who once flocked to his father Ron Paul's banner, then outlined how the history of the Republican party had frequently coincided with the civil rights struggles of black Americans. He also addressed the long-standing controversy of previous comments he made, in which he criticised portions of the historic 1963 Civil Rights Act. Paul has said repeatedly that parts of the act that forced private businesses to serve all races equally were an infringement of the free market best left outside the control of federal law.

"No Republican questions or disputes civil rights. I have never wavered in my support for civil rights or the Civil Rights Act," he said. "The dispute, if there is one, has always been about how much of the remedy should come under federal or state or private purview."

Paul has rapidly emerged as one of the leading lights of the Republican right wing as it comes to terms with defeat in 2012 and looks forward to the 2014 mid-term elections. His name is frequently mentioned as a likely contender for the Republican presidential nomination in 2016. At Howard, Paul stuck to his central anti-government, anti-big spending, anti-tax political beliefs.

"I would argue that the objective evidence shows that big government is not a friend to African Americans," he said. But he also spoke out against a bellicose US foreign policy and criticised military intervention abroad. "I want you to know that all Republicans do not clamour for war," he said.

The speech can be seen in the context of Paul's attempts to reach out to non-traditional areas of Republican support. It also comes after a well-received speech that he gave to the Hispanic Chamber of Commerce last month. In that speech, Paul used Spanish phrases, quoted famous Latin American authors and made a clear pitch for the fast-growing Hispanic demographic that many experts believe is vital to future electoral success in America.