(CNN) -- A new study that surveyed racial attitudes suggests that racial prejudices could tip the balance in the upcoming presidential election.

A poll finds a small percentage of voters said they may turn away from Sen. Barack Obama because of his race.

If there were no racial prejudice among voters, Sen. Barack Obama would receive about 6 percentage points more support, according to an AP-Yahoo News poll, designed in partnership with Stanford University.

The results suggest that 40 percent of white Americans hold at least a partly negative view toward blacks, including more than a third of white Democrats and independents. A small percentage of voters -- 2.5 percent of those surveyed -- said they may turn away from Obama because of his race.

A CNN/Opinion Research Corp. survey also indicates that race could play a big role in November. Asked if race would be a factor in their vote, 37 percent of respondents said yes. But of that group, many are Republicans who are not likely to vote for any Democrat, and some are Democrats who may vote for Obama because of his race.

Of the 8 percent of Democrats who told CNN they plan to vote for Obama's GOP rival, Sen. John McCain, half said race was a factor.

The survey, conducted August 29-31, questioned 1,031 people and has a sampling error of plus or minus 3.5 percentage points.

Experts point out that it's hard to quantify racial prejudice because many people who hold prejudices are not going to admit to it. Watch how race could affect the election »

"The hardest thing in the world for pollsters to poll for, with the exception of sexual behavior, is racial attitudes and how it affects behavior," said Walter Shapiro, Washington bureau chief for Salon.com. Shapiro said while people might say things differently off the record, racial issues cannot easily be quantified.

The pollsters for the AP/Yahoo survey used techniques that they thought would be more likely to lead to honest results -- such as conducting the poll online and using subtle methods and formulas to calculate racial attitudes.

That study also suggests that the number of people who may turn away from Obama because of his race could be larger than what the margin of victory was in the 2004 election.

Jeff Johnson, host of BET's "The Truth With Jeff Johnson," said, "I think there is a concern clearly about the number of people who will vote based on race.

"But I agree -- how you quantify that number, I think, is very difficult."

According to CNN's average of recent national polls, Obama holds a lead of 5 percentage points over McCain.

Johnson said one misconception is that racial prejudices are unique to conservatives or people in "Middle America."

"There are liberals also in many cases that are racist. I don't think we know yet how it's going to play out," he said.

In an interview that aired Sunday on CBS' "60 Minutes," Obama said while some people might not vote for him because he is black, others might vote for him just because he is.

"Are there going to be some people who don't vote for me because I'm black? Of course. There are probably some African-Americans who are voting for me because I'm black or maybe others just inspired by the idea of breaking new ground, and so I think all that's a wash," he said.

Democrats, however, typically get close to 90 percent of the African-American vote anyway.

Salon.com's Shapiro said Democrats can work on increasing turnout among black voters but that it will be hard to make gains on the percentages they already see.

Johnson said he thinks race will matter, and the best way for Obama to balance out any negative effect is to just stay on message.

"I don't know if I believe it's going to be a wash. I think it's going to matter. This race is extremely close, and so every single demographic and every single point is going to count," he said.

"I think he has to speak to the issues of people in Middle America, and by that, it can counterbalance some of these racial issues."

The AP-Yahoo News poll surveyed 2,227 adults. It was conducted August 27-September 5, and has a margin of error of plus or minus 2.1 percentage points.

All About U.S. Presidential Election