A recently released report by the Pew Center is a belated Valentine’s Day gift to interracial families. The report indicates that intermarriage across racial and ethnic lines continues to be on the rise in the U.S. and the change is a sign that acceptance is growing. Although this is definitely cause for celebration and a reason to continue the fight for marriage equality everywhere, we should remember that a fuller and more accurate historical account of interracial sex and marriage in the U.S. should focus on social and legal constraints along with demographic patterns.

One reason why is the large-scale psychological distress experienced by all racial groups resulting from a social and legal history around interracial sex and marriage that’s been fraught with challenges. Legal history tells us that interracial sexual relations have been a troubled issue since the days of colonialism and enslavement, when many African-American women were forced to give birth to mixed race children to increase the enslaved population. This means that a large number of people who can claim interracial heritage do not because they are what multiracial activist Glenn Robinson calls “mixed by force” rather than “mixed by choice.” We must also consider the many free “mixed by choice” families of various backgrounds whose marriages were not recognized in the census records because miscegenation laws got even stricter after the demise of slavery.

Then, there were female members of interracial marriages, such as New York’s Alice Rhinelander in 1925 or California’s Marie Antoinette Monks in 1939, who were accused of fraud so that their marriages could be annulled and so that they could be disinherited. So, we must remember that before the 1967 case Loving v. Virginia ended bans on interracial marriage in all territories where it was outlawed, interracial coupling was a common practice. That means there may be some validity to the critique that today’s demographic patterns may not represent as much of an increase from historical trends as is being reported. Sadly, this is difficult to prove because there are few historical records to document the trend through its centuries-long history in the U.S.

There’s also the issue of psychological distress faced by members of interracial romantic relationships today. This more personal anguish is experienced differently by various races, ethnicities and genders. A study conducted by sociologists at Rice University found that white women married to black men suffer the greatest psychological distress and social stigma. Next are Native American men, who actually out-marry the most. Latino and Asian men and women who are married to African-Americans reported the next highest levels of anguish. Interestingly, and despite mediated depictions to the contrary, African-American men and Asian women marrying whites experience the least distress when engaged in interracial relationships.

The issue of psychological impact suggests that although interracial relationships are on the rise, they are neither gaining equally nor is the increase a sign of equal acceptance in a larger social sense. In fact, according to census data, interracial couples have a slightly higher divorce rate than same-race couples. These facts add a layer of meaning that needs to be considered in the history of interracial marriage and sex so that we can better understand why there are significant differences by gender in the tendency to marry outside one’s racial group. For instance, according to the Pew Center, “Black men are more than twice as likely as black women to marry someone outside their race, and the reverse pattern holds true for Asian men and women. In 2010, nearly one-quarter of black male newlyweds (24 percent) married outside their race, compared with just about 9 percent of black female newlyweds. In contrast, more than one-in-three (36 percent) Asian female newlyweds in 2010 married someone who is non-Asian, compared with only about one-in-six Asian male newlyweds (17 percent).”

Finally, there are the issues of race and ending racism. What happens when we look simply at demographic shifts in interracial marriage and not at the psychological effects of legal and social histories is that we promote them as the only way to dismantle historical taboos. We shift the responsibility of solving racism and reimagining racial identity off of law and social custom and onto interracial families. This is not only unfair — it’s unrealistic. Charting the demise of racism by the rising number of interracial marriages is probably not the most reliable indicator that it’s ending. Wouldn’t the elimination of disparities in income, employment, health care, education, crime, punishment and family structure be more accurate indicators?

Let me put it differently. I once heard someone say that if marriage were the cure for our social ills then certainly sexism would be dead by now. Obviously sexism is still with us. What this says to me is that if the mere presence of interracial intimacy were enough to bring about racial harmony, it would have happened long ago. Instead, as we’ve noted, laws were passed to keep races apart and punishments, including fines, imprisonment and death, were instituted to keep people from crossing the color line. Although we should celebrate changing demographic patterns as one sign of progress when we should be looking for many, we should also remember that the racism inspiring such laws and punishments lives on in many communities. As Diane Farr put it recently, some of us continue the interracial struggle having “been told there was a right and an ‘over my dead body’ [racial] choice for love.”

Before we rush to judgment on whether interracial sex and marriage are doing away with historical taboos, we would be wise to remember that talking about them in demographic terms only forces us to hang on to the concept of race and makes ending racism by legal means all the more difficult to imagine.