(CNN) -- Congressman Barney Frank is taking some major heat for making a serious accusation against Supreme Court Justice Antonin Scalia.

Rep. Barney Frank cites a Supreme Court justice's legal opinions in calling him a homophobe.

"I do think this argument that it is unconstitutional for the federal government to pick and choose as to which marriages it will accept is a good one. At some point that's going to have to go to the United States Supreme Court," Frank told the Web site 365gay.com on Friday.

"I wouldn't want it to go to the United States Supreme Court now because that homophobe Antonin Scalia's got too many votes on this current court."

Frank, D-Massachusetts, explained his remark during an interview Wednesday with host John Roberts on CNN's "American Morning."

John Roberts: Congressman Frank, you made a pretty pointed accusation there. What's the basis for that charge?

Rep. Barney Frank: Two opinions written by Justice Scalia in which he makes it very clear that he thinks it's a terrible idea for people who are gay or lesbian to have equal rights. It's not based on his views on marriage. Obviously, there's a legitimate debate about marriage. Watch Frank make his case against Scalia »

A few years ago, the state of Texas tried to send to prison two men who had private, consenting sex in their own bedroom. And the majority on the court said, "That's unconstitutional. That's an invasion of privacy."

Justice (Clarence) Thomas said, "Well, I think it's a silly law. I would have voted against it, but I don't think the Constitution prohibits it."

Justice Scalia wrote a long, angry dissent in which he made it very clear he thought it was a perfectly good law and that, in fact, homosexuals, as he refers to us, are bad people, and the notion that there ought to be any kind of legal protections is a mistake.

In an earlier case in Colorado, in which he again vigorously denounced the majority in the court for finding that it was unconstitutional to discriminate against people, again, not in marriage but a basis of their political rights, he said, "Well, of course, we disapprove this. We often disapprove of things like murder." I mean, literally, when he was looking for comparisons to the public disapproval of homosexuality, the first thing he said was murder.

So unlike many people who have different legitimate views on this, I urge people to read those two opinions in the Colorado case, the Romer case and the Lawrence case. And, again, there is just no question about his absolute view that ... homosexuals are bad people that shouldn't be treated equally.

amFIX: React to Frank's comments on Scalia

Roberts: Well, let's take the Lawrence v. Texas case then, if we could. In his dissenting opinion he said, "Today's opinion is the product of a court that has largely signed on to the so-called homosexual agenda, by which I mean the agenda promoted by some homosexual activists directed at eliminating the moral opprobrium that is traditionally attached to homosexual conduct."

But what he was doing in that dissent, according to him, was complaining about judges, not elected officials, deciding questions about morality.

Frank: No.

Roberts: Scalia went on to write, "It is clear from this that the court has taken sides in the culture war."

Frank: Yes.

Roberts: So if he is talking here about -- he is upset that the court is deciding this as opposed to elected officials deciding it, how is that homophobic?

Frank: Because ... you forgot to mention the issue here. It isn't whether or not homosexuality is a nice thing. It is whether two men in the privacy of their own bedroom, two men who live together, two adults, should be facing prison because they had intimate relations.

Now, Justice Thomas said it's a silly law. Justice Scalia said, "Oh, no, because we" -- the phrase you quoted is in the context of justifying sending people to prison because they have consenting sex in private because he disapproves of homosexuality. The court wasn't taking sides in a culture war. The very way that he frames it is that.

In other words, he believes that when six of his colleagues said, "No, the Constitution protects privacy, and you do not allow people to go to jail for private, consenting sex," that that's taking sides in the culture war.

So the very fact that he frames it that way is his argument. He is so angry about those of us who are maybe gay or lesbian that he thinks it takes sides in the culture war to say people shouldn't be imprisoned for private sexual acts.

Roberts: And let me get to the point that you were making when you gave that interview, the Defense of Marriage Act. You say that it's going to come before the court at some point.

The Defense of Marriage Act passed overwhelmingly in the House and the Senate. It was signed by a Democratic president.

The National Review online, talking about what you said regarding Justice Scalia, pointed that out and went on to add, quote, "Millions and millions of voters in state after state have acted to preserve traditional marriage. Does Congressman Frank regard all of these Americans as homophobes?

Frank: No, as I just said -- no. I've already answered your question.

No, I just differentiated between Justice Scalia and Justice Thomas. Justice Thomas said it's a silly law to send people to prison because of private, consenting sex. Scalia basically says "Yes, that's where they belong." So that's exactly the case.

No, people have, as I said, legitimate rights to differ over the constitutionality of marriage. My point is that Scalia is so anti people who are homosexual that he will argue so vehemently there that I'd rather not have him have the influence.

But I was talking about his attitude in cases which had to do with political rights in Colorado and sending people to prison. That's very different than the question about marriage.

Roberts: And we should point out, too, that we reached out to Justice Scalia for comment. They declined comment on this, but perhaps we'll hear from him at some point down the line.

Frank: Well, in fairness to Justice Scalia, I obviously was very critical of him. But Supreme Court justices are supposed to argue through their opinions. And that's why I'm not referring to any private comments he made but to his opinions.

And, again, the issue in my mind is his vehement denunciation. Let's put it this way: He says it's taking sides in the culture war if you don't send people to jail for having private, consenting sex. It's got nothing to do with marriage.

All About Barney Frank • Antonin Scalia • Gay and Lesbian Relationships