Warren Buffett is getting some feedback from his billionaire club colleagues to his million-dollar tax challenge, and it's not all that positive.

In an interview with NBC's Tom Brokaw that aired last week, Buffett took his "I'm not paying enough in taxes, and neither are my fellow billionaires" campaign to a new level, highlighting his contention that he pays a lower tax rate than all of his office employees.

He told Brokaw: "I'll bet a million dollars against any member of the Forbes 400 who challenges me that the average (federal tax rate including income and payroll taxes) for the Forbes 400 will be less than the average of their receptionists." (You can see and read the whole conversation in the aptly-named WBW Post Warren Buffett and NBC's Tom Brokaw: The Complete Interview.)

In its new issue just out, Forbes quotes an email message it received from Buffett (#2 with $52 billion):

"So far only three close friends, all 400 members, have made the calculation for me. They all came up with results similar to mine but have no interest in being identified."

Forbes says it "couldn't pin down many other Forbes 400 members to a real million-dollar wager" but did get some quotes from the likes of Mark Cuban (#161 with $2.6 billion), George Kaiser (#26 with $11 billion) and John Catsimatidis (Tied at #220 with $2.1 billion).

Several argued that Buffett isn't taking estate taxes into account. Casino owner Phillip Ruffin (Tied at #220 with $2.1 billion) told Forbes, "He is forgetting about the 55% estate tax at death that goes along with making the money. Maybe he is getting senile?" While not interested in the million-dollar challenge, Ruffin did say he'd play Buffett in poker.

And B. Thomas Glissando (Tied at #239 with $2 billion) brought out the often-heard argument that while the very rich may pay lower tax rates, they're carrying most of the burden when taxes are measured in dollars:

"There are some facts I'd like to remind Mr. Buffet of. The top 1% of earners in this country pay 30% of the income taxes, and the top 5% pay 50% of the income taxes. Other than that, I don't think he's crazy."

If you're interested in some comments from non-billionaires, check out the responses posted on to WBW'sBrokaw interview post. (Thanks to DeliciousIrony for the submission.)

Questions? Comments? Email me at buffettwatch@cnbc.com