In one of the country's richest towns — where Mercedes, BMWs and Land Rovers cruise tree-lined streets of multimillion-dollar homes — a man who proposed building more accessible housing says angry neighbors took out their frustration on his son: a 9-year-old boy who was demoted to a lower-level Little League team.

Christopher Stefanoni says in a federal lawsuit that residents of Darien are so worried that affordable housing will draw black people to town that they'll do just about anything to stop it, including using his son to retaliate against him. Town and Little League officials say that's completely false.

"Darien is a little white enclave, sort of a holdout segregated town," said Stefanoni, 50, a Harvard-educated father of five who has lived in town since 2000. "The attitudes that people in Darien have are very exclusionary, demeaning. When they go after your kids, they've crossed the line."

The town of nearly 21,000 people on Connecticut's Gold Coast consistently appears in Top 10 lists of America's wealthiest towns, with a per-capita income around $95,000. About 94 percent of the population is white, with about 620 Hispanics and 70 blacks, according to the latest U.S. Census Bureau estimates.

The lawsuit and a federal housing investigation reopened old wounds in Darien, a New York City suburb depicted in the 1947 Oscar-winning movie "Gentleman's Agreement" starring Gregory Peck where residents conspired not to sell their homes to Jews.

Stefanoni and his wife, Margaret, filed the lawsuit in 2013 against the Darien Little League and its leaders over the demotion of their son for the fall 2010 season, just days after Stefanoni filed an affordable housing application for property right next to the home of a former league official. Several months later, Stefanoni was banned indefinitely from coaching in the league.

Lawyers for the defendants deny the allegations. A federal judge in Bridgeport is now mulling a motion to dismiss the lawsuit. According to court documents, league officials say they made a mistake placing the Stefanonis' son on a higher-level team, after the housing application was filed, and corrected the error by moving him to another team.

"Mr. Stefanoni is pursuing a baseless litigation as a means to harass and retaliate against defendants for an imaginary slight that has no connection to reality or to the civil rights laws that he purports to vindicate," the defendants' lawyers, Michelle Arbitrio and Fred Knopf, wrote in the motion to dismiss. Knopf has since withdrawn from the case.

Former Little League board members named in the lawsuit declined to comment.

Stefanoni said he has had three affordable housing proposals rejected by the town. They include a 16-apartment complex with five affordable units and a 30-apartment development with nine affordable units. A court sent both of those back to the town's planning and zoning commission for review and approved a third. The commission cited traffic safety and other concerns.

The lawsuit includes allegations about city officials blocking affordable housing applications to keep blacks from moving into town, claims identical to those in another pending federal lawsuit against the town by a different affordable housing developer whose project was rejected.

The U.S. Department of Justice in 2010 began investigating whether the town was violating the Fair Housing Act with a zoning policy approved in 2009 that gave top priority for new affordable housing to Darien residents and other people with ties to the town, including town employees. The planning and zoning commission rescinded the policy later in 2010, and the Justice Department closed the investigation in 2012 without taking any action, the Darien Times reported.

According to state data, 2.6 percent of Darien's nearly 7,100 housing units qualify as affordable. Gov. Dannel P. Malloy has called affordable housing one of the state's most pressing needs and has committed hundreds of millions of dollars for more affordable housing.

In 2010, Darien won a four-year exemption to a state law making it easier for developers to build in towns with less than 10 percent affordable housing, and town officials expect to win another after resolving a dispute with the state. The town says it is entitled to the exemption under a complicated formula involving existing affordable housing units.

Darien First Selectman Jayme Stevenson said the town has made significant efforts to increase affordable housing and its housing practices aren't discriminatory.

"The Darien of today bears no resemblance to the allegations that the Stefanonis ... are intending to propagate," she said. "These folks are developers and they're looking to develop housing and make some money."

Rob Williamson, owner of Uncle's Deli in downtown Darien, said he doesn't believe the town is being discriminatory in rejecting affordable housing applications.

"The town's small, very tight knit," the resident of nearby Stamford said. "That doesn't mean we want to keep anyone out. It's a small, little New England town and I think they want to keep it that way."