For the winning punters chancing their luck at Hawaiian Gardens' charity bingo hall in the heart of one of California's poorest towns, the big prize is $500. The losers walk away with little more than an assurance that their dollars are destined for a good cause.

But the real winners and losers live many thousands of miles away, where the profits from the nightly ritual of numbers-calling fund what critics describe as a form of ethnic cleansing by extremist organisations.

Each dollar spent on bingo by the mostly Latino residents of Hawaiian Gardens, on the outskirts of Los Angeles, helps fund Jewish settlements on Palestinian land in some of the most sensitive areas of occupied East Jerusalem, particularly the Muslim quarter of the old city, and West Bank towns such as Hebron where the Israeli military has forced Arabs out of their properties in their thousands.

Over the past 20 years, the bingo hall has funnelled tens of millions of dollars in to what its opponents — including rabbis serving the Hawaiian Gardens area — describe as an ideologically-driven strategy to grab land for Israel, as well as contributing to influential American groups and thinktanks backing Israel's more hawkish governments.

But the bingo operation, owned by an American Jewish doctor and millionaire, Irving Moskowitz, has taken on added significance in recent weeks as President Barack Obama has laid down a marker to Israel in demanding an end to settlement construction, which the White House regards as a major obstacle to peace. "Moskowitz is taking millions from the poorest town in California and sending it to the settlements," said Haim Dov Beliak, a rabbi serving Hawaiian Gardens and one of the Jewish religious leaders in California who have campaigned to block the flow of funds to the settlers.

"The money Moskowitz puts in to the settlements has changed the game. Moskowitz has helped build a hardcore of the settler movement that may number 50-70,000.

"He's not paying for all of it but he puts the money up front for the vanguards that get things off the ground. That ties Israel's hands. That ties the hands of the Obama administration. If the administration wants to be serious about stopping the settlers it has to begin in Hawaiian Gardens."

Moskowitz is an 80-year-old retired doctor and orthodox Jewish millionaire who built a fortune buying and selling hospitals. In 1988 he also bought the faltering bingo hall in Hawaiian Gardens which, under California law, can only be run as not-for-profit operation so Moskowitz brought it under the wing of a charitable foundation he had established in his own name.

The foundation, which did not respond to requests for an interview, bills the bingo operation as of great benefit to the local community through donations to a number of groups, such as the Hawaiian Gardens food bank, as well as scholarships. It has also given money for disaster relief in Central America, Kosovo and parts of the US.

But tax returns show that the bulk of the donations go to what the foundation describes as "charitable support" to an array of organisations in Israel.

"The loss of many of Dr Moskowitz's relatives during the Holocaust strengthened his conviction that Israel must be maintained as a safe haven for Jewish people from all over the world. In Israel, the Foundation supports a wide array of religious, educational, cultural and emergency services organisations," the foundation says on its website.

What it does not say is that the focus of the donations is a number of Jewish organisations intent on claiming Palestinian territory for Israel and ensuring that occupied East Jerusalem remains in the Jewish state's hands.

Beliak calculates that the foundation has given Jewish settlers well over £100m, beginning with the construction 20 years ago of 133 houses on land confiscated from Palestinians by the Israeli government.

Beliak helped launch the Coalition for Justice in Hawaiian Gardens & Jerusalem to stop the flow of money from the bingo hall to the settlements. Its investigations of tax records show that the Moskowitz Foundation's donations include grants to Beit Hadassah, a militant Jewish settlement in the heart of the West Bank city of Hebron.

Thousands of Arabs have been forced from their homes and businesses around Beit Hadassah ostensibly for their own security after an American-born settler Baruch Goldstein murdered 29 Palestinians nearby in 1994. Goldstein was himself shot dead and his grave is regarded as a shrine by some settlers. Moskowitz has made excuses for Goldstein's actions by blaming Palestinians for pushing him too far.

The foundation has also given more than £3.5m to Ateret Cohanim, a right wing group that houses Jews in the Muslim quarter of Jerusalem's old city. In other parts of East Jerusalem, Moskowitz has funded Jewish colonies situated to box in or cut off Palestinian neighbourhoods that fits in with a broader government strategy to ensure total Israeli control over the city.

"What Moskowitz pioneered was trying to break up the continuity of the Arab population centres in Jerusalem," said Beliak. "The consequences are radically different from just mom and pop buying a little piece of land. These are political statements and facts on the ground, and every [US] administration has allowed him to do this."

Among the most contentious of the organisations backed by Moskowitz is the City of David Foundation in the heart of an Arab neighbourhood of Jerusalem, where about 1,500 Palestinians are facing expulsion ostensibly in the name of archaeological preservation of a site where the organisation says King David established a city 3,000 years ago.

Four years ago, the City of David Foundation director, Doron Spielman, told the Guardian that "the goal of our organisation is to increase the presence of Jews in the neighbourhood as much as possible … We cannot trust that if this is an Arab neighbourhood, Jews will be safe here".

To that end Palestinians have been driven from their homes, sometimes at gunpoint, while others are fighting in the courts to cling on to their properties.

Moskowitz has made no secret of his hostility toward the Palestinians. He opposed the Oslo peace accords, likening them to the appeasement of the Nazis. In 1996 he told the Los Angeles Times that peace talks represented a "slide toward concessions, surrender, and Israeli suicide".

He was also an outspoken opponent of Ariel Sharon's removal of Jewish settlers from the Gaza strip four years ago and provided the settlers with funds to fight the pullout.

Now Moskowitz is building a much bigger bingo hall in Hawaiian Gardens which will increase the flow of cash.

Beliak is particularly angered that the fundraising takes place without interference from the American authorities. In contrast, he says, Muslim charities which raise money to help Palestinians have been targeted for investigation, shut down and some of their administrators jailed because providing welfare to Gaza indirectly helps Hamas.

"After 2001 there was a whole discourse about how supposedly Muslims [in America] used these charitable donations to support violence.

"There was never ever in the US anything substantially that made that case. But here they did have a case where somebody was using money to support settlers, money that fosters extremism and violence, and they completely ignored it," said Beliak.