A clear majority of British people think current immigration levels are too high, according to many polls. A major fear is that too many newcomers will stretch Britain's already shrinking welfare state. According to some reports, Downing Street is considering negative advertising, to persuade Bulgarians and Romanians to stay away. It doesn't help quell these fears when new migrants from poorer parts of the world, such as Somalia, show high rates of unemployment.

But what if we were told that thousands of people from Africa we've seen arriving here are not, in fact, fleeing poverty at all? Or that, legally speaking, they're not even Africans, but rather nationals of such generous welfare utopias as Sweden, Denmark and Holland?

For BBC Radio 4, I've been searching for some idea of what the modern "British Dream" could be, through a series of frank interviews with some of the almost 3 million new arrivals to England and Wales since 2001 (the first of three programmes is broadcast at 8pm starting tonight). Time and again, our team uncovered Africans who were not any longer from Africa. They were EU citizens and actually giving up welfare rights in places like Scandinavia to come to the UK.

Indeed, we met so many Nigerians from Germany, or Somalis from Denmark, that we asked Oxford University's Migration Observatory to crunch the numbers on how many EU migrants are not originally from Europe. They found that 141,000 people, 7% of those who came to the UK under EU rules were born outside the continent. Somalis are one of the biggest such groups, with an estimated 20,000 coming to the UK from the Netherlands alone. Studies show that between one third and a half of the entire Dutch Somali community has moved to the UK.

But why abandon the good life in Sweden, or the Netherlands, to start again from scratch in Britain?

"It's great to have a decent house," I was told by Quman Akli, who was three and a half when her family fled Somalia for Bergen, in North Holland, in 1989. "But also you want more in life than a decent house. You want to be able to progress."

Quman's family had been housed by the Dutch government. She grew up there and spoke Dutch fluently. After finishing school, she would have been entitled to a subsidised university education; even her bus passes would have been paid for by the state.

Instead, in 2003, she told her father Jibril that she wanted to move to Britain, where she would have to pay for university. He wasn't upset – in fact he decided to quit his job in a printing firm and bring the whole family to London.

"I think the UK is more open than other European countries," says Jibril, who is now a London bus driver. He and many other Somalis told me they admired the success of non-white people in Britain – which was conspicuously absent, they felt, on the continent. Jibril mentions the Asian community who came to the UK from Uganda. "They are landlords, they are businessmen, lawyers," he enthuses. "It's amazing."

How should we feel about this admiration of Britain from non-white people across Europe? Some may find it uncomfortable that Jibril and other Somalis I met were partly attracted by the larger number of mosques in Britain. On the other hand, for them, it's often about racism. Quman remembers how well-meaning Dutch people constantly asked her when she was "going home" after 9/11.

"No one has asked me that in London," she says. She enthuses about Britain's education system and the number of minority MPs.

Others are even more direct in their praise. "London has become a place where black people can live," says Kevin Obudako, who was born in Nigeria but came to Britain from Germany. Having been racially taunted when he first got to Germany, he says an awareness of British post-colonial migration drew him here. "You had Nigerians, Jamaicans," he says. "Everyone was accommodated."

Of course, it's no surprise when immigrants flatter Britain with their comments. People from brutal dictatorships admire our freedoms. The global poor want our prosperity. Nothing unusual there.

But this sub-category of EU migrants is different. With all of western Europe at their feet, they are drawn to Britain – even if they are, at least as first, poorer as a consequence. Stories like theirs are a powerful corrective against the tendency to over-simplify when it comes to describing the "influx" of newcomers to Britain. Migrants are neither all out to exploit us, nor all straightforward victims. But their admiration of our society does seem to indicate some kind of "British dream" that has drawn in many millions. Even as we worry about immigration levels, we should at least take the compliment these new Europeans are paying us, by voting with their feet.