In these times of economic implosion, it seems there is one industry that the government is actually keen on crushing. The home secretary, Jacqui Smith, recently unveiled a proposal for new legislation aimed at bringing the sex industry to its knees (metaphorically speaking). If we tackle the demand, Smith proclaimed, then supply will diminish. In other words, Smith wants to penalise punters.

Under the proposal, anyone who buys sex or other erotic services from someone who is "controlled for another person's gain" could be fined and receive a criminal record. Ignorance of the circumstances would be no defence. Harriet Harman, the minister for women, believes the proposed legislation will help stamp out sex trafficking, which she has described as a "modern-day slave trade".

Yet if speakers at a panel debate this week on sex trafficking held at London's Institute of Contemporary Arts are to be believed, most sex workers – including migrant ones – do not see themselves as slaves, and few want to be "saved" by the likes of Smith and Harman. Scaring away potential punters will only rob those who work within the sex industry of their livelihood. (And this includes everything from charging for sex to pole-dancing, providing attentive dinner company and selling erotic lingerie, literature or DVDs.)

Laura María Agustín, anthropologist and author of the controversial Sex at the Margins: Migration, Labour Markets and the Rescue Industry, told the ICA audience that politicians like Smith and Harman are promulgating abolitionism as a benevolent, feminist project. But, Agustín says, "this is state feminism which has nothing to do with gender equality. It's about the state identifying a proper way for its citizens to behave and defining millions of women as victims."

Anyone who does not comply with the political elite's officially sanctioned lifestyle is seen as deviant. In relation to sex, non-conformists become defined either as victims or perpetrators, as abused or abusers.

Earlier this month, on International Sex Workers Rights day, I attended an event in Manhattan organised by the New York-based Sex Workers Outreach Project (Swop). There, I met sex workers and activists who challenged the idea that they are by definition exploited or abused. A transgender woman called Savannah announced that she was proud to be a call girl. She told me: "I've worked as a streetwalker, escort, model, dominatrix, in dungeons ..." Being a call girl is easier, she said, because it means she can avoid police harassment.

For Savannah, the main threat is not from punters, but from the authorities. She and everyone else I spoke to acknowledged that some within the sex industry experience assault and can find themselves in vulnerable situations. At the ICA event, Catherine Stephens, who has worked in the sex industry for 10 years, also acknowledged this, but she believes decriminalisation is the best way of ensuring sex workers avoid harm.

It would be strange to romanticise sex work as something exotic or empowering. But we would also do well to go beyond puritanical rescue missions such as that proposed by Smith and Harman and acknowledge that for many, working within the sex industry is simply an economic decision. After all, for a majority of people, salary is a prime factor in determining what job we pursue. Moreover, some apparently enjoy working within the sex industry. According to Savannah, "some are proud to be sex workers and chose to do it just like others chose to become physicians and are proud of being that."

Georgina Perry, service manager for Open Doors, an NHS initiative that delivers outreach and clinical support to sex workers in east London, has also met women in vulnerable positions and women who have paid to be brought to the UK. These migrants would likely be defined as "trafficked" by the government and various institutions and organisations that work to stamp out "people smuggling".

In Perry's experience, such women happily accept some of Open Doors' services, like free condoms and vocational training advice, but they do not want to be "rescued", thanks very much. They have debts and student loans to pay off, families to support and savings accounts to maintain. They just want to be left alone to get on with their work.

Perry, on her part, is not interested in "forcibly empowering anyone". She said much of the debate around the sex industry is infused with moral panic and pointed out that when women are presented as victims, they elicit sympathy; when they assert their agency, however, they are viewed as a threat to the moral fabric.

Even Jon Birch, inspector at the Metropolitan police clubs and vice unit, acknowledged at the ICA event that not all individuals employed within the sex industry have been coerced into it. He said the vice unit does not aim specifically to target migrant sex workers. Yet this seems disingenuous considering that, according to its website, the vice unit places emphasis on "rescuing trafficked and coerced victims".

It is curious that a term that is impossible to define or quantify, that is often described as a "hidden" or "covert" activity, motivates so much legislation, policy and activism. Individuals who have been defined as "trafficked" or "enslaved" have worked in everything from mining to agriculture, in housekeeping, elderly care and, indeed, in the sex industry. Of course kidnapping – whether within or across national borders – should be clamped down upon. The problem is that today the term trafficking is being applied to more and more forms of migration – and this is making life difficult and miserable for those who must, or who choose to, move across borders for work.

Foreigners who wish to visit or work in the UK have very few legal options available, and so they end up paying strangers to take them on long and risky journeys across the world. When they come over here, they are forced to take jobs in the shadow economy where they are, indeed, vulnerable to exploitation. Yet anti-traffickers rarely reach the sensible conclusion that Britain's and Europe's stringent immigration laws should be revised to allow people to come here to work and contribute to our economies, send remittances to their home countries and go back there when they choose to.

Instead, anti-trafficking campaigners see it as their duty to rescue victimised individuals who may have been trafficked, and to care for them. This does nothing to challenge immigration laws that force some people into the hands of dodgy employers, but it does a lot to paint immigrants as victims who need to be monitored ever more closely.