Hugh Muir talks to families involved in adoption and critics who worry about plans to place more black and Asian children with white families. guardian.co.uk

Morale among social workers has been driven to rock bottom by cuts, targets and ministers making the issue of race and adoption a "political football", according to the biggest professional association.

A Lords committee will hear claims that politicians fuelled stereotypes for political gain, hampering the efforts of social workers to assist vulnerable children.

Nushra Mansuri, of the British Association of Social Workers, is expected to criticise the education secretary Michael Gove, who accused social workers of condemning black and Asian children to a life in care rather than see them adopted by white couples.

"Leftwing prescriptions are denying children the love they need," Gove said in February. "This misguided nonsense punishes those who most need our help." The coalition later amended practice guidelines to speed up the adoption process and emphasise the determination of ministers to see more trans-racial adoptions.

"We are angry being misrepresented in this way," said Mansuri, who speaks for 14,000 social workers. "It is easy to talk about children in care who need to be with loving families and we are concerned about that. But we also know that only a minority can be adopted."

Mansuri, who will address the Lords select committee on adoption legislation led by Lady Butler-Sloss, said ministers increased their popularity by highlighting difficulties in adoption but did not take responsibility: "They set the policies and release the resources. What Gove said is an example of adoption as a political football, of the misrepresentation and distortion that gets people's backs up."

The changes relating to race and adoption had the backing of significant stakeholders, many of whom sat on a working group led by Gove's adoption tsar Martin Narey. But there is continuing disquiet among some minority social workers and activists who say local authorities will feel less obliged to find the most advantageous match. This summer, around 60 people met to protest that ministers were downgrading the importance of race and heritage in the adoption process. The Race Equality Foundation subsequently sent a critique of ministers' adoption intervention to Butler-Sloss's select committee.

Sally Baffour, a black adoptive mother and a former government adviser who was for six years a trustee of the British Association for Adoption and Fostering, said: "There are going to be cavalier decisions made. If it is possible to make a quick match and take the pressure off, why keep looking for a better one?"

Baffour, who sits on adoption panels, said trans-racial adoptions are hard to get right. "Race and heritage and culture are important, but ministers seem totally dismissive. A lot of people think the repercussions are going to be very damaging."

Marlene Ellis, a black Londoner raised for 18 years by white foster parents in the home counties, said the complexities should not be underestimated. "It is impossible to come out really clear and comfortable about who you are in a society that still has very clear classifications for race and culture," she said. "My parents did the best they could do but there are subtle things that happen that erode your confidence. My real memory is loneliness; of not knowing."But minsters can say, with justification, that some social work professionals and trans-racial adoptees fully back the government's stance on race and adoption. Jo Bonnett, a black police officer raised in rural Leicestershire and east London by white English adoptive parents, is one of them. "I didn't find it a negative experience," she says "I think I was very lucky. I had an older brother who was their birth son; a brilliant childhood and fantastic friends. My challenges came at 17, but when you get to that age, and have been brought up in a loving household, you are strong enough to deal with racism or any issues you might have."She said the benefits greatly outweigh the drawbacks. "I don't think race matters in adoption as long as you have loving parents and have all the things a child needs."

Bonnett, 40, said she and her husband, who is white, tried themselves to adopt a black child. "But we were told the child must be mixed race. Ridiculous!"

The figure most directly identified with the adoption review was Tim Loughton, the coalition's first children's minister, but he lost his job in the reshuffle. He was replaced by Edward Timpson. Loughton will also appear before the Lords select committee as will officials from the the College of Social Work. Having conducted a survey, the College of Social Work will say ministers are right to speed the adoption process but must be careful to ensure that haste doesn't lead to bad decisions and adoption breakdowns.

Nearly 90% of social workers polled say adoptions take too long. And more than half think it is right to tackle delays caused by caused by 'excessive emphasis' on an ethnic match between the child and its adoptive parents. But the college will tell peers that most do not feel adequately trained to deal with adoption. Alongside the parliamentary discussions, there will be initiatives incoming days to raise the profile of adoption as part of National Adoption Week.

A Department for Education spokeswoman said the government's motivation was to "make it as easy as possible for people to foster and adopt so that more children in care find loving and stable homes." She added: "We have also introduced measures to tackle delay, sharpen accountability, and hold the local authorities slowest to act to account."

• This article was amended on 6 November 2012. The original said Edward Timpson was adopted. That is not the case.