A couple in Montana is filing a discrimination suit after they say state social workers took their foster child away because they are gay, the Bozeman Daily Chronicle reports.

“It seemed to us they really worked hard to find somebody else to take him instead of leaving him with us,” Joseph told the Chronicle.

Montana’s Department of Public Health and Human Services on Friday denied that its workers discriminated against the gay couple, noting that it “has placed and continues to place foster children with same-sex couples.”

Luis and Joseph Serrano picked up the a 5-week-old baby boy the newspaper calls TJ (not his real name) in March. Like most of Montana’s 3,200 foster children — a number that has more than doubled since 2008— the boy ended up in the system as a result of drug abuse.

The DeSerranos were told that he tested positive for marijuana and methamphetamine at birth and had been living with his mother in a car while his father was in jail for drugs. Despite the rough conditions in which the child started its life, the couple say they saw dramatic improvements after he started living with them.

The couple initial received positive feedback, with one social working telling them not to worry, that they were doing a good job. But other employees made comments that gave them pause.

Joseph tells the Chronicle that the DPHHS employee responsible for signing off on all placements told the couple, in front of a foster care class full of people, they would always be at the bottom of the list for foster child placements.

“[She said] that other social workers wouldn’t want to work with us because we’re a gay couple, and because we’re two males,” Joseph says, adding that she covered herself by adding that she was “kind of joking.”