SALT LAKE CITY -- A Utah judge who had ordered a baby girl taken away from her lesbian foster mothers and placed in a heterosexual home removed himself from the case Monday as criticism turned into calls for his impeachment.

Though Judge Scott Johansen had reversed his decision and allowed the 9-month-old baby to stay with the married women recommended by state welfare authorities, there were concerns he could still have the baby removed from their home in Price later on.

April Hoagland and Beckie Peirce asked for the judge to be disqualified, saying that the decision revealed a potential bias that broke the rules of judicial conduct, their lawyer Jim Hunnicutt said.

Get Breaking News Delivered to Your Inbox

Last week, Hoagland and Peirce told CBS News correspondent Carter Evans the judge on Tuesday ignored pleas from the baby's biological mother to grant them custody. They believe the judge, a bishop in the Mormon church, imposed his religious beliefs over the law. The couple, legally married in Utah, planned to adopt the child

"This is all about sexual orientation, not what is best for the child," Peirce said.

"He has no other grounds but that," Hoagland added.

April Hoagland, left, and Beckie Peirce say a Utah judge is ordering their foster daughter to be removed from their home because they are gay. KUTV

While Johansen disputed their legal standing to call for his removal, he nevertheless stepped aside nearly a week after the Nov. 10 order criticized by national gay rights groups, the state's Republican governor and others.

The couple applauded Johansen's decision and said they're happy the family is being treated equally.

"Our greatest concern now is taking care of our beautiful baby foster daughter," Hoagland and Peirce said in a statement.

Utah officials are pleased the child will stay in a nurturing home with the "very capable" parents, said Ashley Sumner, a spokeswoman for the Utah Division of Child and Family Services.

The case will be referred to presiding juvenile Judge Mary Manley.

In his initial decision, Johansen mentioned research showing children do better when raised by heterosexual families. The American Psychological Association, however, has said there's no scientific basis for believing that gays and lesbians are unfit parents based on sexual orientation.

A gay rights group has filed a complaint with state judicial officials, and the watchdog group Alliance for a Better Utah on Monday called for state lawmakers to impeach the judge.

Johansen is barred from speaking about pending cases and a call to his publicly listed phone number went unanswered Monday.

Josh Kantor, the founder of the progressive-leaning Alliance for a Better Utah, said the judge's move would not change the group's push to get state lawmakers to remove him from the bench.

"This guy now has a pattern of doing these kinds of outrageous things," Kanter said.

Johansen, who has been a state judge since 1992, has had previous questions about his conduct. He was given a reprimand from the Utah Judicial Conduct Commission after he slapped a 16-year-old boy who allegedly became belligerent and insulting in his chambers in 1995.

Three years ago, a woman filed a complaint against the judge after he told her to cut off her 13-year-old daughter's ponytail in court in order to reduce the girl's sentence for cutting off a 3-year-old girl's hair.

The Human Rights Campaign has also filed a complaint with the Judicial Conduct Commission, which can recommend a judge's removal. That group alleged Johansen discriminated against the couple based on sexual orientation and called for a quick decision ahead of the custody hearing next month.