For nearly four decades, couples wishing to adopt American Indian children out of troubled situations have faced several hurdles, including giving the child’s tribe a chance to find suitable tribal parents first.

Now some prospective adoptive parents, Indian birthparents and members of the adoption industry are challenging the laws and regulations involved.

“The...