The Boy Scouts of America is considering filing for bankruptcy protection as it faces dwindling membership and escalating legal costs related to lawsuits over how it handled allegations of sex abuse.

Leaders of the Boy Scouts, one of the country’s largest youth organizations, have hired law firm Sidley Austin LLP for assistance with a possible chapter 11 bankruptcy filing, according to people familiar with the matter.

Founded in 1910, the Boy Scouts group says that more than 110 million people have participated in its educational programs, which promote outdoors skills, character-building and leadership.

The Boy Scouts have been at the center of sexual-abuse scandals in the past, and the organization is facing a number of lawsuits that allege inappropriate conduct by employees or volunteers in incidents dating back as far as the 1960s. Filing for bankruptcy would stop the litigation and would give the nonprofit a chance to negotiate with those who have sued.

Other organizations facing similar legal pressure have also turned to bankruptcy protection in recent years. More than 20 Catholic dioceses and religious orders have filed for chapter 11 protection to negotiate payouts to thousands of victims. And last week USA Gymnastics, the governing body for the sport, filed for bankruptcy as it faces lawsuits from decadeslong sexual abuse by the national team’s former doctor Larry Nassar.

The Boy Scouts released a letter to its employees Wednesday that said it plans to “explore all options available to ensure that the local and national programming of the Boy Scouts of America continues uninterrupted.”

Participation in the organization’s programs has fallen in recent years, though the group opened some of its programs to girls and transgender boys. The Boy Scouts currently have more than 2.3 million youth members. The Church of Jesus Christ of Latter-day Saints, formerly one of the group’s largest sponsors, has said it will withdraw from Boy Scout programs. The church said it would develop its own program for young men.

The Boy Scouts group has drawn scrutiny over its slow pace to become more inclusive, including by lifting a ban in 2015 on gay men and lesbians serving in leadership roles.

The group has also dealt with fallout from its decision last year to expand its recruitment of girls, putting it in competition with the Girl Scouts of the USA, a separate group that offers similar programming and has also seen membership fall.

In November, the Girl Scouts filed a trademark lawsuit against the organization, saying its push for girls to join caused confusion and led to instances in which parents mistakenly signed their children up for Boy Scouts programs.

A Boy Scouts spokeswoman said the group is reviewing the lawsuit and that the expansion came “after years of requests from families who wanted” to participate in its programs.

In recent years, the Boy Scouts group’s legal bills for work done by some outside law firms have grown. In 2017, the organization paid $7.6 million to labor and employment law firm Ogletree Deakins, according to public filings. It paid Ogletree $3.5 million in 2016 and $859,347 in 2015, the filings show.

In the group’s latest annual report, Boy Scout officials said its future financial situation will partly depend on the outcome of sex-abuse-related litigation and future damages awarded. One lawsuit unfolding in Idaho over alleged abuse by several former leaders is set for a jury trial in May.

The organization has said that it has never knowingly allowed a sexual predator to work with youth members and has put rules in place to strengthen protections.

In the annual report, officials said they were “aware of threatened and expanding litigation of a similar nature.” They added that the group’s financial health will also depend on how much of the litigation costs will be covered by insurance.

The organization has sued its insurers at least twice since 2013, accusing them of failing to cover costs related to previous sex-abuse lawsuits. In its annual report, organization leaders said the latest lawsuits could force the group “to pay damages out of its own funds to the extent the claims are not covered by insurance or if the insurance carriers are unable or unwilling to honor the claims.”

—Jonathan Randles contributed to this article.

Write to Katy Stech Ferek at katherine.stech@wsj.com