Some older American Catholics might remember a time when people thought of the Catholic Church like a family: hard to ignore and even harder to leave. But a new Pew Research Center survey of U.S. Catholics shows that at least some of these perceptions may no longer be entirely true.

Indeed, about half (52%) of all U.S. adults who were raised Catholic have left the church at some point in their lives. A significant minority of them returned, but most (four-in-ten of all those raised Catholic) have not.

Roughly two-thirds of those who have not returned (28% of all those raised Catholic) are now ex-Catholics. These are people raised in the church who no longer consider themselves Catholic in any way. We classified the remaining 13% as “cultural Catholics” – those who now have no religion or are part of another religious tradition (other than Catholicism), but nevertheless still identify as Catholic in some way.

Despite the temporary and lasting departures from the church, there still are deep reservoirs of loyalty to the institution. Nearly three-quarters (73%) of current Catholics say they have been part of the church their whole life. A similar share, seven-in-ten, also say they could never imagine leaving Catholicism, no matter what.

Even among younger generations, there is significant loyalty to the church: About half of Catholic adults (56%) under age 30 say they can’t imagine leaving the church.

There has been speculation that the popularity of Pope Francis and the atmosphere of change around his 2-year-old papacy might inspire many former Catholics to return to the church.

But when it comes to ex-Catholics, Francis has his work cut out for him. Only 8% say returning to the church is something they could imagine doing. More open to coming back are those raised in the church but who are now cultural Catholics: 43% say they can imagine returning to the Catholic Church someday.

So American Catholics – both in and out of the church – are still like a family, just one where not everyone is living under the same roof.