Many American Catholics are worried about the apparently weak religious faith, practice and commitment of Catholic youth today and what it portends for the church’s future. Some observers are less concerned, however. Four Catholic sociologists -- William D'Antonio, James Davidson, Mary Gautier, and Katherine Meyer -- for example, have suggested that some scholars (including yours truly) overstate the magnitude of the problem. The title of their Dec. 6 NCR essay, for instance, summarizes their view about my recent book, Young Catholic America: “Assumptions in study on young Catholics lead to unnecessarily grim outlook.” I wish they were correct. But they are not. The situation, in fact, is grim.

I agree with D'Antonio, Davidson, Gautier and Meyer that our respective research studies are methodologically sound and produce reliable data -- at least for answering certain research questions. The problem in this case is that my colleagues are relying upon data that are incapable of addressing the issues about the youth in question here. In their piece, my colleagues did not mention two crucial facts about our differing studies, the understanding of which helps to explain why we come to such different conclusions about Catholic youth.

First, the research of my four colleagues is based upon cross-sectional survey data of Americans who self-identified as Catholics at the time of the surveys. This enables them to study generational differences among Catholics, which is a valuable analytical concern, as far as it goes. However, this methodology systematically ignores ex-Catholics. By design, it is blind to “defections” of youth (and adults) from the faith, focusing only on those who are still in the fold. But that is like trying to evaluate college retention or company job satisfaction by only studying students who finish their degrees or studying current employees while ignoring the college drop-outs and disgruntled workers who already left the firm. The view produced is badly distorted.

By contrast, my National Study of Youth and Religion is a longitudinal panel study that tracked a large sample of the same youth respondents over 10 years of their lives. We are thus able to identify at any given time not only current Catholics, but also ex-Catholics. And we find (and will publish in a forthcoming report) that fully one-half of youth who self-identified as Catholic as teenagers no longer identified as Catholics 10 years later in their 20s. That is a 50 percent loss through attrition in one decade. If that number is not grim, I do not know what is.

This helps explain why, according to research by the Pew Research Center on Religion and Public Life, one in 10 adult Americans (13 percent) today is an ex-Catholic. So the reason D'Antonio, Davidson, Gautier and Meyer can disagree with my conclusions is not because they have relevant evidence that says something different; they do not even have the empirical evidence necessary to speak to the question at hand.

A second difference between our two studies may help to explain why my colleagues reach such different conclusions. While their research relied on analyses of survey data alone, my study collected and analyzed not only survey data but also about 1,000 total in-depth personal interviews with teenagers and emerging adults around the country. Social surveys are good for describing simple proportions and distributions at the big-picture level. But they suffer severe limitations in representing the intensity and complexity of the feelings, attitudes and moods of real people on the ground.

Personal interviews are necessary for learning about such outlooks and emotions. By conducting so many personal interviews, my study provided powerful gut-level insights into where youth really are in their views about faith and church, which led to our dismal conclusions. D'Antonio, Davidson, Gautier and Meyer’s survey-only research design could not produce that kind of immediate research experience and insight. So the reality of the church’s failure among so many youth likely was not as apparent to them.

Here, then, is the moral of this story. What may seem to be minor distinctions in research design often make huge differences in the kinds of evidence, comparisons and conclusions that research can produce and support. The survey studies of D'Antonio, Davidson, Gautier and Meyer are good for comparing differences in attitudes among existing adult American Catholic across generations (at least, the minority that is willing to answer surveys). But they are simply incapable of addressing and answering the question of the church’s more recent (lack of) success in forming and retaining its youth. Studies that are designed specifically to answer that question, by contrast, make this fact very clear, like it or not: The situation regarding Catholic youth and the church is indeed very grim.

[Christian Smith is the William R. Kenan, Jr. Professor of Sociology and Director of the Center for the Study of Religion and Society at the University of Notre Dame.]