First a short overview, then my comments.

Alice Goffman’s book On the Run: Fugitive Life in an American City, is one of the major events in sociology of the last few years. How unusual is it for a book based on a sociology dissertation to get this treatment?

Cornel West endorsed it as “the best treatment I know of the wretched underside of neoliberal capitalist America.” Writing in theNew York Times Book Review, Alex Kotlowitz said it was “a remarkable feat of reporting” with an “astonishing” level of detail and honesty. The New Yorker’s Malcolm Gladwell called it “extraordinary,” and Christopher Jencks, in the New York Review of Books, predicted that it would “become an ethnographic classic.” Tim Newburn, a highly regarded criminologist at the London School of Economics, hailed On the Run as “gloriously readable” and “sociology at its best.”

On the other hand, the book has lots of critics — here’s James Forman in the Atlantic — who think Goffman’s research subjects aren’t representative of the poor Black communities she wants to describe. And then there’s the exploitation/privilege/outsider argument, summarized by Claude Fischer (I added the links):

A typical line of criticism charges that outsiders cannot accurately describe their subjects of study. For example, one highly circulated review of Goffman’s book, alarmed at her white privilege, describes the study as “theft,” abetting “fantasies of black pathology,” and possibly causing harm by revealing to police the tricks of hiding. “Inner city Philadelphia isn’t Alice Goffman’s home,” another reviewer writes, “and it’s not her job to turn it into a jungle that needs interpreting.” A Buzzfeed writer simply tweeted, “Ban outsider ethnographies.”

Comments

On the morals and ethics, I’m not going to draw conclusions, except to say that I agree Goffman was wrong to help try to find and kill the guy who killed her friend, if what she says is true.

On the social science, I also have a limited perspective, because I don’t really read the book as social science; I think it’s much more a sociological memoir — and I don’t mean that as a criticism of ethnography, to which I would not apply that label in general.

In fact, the book is least persuasive when she tries to be most dispassionate. On the issue of representativeness, for example, Goffman clearly is wrong when she writes:

Initially I assumed that Chuck, Mike, and their friends represented an outlying group of delinquents: the bad apples of the neighborhood. After all, some of them occasionally sold marijuana and crack cocaine to local customers, and sometimes they even got into violent gun battles. I grew to understand that many young men from 6th Street were at least intermittently earning money by selling drugs, and the criminal justice entanglements of Chuck and his friends were on a par with what many other unemployed young men in the neighborhood were experiencing.

That’s a non sequitur, with the typical slippery use of “many” (which I have probably fallen into myself). The fact is her entire project was shaped by the guy she initially fell in with, and his friends, and they were by any measure an “outlying group.” If you don’t see people who have multiple running gun battles as unusual, your perspective may be a little skewed.

Patrick Sharkey’s review puts their outlier status in the context of declining violent crime — including in Philadelphia’s most violent neighborhoods:

The decision to engage in violence can be thought of as a rational response to the pressures and threats that young men perceive in this environment. But the decision to fire a gun in public space is one that is now universally rejected by every segment of American society, and it is a decision that comes with clear, long-term consequences that are understood by all of the young men in the neighborhood that Goffman studies. … As callous as it sounds given the hardships faced by this young man and the lack of choices available to him in this situation, my sense is that most residents of the block where he chose to fire his weapon would consider attempted murder to be an appropriate charge and would have been happy if he were located by the police and sentenced to prison.

In a number of passages it’s impossible to differentiate what Goffman knows versus what she was told. When the facts are wrong, that’s unfortunate. For example, she writes, “By the time Chuck entered his senior year of high school in 2002, young women outnumbered young men in his classes by more than 2:1.” Was that his perspective? People are reliably bad at estimating group sizes. The actual ratio of women to men among Black 17-year-olds living below the poverty line and enrolled in school in Philadelphia was skewed, but only 1.65-to-1.

Nevertheless, her description of the many ways the incarceration empire impinges on the daily lives of poor Black people in Philadelphia is sometimes insightful and useful. I mostly agree with Goffman’s political description and conclusions about the injustices here, and if the book does some harm it also will do some good. I would be happy to see the volume come down in this discussion and for us to treat this is a regular book — despite the inordinate attention it gets outside of academia. It’s deeply flawed but it’s worth reading. Its systematic evidence is weak but it’s thought-provoking and offers lots of food for thought in research and policy debates. It’s well-written and its topic is important. I have nothing against her and look forward to what she will do next.

That said…

My extremely shallow expertise in qualitative research is unfortunately not balanced by a vast knowledge of survey research methods. But that is my relative strength here, and on that I’d like to register an objection to Goffman’s study — both the book and her 2009 article in American Sociological Review (official link here, Google for a free copy).

There is an understandable problem with reproducibility even in the best ethnographies. As evidence for her conclusions, I simply discount her ostensibly meticulous counting of events, like this:

In that same eighteen-month period, I watched the police break down doors, search houses, and question, arrest, or chase people through houses fifty-two times. Nine times, police helicopters circled overhead and beamed searchlights onto local streets. I noted blocks taped off and traffic redirected as police searched for evidence— or, in police language, secured a crime scene— seventeen times. Fourteen times during my first eighteen months of near daily observation, I watched the police punch, choke, kick, stomp on, or beat young men with their nightsticks.

To me this just means “a lot.” First, there is no denominator. That is, no way to gauge how prevalent these events were compared with anything else. “A lot” is a fine metric for this kind of observation, because all the insights come from the details that follow, not from the recitation of frequencies. (There also is a frustrating lack of precision in these passages. Consider: “I watched the police break down doors, search houses, and question, arrest, or chase people through houses fifty-two times.” What exactly happened 52 times? What do “and” and “or” mean in that sentence?) However, if you think of it as sociological memoir, those numbers mean something because they tell you about her evolving perspective and experience. Wow, I think, if I witnessed 14 police beatings first-hand that would really affect me.

That part of the study is not reproducible. But if you do a survey while you’re doing an ethnography, you’re still doing a survey. The survey is not an ethnography. That means you should (my opinion, not a rule) make available your instrument, methods employed, and the data. Goffman has said she destroyed her field notes so they couldn’t be subpoenaed, but I don’t see why that should apply to her survey data, which could have been stripped of identifying information and processed like any other survey with sensitive material. Unlike the recitation of event counts, the survey is potentially reproducible. She reports (a few) percentages. Other researchers could conceivably conduct similar research in a different place or time and draw useful comparisons; someone might even attempt to reproduce her survey just to see if they get the same result.

Let me back up. I can, without violating the copyright rules regarding quotation length, reproduce everything she wrote about the survey portion of her study. (The much remarked upon 50-page “methodological note” at the end of the book never mentions the survey.) Here I will compare what she said in ASR versus the book, not as a gotcha exercise but because there is so little information in either that the variation between the two may be informative.

In ASR (passages excerpted across several pages of the article):

The five blocks known as 6th Street are 93 percent Black, according to a survey of residents that Chuck and I conducted in 2007. … Of the 217 households surveyed, roughly one fourth received housing vouchers. In all but two households, members reported receiving some type of government assistance in the past three years. … In the survey that Chuck and I conducted in 2007, of the 217 households that make up the 6th Street neighborhood, we found 308 men between the ages of 18 and 30 in residence. Of these men, 144 reported that they had a warrant issued for their arrest because of either delinquencies with court fines and fees or for failure to appear for a court date within the past three years. Also within the past three years, warrants had been issued to 119 men for technical violations of their probation or parole (e.g., drinking or breaking curfew).

The footnote to that last passage clarifies that “in residence” doesn’t really mean living there:

I counted men who lived in a house for three days a week or more (by their own estimates and in some cases, my knowledge) as members of the household. I included men who were absent because they were in the military, at job training programs (like JobCorp), or away in jail, prison, drug rehab centers, or halfway houses, if they expected to return to the house and had been living in the house before they went away.

From this we learn that the survey included 217 households — and also that those 217 households make up the entire 6th Street neighborhood (not its real name). That seems to imply a 100% response rate, because the 217 households surveyed corresponds to the 217 that “make up” the neighborhood. (That is an excellent response rate.) That is also important because of what comes next. If there were 217 households in the neighborhood, but she had only done interviews with, say, half of them, it would have been very surprising to find 308 men ages 18-30 living there. As it is, it’s extremely unlikely there were 308 men 18-30 living in all 217 households. The footnote says she included men who only lived there part time, or who were away for prison or other institutional spells. Frustratingly, it doesn’t say how many in the sample this applied to.

Here’s what tripped me up about that: 308 men 18-30 in 217 households is 1.4 per household — too many, I thought. So I looked at the 45 census tracts in West Philadelphia that are 75% Black or more (based on this map) in the 2005-2009 American Community Survey, and calculated the average number of men in that age range as .57 per household. (Actually I did the estimate for 18-29 because of how the data are published.) There is one tract with 1.04 men per household, but that’s an outlier. If she’s getting almost 2.5-times more than the average in her sample (1.4 versus .57), there must be a lot of missing men living in the neighborhoods not counted by the Census Bureau. Besides wondering if that’s accurate, you also ought to wonder who answered the questions about those men who weren’t actually residing in their residences.

The book’s version, unfortunately, doesn’t help clear that up:

In 2007 Chuck and I went door to door and conducted a household survey of the 6th Street neighborhood. We interviewed 308 men between the ages of eighteen and thirty. Of these young men, 144 reported that they had a warrant issued for their arrest because of either delinquencies with court fines and fees or failure to appear for a court date within the previous three years. For that same period, 119 men reported that they had been issued warrants for technical violations of their probation or parole (for example, drinking or breaking curfew).

This says they interviewed all 308 men, which seems like just an editing mistake. But then of the 144 who “reported that they had a warrant,” was that out of 308 or out of some smaller number of men whom they actually interviewed? Did someone else answer for the men who weren’t there?

Later in the book she describes the women she interviewed in the same survey:

In our 2007 household survey of the 6th Street neighborhood, 139 of 146 women reported that in the past three years, a partner, neighbor, or close male relative was either wanted by the police, serving a probation or parole sentence, going through a trial, living in a halfway house, or on house arrest. Of the women we interviewed, 67 percent said that during that same period, the police had pressured them to provide information about that person.

If she had limited the sample of women to the same age range as the men (which there’s no indication she did) then you would expect, based on the Black West Philadelphia census tracts, about 1.2 men for every woman, or about 250 women in the sample. However, if she included women of all ages, it should have been even more. Why are there only 146 women? Without any more information — and there is no more information in the book — it’s impossible to figure this out.

The last thing you might like to know if you wanted to pursue these very interesting and potentially important results, is what the survey instrument was like, and how it was administered. This was more than 450 people, a decent size survey, with good potential to yield useful results. We know they asked race, housing and other government assistance, warrant status (including the type of warrant) for the men, and criminal justice status for the partner/neighbor/relative (asked separately?) of the women. What strikes me as challenging here is asking these very sensitive questions and getting such a high response rate. Especially given all we learn from the book, knocking on doors and asking people if they have any warrants seems like it wouldn’t always be welcome. How long were the interviews? Did they make multiple visits when people weren’t home? So knowing how they did it would be very helpful for future work.

In the end, besides what we learn from On the Run itself, I hope we learn from the debate over it how we can better balance the need for protecting research subjects — while learning a lot from them — with the imperative to conduct research that is transparent, verifiable, and (as much as possible) reproducible.

ADDENDUM: The importance of the survey

The survey plays a very small part in the book and ASR article, only mentioned a few times, but it is important because it does offer a hint of generalizability to her research. James Forman wrote in the Atlantic:

The fact that Goffman’s subjects have serious criminal histories impairs our ability to generalize from some of her findings. For example, her central characters are all wanted on warrants at one time or another, some of them repeatedly (Mike has 10 warrants altogether). To Goffman, this indicates that Philadelphia’s criminal-justice system issues too many warrants. But it may simply indicate that Mike and his friends are unusually criminally active. Perhaps anticipating this challenge, Goffman extends her inquiry beyond the most criminally active members of the community. When she conducts a door-to-door survey of Sixth Street, she finds that about half the men there were wanted on warrants over a three-year period. This is astounding; no previous researcher has reported such a high concentration of fugitives living in one community. This raises questions that Goffman doesn’t answer with precision, but that I hope she and others will explore in the future: How many of these warrants were for failure to pay court costs—which should rarely if ever be imposed on poor people in the first place—versus something more serious, such as skipping a court date? Does fugitive status affect the lives of less criminally involved young men in the same ways it affects the lives of Mike and his friends? If it does, and if other communities harbor equally large proportions of fugitives, Goffman has discovered a profound social problem that deserves further research and a policy response.

The survey thus represents an important avenue for the book’s impact on future research.

ADDITIONAL ADDENDUM: Goffman’s response

In a statement on her website, Goffman responds to several of the recent criticisms. The first is that she participated in a conspiracy to commit murder. Her description there doesn’t change my impression of the scene she describes. But it is interesting for what it reveals about her sense of the men at the core of her story: Chuck and Mike. She writes:

In the months before he died, Chuck was actively working to preserve a precarious peace between his friends and a rival group living nearby. His sudden death was a devastating blow not only to his friends and family, but to the whole neighborhood. After Chuck was shot and killed, people in the neighborhood were putting a lot of pressure on Mike and on Chuck’s other friends to avenge his murder. It seemed that Chuck’s friends were expected to fulfill the neighborhood’s collective desire for retribution. Many of the residents were emphatic that justice should be served, and the man who killed Chuck must pay.

This surprises me, because I did not get the sense from the book that the gang rivalry Goffman described had motivated a collective conflict between entire neighborhoods. That contradicts the comment from Patrick Sharkey above, which speculates that most people in the neighborhood would have been glad to see the police arrest and incarcerate the people who conduct running gun battles in the street. Maybe Skarkey and I are victims of do-gooder liberal attitudes, and really there are neighborhoods in Philadelphia that see themselves at war with whole other neighborhoods and clamor for more shooting.

As an aside, I recently read Ghettoside: A True Story of Murder in America, in which journalist Jill Leovy argues that poor Black communities are at once over-policed — as innocent people are harassed, violated, prosecuted, and incarcerated for minor (or no) offenses — and under-policed such that murderers in Black communities are much less likely to be arrested and convicted than are those who kill Whites. This latter neglect by police contributes to the real problem of violence because it encourages informal (violent) means of addressing conflict in the community. By Leovy’s reasoning, and Sharkey might agree, majorities of people in violent Black neighborhoods would like fewer drug users incarcerated, but more of the people like Mike and whoever killed Chuck incarcerated.

Of course, I don’t know the situation in the “6th Street” neighborhood. But it’s one thing for an activist or community leader to say, “Our neighborhood stands united!”, and another for a sociologist to speak of “the neighborhood’s collective desire.” The former may be effective politics, but the latter is likely an overly-simplified research conclusion.

Which brings me finally back to the issue of Goffman’s survey. In her response, Goffman says Chuck was actively working on neighborhood peace in the months before his death, which occurred in the summer of 2007. Remarkably, that is the same time that she and Chuck conducted their survey of all 217 households in the neighborhood, with a reported 454 interviews. The timing of the survey is not completely clear, but she wrote that it was in 2007. and in an interview she said the survey was in the summer (“Yeah, so we did this survey, Chuck and I, one summer. We interviewed the households in this four-block radius…”). So it’s either a head-scratching story or an incredibly impressive image: Chuck and Goffman conducting a door-to-door household survey at the same time that he’s negotiating a delicate gang truce, interviewing every single household in the neighborhood in the weeks leading up to Chuck’s death, gathering the information that would allow Goffman to in fact accurately speak of their “collective desire.”