Are Ethnographers Ever Wrong?
There are few law schools in the U.S. (and likewise, I would venture, in the U.K.) where the curriculum fails to address the importance of social science to law practice. Regrettably, social scientists do not always understand that the lessons can run in both directions, especially when it comes to the marshaling of facts. This became apparent to me when I read Robert Dingwall’s review of my recent book, Interrogating Ethnography, on Social Science Space. Dingwall, who is a consulting sociologist and a co-editor of the Sage Handbook of Research Management, panned my book, which of course happens to almost every author. But he also missed the entire point of the subtitle – Why Evidence Matters – which I fear may tell us something more troubling about the practice of sociology.
Steven Lubet is Williams Memorial Professor at the Northwestern University Pritzker School of Law and author of Interrogating Ethnography: Why Evidence Matters, and other books.
Dingwall begins by calling Interrogating Ethnography a “work of legal imperialism” which asserts “that the trial process establishes truth in a quite fundamental sense.” In fact, I said the opposite. Recognizing that “there is an uneasy relationship between truth and trials,” I pointed out that “wrongful convictions and other erroneous verdicts are an enduring problem, for which the adversary system has no absolute solution.”
Contrary to Dingwall’s description, I did not offer trials as a “gold standard” for determining “what really happened,” but rather as an extended metaphor for critically examining the reliability of factual claims. My premise is that ethnography tends largely – although not entirely – to lack a tradition of dialectic testing, and that the field could be strengthened if ethnographers would more often challenge the accuracy of each other’s observations. I admitted, of course, that lawyers and courts often get things wrong, but at least they do not “routinely allow questionable assertions to go unexamined.”
The concept of the “ethnographic trial” was actually originated by Princeton’s Mitchell Duneier, author of the esteemed ethnographies Sidewalk and Slim’s Table. Far from imperializing, my own contribution was to elaborate and expand on Duneier’s insights, by showing how the techniques of cross-examination and documentation can be adapted for use in social science, and particularly in ethnography.
Dingwall is highly skeptical of documentation, saying that such fact-checking can lead to “second-best” results. Here we simply disagree. Yes, documentation has its limitations, but so do human informants and interview subjects, whose statements are predictably affected by poor memory, self-interest, skewed perspective, implicit biases, the desire to please, and sometimes outright prejudice or deception. Dingwall may believe that ethnographers can effectively weed through all that, but the best of them – for example, Matthew Desmond, Mary Pattillo, Terry Williams, Kathryn Edin, and, again, Mitchell Duneier – realize that documentation is an essential tool for social scientists. Interrogating Ethnography therefore illustrated various means of fact-checking ethnographies, with results that ranged from disproof (to the extent possible, given the convention of anonymization) to solid verification.
This brings us to the heart of Dingwall’s review, which is his defense of Alice Goffman’s On the Run. I had no plans to revisit these issues – having already said my piece – but Dingwall’s argument presents a good opportunity to elucidate the role of factual accuracy in social science. In any case, he is right that my project began with a critical review of On the Run, although he tellingly mischaracterizes both my critique and Goffman’s book itself.
To show that my methods of verification are unacceptable, Dingwall says, “We can illustrate this by reference to a section of Goffman’s account that Lubet makes great play with, namely the avoidance of emergency rooms by young black men with injuries or other medical conditions for fear of screening or arrest.” He then claims to refute my critique by positing that “Goffman could be reporting a genuine belief based on a mistaken inference” which he himself has “no problem in imagining.” Here we have a straw person.
There is little doubt that Goffman’s informants really believed that they risked arrest in emergency rooms, which I acknowledged several times in Interrogating Ethnography. The problem, however, is that Goffman claims to be reporting on more than mistaken perceptions. She states that the police do in fact set up cordons at the emergency room door:
To round up enough young men to meet their informal quotas and satisfy their superiors, the police wait outside hospitals serving poor Black communities and run the IDs of the men walking inside. (55)
Goffman says this twice more in On the Run, both times as truth claims about what she calls a “standard practice,” and not as an account of mistaken inferences. I doubted the story and therefore set out to investigate it. Because Goffman refused to identify the hospitals she observed, I contacted personnel from every hospital that could reasonably be described as serving the poor Black communities of Philadelphia (none of whom were “counsel,” contrary to Dingwall’s supposition). Everyone denied the existence of police gauntlets at the door.
Dingwall brushes off my fact checking with a jaunty Britishism that usually implies mendacity: “Well, they would say that, wouldn’t they?” Not that my informants were necessarily lying, he quickly adds. Perhaps they were only “buffered from knowledge” of the true goings on.
“Lubet is simply uncritical about his own ‘evidence,’” Dingwall charges, ignoring the fact that my “evidence” (the scare quotes are his) included a statement from Philadelphia’s former deputy mayor for public security – an African American who had earlier been a public defender for over twenty years – to whom I provided the relevant excerpts from On the Run. “The passage about hospitals is not in any way a standard practice,” he said, “It is not a practice, period.” I also interviewed two other Philadelphia city officials, a high-ranking police officer, and two long-time public defenders, all of whom refuted Goffman’s claim. In addition, I cited three highly regarded ethnographies of similar populations in the same Philadelphia neighborhoods – two of them by African Americans, one of which was conducted in hospital emergency rooms – none of which mentioned police cordons. Finally, I cited five other journalists – writing in Philadelphia Magazine, New York magazine, the Chronicle of Higher Educations, Slate, and The Atlantic – who examined Goffman’s hospital story and found no trace of it in reality.
The question must be asked: If this evidence is insufficient for Dingwall, what could possibly satisfy him? On the Run relies exclusively on anonymous sources in masked locations, with dates and events intentionally jumbled, and with the author’s field notes burned. In contrast, I identified my principal informants, provided the dates and details of our communications, documented other evidence, and had everything vetted by an independent fact checker.
It takes impressive intellectual gymnastics to conclude that the ethnographer is presumptively right and everyone else may be so “buffered from knowledge” that there is no point to investigating further. Dingwall actually posits a possible police conspiracy to falsify arrest locations, thus keeping the hospital ID checkpoints a secret, to this day, from everyone except Goffman. (Tip: If your analysis involves a hidden conspiracy, there is probably something wrong with it. See, e.g., Trump, Donald and the “Deep State.”) The purpose of an ethnographic trial – analogous, though not identical, to a court trial or the replication of a randomized controlled trial – is to seek out and evaluate potentially disconfirming evidence. Sociologists, too, should apply a standard of proof more rigorous than “I can imagine it.”
A great virtue of ethnography lies in the close observation and thick description of real people in the real circumstances of their lives. This may of course include informants’ genuine beliefs, but distinctions must still be drawn between reported inferences and observed facts. That is where Goffman went wrong, by depicting non-existent (even if mistakenly inferred) police checkpoints as though they were real. Dingwall’s ex post rationalizations – inventing scenarios that might or could have occurred – are interesting, but they are not evidence.
As I point out in the conclusion of my book, there is no scholarly discipline other than ethnography that relies so heavily on anonymous sources and undisclosed locations, reported by a single investigator whose data remain unseen. Recognizing this, my goal in writing Interrogating Ethnography was not to attack the field, but rather to improve it by offering some useful means of testing and verification. I did not propose that law should overwhelm or dominate ethnography, but only that ethnographers can learn something from methods that have proved useful to attorneys. It is revealing that Dingwall’s review never once mentions that the book’s subtitle is Why Evidence Matters. Then again, he has rather unintentionally proven that it does.
