Consider the last business book you read. What kind of evidence did it provide to support its claims? True, examples from the real world can be illuminating, but case studies have their limits. You need to know what those limits are to avoid getting duped.
In many management books, apparently successful companies, like Southwest Airlines (LUV), are dissected ad nauseum in an attempt to discover what makes them tick. Yet the many different accounts often disagree. Is Southwest recognized because of its strategy? That’s certainly a plausible view, but what about its culture? That is in many ways an equally compelling position, although one based on very different information. Could it be because of its strategy and its culture? And, oh yes, its leadership, an attribute that seems critical when the company is viewed from yet another perspective. When everything seems important, how are we supposed to know what to focus on?
This problem comes from an over-reliance on case studies to make conclusions, almost to the point of excluding other types of evidence. Careful observation can offer recommendations that might help you repeat — or avoid — a particular result. But we run into problems when we don’t take into account the limitations of case studies, which are as follows:
Description: What happened
Alfred Sloan’s 1963 book My Years with General Motors is representative of just how careful you need to be in order for a descriptive case study to be useful. Having served as president, CEO, and chairman of General Motors (GM) in a few different combinations from 1923 to 1956, Sloan’s book provided his perspective on his role at the company and the principles he used to fulfill that role.
Sloan leaves it to the reader to draw generalizations beyond his experiences, and is careful to make no claims to underlying or enduring insights. His observations, however, have inspired generations of managers and researchers as they have sought to specify ways to cope with challenges similar to the ones Sloan confronted.
For descriptions to go beyond the constraints of Sloan’s approach, we must be sure that the cases are broadly applicable. Once again, popular business books typically overreach on this front. There are some worthwhile exceptions but, in general, most of these books show one of two deep flaws.
One is to illustrate a theory that is based on lots of data with carefully chosen cases, but leave the larger population that justifies the theory invisible. In other words, we are offered “for instance” instead of proof. The second is to examine only carefully chosen outliers, usually high-performing firms. This is especially problematic since, without showing any connection between the sample and some larger population, there is no good reason to conclude that the sample is representative of anything other than itself.
Explanation: Why it happened
What happened is one thing; understanding why it happened is something else. Alfred Chandler, one of the great business historians, used case studies to help us understand the emergence of the multi-divisional organization. He focused on four companies: DuPont (DD), General Motors, Sears (SHLD), and Standard Oil in his seminal classic Strategy and Structure. Out of that research emerged a set of hypotheses that have been tested and found valid hundreds of times in hundreds of subsequent case studies, which have supported Chandler’s idea that in sufficiently diverse companies, organizing around business functions (accounting, production, marketing) is less effective than organizing around markets.
When it comes to the explanations offered by popular business analysis, however, there is rarely such care. Companies with great track records are immediately lionized as great companies for all sorts of reasons, and the business press always has a favorite. For a decade or more, it was Southwest Airlines; today it’s Apple’s (AAPL) turn: there are any number of competing arguments that can attribute success to varying combinations and types of strategy, culture, leadership, and other elements.
All of these explanations make sense, but the right explanation needs to make sense of all the pertinent facts, and it needs to make sense of them better than the alternatives. This is a standard that Chandler’s work rose to. But when it comes to today’s business case studies, unfortunately, it is often left to the reader to determine what the competing explanations might be and to weigh the evidence.
Prediction: What happens next
When it comes to prediction, it gets worse. Most business books move from overstated claims of explanation to entirely unjustified predictions based on the notion that because something seems to have worked for them over there it will work for you over here. This belief is based on a flawed assumption that using ideas that worked out in the past will somehow allow you to shape the future in a desired way.
The only way to justify a prediction is to make one and see if it holds up under repeated tests. I don’t know of any business book that has subjected its prescriptions to tests of predictive accuracy that bear even a distant relationship to the scientific method. (Well, I know of one. But I wrote it, so I’m not in a position to offer an opinion on its merits.)
The real world is a messy place, which can make controlled experiments impossible. But the inability to generate evidence of predictive accuracy does not allow us to change the rules of evidence. And it is dangerously misleading to treat fables as fact. Only when case studies are elaborations of a validated connection between cause and effect can they contribute to our ability to predict outcomes accurately.
Case studies can be useful in support of description, explanation or prediction, but in different ways and with different degrees of confidence. So the next time someone begins a sentence with “for example,” think twice.
Michael Raynor is the author of The Innovator’s Manifesto: Deliberate Disruption for Transformational Growth and is a director at Deloitte Consulting LLP. He lives just outside of Toronto.
