An experimental approach to the right answers
By Hal Weitzman
Published: April 20 2009 03:00 | Last updated: April 20 2009 03:00
Steven Levitt has an unusual admission to make for someone who has just finished teaching a class at the University of Chicago's Booth School of Business.
"I don't know that much about business," says the economics professor whose 2005 book, Freakonomics , made him an unexpected household name.
"Four years ago, I especially didn't know much about business," Prof Levitt says, "but, when Freakonomics came out, somebody decided to call it a business book - which it wasn't meant to be - and, when you write a best-selling business book, you automatically become a business expert. That makes businesses want to talk to you - so, over the past four years, my colleagues and I have spent a lot of time talking to businesses."
As an economist suddenly thrust into the world of corporate America, Prof Levitt says he was struck by the anomaly between economic textbook descriptions of business behaviour and what actually goes on inside companies.
"What's so strange to an economist who walks into a business is that economists have a set of models that describe how businesses should optimally respond. But that's not how businesses make decisions.
"That's not to say the economists' models are necessarily right. The business model can be sensible. It's usually pretty seat-of-the-pants, built round a set of rules of thumb, but that makes sense because the world is so complex and they have to make so many decisions that they can't optimise every one. But there are some decisions that are too important to make guesses on - and, in those cases, you either need to find data to help you or to generate your own data through experiments."
That view led Prof Levitt to team up with John List, a fellow economist at the University of Chicago who has been conducting field experiments and teaching experimental techniques for 15 years. Prof List says that, after working with companies such as United Airlines and Chrysler, he reached the same conclusion.
"The level of experimentation is abysmal," says Prof List. "These firms do not take full advantage of feedback opportunities they're presented with. After seeing example after example, we sat down and said, 'We have to try to do something to stop this.' One change we could make is to teach 75 to 100 of the best MBA students in the world how to think about feedback opportunities and how to think about designing their own field experiments to learn something that can make their company better."
The two economists decided to team up to develop a course for Booth students on "Using Experiments in Firms" - the first time either had taught at the business school.
They say data-gathering experiments at companies fall into two broad cate-gories: accidental experiments - for example, in which companies automatically generate data about how responsive customers are to changes in prices - and those that are deliberately designed and carried out, such as constructing an experimental framework to find out what optimal pricing levels would be.
Although they clearly favour the latter, Prof Levitt says they also want students to recognise and analyse the accidental experiments their companies are already undertaking. "The nice thing about accidental experiments is that they're free.
"If you can figure out the answer without having to design and conduct an experiment, that's wonderful. But a lot of everyday activities that businesses undertake could be transformed into experiments with almost no effort and almost no cost. The way businesses operate more and more lends itself to being able to run these real-world experiments. The lessons are enormous and the costs are often trivial.
The intent is not necessarily to churn out MBA graduates who will insist on conducting experiments when they return to corporate life, says Prof Levitt, so much as to provoke them to think about the issues.
"We teach the students that there's a hierarchy of decision making," he says. "One thing you can do is what's always been done. The second thing is you could just think about whether what you're doing is the right thing. The third step in the hierarchy is you could take the data you have and try to analyse it. But the problem is that it's often not the right data. The fourth rung would be an accidental experiment. The fifth would be to go out and generate the idealised data you would like to have."
Realistically, companies may not have the resources or the know-how to design experiments and analyse data, so when they look for information, they often turn to outside experts. Prof Levitt says he impressed on the students that they should be sceptical of such analysis.
"Although it seems incredibly simple, even analysing the data trips people up and the way you design the experiment affects how much data you get out of it and how generalisable those data will be to other settings," he says.
Prof Levitt gives the example of a company that was told by consultants that sales surged when it advertised on television - without realising that its advertisements appeared just before big holiday sales days. "The consultants had missed an obvious point," he says. "It wasn't that advertising caused their sales to go up but that knowing their sales were going to go up caused them to advertise a few weeks earlier."
The message seems to have got through.
"We learnt not to take data at its face value," says Hannah Levine, one of the Booth students on the course. "You really have to understand what tests were run to test the hypothesis and work out whether the analysis was correct."
Prof Levitt suggests that the lesson is just one of a number of fresh perspectives that he and Prof List have brought to the Booth MBA. "Even if our students don't become experts at data-crunching, at least they've thought about how you can be misled with data, how to think about correlation and causality - issues that have tended not to be central to an MBA's education."
Meghan Sheehan, who will join the Global Leaders Programme at Barclays Bank in London after she graduates from Booth this year, says she can imagine using the techniques she learnt to look at issues such as how the bank can discover the best way for its customers to keep in touch.
"You could design experiments to test the relative virtues of telephone, internet and branch visits," says Ms Sheehan. "How could you incentivise people to bank online rather than in a branch?
"Do you use a negative incentive to make in-person banking more difficult or do you offer an added benefit?"
These are exactly the sorts of questions Profs List and Levitt are trying to provoke. "We're on a proselytising mission of bringing a different way of thinking," says Prof Levitt.
"We're trying to bring about a revolution in business, so this is the first shot over the bows. This is an important battle to change the way businesses think."
Changing business thinking means placing datagathering at the heart of business practice.
"With the right data, you can improve your decisions," says Jasper Platz, another Booth student.
Copyright The Financial Times Limited 2009