Investigations into a case of alleged scientific misconduct have revealed numerous holes in the oversight of science and scientific publishing
Sep 10th 2011 | from the print edition
ANIL POTTI, Joseph Nevins and their colleagues at Duke University in Durham, North Carolina, garnered widespread attention in 2006. They reported in the New England Journal of Medicine that they could predict the course of a patient’s lung cancer using devices called expression arrays, which log the activity patterns of thousands of genes in a sample of tissue as a colourful picture (see above). A few months later, they wrote in Nature Medicine that they had developed a similar technique which used gene expression in laboratory cultures of cancer cells, known as cell lines, to predict which chemotherapy would be most effective for an individual patient suffering from lung, breast or ovarian cancer.
At the time, this work looked like a tremendous advance for personalised medicine—the idea that understanding the molecular specifics of an individual’s illness will lead to a tailored treatment. The papers drew adulation from other workers in the field, and many newspapers, including this one (see article), wrote about them. The team then started to organise a set of clinical trials of personalised treatments for lung and breast cancer. Unbeknown to most people in the field, however, within a few weeks of the publication of the Nature Medicine paper a group of biostatisticians at the MD Anderson Cancer Centre in Houston, led by Keith Baggerly and Kevin Coombes, had begun to find serious flaws in the work.
Dr Baggerly and Dr Coombes had been trying to reproduce Dr Potti’s results at the request of clinical researchers at the Anderson centre who wished to use the new technique. When they first encountered problems, they followed normal procedures by asking Dr Potti, who had been in charge of the day-to-day research, and Dr Nevins, who was Dr Potti’s supervisor, for the raw data on which the published analysis was based—and also for further details about the team’s methods, so that they could try to replicate the original findings.
A can of worms
Dr Potti and Dr Nevins answered the queries and publicly corrected several errors, but Dr Baggerly and Dr Coombes still found the methods’ predictions were little better than chance. Furthermore, the list of problems they uncovered continued to grow. For example, they saw that in one of their papers Dr Potti and his colleagues had mislabelled the cell lines they used to derive their chemotherapy prediction model, describing those that were sensitive as resistant, and vice versa. This meant that even if the predictive method the team at Duke were describing did work, which Dr Baggerly and Dr Coombes now seriously doubted, patients whose doctors relied on this paper would end up being given a drug they were less likely to benefit from instead of more likely.
Another alleged error the researchers at the Anderson centre discovered was a mismatch in a table that compared genes to gene-expression data. The list of genes was shifted with respect to the expression data, so that the one did not correspond with the other. On top of that, the numbers and names of cell lines used to generate the data were not consistent. In one instance, the researchers at Duke even claimed that their work made biological sense based on the presence of a gene, called ERCC1, that is not represented on the expression array used in the team’s experiments.
Even with all these alleged errors, the controversy might have been relegated to an arcane debate in the scientific literature if the team at Duke had not chosen, within a few months of the papers’ publication (and at the time questions were being raised about the data’s quality) to launch three clinical trials based on their work. Dr Potti and his colleagues also planned to use their gene-expression data to guide therapeutic choices in a lung-cancer trial paid for by America’s National Cancer Institute (NCI). That led Lisa McShane, a biostatistician at the NCI who was already concerned about Dr Potti’s results, to try to replicate the work. She had no better luck than Dr Baggerly and Dr Coombes. The more questions she asked, the less concrete the Duke methods appeared.
In light of all this, the NCI expressed its concern about what was going on to Duke University’s administrators. In October 2009, officials from the university arranged for an external review of the work of Dr Potti and Dr Nevins, and temporarily halted the three trials. The review committee, however, had access only to material supplied by the researchers themselves, and was not presented with either the NCI’s exact concerns or the problems discovered by the team at the Anderson centre. The committee found no problems, and the three trials began enrolling patients again in February 2010.
Finally, in July 2010, matters unravelled when the Cancer Letter reported that Dr Potti had lied in numerous documents and grant applications. He falsely claimed to have been a Rhodes Scholar in Australia (a curious claim in any case, since Rhodes scholars only attend Oxford University). Dr Baggerly’s observation at the time was, “I find it ironic that we have been yelling for three years about the science, which has the potential to be very damaging to patients, but that was not what has started things rolling.”
A bigger can?
By the end of 2010, Dr Potti had resigned from Duke, the university had stopped the three trials for good, scientists from elsewhere had claimed that Dr Potti had stolen their data for inclusion in his paper in the New England Journal, and officials at Duke had started the process of retracting three prominent papers, including the one in Nature Medicine. (The paper in the New England Journal, not one of these three, was also retracted, in March of this year.) At this point, the NCI and officials at Duke asked the Institute of Medicine, a board of experts that advises the American government, to investigate. Since then, a committee of the institute, appointed for the task, has been trying to find out what was happening at Duke that allowed the problems to continue undetected for so long, and to recommend minimum standards that must be met before this sort of work can be used to guide clinical trials in the future.
At the committee’s first meeting, in December 2010, Dr McShane stunned observers by revealing her previously unpublished investigation of the Duke work. Subsequently, the committee’s members interviewed Dr Baggerly about the problems he had encountered trying to sort the data. He noted that in addition to a lack of unfettered access to the computer code and consistent raw data on which the work was based, journals that had readily published Dr Potti’s papers were reluctant to publish his letters critical of the work. Nature Medicine published one letter, with a rebuttal from the team at Duke, but rejected further comments when problems continued. Other journals that had carried subsequent high-profile papers from Dr Potti behaved in similar ways. (Dr Baggerly and Dr Coombes did not approach the New England Journal because, they say, they “never could sort that work enough to make critical comments to the journal”.) Eventually, the two researchers resorted to publishing their criticisms in a statistical journal, which would be unlikely to reach the same audience as a medical journal.
Two subsequent sessions of the committee have included Duke’s point of view. At one of these, in March 2011, Dr Nevins admitted that some of the data in the papers had been “corrupted”. He continued, though, to claim ignorance of the problems identified by Dr Baggerly and Dr Coombes until the Rhodes scandal broke, and to support the overall methods used in the papers—though he could not explain why he had not detected the problems even when alerted to anomalies.
At its fourth, and most recent meeting, on August 22nd, the committee questioned eight scientists and administrators from Duke. Rob Califf, a vice-chancellor in charge of clinical research, asserted that what had happened was a case of the “Swiss-cheese effect” in which 15 different things had to go awry to let the problems slip through unheeded. Asked by The Economist to comment on what was happening, he said, “As we evaluated the issues, we had the chance to review our systems and we believe we have identified, and are implementing, an improved approach.”
The university’s lapses and errors included being slow to deal with potential financial conflicts of interest declared by Dr Potti, Dr Nevins and other investigators, including involvement in Expression Analysis Inc and CancerGuide DX, two firms to which the university also had ties. Moreover, Dr Califf and other senior administrators acknowledged that once questions arose about the work, they gave too much weight to Dr Nevins and his judgment. That led them, for example, to withhold Dr Baggerly’s criticisms from the external-review committee in 2009. They also noted that the internal committees responsible for protecting patients and overseeing clinical trials lacked the expertise to review the complex, statistics-heavy methods and data produced by experiments involving gene expression.
That is a theme the investigating committee has heard repeatedly. The process of peer review relies (as it always has done) on the goodwill of workers in the field, who have jobs of their own and frequently cannot spend the time needed to check other people’s papers in a suitably thorough manner. (Dr McShane estimates she spent 300-400 hours reviewing the Duke work, while Drs Baggerly and Coombes estimate they have spent nearly 2,000 hours.) Moreover, the methods sections of papers are supposed to provide enough information for others to replicate an experiment, but often do not. Dodgy work will out eventually, as it is found not to fit in with other, more reliable discoveries. But that all takes time and money.
The Institute of Medicine expects to complete its report, and its recommendations, in the middle of next year. In the meantime, more retractions are coming, according to Dr Califf. The results of a misconduct investigation are expected in the next few months and legal suits from patients who believe they were recruited into clinical trials under false pretences will probably follow.
The whole thing, then, is a mess. Who will carry the can remains to be seen. But the episode does serve as a timely reminder of one thing that is sometimes forgotten. Scientists are human, too.
Correction: This article originally stated that by the end of 2010 officials at Duke University began the process of retracting five papers. That should have been three papers. This was corrected on September 8th.
Published: April 5 2011 22:51 | Last updated: April 5 2011 22:51
The paradox of large organisations is that they have virtually all the assets – money, brands, intellectual property, facilities, momentum – yet their natural dominance is frequently overturned by upstarts. How is it that emerging companies, with minimal resources, can defeat huge, established players?
The answer is that big corporates fall prey to a number of diseases, the pathology of which I describe below:
● Sunk cost fallacy: the idea that you should throw good money after bad. I have witnessed boards that continued to waste money on doomed projects because no one was prepared to admit they were failures, take the blame and switch course. Smaller outfits are more willing to admit mistakes and dump bad ideas.
● Groupthink: a condition whereby an error is perpetuated thanks to peer pressure, so it becomes orthodoxy. The belief that subprime mortgages were a sound investment was a classic. Virtually the entire finance industry fell for this myth, thanks to the perpetuation of false assumptions, and an unwillingness to break ranks and question the “experts”.
● An obsession with governance: increasingly, institutions favour compliance over competence, and box-ticking over practical solutions. I resigned from the board of a Nasdaq-traded corporation because – thanks to Sarbanes-Oxley – the audit committee meetings lasted longer than the actual board meetings.
● Institutional capture: the phenomenon whereby management end up running an enterprise for their own benefit, rather than for the real owners. This is also known as the principal/agent problem, and is epitomised by behaviour at banks. There, senior executives still think they are entitled to millions of dollars remuneration for an average job. In reality those executives are easily replaceable, while the value and market shares of their banks have actually been built up over decades by many others.
● Office politics: self-destructive infighting for power within large businesses is endemic, and perhaps the biggest value destroyer of all. A brilliant New York Times article last year by an ex-Microsoft executive revealed how the world’s largest software business had a viable tablet PC 10 years ago, which could have pre-empted the smash hit iPad from Apple. But other Microsoft divisions conspired to kill the project because they did not want resources being diverted away from their own imperial ambitions.
● Lack of proprietorship: when employees have no effective capital stake in an organisation, they tend to be less cost-conscious, and take a more cavalier attitude to waste, personal expenses and the like.
● Risk aversion: in most large corporates, the punishment for management failure is greater than the rewards for success. So, rational individuals pursue cautious strategies because they do not want to damage their career prospects. Entrepreneurs often have nothing to lose, and hence are more willing to take the plunge with innovations. Thus a solitary inventor such as James Dyson can beat the rich market leader Hoover because it saw only danger in new technology, while he saw a better vacuum cleaner.
● The burden of history: many older companies have legacy issues such as pension scheme deficits, union contracts, inefficient equipment and so on. By contrast, newcomers can outsource and use the latest technology.
● Anonymous mediocrities: there is nowhere to hide in a small company – if you can’t deliver, you’re out. But in a large outfit, also-rans can get away with poor work for years.
● Commodity products: large companies need large markets, which tend to be mature, more competitive and lower margin. Small firms are happy to sell niche merchandise, where returns are often higher.
The deck is stacked in favour of bigness. But, just as the lumbering dinosaurs could not adapt and became extinct, so giant businesses are likely to be less flexible, motivated and focused. The truth is that there are many intangible diseconomies of scale, which means leviathans are often less profitable, and usually much less fun.
Published: April 6 2011 23:14 | Last updated: April 6 2011 23:14
Classic strategy: Bob Deluce, Porter’s founder
The story. In 2006, aviation entrepreneur Bob Deluce launched Porter Airlines, a low-cost carrier, into the already crowded North American market.
The challenge. Several small, low-fare airlines had started in Canada to compete with the two dominant carriers: Air Canada, the national flag-carrier, and WestJet, a low- cost airline modelled on Southwest Airlines. Most start-ups focused on high-density routes and tried to gain market share by pricing lower than incumbents. This triggered fatal price wars with incumbents, and most observers were sceptical about Porter’s ability to take them on.
The strategy. Porter executed a classic strategy of finding an untested market space that avoided competing on price. Porter’s slogan, “Flying Refined”, reflected its tactic of providing a high-quality flying experience for economy-class fares. Perks included free wine and snacks on board, large airport lounges – with free WiFi, computers and espresso machines – and a frequent-flyer programme.
Much like Southwest, Porter recruited crew and ground staff with the right attitudes to customer service first and then training them for the job.
Porter’s best-value proposition was to base itself at Billy Bishop Toronto City Airport, a small airport located on an island in Lake Ontario, just five minutes’ drive – including a ferry trip – from downtown Toronto. Porter targeted business passengers travelling to and from the commercial capital of Canada during the week. It highlighted the contrast in terms of time and money with using the main Toronto airport – nearly an hour’s drive on a busy highway.
Porter purchased the shabby terminal building at the airport from the airport authority and, after a $50m refurbishment, converted it into a modern facility optimised to speed travellers through check in.
Until recently, it kept other airlines from using the terminal. It was embroiled in a dispute with Jazz, an Air Canada affiliate, which claimed it was unfairly being stopped from using the airport.
Air Canada recently received approval to launch from the airport and Jazz is poised to start a service to Montreal.
Business travellers came to rely on being on board less than 45 minutes after leaving downtown Toronto. Porter estimates that 1.3m passengers used the terminal in 2010.
Finally, Porter’s fleet of new Bombarbier Q400 turboprops were fitted with leather seats and extra leg room. Commercial jets are not allowed at the airport because of noise restrictions. Porter’s Q400 aircraft used vibration-cancelling technology to provide ride quality comparable to jet aircraft. The Q400 also has superior fuel efficiency to jets of comparable size.
Results. Porter launched in the high-density Toronto-Ottawa-Montreal market and offered high frequencies on these routes. It used heavy advertising in the target markets to woo early adopters who would become advocates. Porter later expanded scheduled services to six cities in eastern Canada as well as New York, Boston and Chicago. It is reported to have reached cash break-even two years after starting up, and is now reported to be profitable.
Porter’s main challenge now is to continue to grow without triggering a price war with Air Canada and WestJet, which may try to stop its erosion of their market share on key domestic and cross-border routes.
Key lessons. Even in a highly competitive market, it is possible for a new entrant to prosper if it can differentiate itself in a way that is difficult to copy.
Porter’s big advantage is its base close to the heart of Toronto, where it has locked up a majority of the landing slots and owns the terminal.
Other advantages were its focus on building loyalty early among its target audience, and the affordable Q400s that are well suited to its route structure of short-haul journeys.
The writer is professor of strategic management at the Rotman School of Management at the University of Toronto
Published: March 24 2011 13:26 | Last updated: March 24 2011 19:18
Li & Fung, a Hong Kong-based consumer goods sourcing and logistics company, warned that “a new era in sourcing with higher prices” has begun, as manufacturers pass on the rising costs of both raw materials and Chinese labour to customers.
The supply chain company, which sources products for companies including Walmart and Gap of the US, and Debenhams of the UK, on Thursday reported a 27 per cent rise in profits to HK$4.28bn ($550m) for 2010.
Bruce Rockowitz, president of Li & Fung Trading, said: “The biggest topic on the minds of everyone in this business is that higher prices are really here to stay. At this point, retailers are not sure what they can pass on to consumers and what they cannot.”
William Fung, the company’s group managing director, said heightened competition for labour in China, which has resulted in wage increases of about 20 per cent this year, heralded the end of China-led deflation for the world economy.
The company said higher prices had historically been good for its trading business. “For the last 20 years, because prices were going down we had to ship more pieces [to keep revenues growing],” said Mr Rockowitz. The company’s core operating margins rose from 3.82 per cent in 2009 to 4.56 per cent in 2010.
The higher labour costs in China have prompted Li & Fung to move labour-intensive work on products such as garments to countries with lower wages, such as Bangladesh, Vietnam and Indonesia.
The company said that China was now responsible for only 25 per cent of Li & Fung’s clothing sourcing, and Mr Rockowitz said Bangladesh and Vietnam were rapidly gaining share of its clothing business.
In spite of this shift, China’s share of the company’s total sourcing rose from 54 per cent in 2009 to 57 per cent in 2010 because of a series of acquisitions last year.
At its results briefing, the company, which uses “stretch” targets in three-year plans, announced a plan under which it hoped to reach $1.5bn in core operating profit by 2013, 50 per cent higher than in the previous plan.
In 2010, the company acquired Visage, a UK-based private-label supplier, Jimlar Corp, a US-based footwear maker, and its sister company Integrated Distribution Services, a logistics provider, which it said would be responsible for $2.8bn in annualised turnover.
It'll take some doing, but Stanford aims to put a T in "engineer"—not an easy fit.
The School of Engineering sees the task as its signature challenge in a rapidly changing world. It wants to reformat students of all shapes and sizes, intellectually speaking, into people who envision themselves as capital Ts—vertically supported by strong math and science training but stretching laterally with extensive business and communication skills. The prototypical 21st-century engineer, says Dean James Plummer, will be someone who combines a deep grounding in the technical fundamentals with an entrepreneurial outlook and an instinct for lifelong learning.
To foster that, Plummer says, "we are pushing the bounds of an engineering education." And that can be tough going, in part because everyone from accrediting examiners to each new wave of students must be willing to buy into some edgy rethinking. Stanford offers and promotes a prominent set of nontraditional courses, such as those involving design-based problem solving—but as electives that have to win acceptance as professionally valuable.
Can an exceptional technical curriculum be combined with a broad interdisciplinary mindset? It has to be, asserts Plummer.
"The half-life of engineering information is probably three to five years," says Plummer, citing the furious pace of technological advances. "We need to teach students how to keep learning. . . . They may even change technical fields multiple times."
Engineers who once would have remained employed for decades with the same large companies now require the business and collaboration skills to work effectively in the small firms and start-ups of the so-called new economy.
Careers will be spent in a global environment, with an international set of customers and clients. Engineers who find themselves living in different parts of the world will benefit from an understanding of diverse cultures and people. With that in mind, a summer internship program places students with companies in China for nitty-gritty experience.
Engineers aspiring to leadership positions will compete in creativity and decision-making abilities as well as technical knowledge and business smarts. And there are ways to tailor undergraduate programs to enhance those qualities, says Plummer.
Can an exceptional technical curriculum be combined with a broad interdisciplinary mindset? It has to be, asserts Plummer.
But every serious effort to integrate all these skills, notes Plummer, is susceptible to "real resistance against cutting into the technical part of the education." Four years ago, he points out, his school's increasing focus on a less traditional approach complicated its accreditation, although the maximum six-year renewal was received.
Thomas Stahovich, an associate professor of mechanical engineering at UC-Riverside, has written about the issues involved in remodeling engineering curricula. He sees the right balance of vertical and horizontal ingredients at Stanford.
"Both skill sets are essential for students to be able to put their knowledge into practice," he said by email. "Stanford's approach of infusing the curriculum with an emphasis on communication skills, innovation, international experiences, teamwork skills and the like should be quite effective at producing engineers who can make an impact on the world."
To implement its vision, Engineering particularly looks to two buzz-worthy assets: the Stanford Technology Ventures Program and the Hasso Plattner Institute of Design, or d.school, which uses interdisciplinary teams of faculty but was organized within the school.
The STVP curriculum, which emphasizes entrepreneurship and innovation in high-tech ventures, draws more than 1,000 students a year to its elective courses. Co-director Tom Byers, a professor of management science and engineering, and executive director Tina Seelig received the National Academy of Engineering's top award for teaching last year.
The d.school, headed by mechanical engineering professor and design-firm IDEO guru David Kelley, encourages the notion of tackling "nearly any challenge" with collaborative expertise. This spring, it moved into renovated space where expansive, highly adjustable work areas reflect the same spirit of innovation as the design projects.
Bulk of faculty have not lived and worked in multiple cities
By LEE U-WEN
BUSINESS schools in general are not doing enough to prepare their students to compete in an increasingly globalised world, according to a new report released by the Association to Advance Collegiate Schools of Business (AACSB).
'We did a formal survey of business schools and found that many of them regarded themselves as international if, say, they sent their students on exchange programmes overseas.' - Prof De Meyer
A large number of institutions, too, have a narrow view of what it means to be truly considered international, said Arnoud De Meyer, a member of the 12-strong task force that spent nearly three years putting together the 342-page report.
The task force is headed by Robert Bruner, dean of the University of Virginia's Darden School of Business. Others in the committee include Indian School of Business professor and dean emeritus M Rammohan Rao and Peter Wolnizer, dean of the University of Sydney Business School. The AACSB is one of the world's leading accreditation agencies for business schools.
The report's findings, which were presented to deans at a conference in Arizona last month, make a strong case for why this is a 'major turning point' in fundamentally changing the way business schools should organise themselves, said Prof De Meyer, president of Singapore Management University (SMU).
'This is much more than simply teaching a little more marketing in the curriculum or adding a course on internationalisation.'
'We did a formal survey of business schools and found that many of them regarded themselves as international if, say, they sent their students on exchange programmes overseas,' said Prof De Meyer, a former director of the Judge Business School at the University of Cambridge in the UK.
'However, this is just giving students the opportunity to have a different experience elsewhere, it does not mean that the school or the university is international.'
As far as faculty is concerned, the bulk of them lack the exposure of having lived and worked in multiple cities. The problem here is two-fold, said the 56-year-old management studies scholar.
First, many universities do not invest as heavily as they should in developing their teaching staff. The other hurdle that needs to be overcome is that academics would largely prefer to stick to their own discipline throughout their career, rather than branch out and become a global specialist in several areas.
'This is a big step for many faculty members, and there is that risk that you will be seen as being superficial,' he explained.
There is also a growing number of universities that have adopted the model of using multiple campuses to expand their reach in more markets, be it within their own country or overseas.
The downside, however, is that there is usually little interaction between the campuses as the different campuses tend to function independently, he said.
What's needed is for a repository to be created to consolidate best practices - in Prof De Meyer's words, a 'database of interesting ideas' - so that mistakes are not repeated and lessons are learnt.
As for SMU's own plans on how to ensure its graduates and staff are ready to cope with the challenges of a global world, Prof De Meyer said that the university had to invest more in the development of pedagogy and to expand its areas of research even further.
He also revealed that the management is exploring the possibility of 'company internships' that would see faculty spend a meaningful amount of time at an international organisation.
'We have to think about how SMU is projected outside of Singapore. Our brand is already very well known locally for our undergraduate education. Our research is good in some cases, but it is patchy and limited to Singapore,' he said.
'We owe it to the country on how we present ourselves to the world, so we are going through a strategic reflection on the internationalisation of SMU.'