|
| |
Home
Statistical Significance versus Real Significance
SIGNIFYING NOTHING?
Too many economists misuse statistics
FIGURES lie, as everyone knows, and liars figure. That should make
economists especially suspect, since they rely heavily on statistics to
try and resolve a wide range of controversies. For example, does a rise
in the minimum wage put people out of work? Are stockmarket returns
predictable? Do taxes influence whether a company pays dividends? In
recent years, helped by cheaper, more powerful computers, and egged on
by policy-makers anxious for their views, economists have analysed
reams of statistics to answer such questions. Unfortunately, their
guidance may be deeply flawed.
Two economists, Deirdre McCloskey of the University of Illinois, and
Stephen Ziliak of Roosevelt University, think their colleagues do a
lousy job of making sense of figures, often falling prey to elementary
errors. But their biggest gripe is that, blinded by statistical
wizardry, many economists fail to think about the way in which the
world really works.
To be fair, statistics can be deceptive, especially when explaining
human behaviour, which is necessarily complicated, and to which iron
laws do not apply. Moreover, even if a relationship exists, the wrong
conclusions can be drawn. In medieval Holland, it was noted that there
was a correlation between the number of storks living on the roof of a
house and the number of children born within it. The relationship was
so striking that, according to the rules of maths that govern such
things, you could say with great confidence that the results were very
unlikely to be merely random. Such a relationship is said to be
"statistically significant". But the Dutch folklore of the time that
storks somehow increased human fertility was clearly wrong.
Examples of similar errors abound. W.S. Jevons, an English economist of
the mid-19th century, thought that sunspots influenced crop yields.
More recently and tragically, British mothers have felt the harsh
effects of statistical abuse. An expert witness frequently called to
give evidence in the trials of mothers accused of murdering their
children argued that the odds of more than one cot death in a family
were statistically so slim that three such deaths amounted to murder.
On this erroneous evidence, hundreds of parents have been separated
from their children and many others have been sent to prison.
A failure to separate statistical significance from plausible
explanation is all too common in economics, often with harmful
consequences. In a past paper*[1] Professors McCloskey and Ziliak
attacked other economists' over-reliance on statistical rather than
economic reasoning, and focused on one case in particular.
In the 1980s, the American state of Illinois launched a programme to
keep people off the dole. Economists asked whether its costs outweighed
its benefits. One study estimated that the programme produced benefits
that were more than four times as large as the costs. Although this
seemed a good deal for taxpayers--and other tests seem to support this
conclusion--the authors of the study rejected such a finding because
they found that their estimate was not statistically significant. In
other words, their results fell just short of 90% certainty--the usual,
though AD HOC, rule of thumb for most economic work--of not being
random.
But far from this being an unusual case, Ms McCloskey and Mr Ziliak
found that 70% of the papers published during the 1980s in the AMERICAN
ECONOMIC REVIEW (AER), one of the most respected journals of the dismal
science, failed to distinguish between "economic" and "statistical"
significance. They relied too much on numbers, and too little on
economic reasoning.
INCREASINGLY INSIGNIFICANT
The two had hoped things might be getting better in recent years. The
reverse seems to be the case. In their latest work**[2], Ms McCloskey
and Mr Ziliak looked at all the AER articles in the 1990s, and found
that more than four-fifths of them are guilty of the same sin. Indeed,
so pervasive is the cult of statistical significance, say the authors,
that ever more economists dispense altogether with the awkward question
of whether the patterns they uncover have anything meaningful to say
about the real world.
Examples are legion, and can be found in the work of very distinguished
economists. In a widely quoted study of the minimum wage two Princeton
University professors, Alan Krueger and David Card, claimed to show
that, contrary to what you might expect, a rise in minimum wages caused
less unemployment, not more. Though their statistics looked compelling,
professors McCloskey and Ziliak say, they seemed to indicate, at best,
a rise in employment so small as to be economically insignificant.
Moreover, the paper did not address why this surprising result might be
true (although the authors have discussed that question elsewhere).
Another paper criticised by Ms McCloskey and Mr Ziliak is one
co-written by Gary Becker, a Nobel-winning economist. This claims to
show that addiction is rational, mainly on the basis that people's
response to changes in price is statistically significant. This is
interesting, but does not really explain much. The three authors
offered little account of why people become addicted--an odd life
choice for a rational person to make.
Most fundamentally, argue Ms McCloskey and Mr Ziliak, the focus on
statistical significance often means that they fail to ask whether
their findings matter. They look, in other words, at things that are
statistically but not economically insignificant. Most people would
prefer their conclusions to be significant in both senses. Failing
that, economic significance is presumably the more important.
* "The Standard Error of Regressions". By Deirdre McCloskey and Stephen
Ziliak. Journal of Economic Literature, March 1996
**"Size Matters: The Standard Error of Regressions in The American
Economic Review". (Forthcoming in the Journal of Socio-Economics)
from: The Economist Jan
31, 2004
|