Filetype PDF | Posted on 20 Aug 2022 | 4 years ago
Authentication
digital resources
290x Filetype PDF File size 0.56 MB Source: www.bitss.org
American Economic Association
Replication in Empirical Economics: The Journal of Money, Credit and Banking Project
Author(s): William G. Dewald, Jerry G. Thursby, Richard G. Anderson
Source: The American Economic Review, Vol. 76, No. 4 (Sep., 1986), pp. 587-603
Published by: American Economic Association
Stable URL: http://www.jstor.org/stable/1806061
Accessed: 09/07/2010 14:14
Your use of the JSTOR archive indicates your acceptance of JSTOR's Terms and Conditions of Use, available at
http://www.jstor.org/page/info/about/policies/terms.jsp. JSTOR's Terms and Conditions of Use provides, in part, that unless
you have obtained prior permission, you may not download an entire issue of a journal or multiple copies of articles, and you
may use content in the JSTOR archive only for your personal, non-commercial use.
Please contact the publisher regarding any further use of this work. Publisher contact information may be obtained at
http://www.jstor.org/action/showPublisher?publisherCode=aea.
Each copy of any part of a JSTOR transmission must contain the same copyright notice that appears on the screen or printed
page of such transmission.
JSTOR is a not-for-profit service that helps scholars, researchers, and students discover, use, and build upon a wide range of
content in a trusted digital archive. We use information technology and tools to increase productivity and facilitate new forms
of scholarship. For more information about JSTOR, please contact support@jstor.org.
American Economic Association is collaborating with JSTOR to digitize, preserve and extend access to The
American Economic Review.
http://www.jstor.org
in Empirical Economics:
Replication
Project
and Banking
Credit
of Money,
The Journal
BY G. THURSBY, AND RICHARD G. ANDERSON*
WILLIAM G. DEWALD, JERRY
research. It
This paper examines the role of replication in empirical economic
and data from
collected programs
study that
presents the of a two-year
findings to Our
their results. research provides
authors and attempted replicate published
and causes of failures to replicate
about the extent
information
new and important
in
errors
Our suggest that inadvertent
published results in economics. findings
than occurrence.
a rather a rare
are commonplace
empirical articles
published
The confirmation of research findings for knowledge-does not fit within the
which defines the
researchers is " paradigm
through replication by other puzzle-solving"
Scien-
research.
an essential part of scientific methodology. in scientific
reward structure
in not awarded
William Broad and Nicholas Wade Be- are
tific and professional laurels
trayers of Truth (1983) present examples for replicating another scientist's findings.
wherein the inability of other researchers to Further, a researcher undertaking a repli-
as imagination
replicate published scientific findings re- cation may be viewed lacking
vealed both inadvertent errors and outright and creativity, or of being unable to allocate
fraud. Replications in the physical and social his time wisely among competing research
sciences are attempted infrequently, how- projects. In addition, replications may be
ever. Thomas Kuhn (1970) emphasized that interpreted as reflecting a lack of trust in
replication-however valuable in the search another scientist's integrity and ability, as a
critique of the scientist's findings, or as a
Final-
personal dispute between researchers.
ly, ambiguities and/or errors in the docu-
*Dewald is Senior Economist, Bureau of Economic may leave
and Business Affairs, U.S. Department of State, mentation of the original research
unable to distinguish between
Washington, D.C. 20520. Thursby and Anderson are the researcher
Associate and Assistant Professors, respectively, De- errors in the replication and in the original
partment of Economics, The Ohio State University, study. Months of effort may yield the repli-
Columbus, OH 43210. We extend our thanks to the inconclusive results the
and Banking and to the pres- cator only regarding
Journal of Money, Credit validity of the original study, and thus no
ent editors of the JMCB for their cooperation and foundation for his future research in the
decision to continue to request that authors submit data area. These circumstances nurture a natural
along with papers for review. We thank the National studies.
replication
contract to undertake
financial under reluctance
Science Foundation for support
SES-8112800. We extend special thanks to the authors of Money, Cred-
who responded to our requests for programs and data, In July 1982, the Journal
many of whom spent significant amounts of time and it and Banking, with financial support from
We thank the National Science Foundation, embarked
project.
for this
resources compiling materials upon the JMCB Data Storage and Evalua-
Hashem Dezbakhsh, Roger Lagunoff, and Joseph
Hohman for research assistance, and Jeanette White, of the the JMCB
tion As part Project,
the JMCB secretary, for supervising the receipt and Project.
distribution of data sets. We thank Ed Kuh and MIT's adopted an editorial policy of requesting
from the and data used in
Center for Computational Research in Economics and authors programs
and
programs
and these
Management Science for loaning us the TROLL com- their articles making
puter program. We also thank Benjamin Friedman, data available to other researchers on re-
Lawrence Goldberg, Edward J. Kane, Thomas Mayer, quest. In a second part of the Project, we
Daniel Newlon, Edward J. Ray, Anthony Saunders, attempted replication of published results
George Stigler, Stephen Stigler, Roger Waud, and of data sets. Our
Geoffrey Woglom for their comments and assistance. for a number the submitted
ours. findings suggest that inadvertent errors in
Responsibility for errors remains
587
THE AMERICAN ECONOMIC REVIEW SEPTEMBER 1986
588
inter-
private
however,
published empirical articles are a common- For other researchers,
place rather than a rare occurrence. While est prevailed and our request was either re-
correction of the errors did not affect the fused or ignored. We note that NSF Policy
authors' conclusions in most studies, the er- Number 754.2 requires that computer pro-
rors make independent replication impossi- grams and data which have been produced
ble unless the replicator errs in precisely the with the assistance of NSF grants be made
either by pub-
same way. available to other researchers
In one crucial aspect, replications in em- lication, duplication, or loan to the re-
pirical economics are simpler than in the searcher. Investigators have the first right of
experimental sciences: given the researcher's publication, but the NSF rule requires that
computer programs and data set, calcula- the programs and data be made available to
tions may be repeated. The required pro- others. It appears that this policy is seldom
grams and data are rarely available for rep- enforced and that investigators either are
lication, however. Many researchers employ unaware of the policy or unafraid of the
proprietary statistical packages such as SAS, penalties for failure to comply with it.
SPSS, TROLL, TSP, etc., which prohibit
Journals
Professional
Role of
copying the program; replication is impossi- I. The
without sub-
ble for the individual researcher authors'
access Professional journals disseminate
university)
sidized (for example, major
to the same computer hardware and soft- findings throughout the world. Our results
ware. Many other researchers utilize pro- suggest that journals take a more active role
assistants the quality of the results pre-
research in assuring
grams which they or their
have written in FORTRAN, Pascal, or other sented in empirical studies.' As editor of
Frisch for such
languages; interpretation and evaluation of Econometrica, Ragnar argued
in of that "In
these programs is difficult at best-and a role the first issue journal:
impossible at worst-without considerable statistical and other numerical work pre-
in the raw data
skill, experience, and the cooperation of the sented Econometrica original
Dean Leimer will, as a rule, be published, unless their
original programmer. and Selig excessive. This is important in
Lesnoy (1982), for example, traced the false volume is
to stimulate control, and fur-
conclusions of Martin Feldstein (1974) to a order criticism,
we discovered ther studies" (1933, p. 3). Today, most eco-
error;
computer programming except the JMCB do not
similar errors in some of our replication nomics journals
studies. Finally, we note that some research have editorial policies which facilitate repli-
of such cation of published results by requesting
projects employ computer programs from
enormous size and complexity as to all but programs and data sets authors.
guarantee that no other researcher will at- It is a matter of public record that errors
tempt replication of the study. The large- exist in published empirical studies. Recent
scale macroeconometric models such as the examples include Leimer and Lesnoy, cited
MPS model are members of this group. We above, and Frederick Siskind's 1977 correc-
discuss below our attempts at replication of tion of Finis Welch's 1974 minimum wage
a study based on the MPS model at Harvard study. Our research suggests that there are
University. many more unrecorded and undiscovered
Similar problems arise with data. Some cases similar to these.
in
data are confidential, having significant pro- The frequency and magnitude of errors
prietary value due to the difficulty and/or empirical articles raise serious questions re-
expense of their collection, while federal law garding the integrity of the refereeing pro-
makes other data available only for the in- cess of professional journals. Referees are
ternal use of employees of government concerned primarily with methodology, the-
such as the Federal Reserve
agencies System.
In the JMCB Project, many authors fur-
data had not
nished their data even when the of journals is
'An argument for the role professional
been fully exploited in their own research. presented by Edward Kane (1984).
76 NO. 4 DEWALD ETAL.: REPLICATION-THEJMCB PROJECT 589
VOL.
oretical specification, statistical estimators, tical tests of a hypothesis, in which
and importance of results; an author's pro- either the confirmation or the con-
grams, data, and calculations are typically tradiction of the author's statistical
assumed to be correct. While our findings tests is reported. For this task to be
suggest that this assumption often is unwar- reasonably economical, any author
ranted, we hesitate to suggest-due to the should be willing to provide his un-
cost).
(at
massive amount of time which would be derlying data to other scholars
required-that referees should be required Indeed, this behavior is a requirement
and for responsible scholarship.
programs
to check an author's computer p. 1295]
data. Our findings suggest that the existence [1975,
of a requirement that authors submit to the
journal their programs and data along with The editors subsequently added a new sec-
each manuscript would significantly reduce tion to the journal devoted to verifications
of We and contradictions of papers first published
the frequency and magnitude errors. this section con-
found that the very process of authors com- in the JPE. Invariably,
piling their programs and data for submis- tained papers employing either new data sets
sion reveals to them ambiguities, errors, or alternative statistical techniques; little at-
the
would be un- tention was paid to replication. Further,
and oversights which otherwise nor facilitated making
detected. JPE neither required
Our experience with authors who had not programs and data available for replication
prepared programs and data for submission attempts.
to the JMCB prior to submission of the The JPE experiment is a classic example
article is indicative of the difficulties. Many of market failure. The benefits of reduced
could not locate the data for the article, frequency and magnitude of errors in em-
many of the characteris-
while others had lost their programming. pirical articles share
the
Even when the programs and data could be tics of public goods: all who read journal
located, authors often had not kept a con- benefit from the knowledge that the research
temporaneous record of the progress of the reported in its articles has been more care-
the the quan-
research and could not reconstruct their re- fully monitored by researcher;
sults. In many cases-particularly in larger tity of benefits available to any single reader
journal;
the
universities-graduate student research as- is not reduced by others reading
sistants had conducted essential parts of the and it is difficult to induce the reader to
value (price) for better
research project, and after their departure it reveal his or her true
high
faces
A researcher
was impossible to reconstruct data sets from quality articles. single
original sources. costs in time and money from undertaking
In the of economic replication of a study and finds no ready
principle, marketplace which the social
research might be expected to provide a marketplace correctly prices
check against careless, undocumented em- and individual value of the good.
the sub-
pirical research. Since the editorial policies An editorial policy which requires
journal
of professional journals had failed to address mission of programs and data to the
the problems of replication of published has two significant advantages relative to a
work, Edgar Feige proposed in 1975 that laissez-faire system wherein interested re-
authors First,
minimum must contact directly.
"6... a editors searchers
as journal
standard, cost borne by a
could explicitly publicize the necessity of it substantially reduces the
full reporting of procedures and data..." researcher seeking to replicate original re-
(p. 1293). In response to Feige, the editors of search. The economic self-interest of the
journal
the Journal Political wrote: author in satisfying the editors of the
of Economy to the
assures that the materials submitted
We believe that the true remedy is journal are more complete and correct than
resort to the powerful force of compe- what if anything might be furnished to an
The provides a
tition. We believe that should individual researcher. journal
journals cost-effective clearinghouse for these materi-
be to accept alternative statis-
prepared
no reviews yet
Please Login to review.
