This article first appeared in the Newsletter of the Royal Economic Society in 2014. It is republished here with thanks.

Michael Joffe

Imperial College London

In a recent edition of this Newsletter, I argued for the centrality of evidence in the context of the teaching of economics.[i] This perspective suggests a second issue: can economics itself be evidence based? The context of teaching is relatively straightforward conceptually, because one can take the standard curriculum and the content of textbooks as representing “economics”, as made manifest to students at undergraduate and postgraduate levels. And there is a clear contrast between that over-schematic portrayal of economics on the one hand, and the work of a large proportion of academic economists on the other, much of which is empirical – and a great deal of which is of excellent quality. But what would it mean to say that economics should be evidence based?[ii]

Evidence based economics in context

The phrase “evidence based” has become current in other branches of knowledge in recent decades. Its first widespread use was in relation to medicine, reflecting the view that medical practice relied too heavily on passed-down traditions and on clinicians’ intuition, derived from experience, which is highly fallible.[iii] Its usage has since spread to encompass not only other clinical disciplines, including psychological therapies, but also education, as well as public policy. These are quite different in many respects, but all share the feature that they are interventions. In such contexts, therefore, “evidence based” is primarily a matter of efficacy/effectiveness, and sometimes also of efficiency. If one allows an extremely wide definition of technology, they are all examples of technology assessment.

By “evidence based economics”, I mean something rather different: that evidence should be used as the basis for economics in the sense of finding out how the economy works. Pursuing the analogy with medicine, it is much more like the body of research underlying modern clinical practice that long predated the evidence based medicine initiative. This is what transformed medicine from something that often did more harm than good, e.g. the indiscriminate use of leeches, to the situation in the mid- to late-twentieth century, by which time a great deal of research ensured that diagnosis and clinical management were based on secure scientific foundations. This included laboratory research (physiology, pharmacology, etc) – both observational and experimental – as well as epidemiology which provides the empirical basis for prevention and which is observational. These types of research continue to play an important role in biomedical science.

The essential point here is that a highly diverse body of scientific knowledge was assembled, largely as knowledge for its own sake rather than directly for practical application. This then facilitated major developments in the practice of medicine and disease prevention. Subsequently, the evidence based medicine movement provided an additional impetus, promoting rigorous assessment of specific interventions.

This context is important for understanding the role of different types of research. Randomised controlled trials (RCTs), and meta-analyses of RCT evidence, are regarded as the gold standard for assessing effectiveness. However, RCTs have contributed very little to the body of scientific knowledge that provides the deeper understanding that has underpinned medical practice for more than half a century. This suggests that in evidence based economics, in the sense of improving our understanding of how the economy works, a range of different types of evidence is appropriate – assuming that the analogy is valid.

If it’s untrue, don’t accept it

The phrase “evidence based economics” implies that our account of economic phenomena should be derived from systematic observation of the world, rather than e.g. from axioms. There are two basic principles, one positive and one negative. The negative one is more straightforward, so I will start with that: things that are known to be untrue should not be accepted as fact. When I have raised this perspective at public meetings, it immediately generates plenty of candidate untruths that should not be taught (if the context is the curriculum), or accepted as scientific truth. One that is frequently suggested is money: its nature, how it is created in a modern economy, and along with that, the role of banks. Others are drawn from labour economics, international trade, industrial organisation, etc.

However, many if not all of these may well be contestable – not everyone agrees that each of them is untrue. A systematic discussion is needed to disentangle these issues, on the basis of evidence not of theoretical prejudice (or ideology). Such discussions may already exist within the context of particular research topics and sub-disciplines, where the discourse does focus on empirical adequacy. If so, the emerging conclusions need to be made available to the economics profession as a whole, and ideally also to be accessible to non-economists, including policy makers, who rely on economists for vital background information. Otherwise, their default belief is likely to revert to the over-schematic stories of elementary textbooks. This suggests the need for a discussion forum, a topic to which I return at the end of this article.

Is it possible to build theories that are based on evidence?

Some people believe that the positive side of evidence based economics is more problematic. Economic life is complicated, and constantly changing. It is not easy to obtain all the relevant evidence, and causal inference can be tricky. We should not, however, be excessively discouraged: rich datasets have started to proliferate in recent decades, and statistical methods of analysis have greatly improved. Indeed, many sub-disciplines of economics are already moving in an evidence based direction.

There are also complementary approaches, including comparative economic history, institutional economics, behavioural economics, field trials, randomised controlled trials, survey analysis, etc. To the extent that evidence of widely different types is found to favour the same hypotheses, the more can secure conclusions be drawn.

The important, and difficult, word in evidence based economics is “based”. There is no lack of good evidence in economics – the issue is its relationship with theory. One aspect is that core theory shows a remarkable ability to survive, irrespective of its relationship to the real world.[iv] The proposition that good theory could be constructed on the basis of evidence requires that (a) there is enough evidence of the appropriate types; (b) it is possible to overcome the intrinsic difficulty of constructing theory from evidence.

Let us look first at the adequacy of evidence. In some cases, there may be a lack of empirical research in an area that is theoretically important. For example, the distribution of profit rates across firms within each sector, or across the real economy more broadly, is at the heart of the evolution of market structure. It also represents the variation in the success of different firms, as measured by their profitability, which is hardly a trivial matter. And yet little evidence exists.[v]

Another possible concern is that empirical work may be undertaken that focuses only on the more superficial aspects of a topic, albeit something that is important from a policy viewpoint, leaving the core of the theory unexamined – from a theoretical perspective, the research may not be ambitious enough. Related to this, when theory predicts a particular outcome, whereas something different is repeatedly observed, these falsifying examples should be used to call into question the theory that made the wrong prediction, systematically using the argument of modus tollens. Thus, bubbles are repeatedly observed, yet some interpretations of orthodox theory have been claimed to predict that they are impossible . This indicates that some element of the theory needs to be modified, because its prediction is known to be invalid. And then, one can go beyond mere criticism, and use that observation as a clue to aid development of better theory.

But suppose that the obstacles relating to evidence can be solved. Is it not inherently problematic to generate good theory on that basis? I contend that the difficulties involved have tended to be greatly exaggerated, and that biology provides instructive examples of good practice.

Where do explanatory categories come from?

Explanatory concepts can arise out of the subject matter, by repeated and systematic observation. Biology is a good model: for instance, the concept of a mammal was introduced by the great classifier Linnaeus in 1758, thereby making sense of numerous previous observations by naturalists. Dolphins and bats were now classified as mammals, rather than respectively fish and birds – and so were humans. Bringing these into the same group as quadrupeds had the potential to make sense of their physiology as well as their evolutionary history. A century later, the over-arching evolutionary explanation was provided by Darwin, and research commenced into mammalian physiology. The fact that such disparate types of evidence now fit well together greatly reinforces the case that “mammal” is a good explanatory category – one of the strengths of biology is the way these different sub-disciplines correspond.[vi]

The standard sequence is: multiple observations, generalisation, classification, explanation, and finally confirmation. There are numerous ways that this can be done in economics. One source of evidence is “data-first” econometric analysis using the best available data and techniques, based on broad economic concepts that are not tightly tied to one particular pre-specified theory, thus allowing the development of new explanatory concepts.[vii] Comparative economic history is another. In economics we would do well to develop the capacity to derive explanatory categories inductively, from evidence, in a more systematic way.

Theories and models

A feature of economics, which may have increased in recent decades, is that theory tends to be equated with modelling. Indeed, what is called economic theory is in fact a modelling approach – a large and elaborate one to be sure – with a substantial number of specific component models. The use of models is not itself problematic; the difficulty arises when they are used as a description of reality.

A model has assumptions as input, and a “story” to make it work. It may well be informed by evidence but its essence is simplicity. A theory, in the sense used here, is more elaborate: a well-substantiated explanation of some aspect of the natural world, based on a body of facts that have been repeatedly confirmed through observation and experiment. It is cumulative and alterable, in that it can be added to or improved as new scientific information is gained. The germ theory of disease is a good example: it has its model-like aspects – an entity that is capable of reproduction, entering the host, and causing symptoms – but the details differ for different diseases, according to evidence on their nature, their mechanism of action and the symptoms they cause. There are models too, for example of the spread of epidemics, informed by evidence about each specific disease.

There is no reason why economics should not also adopt a similar structure: a theory that integrates evidence of different kinds, ideally setting out the causal mechanisms and the effects they have, and that is able to adapt to new evidence. And then embedded within that, a series of models each of which selects particular features, so that one can readily see what has been included in the model, and more importantly, what has been omitted. Under those conditions, it would be harmless to use simplifying assumptions for modelling – temporarily as it were – not mistaking the simplified version for a description of the world. That this is a danger in the current practice of economics is shown by the use of DSGE models that omitted possible financial frictions, which are widely considered to have proven extremely dangerous in the build up to the financial crisis of 2007/08.

As already indicated, evidence plays multiple roles in developing and confirming theories. Ideally one has evidence for the inputs, the mechanism, and the difference that the mechanism makes to the inputs, i.e. the output. In addition, as I have argued before, many different types of evidence can be useful – statistical, behavioural, historical (narrative and comparative), surveys, etc.

Conclusion

It may well be that what I am advocating already applies in particular research topics or sub-disciplines. In such a case there is likely to be a need for the participants to make that information more widely available to colleagues in other sub-disciplines, and indeed to non-economists who may depend on economics for an important part of their background information, e.g. policy makers. After all, such sub-disciplines as international trade or labour economics have wider importance than just to the economists who specialise in those areas. Without that, the outsiders are likely to be misinformed, e.g. by having an over-schematic view of that research area.

We have now reached the point where the availability of datasets, sophisticated analytic methods, etc means that economics could become a fully empirically based discipline. This might entail some alterations in how theory is perceived, but there is no intrinsic barrier – and some useful exemplars to follow in the natural sciences, especially biology. In addition, the generation of theory from evidence should be accorded substantially higher professional status than is currently the case.

To fulfil the need for a forum, the website http://www.evidence-based-economics.org has been set up. It welcomes contributions from all types of economists, and indeed also from non-economists who are engaged in relevant work. There is no length requirement, but brevity is greatly welcomed, as is clarity. The content should address some aspect of the relationship between evidence and theory in economics, preferably with a primarily substantive orientation (rather than philosophical – there are other fora for that). For example, contributions could present a body of evidence that contradicts or confirms existing theory – not only mainstream theory. Or it could be: a body of evidence that is “in search of a theory”, i.e. consistent observations that currently lack a good explanation; clarification of a basic classificatory term (e.g. investment, or capital); a concept that is indispensable in thinking about the economic world yet is missing from current economic theory; or a careful descriptive study on how a particular sub-system of the economy works. Another possibility could be the evidence for and/or against a particular type of policy, e.g. quantitative easing, or privatisation. In such a case they should trace the implications “backwards” into theory, not just be pragmatic, to see what the more general implications are – most such policies have been justified on the basis of some theory, and their success or lack thereof can throw some light on that. Each contribution will be open to online comments in the usual way. Potential contributions should be sent to me at m.joffe@imperial.ac.uk.

Notes:

[i] http://www.res.org.uk/view/art6Oct13Features.html

[ii] The phrase “evidence-based economics” has previously been used, notably by the philosopher of economics Julian Reiss: see in particular his paper “Evidence-based economics – issues and some preliminary answers” (2004) Analyse & Kritik 26: 346-63; and his book Error in economics: towards a more evidence-based methodology (2008) Abingdon, Routledge. There is also a doctoral programme on the subject, based in Munich: see http://www.evidence-based-economics.de/home.html and a letter in the last issue of this Newsletter. Another contribution is by Sean Harkin, in World Finance: http://www.worldfinance.com/home/contributors/evidencebased-economics. The phrase “evidence based economics” has also been used in a number of articles, to bolster a particular viewpoint, sometimes without any actual evidence being presented. On the other hand, many authors have embraced a position close to that argued here but without using the phrase “evidence-based economics”, e.g. Katarina Juselius (2011) ‘Time to reject the privileging of economic theory over empirical evidence? A reply to Lawson’ Cambridge Journal of Economics 35: 423–36.

[iii] The founding text was Cochrane AL. Effectiveness and Efficiency. Random Reflections on Health Services. London: Nuffield Provincial Hospitals Trust, 1972. (Reprinted in 1999 for Nuffield Trust by the Royal Society of Medicine Press, London (ISBN 1-85315-394-X).) The Evidence Based Medicine movement started in the 1980s, mainly at McMaster University, and in the 1990s resulted in the foundation of The Cochrane Collaboration (http://www.cochrane.org/). A classic paper is Sackett DL, et al. (1996) ‘Evidence based medicine: what it is and what it isn’t.’ BMJ 312 (7023): 71–2. doi:10.1136/bmj.312.7023.71.

[iv] This applies to a number of heterodox traditions, not just to neoclassical theory. There is sometimes a tendency to restrict empirical work to the testing of hypotheses that are heavily theory-dependent. On the other hand, it is obviously desirable to keep the aspects of any existing theory that describes the actual economy well.

[v] This has been investigated by Julian Wells (2007) in ‘The rate of profit as a random variable’ – see http://eprints.kingston.ac.uk/9490/.

[vi] Joffe M (2013) ‘The concept of causation in biology’ Erkenntnis DOI 10.1007/s10670-013-9508-6.  http://link.springer.com/article/10.1007%2Fs10670-013-9508-6.

[vii] Katarina Juselius, op cit.