Evidence-based economics: the fundamentals

Michael Joffe

Imperial College London (m.joffe@imperial.ac.uk)

The possibility of evidence-based theory

Economics has been transformed in recent decades. Evidence is now abundant, resulting from the ever-increasing availability of datasets of diverse types, and improvements in econometric and statistical methods as well as in causal inference. Most academic studies now involve evidence –although it is true that this is mainly confined to econometric and statistical evidence. Does this mean that economics is already, or is becoming, evidence-based?

No, this is not happening in any systematic way – although there are some examples of good practice (see below). Increasingly, evidence is used as the basis of policy advice – although whether or not this is acted upon is a separate question. But economics, in the sense of providing an account of how the economy works, still has an uneasy relationship with evidence. What would evidence-based economics, in this sense, mean? How would it differ from the present situation?

The basic issue is, where does theory come from? I am using “theory” here in the way that natural scientists, such as biologists, use it. This is different from its usage in economics, where theory is taken to be a model, a collection of models, or a modelling approach. The prevailing situation in much of economics is that abundant evidence coexists with “standard theory”, the traditional modelling approach. The main problem with standard theory is that because it is derived from assumptions and axioms rather than from real-life observations, it has no systematic connection with reality. Much of it may be a good representation of how the economy works, but much of it is not. Often it is schematic and out-dated. But too often, the abundant availability of evidence has not yet been sufficient to confine inadequate theory to oblivion – it survives as a relic of traditional belief.

Lessons from the natural sciences

One fundamental barrier is that the habitual way of thinking shared by many economists, including heterodox ones, is that one cannot think seriously and systematically about any topic unless one has a model. Natural scientists such as biologists use models, but their primary language is causal, and this applies also to evidence-based economics. The basic elements of a science are accurate description and classification of phenomena, and identification of the causal processes that bring them about and maintain them, plus the way that these causes interrelate. It is a combination of empirical work and causal explanation, developed from an iterative process; explanations may be derived from generalizations directly based on the evidence, or they may involve an inspired imaginative leap that explains the empirical observations. The result is an empirically-based causal theory that identifies the “correct” causal processes – the ones that correspond with those operating in the real world.

Typically, such a theory involves heterogeneity, incompleteness and multiple causation. The reason for heterogeneity is that universal laws are rare – this is as true of biology and geology as it is of economics. For example, the core principles of the germ theory of disease apply to measles, rabies, tuberculosis and malaria, but in very different ways because the causal processes are so different. The same applies to the economy: e.g. institutional and cultural differences between countries are profound, and they can change over time. Empirically-based causal theories are incomplete: not all diseases are caused by infectious agents, and not all micro-organisms cause disease – most are harmless, and some are even beneficial. And multiple causation is the rule, e.g. the impact of an invading horde of micro-organisms may depend on unrelated factors such as nutritional status or genetics. These three features are likely to apply to any complicated open-ended system, not only living systems, including the economy.

Can economics really be evidence based?

Can such a way of working be applied to economics? It is not as difficult as it may sound! In economics, one example of good practice is the account of how money is created in a modern economy. This was generated by a systematic description of the way that money is moved around the economy, and how it is created and destroyed. Close attention was paid to basic accounting principles, using balance sheets, and to the actual behaviour of the various types of participant. After the new economics foundation (nef) brought out a book describing the process, the Bank of England published essentially the same account. And it transpired that those working in central banks had known all along that the standard version as presented in textbooks was wrong, and that the nef account was essentially correct. Its obviousness to participants was not a sign of naivety, but rather – as so often – a confirmation of its validity.

Another example is from the analysis of international capital flows. The copious flow of money from China to the US that started at the beginning of the century has been a conspicuous feature of recent decades. Under the term “global imbalances”, some economists believe it was an important factor in the financial crisis of 2008. In understanding its causes, one can start from various types of evidence: institutional/historical, information on the ability of Chinese manufacturing to compete internationally, data on profitability and saving rates, data on international capital movements, etc. Or one can start from standard theory. The first of these approaches was adopted by high-quality journalists in such orthodox publications as The Economist, and is also how evidence-based economics would proceed. The second is the standard procedure of academic economists, who have missed the obvious, and correct, account. Instead, they have opted for an explanation not of the phenomenon itself, but of its deviation from what is considered “typical” according to standard theory. Part of this is the assumption that the same causal processes apply universally, whereas it is well established that in China, and elsewhere in East Asia, the patterns of capital movement have been quite different from those familiar from European and American experience. This wild goose chase has led to a notion involving financial under-development and its effect on saving – which is not even correct empirically for East Asian economies.

Methodological and normative issues

Various quasi-philosophical arguments are frequently put forward as reasons why such evidence-based accounts of aspects of the economy are impossible. One is that experiments are impossible in economics, in contrast with the natural sciences. This ignores both the fact that experiments are now common in economics, e.g. laboratory “experimental economics” and randomized controlled trials, and that most branches of biology and geology are purely observational, not experimental.

A second is that the economy is too complicated, unlike the physical world. But the living world is hugely complicated too – and like the economy, open-ended and therefore not susceptible to a deterministic approach. The difference is that the way biologists have gone about finding causal explanations has been more successful than the standard methodology of economics.

A third objection is that all observations are theory laden, so one has to start from theory. The problem here is that this is taken as a requirement that a tight specification of a hypothesis should be brought to the data – but this approach may prevent the true account from emerging. One needs enough theory to specify what variables could be relevant, but a straightjacket is counter-productive.

A fourth suggested obstacle is “the under-determination of scientific theory by the evidence”, in other words that there are many possible explanations of any phenomenon – so one cannot choose which evidence-based explanation is correct. This basic idea is nowadays disputed by philosophers of science, and in any case in economics the problem is often the reverse: that none of the suggested hypotheses can account for the main observed features of the phenomenon. For example, in trying to explain why the unprecedented “capitalist” growth of the past 200 years has its particular observed spatial and temporal distribution, none of the standard theories comes close.

A fifth suggested problem is less technical and more normative: that economics is and should be a “moral science” – that it is inevitably value-laden, so that disagreements about how the economy works will be based on ideology. But although one’s values may influence one’s priorities, e.g. how important inequality is, or stability, the way the economy works is a question of what the causal processes are, not of how one would like them to be – when these two conflict, to choose truth over wishful thinking is necessary, and may require moral courage.

Some of the nuts and bolts

Many economists still think that “evidence” is only of one kind, i.e. statistical/econometric analysis. Whilst this is important, it is not enough on its own. One reason for its privileged position may be that it is typically contrasted with “anecdotal evidence”, which is unreliable. But the truth is richer than that.

It is true that basing one’s thinking about the economy on one or more stories carries the danger that one will just favour the narrative that suits one’s preconceptions. Any item of evidence needs to be representative of the underlying reality in some way – in fact this is true also of statistical analyses. And subjective bias (wishful thinking) needs to be avoided, whatever the type of evidence. In addition, any kind of evidence is fallible, so that caution is required. This applies as much to statistical analyses as to any other type – in medicine, it is commonplace to find that the results of epidemiological studies fail to be replicated – the rule of thumb is that a plurality of studies, as with cigarette smoking and lung cancer, is required before one accepts a finding as truly established.

The history of science shows clearly that secure knowledge derives not only from iteration between evidence and theory, but also that it typically depends on a variety of types of evidence – the more diverse the better. The implication for economics is that econometrics needs to be augmented with other approaches. A particularly valuable one is comparative economic history – highlighting the similarities and the differences between the experiences of different economies. Other important methods include behavioural and experimental economics, field trials, randomised controlled trials, institutional analysis, survey analysis, etc.

Ideally, the evidence base should encompass evidence both of the “difference-making” variety – e.g. that lung cancer rates are far higher among cigarette smokers than in non-smokers – and evidence that throws light on the mechanisms or capacities involved.

Models can be extremely useful. They are necessarily simplifications – “as simple as possible but no simpler”. This means that there is a danger that important causal relationships will be omitted, and it may not even be realised that this has been done, as notoriously happened with the pre-crisis DSGE models that ignored the financial sector, thereby implicitly assuming it worked perfectly well. In the natural sciences such as biology and geology, models are typically embedded in an empirically-based causal theory, which has been generated by iterating evidence and theory as outlined above. The advantages are (a) a richer causal understanding is obtained, and made explicit; and (b) it is clear what variables have been omitted in the modelling process.

Conclusion

With a different approach to research, economics could move to a situation where our account of how the economy works is based on evidence. Some exemplars of good practice can be derived from the natural sciences. This would have two important consequences, one negative and one positive: (i) things that are known to be untrue would no longer be presented as fact, e.g. in textbooks; (ii) economic theories would be based on evidence.