Will big data keep its promise?

## Will Big Data Keep Its Promise?

Speech given by

Andrew G Haldane Chief Economist Bank of England

Data Analytics for Finance and Macro Research Centre, King's Business School 19 April 2018

The views expressed here are not necessarily those of the Bank of England or the Monetary Policy Committee. I would like to thank Shiv Chowla for his help in preparing the text. I would like to thank David Bholat, David Copple, Andi Joseph, Perttu Korhonen, Katie Low, Clare Macallan, Paul Robinson, Michael Saunders, Misa Tanaka, Silvana Tenreyro, Ryland Thomas, Arthur Turrell, Arzu Uluc and Michalis Vasios for their comments and contributions.

I am delighted to be here to launch the Data Analytics for Finance and Macro (DAFM) Research Centre at King's College Business School. I would like to congratulate Professors George Kapetanios and Georgios Chortareas, as Co-Directors (as well as former colleagues), for getting the centre to the launch pad and primed for take-off.

To get my punchline in early, I believe the application of data analytic techniques to the many pressing questions in finance and macro holds great promise. That is the reason the Bank of England, around four years ago, set up its own data analytics division. And that is why I very much welcome the setting up of this new centre, as a means of realising that promise.

But will Big Data keep its promise? I want to try and illustrate some of that promise of Big Data, as well as the potential pitfalls, by drawing on examples from recent Bank of England research on the economic and financial system. I will conclude with some, more speculative, thoughts on future Big Data research. 1

## The Path Less Followed

The first thing to say is that Big Data and data analytic techniques are not new. Nonetheless, over recent years they have become one of the most rapidly rising growth areas in academic and commercial circles. Over that period, data has become the new oil; data analytic techniques have become the oil extraction and refining plants of their time; and data companies have become the new oil giants. 2

Yet economics and finance has, to date, been rather reticent about fully embracing this oil-rush. For economics and finance, the use of data analytic techniques has been the path less followed, at least relative to other disciplines. One simple diagnostic on that comes from looking at the very different interpretations put on the expression 'data mining' by those inside and outside of economics and finance.

For economists, few sins are more heinous than data-mining. It is the last resort of a scoundrel to engage in 'regression-hunting' - reporting only those regression results which best fit the hypothesis the researcher first set out to test. It is what puts the 'con' into econometrics. 3 For most economists, such data-mining has unfortunate similarities with oil-drilling - a dirty, extractive business which comes with big health warnings.

For data scientists, the situation could not be more different. For them, the mining of data is a means of extracting valuable new resources and putting them to use. It enables new insights to be gained, new products to be created, new connections to be made, new technologies to be promoted. It provides the raw material for a new wave of productivity and innovation, an embryonic Fourth Industrial Revolution. 4

1 Cœuré (2017) offers an excellent summary of the potential for Big Data to improve policymaking, in particular in central banks.

3 Leamer (1983).

2 For example, the Economist (2017), Henke et al (2016).

4 See, for example, Schwab (2017).

What explains some economists' caution about Big Data? The answer lies, in part, in methodology. 5 A decent chunk of economics has followed in the methodological footsteps of Karl Popper in the 1930s. Popper championed a deductive approach to scientific advance. 6 That started with axioms, moved from axioms to theory and then and only then took hypotheses to the data. Theory, in other words, preceded measurement.

There is an alternative, inductive , approach. This has even deeper roots, in the work of Francis Bacon from the early 1600s. 7 This turns the telescope around. It starts with data, unconstrained by axioms and hypotheses, and then uses this to inform choices about models of behaviour. Data, in other words, precedes theory. Indeed, some data scientists have suggested such an approach could signal the 'End of Theory'. 8

So where some economists have tended to see the pitfalls in Big Data, data scientists have seen promise. Where some economists have tended to see the ecological threat it poses, data scientists have seen the economic potential. I am caricaturing a little, but only a little. So who is right? And does the era of Big Data signal an oil-rush or an oil-spill?

The truth, as often, probably lies somewhere in between. Both deductive and inductive approaches can offer insights into understanding the world. They are better seen as methodological complements than as substitutes. Put differently, using one approach in isolation increases the risk of making faulty inferences, and potentially serious mistakes, in understanding and policy. Let me give a couple of examples to illustrate.

During the global financial crisis, it is now fairly well accepted that the workhorse Dynamic Stochastic General Equilibrium (DSGE) model of the macro-economy fell at the first fence. 9 It was unable to account for business cycle dynamics, during or after the crisis. While theoretically pure, it proved empirically fragile. That empirical fragility arose, I would suggest, from a methodological over-reliance on deductive methods. Or, put differently, from too little focus being placed on real-world data from the past, including on crises.

As a counter-example, in 2008 Google launched a predictive model of flu outbreaks based on searches for phrases such as 'indications of flu'. 10 This did a terrific job of tracking flu outbreaks in the US in 2009-10. But that model's predictions broke down spectacularly in the following years. 11 That empirical fragility arose, I would suggest, from an over-reliance on empirical regularities and an over-adherence to inductive methods. Or, put differently, from too little focus being placed on the deep medical causes of past flu outbreaks.

In the first case, empirical fragility arose from too tight a set of axioms and restrictions, from placing too great an emphasis on theory over real-world correlations and historical experience. In the second, empirical

5 Haldane (2016).

7 Bacon (1620).

6 Popper (1934) and Popper (1959).

8 Anderson (2008).

- 10 Ginsberg et al (2009).

9 For example, Stiglitz (2018).

11 Lazer et al (2014).

fragility arose from too loose a set of axioms and restrictions, from observed empirical correlations being given too great a role relative to theory and causality.

In both cases, these mistakes might have been reduced had inductive and deductive approaches been used in a complementary, or iterative, fashion. This iterative approach has a strong pedigree in other disciplines. The history of advance in many scientific disciplines has involved a two-way process of learning between theory and empirics, with theory motivating measurement at some times, and measurement motivating theory at others, in a continuous feedback loop. 12

One example of this approach, discussed by Governor Carney at the time the Bank's own data analytics programme was launched, concerns the dynamics of planetary motion. 13 It was Sir Isaac Newton (a fellow money-printer, as former Master of the Royal Mint) who developed the physical theory of celestial motion. But this theory was built on the empirical shoulders of another scientific giant, Johannes Kepler. When it comes to planetary motion, empirics first led theory, the inductive led the deductive.

The same has been true, at times, when understanding the motion of economies and financial markets. Keynesian and Monetarist theory were both built on empirical experience during the Great Depression. The Phillips curve began life as a Kepler-esque empirical regularity, which was only subsequently given a Newtonian theoretical basis. Many puzzles in finance, which have taxed theorists for decades, began as empirical anomalies in asset markets. 14 In each case, empirics led theory, the inductive led the deductive.

My lesson from all of this is clear. If that iterative learning process between empirics and theory is to continue to bear fruit in economics, then deductive and inductive approaches may need to share a broadly equal billing. If so, I think there are high returns to economics and finance making a further intellectual investment in Big Data and accompanying analytical techniques in the period ahead, the path less followed.

## The Definition of Big Data

If Big Data holds promise, it is probably useful to start by defining just what it is. This is not entirely straightforward. Like beauty, what counts as Big Data lies in the eye of the beholder. It is also a fluid concept. For example, it is clear that data no longer means just numbers; it means words too. Indeed, growth in research on semantics has taken off over recent years, including in economics and finance.

What is less contentious is that there has been the most extraordinary revolution in the creation, extraction and capture of data, broadly defined, over the course of the past decade or so. This has been, in part, the

12 Bacon (1620) summarises this well: 'Those who have handled sciences have been either men of experiment or men of dogmas. The men of experiment are like the ant, they only collect and use; the reasoners resemble spiders, who make cobwebs out of their own substance. But the bee takes a middle course: it gathers its material from the flowers of the garden and of the field, but transforms and digests it by a power of its own.'

13 Carney (2015).

14 Obstfeld and Rogoff (2001) discuss six major puzzles in international macroeconomics, such as the excess volatility of exchange rates relative to fundamentals.

result of Moore's Law and accompanying advances in information technology. 15 Unlike oil, whose resources are finite, new data is being created at an unparalleled rate and has almost limitless supply.

It is estimated that 90% of all data ever created occurred in the past two years. 16 A good chunk has come courtesy of social media. Around 1.5 billion people use Facebook daily and 2.2 billion monthly. In 2017, there were 4.4 billion smartphone subscriptions, more than one for every second person on the planet. By 2023, there are projected to be 7.3 billion smartphone subscriptions, almost one for every person. 17 An estimated 1.2 trillion photos were taken in 2017, as much as 25% of all photos taken ever. 18

A different window on this information revolution is provided by looking at numbers of data scientists. Using vacancies data from the job search website Reed, there were recently over 300 UK job adverts for data scientists. 19 As recently as 2012, there were hardly any. Estimates based on self-identification on social networking site Linked-In suggest there may be upwards of 20,000 data scientists globally. 20

There has, at the same time, been rapid growth in new techniques for handling, filtering and extracting information from these data. Machine learning techniques are developing rapidly. So-called 'deep learning' techniques are complementing existing approaches such as tree-based models, support vector machines and clustering techniques. 21 Within text mining, dictionary techniques, vector space models and semantic analysis are rapidly gaining traction. 22

All of these methods offer different means of teasing out information, and making robust inferences, in situations where empirical relationships may be complex, non-linear and evolving and where data may be arriving at different frequencies and in different formats. These approaches differ significantly from classical econometric techniques for inference and testing often used in economics and finance.

This revolution in data provision, and in techniques to understand it, offers analytical riches. Mining those riches requires, however, considerable care. For example, issues of data privacy loom much larger with granular, in some cases personalised, data. These issues have, rightly, risen in prominence recently. At the same time as putting it to use, safeguarding Big Data is a key preoccupation of the Bank in its research.

## The Promise of Big Data

To the extent Big Data can be characterised, this is usually done using the 'three V's': volume , velocity and variety . Using the three V as an organising framework, let me discuss some examples of how these data

15 Moore (1965) noted the annual doubling in the number of components per integrated circuit.

- 17 Ericsson Mobility Report (2017)

- 16 SINTEF (2013).

- 18 See https://www.statista.com/chart/10913/number-of-photos-taken-worldwide/

- 20 Dwoskin (2015). The true number of data scientists worldwide is highly uncertain. Many individuals work on data science without necessarily using that job title, but the opposite is also true.

- 19 Using dataset in Turrell et al (forthcoming).

- 21 Chakraborty and Joseph (2017).

- 22 Bholat et al (2015).

and techniques have been used in recent Bank research to improve our understanding of the functioning of the economy and financial system.

## Volume

The statistical bedrock of macro-economic analysis, since at least the middle of the 20 th century, has been the National Accounts. The National Accounts have always relied on an eclectic range of data. 23 In the past, manor accounts on land use, crops and livestock were used to estimate agricultural output. Industrial production was measured by sources as varied as numbers of iron blast furnaces and books listed by the British Library. And services output was estimated based on merchant shipping tonnage. 24

With more data than ever coming on stream, that use of new and eclectic data sources and methods is, if anything, gaining further ground within statistical agencies. In the area of consumer price measurement, MIT's 'Billion Prices Project' uses data from over 1,000 online retailers in around 60 countries to collect 15 million prices on a daily basis. This approach has been found to offer a timelier (and cheaper) reading on consumer prices than traditional surveys. 25 Online price data have also been found to improve short-term forecasts for inflation in certain markets. 26

In the same spirit, the UK Office for National Statistics (ONS) is exploring the use of 'web scraping' to complement existing price collection methods. To date, they have focussed on items such as groceries and clothing. Although early days, the potential benefits in terms of increased sample sizes and granularity seem to be considerable. For example, the ONS have so far collected 7,000 price quotes per day for a group of grocery items, which is larger than the current monthly collection for those items in the CPI. 27

For GDP measurement, new sources and methods are also gaining ground. One recent study used satellite imaging to measure the amount of non-natural light emitted from different regions of the world. This has been found to have a statistically significant relationship with economic activity. 28 This approach could potentially help in tracking activity in regions that are geographically remote, where statistical surveying techniques are poor or where mismeasurement problems are acute.

A more down-to-earth example, used by the UK's ONS and other statistical agencies, is so-called administrative data. This includes data collected by government agencies as part of their activities - for example, on tax revenue and benefit payments. In the UK, some of these data have recently become available for wider use through the government's Open Data initiative, albeit subject to important checks.

23 Coyle (2014).

24 Fouquet and Broadberry (2015).

26 Cœuré (2017).

25 Cavallo and Rigobon (2016).

27 See https://www.ons.gov.uk/economy/inflationandpriceindices/articles/researchindicesusingwebscrapedpricedata/august2017update

28 Henderson, Storeygard and Weil (2011).

As one example, VAT data from SMEs in a subset of industries have recently been used by the ONS when constructing output-based estimates of GDP. As with prices, the gains in sample size and granularity from using such administrative data are potentially large. The ONS's Monthly Business Survey of activity typically relies on a sample of around 8,000 firms to represent this subset of SMEs. This is now being complemented by VAT returns from around 630,000 reporting units. 29

These new data augment, rather than replace, existing survey methods. They have the potential to improve the timeliness and accuracy of National Accounts data on aggregate economic trends. The ONS has its own Data Science Campus to spearhead these efforts. And new research organisations, such as the Alan Turing Institute, are doing excellent work applying new data and techniques to economic measurement.

Another potentially fruitful area of exploration, when tracking activity flows in the economy, is financial data. Almost all economic activity leaves a financial footprint on the balance sheet of some financial institution. Tracking the flow of funds between financial institutions can help in sizing that footprint and thus, indirectly, in tracking economic activity.

At the Bank, we have over recent years drawn on the Financial Conduct Authority's Product Sales Database (PSD). This is a highly granular source of administrative data on owner-occupier mortgage products taken out in the UK. It contains data on close to 16 million mortgages since mid-2005. The PSD has provided the Bank with a new, higher resolution lens on household and housing market behaviour.

For example, in 2014 the PSD was used by the Bank's Financial Policy Committee (FPC) to help inform and calibrate its decisions on setting macro-prudential restrictions on high loan-to-income mortgages to UK households. 30 Since then, we have used these data to track the characteristics of existing mortgagors with high loan-to-income and high loan-to-value mortgages over time. 31 PSD data have been used to understand pricing decisions in the UK housing market. 32 And they have also been used to calibrate a multi-sector, agent-based model of the UK housing market. 33

The Bank and the ONS have over recent years been developing a more comprehensive set of data on flows of funds between institutions. The hope is that these data help in tracking not only portfolio shifts but also how these might affect financial markets and the wider economy. For example, do portfolio reallocations by institutional investors affect asset markets and have knock-on effects for spending? 34 Answering questions like this helps, for example, when assessing the efficacy of quantitative easing. 35

29 https://www.ons.gov.uk/economy/grossdomesticproductgdp/articles/vatturnoverinitialresearchanalysisuk/december

30 June 2014 Financial Stability Report.

32 Bracke and Tenreyro (2016) and Benetton, Bracke and Garbarino (2018).

31 Chakraborty, Gimpelewicz and Uluc (2017).

33 Baptista et al (2016).

35 For example, Albertazzi, Becker and Boucinha (2018) show evidence of the portfolio rebalancing channel from the ECB's asset purchase programme.

34 Bank of England and Procyclicality Working Group (2014).

New, highly granular data are also coming on stream on payment, credit and banking flows. Some of these have been used to help predict, or track, movements in economic activity. They have had some success. For example, in the US a dataset of over 12 billion credit and debit card transactions over a 34 month period has recently been used to analyse consumption patterns by age, firm size, metropolitan area and sector. 36

In time, it is possible these sorts of data could help to create a real-time map of financial and activity flows across the economy, in much the same way as is already done for flows of traffic or information or weather. Once mapped, there would then be scope to model and, through policy, modify these flows. This is an idea I first talked about six years ago. Today, it looks closer than ever to being within our grasp. 37

These are all areas where DAFM could contribute importantly to efforts to improve the quality and timeliness of data on the macro-economy and financial system. As is well-recognised, the scope for improvements in the quality of National Accounts data is considerable. 38 And those measurement challenges will only become greater as we move towards an increasingly digital and service-oriented economy.

## Velocity

A second dimension of the Big Data revolution is its greater frequency and timeliness. Higher-frequency data can give new or timelier insights into trends in financial markets and the economy. It can also sometimes help with thorny identification problems, which otherwise plague both Big Data (as the Google flu example demonstrated) and classic econometric methods (as the DSGE example demonstrated).

The crisis revealed that, in situations of stress, some of the world's largest and deepest financial markets could be drained of liquidity. This resulted in some of these markets seizing up. In response, as one of their first acts, the G20 agreed in 2009 to collect a far greater amount of data on transactions in these markets, to help better understand their dynamics in situations of stress. 39 These data are stored in Trade Repositories.

Over recent years, these trade repositories have begun collecting data on a highly granular, trade-by-trade basis. This has meant they have quickly accumulated a large stockpile of data. For example, around 11 million reports are collected in the foreign exchange market each working business day. These provide a rich data source when making sense of high-frequency financial market dynamics and dislocations.

One example of such a dislocation came when the Swiss franc was de-pegged in January 2015. This unexpected move caused large shifts in asset prices. The franc exhibited a sharp V-shaped movement in

36 Farrell and Wheat (2015).

38 For example, Bean (2016).

37 Ali, Haldane and Nahai-Williamson (2012).

39 See, for example, FSB (2010).

the hours immediately after the de-pegging. By analysing trade repository data on forward contracts for the Swiss franc-euro exchange rate, some of the drivers behind these moves could be detected. 40

For example, high-frequency movements in the Swiss exchange rate can be compared to the volume of trades in forward contracts. These trades can be further decomposed by counterparty - for example, large dealer banks versus end-investors. This type of decomposition technique shows that it was the withdrawal of liquidity by the large dealer banks that caused the overshoot of the franc - a classic sign during times of market turmoil. 41 This move partially reversed once dealers resumed market-making.

The trade repository data can also be used to assess whether the franc de-pegging had any lasting impact on market functioning. The Bank's research found it did, with a persistent fragmentation in the franc forwards market. Liquidity and inter-dealer activity was structurally lower, and market volatility persistently higher, after this episode.

The extra granularity of these data makes it possible to tell a quasi-causal story about the drivers behind the V-shaped movement in asset markets after the de-pegging. Using tick-by-tick and trade-by-trade data side-by-side allows identification of the triggers and amplifiers, in a way which would not otherwise be possible.

A second example of research using higher-velocity data to improve our understanding of economic dynamics comes from the labour market. Understanding the joint behaviour of employment and wages remains one of the central issues in modern macro-economics. These dynamics have been complicated recently by changes in the world of work, with automation changing both the nature and structure of work.

Recent Bank research has used granular data on advertised job vacancies to shed light on these dynamics. 42 The research analyses around 15 million job vacancies over a ten year period. Instead of classifying jobs by sector, occupation or region, it uses machine learning techniques on the text describing jobs to classify and cluster vacancies. What results is a more 'job description-based' classification scheme for labour demand.

This approach provides a different way of classifying and describing how the world of work is evolving - for example, the types of skills required in the face of automation. The classification scheme has also been useful when identifying the relationship between labour demand and wages. Using the job description-based classification helps identify a clearer link between labour demand and offered and agreed wages.

40 Cielinska et al (2017). Other recent research papers using trade repository data include Abad et al (2016) and Bonollo et al (2016).

41 See, for example, Duffie, Gârleanu and Pedersen (2005) and Lagos, Rocheteau and Weill (2011).

42 Turrell et al (forthcoming).

## Variety

One of the potentially most productive avenues for Big Data research, in the macro and finance sphere, involves using not numbers but words as data. Semantic data and semantic search techniques have a rich pedigree in other social sciences, such as sociology and psychology. But, so far, their application in economics and finance has been relatively limited. 43

Like other social sciences, economics and finance involves human choice. And we know humans often rely on heuristics or stories, rather than statistics, when making sense of the world and when making decisions. Capturing these stories, semantically, is thus important for understanding human behaviour and decisions.

As an example, the Bank has recently begun looking at the language it uses when communicating externally, whether to financial firms or the public at large. For example, Michael McMahon at the University of Oxford and I have recently assessed how the simplification of the Monetary Policy Committee's (MPC) language in the Inflation Report at the end of last year boosted public understanding of monetary policy messages. 44

A second example considers a far less well-explored aspect of the Bank's decision-making - its supervision of financial firms. 45 This draws on a text-based analysis of the Bank's confidential Periodic Summary Meeting (PSM) letters to financial firms. These are arguably the single most important letters the Prudential Regulation Authority (PRA) regularly sends to firms, setting out supervisors' assessment of firms' risks and requested actions to mitigate these risks. Using a machine learning technique called random forests, the research analyses these letters and extracts data on their tone and content.

This type of analysis has a number of policy applications. It can be used to assess whether the letters convey a clear and consistent supervisory message to firms. For example, it is possible to compare the strength and content of these letters with the Bank's internal assessment of firms' strengths and shortfalls. Are the two, and the Bank's supervisory messaging, consistent? In general, the research found they are.

This approach can also be used to assess how the style of supervision has evolved over time. For example, how has it changed since the transition in supervisory models from the Financial Services Authority (FSA) to PRA? The research finds that supervisory messaging has become more forward-looking, formal, and content-rich comparing the two regimes, consistent with the PRA's new supervisory model.

This exercise is, I think, a good example of a new technique (random forests) being applied to an entirely new database (the Bank's supervisory assessments) in a policy area virtually unexplored previously by researchers (financial firm-specific supervision). It reaches conclusions that have a direct bearing on policy questions. As such, I think it highlights nicely the promise of Big Data.

43 Notable examples include Schonhardt-Bailey (2013) and Goldsmith-Pinkham, Hirtle and Lucca (2016).

44 Haldane and McMahon (forthcoming).

45 Bholat et al (2017).

My final example uses not newly-created but old data. Nonetheless, I think it provides a good illustration of how new techniques can also be used to understand the past. Long before it was responsible for monetary policy and financial stability, one of the Bank's key roles was the provision of last resort lending to commercial banks facing liquidity pressures.

It is hard to date precisely, but the Bank began undertaking such operations in earnest probably around the time the UK faced a steady sequence of banking panics in 1847, 1857 and 1866. The Bank responded to these panics by making liquidity available to support banks. Last resort lending, as Bagehot came subsequently to call it, was born. 46 Indeed, Bagehot later defined principles for such lending: it should occur freely, at a penalty rate against good collateral.

An interesting historical question, with relevance to the present day, is whether in fact the Bank abided by these principles in its last resort lending during the 1847, 1857 and 1866 panics. To assess that, we took data from the giant paper ledgers recording changes to the Bank's balance sheet where these interventions were recorded on a loan by loan, counterparty by counterparty, interest rate by interest rate basis. 47

The transcription of these data benefitted from the fact that the hand-writing in the ledgers was from a small number of clerks across the three crises - one of the indirect benefits of job continuity. While the data was mostly transcribed manually, the project developed an image recognition system using a neural network algorithm which we will use in future to turn historic ledger activities into 21 st century machine readable data.

The data on the Bank's historic last resort lending is new and highly granular, Big Data from a bygone era. It shows that the Bank's approach to last-resort lending evolved considerably during the mid-19 th century crises. That meant, by the time of the 1866 crisis, the Bank was more or less following the principles for last-resort lending subsequently set out by Bagehot. It is another example of empirics leading theory.

Machine learning techniques are being applied to the statistics regularly collected and reported by the Bank. In particular, these techniques are being used to spot errors or anomalies in the raw data provided to the Bank. This makes cleaning the data much more systematic and efficient than is possible using manual processes. Data science methods can also be used to match new sources of granular data. This not only provides another means of checking the plausibility of data, but can offer insights that individual data sources cannot reveal by themselves. 48 At the Bank of England, as elsewhere, the robots are on the rise.

46 Bagehot (1873).

48 Bahaj, Foulis and Pinter (2017), for example, match firm-level accounting data, transaction-level house price data and loan-level residential mortgage data to show how the house price of the director of an SME can affect their firm's investment and wage bill.

47 Anson et al (2017).

## Looking to the Future

Looking to the future, there are many potential areas where these new sources and new techniques could be expanded to improve the Bank's understanding of the economic and financial system. From a long list, let me discuss one which I think holds particular promise.

Behavioural economics has, rightly, made a big splash over the past several years in re-shaping economists' thinking about how human decisions are made. Human decisions and actions deviate, often significantly and consistently, from the rational expectations norm often assumed. 49 Rules of thumb and heuristics dominate human decision-making. And the expectations formed by people are often shaped importantly by history, emotion and others' actions, every bit as much as by rational calculation.

These behaviours appear to be important both for individuals (micro-economic) and for societies (macro-economic). For example, the popular narratives which develop in financial markets and in everyday public discourse have been found to be important empirical drivers of fluctuations in asset prices and in economic activity. 50 These narratives may be particularly important at times of economic and financial stress, when emotions run high and social stories take on added significance.

Yet when it comes to measuring these behaviours, at either the micro- or macro-economic level, our existing methods are often poorly-equipped. Capturing people's true sentiments and preferences is devilishly difficult. Traditional surveys of market participants or the general public tend to be biased in their sampling and framed in their responses. As in quantum physics, the act of observing can itself alter behaviour.

These realities may call for exploring non-traditional means of revealing people's preferences and sentiments. To give one recent example, data on music downloads from Spotify has been used, in tandem with semantic search techniques applied to the words of songs, to provide an indicator of people's sentiment. Intriguingly, the resulting index of sentiment does at least as well in tracking consumer spending as the Michigan survey of consumer confidence. 51

And why stop at music? People's tastes in books, TV and radio may also offer a window on their soul. So too might their taste in games. Indeed, I am interested in the potential for using gaming techniques, not just to extract data on people's preferences, but as a means of generating data on preferences and actions.

Existing models, empirical and theoretical, often make strong assumptions about agent behaviour.

Theoretical models are based on axiomatic assumptions. Empirical models are based on historical

49 Rotemberg (1984), for example, discusses the statistical rejection of rational expectations models for consumption and labour demand.

50 51 Tuckett and Nyman (2017), Shiller (2017) and Nyman et al (2018). Sabouni (2018).

behaviours. These restrictions may, or may not, be borne out in future behaviour. If they are not, the model will break-down out of sample, as did the (deductive) DSGE model and the (inductive) Google flu model.

A gaming environment could be used to understand behaviour in a way which placed fewer restrictions. People's behaviour would be observed directly in the act of game-playing which, provided this behaviour was a reasonable reflection of true behaviour, would give us new data. Because this is a virtual rather than real world, with shocks controlled and regulated, that could make it easier to address issues of causality and identification in response to shocks, including policy shocks.

There are already multi-person games with primitive economies attached to them, which allow goods and monies to change hands between participants. These include EVE Online and World of Warcraft. Some economists have begun to use gaming technologies to understand behaviour. 52 For example, Steven Levitt (of Freakonomics fame) has used gaming platforms to understand the demand curve for virtual goods. 53

The idea here would be to use a multi-person dynamic game to explore behaviour in a virtual economy. This would include player interactions - for example, the emergence of popular narratives which shape spending or saving. And it could include player reactions to policy intervention - for example, their responses to monetary and regulatory policies. Indeed, in the latter role, the game could serve as a test-bed for policy action - a large-scale, dynamic, digital focus group. 54

Artificial intelligence experts are creating virtual environments to speed up the process of learning about system dynamics. 'Reinforcement learning' allows algorithms to learn and update by drawing on interactions among virtual players, rather than drawing on limited runs of historical experience. 55 At least in principle, a virtual economy would allow policymakers to engage in their own reinforcement learning, speeding up their process of discovery about the behaviour of the complex economic and financial system.

## Conclusion

So will Big Data keep its promise? I am optimistic it will. Economics and finance needs to make an on-going investment in Big Data and data analytics if it is to rebalance the methodological scales. And early research, including at the Bank, suggests the returns to such activity could be high, deepening our understanding of the economy and financial system.

These returns will best be harvested if there is strong collaboration between statistical authorities, policymakers, the commercial sector, research centres and academia. The Bank of England can play a

52 For example, Lehdonvirta and Castronova (2014).

53 Levitt et al (2016).

54 Yanis Varoufakis has previously been involved with a similar idea: http://uk.businessinsider.com/yanis-varoufakis-valve-game-

economy-greek-finance-2015-2

See https://deepmind.com/blog/deep-reinforcement-learning/ for a discussion.

catalytic role in bringing this expertise together. forward to working in partnership with you. So too can DAFM. I wish DAFM every success and look

## References

Abad, J, Aldasoro, I, Aymanns, C, D'Errico, M, Rousová, L F, Hoffmann, P, Langfield, S, Neychev, M and Roukny, T (2011) , 'Shedding light on dark markets: First insights from the new EU-wide OTC derivatives dataset', ESRB Occasional Paper Series, No. 11.

Albertazzi, U, Becker, B and Boucinha, M (2018) , 'Portfolio rebalancing and the transmission of largescale asset programmes: evidence from the euro area', ECB Working Paper Series , No. 2125.

Ali, R, Haldane, A and Nahai-Williamson, P (2012) , 'Towards a common financial language', paper available at https://www.bankofengland.co.uk/paper/2012/towards-a-common-financial-language

Anderson, C (2008) , 'The End of Theory: The Data Deluge Makes The Scientific Method Obsolete', Wired Magazine , 23 June.

Anson, M, Bholat, D, Kang, M and Thomas, R (2017) , 'The Bank of England as lender of last resort: new historical evidence from daily transactional data', Bank of England Staff Working Paper , No. 691.

Bacon, F (1620) , Novum Organum .

Bagehot, W (1873) , Lombard Street: A Description of the Money Market , Henry S. King & Co.

Bahaj, S, Foulis, A and Pinter, G (2017) , 'Home values and firm behaviour', Bank of England Staff Working Paper , No. 679.

Bank of England and Procyclicality Working Group (2014) , 'Procyclicality and structural trends in investment allocation by insurance companies and pension funds', Discussion Paper , July.

Baptista, R, Farmer, JD, Hinterschweiger, M, Low, K, Tang, D and Uluc, A (2016) , 'Macroprudential policy in an agent-based model of the UK housing market', Bank of England Staff Working Paper , No. 619.

Bean, C (2016) , 'Independent Review of UK Economic Statistics', available at https://www.gov.uk/government/publications/independent-review-of-uk-economic-statistics-final-report

Benetton, M, Bracke, P and Garbarino, N (2018) , 'Down payment and mortgage rates: evidence from equity loans', Bank of England Staff Working Paper , No. 713.

Bholat, D, Brookes, J, Cai, C, Grundy, K and Lund, J (2017) , 'Sending firm messages: text mining letters from PRA supervisors to banks and building societies they regulate, Bank of England Staff Working Paper , No. 688.

Bholat, D, Hansen, S, Santos, P and Schonhardt-Bailey, C (2015) , 'Text mining for central banks', Bank of England Centre for Central Bank Studies Handbook .

Bonollo, M, Crimaldi, I, Flori, A, Gianfanga, L and Pammolli, F (2016) , 'Assessing financial distress dependencies in OTC markets: a new approach using trade repositories data', Financial Markets and Portfolio Management , Vol. 30, No. 4, pp. 397-426.

Bracke, P and Tenreyro, S (2016) , 'History dependence in the housing market', Bank of England Staff Working Paper , No. 630.

Carney, M (2015) , speech at Launch Conference for One Bank Research Agenda, available at https://www.bankofengland.co.uk/speech/2015/one-bank-research-agenda-launch-conference

Cavallo, A and Rigobon, R (2016) , 'The Billion Prices Project: Using Online Prices for Measurement and Research', Journal of Economic Perspectives , Vol. 30, No. 2, pp. 151-78.

Chakraborty, C, Gimpelewicz, M and Uluc, A (2017) , 'A tiger by the tail: estimating the UK mortgage market vulnerabilities from loan-level data, Bank of England Staff Working Paper , No. 703.

Chakraborty, C and Joseph, A (2017) , 'Machine learning at central banks', Bank of England Staff Working Paper , No. 674.

Cielenska, O, Joseph, A, Shreyas, U, Tanner, J and Vasios, M (2017) , 'Gauging market dynamics using trade repository data: the case of the Swiss franc de-pegging', Bank of England Financial Stability Paper , No. 41.

Cœuré, B (2017) , 'Policy analysis with big data', speech at the conference on 'Economic and Financial Regulation in the Era of Big Data'.

Coyle, D (2014) , GDP: A Brief but Affectionate History , Princeton University Press.

Duffie, D, Gârleanu, N and Pedersen, L (2005) , 'Over-the-Counter Markets', Econometrica , Vol. 73, No.6, pp. 1815-1847.

Dwoskin, E (2015) , 'New Report Puts Numbers on Data Scientist Trend', Wall Street Journal , 7 October.

Economist (2017) , 'The world's most valuable resource is no longer oil, but data', article on 6 May 2017.

Ericsson (2017) , Ericsson Mobility Report , November 2017.

Farrell, D and Wheat, C (2015) , 'Profiles of Local Consumer Commerce', JPMorgan Chase & Co. Institute .

Financial Stability Board (2010) , 'Implementing OTC Derivatives Market Reforms', Financial Stability Board .

Fouquet, R and Broadberry, S (2015) , 'Seven Centuries of European Economic Growth and Decline', Journal of Economic Perspectives , Vol. 29, No. 4, pp. 227-244.

Ginsberg, J, Hohebbi, M, Patel, R, Brammer, L, Smolinski, M and Brilliant, L (2009) , 'Detecting influenza epidemics using search engine data', Nature , Vol. 457, pp. 1012-1014.

Goldsmith-Pinkham, P, Hirtle, B and Lucca, D (2016) , 'Parsing the Content of Bank Supervision', Federal Reserve Bank of New York Staff Reports , No. 770.

Haldane, A (2016) , 'The Dappled World', speech available at https://www.bankofengland.co.uk/speech/2016/the-dappled-world

Haldane, A and McMahon, M (forthcoming) , 'Central Bank Communication and the General Public', American Economic Review: Papers & Proceedings .

Henderson, V, Storeygard, A and Weil, D (2011) , 'A Bright Idea for Measuring Economic Growth', American Economic Re view : Papers & Proceedings , Vol. 101, No. 3, pp. 194-99.

Henke, N, Bughin, J, Chui, M, Manyika, J, Saleh, T, Wiseman, B and Sethupathy, G (2016) , 'The Age of Analytics: Competing in a Data-Driven World', McKinsey Global Institute .

IMF (2018) , 'Cyclical Upswing, Structural Change', World Economic Outlook , April 2018.

Lagos, R, Rocheteau, G and Weill, P-O (2011) , 'Crises and liquidity in over-the-counter markets', Journal of Economic Theory , Vol. 146, No. 6, pp. 2169-2205.

Lazer, D, Kennedy, R, King, G and Vespignani, A (2014) , 'The Parable of Google Flu: Traps in Big Data Analysis', Science , Vol. 343, pp. 1203-1205.

Leamer, E (1983) , 'Let's Take the Con Out of Econometrics', American Economic Review , Vol. 73, No. 1, pp. 31-43.

Lehdonvirta, V and Castronova, E (2014) , Virtual Economies: Design and Analysis , MIT Press.

Levitt, S, List, J, Neckermann, S and Nelson, D (2016) , 'Quantity discounts on a virtual good: The results of a massive pricing experiment at Kind Digital Entertainment', Proceedings of the National Academy of Sciences of the United States of America , Vol. 113, No. 27, pp. 7323-7328.

Moore, G (1965) , 'Cramming more components onto integrated circuits', Electronics , Vol. 38, No. 8.

Nyman, R, Kapadia, S, Tuckett, D, Gregory, D, Ormerod, P and Smith, R (2018) , 'News and narratives in financial systems: exploiting big data for systemic risk assessment', Bank of England Staff Working Paper, No. 704.

Obstfeld, M and Rogoff, K (2001) , 'The Six Major Puzzles in International Macroeconomics: Is There a Common Cause?', NBER Macroeconomics Annual , Vol. 15, MIT Press.

Popper, K (1934) , Logik der Forschung , Akademie Verlag.

Popper, K (1959) , The Logic of Scientific Discovery , Routledge.

Rotemberg, J (1984) , 'Interpreting the Statistical Failures of Some Rational Expectations Models', American Economic Review , Vol. 74, No. 2, pp. 188-193.

Sabouni, H (2018) , 'The Rhythm of Markets', mimeo.

Schonhardt-Bailey, C (2013) , Deliberating American Monetary Policy: A Textual Analysis , MIT Press.

Schwab, K (2017) , The Fourth Industrial Revolution , Portfolio Penguin.

Shiller, R (2017) , 'Narrative Economics', American Economic Review , Vol. 104, No. 4, pp. 967-1004.

SINTEF (2013) , 'Big Data, for better or worse: 90% of world's data generated over last two years', ScienceDaily , 22 May.

Stiglitz, J (2018) , 'Where modern macroeconomics went wrong', Oxford Review of Economy Policy , Vol. 34, No. 1-2, pp. 70-106.

Tuckett, D and Nyman, R (2017) , 'The relative sentiment shift series for tracking the economy', mimeo.

Turrell, A, Speigner, B, Thurgood, J, Djumalieva, J and Copple, D (forthcoming) , 'Using Online Vacancies to Understand the UK Labour Market from the Bottom-Up', Bank of England Staff Working Paper .