Model use and misuse

## Model use and misuse

Speech given by

David Rule, Executive Director of Insurance Supervision

Association of British Insurers Prudential Regulation Seminar

14 May 2019

I am grateful to the following for contributions to this speech: Liam Clarke, Chintan Patel, Paul Collins, Amanda Istari, Brian Howell, Sarah O'Sullivan, Anu Ralhan, Peter McIntyre and David Phillips.

I would like to start by thanking the ABI for the invitation to make this keynote address. Today I want to talk about insurance models and the growing importance of model risk management. In doing so, I will describe some recent findings from the PRA's work to guard against weakening over time of capital requirements calculated from internal models or 'model drift'.

A model applies theories, techniques and assumptions to process input data into quantitative estimates for a particular purpose. 1 Make that purpose providing insurance and it is close to a definition of what insurers do. No surprise therefore that insurers are big users of models. But models can go wrong. They might have fundamental errors. Probably more commonly, they can be misused: for example, because management do not understand the limitations of a model. In insurance, a model going wrong might mean under-pricing of products, under-reserving against risks or treating certain customer groups unfairly. In the particular case of internal models used to calculate regulatory capital requirements, it could result in insufficient capital to protect policyholders.

## Insurance Capital Models

Let us start with capital models. Solvency II allows insurers to apply to the PRA for permission to use a model to calculate their capital requirements. The purpose of these models is to estimate the amount of capital needed to protect the insurer against losses in a '1 in 200' scenario over one year. They are highly complex. Taking the example of a life insurer, a typical model might involve:

-  Producing a base balance sheet. Many assets and liabilities can be valued without models: for example using market prices. But other valuations are themselves based on models;

-  Estimating a distribution of outcomes over one year for the material risks to which the insurer is exposed: such as, changes in interest rates, credit spreads, inflation, property prices, longevity, etc;

-  Assessing to what extent these risks are inter-dependent: for example, asset prices are related to interest rates;

-  Estimating the impact of the combination of risks to which the insurer is exposed on its balance sheet across many different scenarios;

-  Taking these outputs and converting them into a simpler set of equations - or proxy model - that relates the insurer's net assets to a series of risk factor inputs. The purpose of estimating a proxy model is to make it tractable to run many more scenarios. The complexity of the underlying valuation models would make this impractical computationally.

-  Running hundreds of thousands of scenarios through the proxy model in order to generate a distribution of probable balance sheet outcomes over one year; and

1 This definition is based on the one in Supervisory Guidance on Model Risk Management , SR Letter 11-7, Federal Reserve Board and Office of Comptroller of the Currency, April 4 2011

-  Focusing on the tail of losses and estimating the 99.5 th percentile of the distribution to identify the minimum amount of capital necessary to protect the insurer against a '1 in 200' scenario.

## (see Chart 1)

The use of models in Solvency II is more ambitious than in banking regulation. Comparing use of models in banking and insurance regulation:

-  The insurance capital model framework encompasses the whole balance sheet whereas banking models focus on particular risks (market, credit, counterparty). In the case of credit risk, models are built for particular portfolios;

-  Insurance models estimate overall capital requirements taking account of assumed diversification benefits across different risks whereas bank capital requirements are determined risk by risk and added together, with no diversification benefits across risks, although diversification is taken into account within risks;

-  Regulation determines or constrains more elements of bank than insurance models;

-  In banking regulation, the standardised approach is calibrated to give higher capital requirements than internal models for a typical bank whereas in insurance the two approaches are calibrated to the same confidence level; and

-  It is more straightforward for banking supervisors to top up capital requirements using so-called Pillar 2 of the bank capital framework. Once an internal model has been approved under Solvency II, supervisors have more limited scope to apply capital add-ons, which can only be temporary.

I make no apology therefore for the exacting nature of our internal model approval and change processes for UK insurers. Models must meet the tests and standards set out in Solvency II. These are both quantitative around calibration, statistical quality and profit and loss attribution - and qualitative - around independent validation, documentation, model governance and the use test. The PRA has approved 24 UK insurers to calculate capital requirements using a model, which is by some way the highest number by EU country. That gives us the advantage of peer comparison. We challenge hard where we think model calibrations are weak based on either comparison with other insurers' models or our own view of a risk. At the same time, though, we recognise that every insurer's risk profile is different. Insurers are many and varied. The standard formula cannot be flexible enough to work well for all of them. For this reason, internal models are an important element of the Solvency II framework.

Part of the reason for our intensive review of insurer models is that the interests of management and shareholders do not necessarily align with our own. Financial performance is measured by return on equity. Especially in a highly competitive trading environment, lowering regulatory capital requirements may seem a more achievable way to boost return on equity than increasing underlying profits. One legitimate approach is to reduce underlying risks. If an insurer's internal model is well designed, lower risks should lead to a fall in

capital requirements. We become concerned when genuine risk reduction tips over into weakening of assumptions or more aggressive modelling of risks. It is one thing we look for when reviewing material model changes proposed by insurers. Another is selectivity. Insurers might seek to make changes to their models in areas where risks are reducing or they have overestimated them but neglect areas where risks are increasing or they have underestimated them.

## Model Drift

Model drift is the risk that the capital requirements calculated using an internal model may gradually weaken over time such that they no longer reflect the risks to which the firm is exposed. No measure of model drift is perfect. The PRA tracks movements in internal model capital compared to the standard formula and the net best estimate of liabilities. 2 The first benchmarks the tailored risk calibrations in an insurer's internal model against the 'off the peg' measures of risk specified in Solvency II. The second compares them to the size of the insurer's balance sheet - a sort of insurance 'leverage ratio'. Supervisors supplement these aggregate measures with 'bottom up' analysis to compare the outputs of internal models, looking at calibrations risk by risk, over time and across insurers. 3

## ( see Chart 2)

In 2016 and 2017, the aggregate capital requirements of UK general insurers with internal models increased more or less in line with the standard formula and best estimate of liabilities. That is the outcome I would expect, other things being equal. No generalised model drift was apparent. For life insurers in aggregate, by contrast, both standard formula capital and best estimates of liabilities rose considerably more than internal model capital. At face value, this looks like considerable model drift.

For a couple of reasons, though, we are not yet ringing the alarm bells. First, it is only two years of data. We will need a longer period to draw strong conclusions about trends. Second, some life insurers with internal models have been growing in areas where the standard formula over-states the risks: in particular, illiquid assets within Matching Adjustment portfolios. Third, life insurers have been expanding unit-linked business and transferring an increasing proportion of longevity risk on new annuities business through reinsurance, both of which would tend to lead to faster growth in the best estimate of liabilities than capital requirements. In summary, the two benchmarks against which we compare internal model capital might not have been fully reliable measures of changing risks. Our 'bottom up' analysis of internal model outputs has also not suggested any generalised dilution of standards. Nonetheless, the significant reduction in internal model capital compared to the standard formula is not a trend we would expect to continue over time. We will be watching it carefully.

2 See Monitoring Model Drift and Standard Formula SCR Reporting for Firms with an Approved Internal Model , Prudential Regulation Authority, Supervisory Statement 15/16.

3 See Solvency II Regulatory Reporting: Internal Model Outputs , Prudential Regulation Authority, Supervisory Statement 25/15.

One counter-balance to any selectivity in insurers' model development is the PRA having its own model change agenda. For example, over the past couple of years, we have asked life insurers to adapt their interest rate modelling to include the possibility of negative rates and we have pushed them to improve their modelling of default and downgrade risks in Matching Adjustment portfolios.

## Proxy Modelling and Use of Management Actions

Two areas we have reviewed recently are proxy modelling and use of management actions in internal models. I mentioned earlier that life insurers build proxy models as an approximation of more complex valuation models. They use them to calculate and rank losses under a very large number of one-year scenarios, to create a distribution of possible losses, to identify the 99.5 th percentile loss and thus to calculate the regulatory capital requirement. If an insurer calibrates a proxy model poorly, regulatory capital will be wrong, even if the underlying valuation models are broadly correct. Weaknesses could also lead to bad decisions in other contexts where insurers use proxy models, including capital allocation, pricing and risk management. For example, insurers often employ proxy models for stress and scenario testing.

In May 2018, the PRA issued a survey to a number of life insurers with a proxy model. The responses revealed that over the previous couple of years, some insurers had improved the quality of their proxy modelling considerably. Others had not made the same investment. Standards of validation also varied. The best firms had increased the number of validation tests by improving the speed of valuation models, were placing the validation tests at points carefully selected to challenge the proxy model calibration and were conducting more validation after the reporting date .

Another area that we have reviewed recently is what future management actions insurers assume when calculating their best estimate of reserves and internal model capital. Typical management actions might include, for example, an insurer assuming that it can replace reinsurance following the default of a reinsurer or at policy renewal, or a life insurer assuming that it could reduce bonus rates on a with-profits fund if it was in financial distress. Solvency II allows insurers to assume management actions provided it would be reasonable to implement them in stressed market conditions. We found that some insurers, however, were inconsistent and insufficiently rigorous in their approach. Good practice is to bring all future management actions together into a single document that, at least at a high level, the board can review. This plan should quantify each management action and show how it is used in the calculation of reserves, capital requirements or otherwise (for example, in stress testing).

## Model Risk Management

I have focused so far on models used to help calculate regulatory capital and reserves. Insurers use many other models though, notably for pricing. For many of these models, new technology is bringing rapid change. Processing power continues to increase, including from use of the Cloud. This is enabling development of models that employ artificial intelligence techniques, such as machine learning, to analyse so-called Big Data - larger, more complex information from new sources, some of it structured but a lot of it unstructured. One example is real time information about customer behaviour taken from computing devices embedded in everyday objects, such as mobile phones, home appliances and cars - the so-called internet of things. Together with digital distribution of insurance, these changes bring huge opportunities. For example, better information may bring improved pricing, wider availability of insurance, more tailored and relevant products, enhanced fraud detection and better customer service.

At the same time, they are raising new questions and risks. Many are ethical and conduct related. For example, when designing a pricing algorithm what characteristics and behaviours of people is it acceptable to include? How transparent do insurers need to be to their customers about these algorithms? Where is the boundary between collecting data on behaviour to improve pricing and unacceptable interference in privacy? Is it acceptable to charge a customer more because their behaviour suggests they might be willing to pay more?

All these questions highlight the growing importance of managing model risk, including these ethical issues. The US Federal Reserve System and Office of the Comptroller of the Currency published the seminal guidance on model risk management in 2011. 4 Although targeted at banks, most of it is also relevant to any organisation making extensive use of models. Its recommendations cover model development, implementation and use; independent model validation, comprising evaluation of conceptual soundness, ongoing monitoring and backtesting; and effective governance, policies and controls, including the importance of a comprehensive model inventory and good documentation.

Model risk can arise in two ways: from fundamental flaws in models and from misuse of models. Turning first to model flaws, the risk is not that the model is imperfect - that is inevitable. Rather it is that errors are so fundamental that the model produces inaccurate or undesirable outputs when viewed against design objectives and intended business uses. For example, an insurer's internal model might produce a capital requirement that is significantly too low; or a pricing model might include analytical methods that lead to outcomes that are outside the insurer's risk appetite or, in the extreme, in breach of regulations or illegal. The Model Risk Working Party of the Institute and Faculty of Actuaries have published two good papers on model risk. 5 The second one 6 relates the case of the loss of the Mars climate orbiter in 1998 as an example

4 Supervisory Guidance on Model Risk Management , SR Letter 11-7, Federal Reserve Board and Office of Comptroller of the Currency,

April 4 2011

5 https://www.actuaries.org.uk/practice-areas/risk-management/disbanded-research-working-parties/model-risk

of flawed model design. The trajectory of the spacecraft had been incorrectly calculated, which meant that the spacecraft had actually been orbiting much closer to Mars than had been targeted causing the space probe to disintegrate in the planet's atmosphere. The problem was as basic as the use of incorrect units in part of the navigation software. An external contractor had produced software to provide Thruster performance data in English units of pound-seconds. The navigation team at NASA's Jet Propulsion Laboratory mistook this data as being in the required metric units of Newton-seconds. This led to errors in the spacecraft's trajectory calculations. One of the root causes was NASA's adoption of a 'faster, better cheaper' philosophy that had not given sufficient prominence to risk management. It had also not carried out an independent, end-to-end validation. Finally, developers had had an insufficient sense of ownership of the whole project as opposed to their bit of it.

The second source of model risk is misuse of models. This can arise when management take decisions based on the outputs of a model without sufficiently understanding the uncertainty around those outputs. Complicated, dynamic problems are more difficult to model than simple, stable problems. Model outputs are therefore more uncertain for problems that are more complex and where the underlying process in the real world is changing. Calculating the capital requirements of an insurer is surely a prime example.

The IFoA Model Risk Working Party has an interesting typology of corporate cultures according to their concern for model uncertainty and the legitimacy of modelling. Although they explain that all are valid viewpoints, I would feel more comfortable if insurers fell into the category of conscientious modellers, with modelling having high legitimacy but management also having a high concern for model uncertainty.

## (see Chart 3)

The Solvency II use test requires models to play an important role in risk management, decision-making and capital allocation. Management should therefore have confidence in their models. However, that confidence must be tempered by a proper understanding of model limitations and assumptions, which the use test indeed also requires. Insurers should try to de-mystify models as much as possible and avoid any sense that they are 'black boxes'. They need to understand for what purpose a model has been developed and the boundaries of its legitimate application. Criticism of the model ought to be welcomed and followed up: for example, arising from independent validation. Stress testing should be used to explore and improve understanding of model weaknesses. Management should challenge the outputs of complex models using other simpler models, rules of thumb and expert judgement. 7 For example, internal model estimates of capital can be compared to simpler standard formula or leverage measures. We encourage insurers to develop their own model drift measures and not limit themselves to the PRA's metrics. To take another example, in a rapidly changing world, it is good practice for insurers to pay close attention to claims

6 Model risk: illuminating the black box , British Actuarial Journal, 22 August 2017, R. Black, A. Tsanakas, A. D. Smith, M. B. Beck, I. D. Maclugash, J. Grewal, L. Witts, N. Morjaria, R. J. Green and Z. Lim.

7 In the Dog and the Frisbee, 2012, Andy Haldane and Vas Madouros explained how simple rules of thumb often outperform complicated models in complex, uncertain environments. https://www.bankofengland.co.uk/paper/2012/the-dog-and-the-frisbee

experience as a risk indicator. If recent claims experience is suggesting increasing risk when models are not, that should raise questions for management.

The risk of misuse of models might be particularly acute when an insurer buys in a model from a third party. Its people have not developed the model and become familiar with its limitations. Management might be tempted to put too much trust in the reputation of the supplier and assume that if many other insurers are using the model, 'it must be fine'.

The Senior Managers and Certification Regime requires insurers to allocate certain prescribed responsibilities to individual senior managers. Responsibilities for managing capital, funding and liquidity and for the Own Risk and Solvency Assessment, for example, entail a good understanding of an insurer's internal capital model. In practice however, it is unrealistic to expect that everyone using a model will fully comprehend how it works. The crux of good model risk management is to ensure that decision makers know enough. My former boss, Andrew Bailey summarised this when discussing our expectations of boards in 2015, 'the challenge is to reduce complexity to simplicity, so that Board members feel that they understand:

-  where is the model expected to work well;

-  in what circumstances is it likely to break down;

-  is the overall model output credible;

-  what 'moves the dial' in terms of key assumptions or judgements; and

-  are those assumptions and judgements reasonable?' 8

## LIBOR Transition

Before I conclude, I would like to spend a moment on a different subject, LIBOR transition. As many of you will be aware, the FCA's agreement with panel banks to provide LIBOR quotes will end in December 2021, now just over 30 months away. Across financial markets, SONIA is the Working Group on Sterling Risk Free Reference Rates' preferred benchmark for the transition to sterling risk-free rates from LIBOR. Liquidity is now building in SONIA markets - both derivatives and cash.

Our Dear CEO letter in September 2018 has had the desired impact of raising the profile of LIBOR discontinuance, with a requirement on the largest firms to provide transition plans, and giving greater visibility of the issue to other firms. We, and the FCA, have been clear that this is a significant change and achieving an orderly transition is a high priority for us. We will shortly publish the main points arising from responses to that Dear CEO letter.

For insurers, the Solvency II discount curves for major currencies are currently LIBOR-based. We are aware that insurers need clarity about when and how these discount curves will transition to replacement risk-free

8 Governance and the role of boards , Andrew Bailey, 2015, https://www.bankofengland.co.uk/speech/2015/governance-and-the-role-ofboards.

rates. We understand the challenges this poses to insurers, and we are working constructively with EIOPA and others to address these issues. We encourage insurers to continue to focus on the actions within your control such as identifying where LIBOR exposure is on your balance sheets, engaging with counterparties, and preparing for operational changes.

## Conclusion

To conclude, as insurers deploy increasing numbers of models, drawing on massive amounts of data and with new analytical techniques, it is vital that they give sufficient attention to effective model risk management. Insurers need to be sufficiently confident to use their models but not so over-confident that they misuse them.

Time =0

Time

## Chart 2: PRA's analysis of selected internal model drift metrics from 1 Jan 2016 to 31 Dec 2017

Source: Solvency II regulatory reporting and Bank calculations

## Chart 3: Alternative perceptions of modelling and its uses

Low legitimacy of modelling

High legitimacy of modelling

Source: Adapted from Model risk - illuminating the black box (https://www.actuaries.org.uk/practice-areas/risk-management/disbandedresearch-working-parties/model-risk)