Management of community financial institutions continue to place even more reliance on models to help guide the decision-making process across all operational activities. With this increased reliance on models, there is a directionally consistent increase in the risk associated with the use of models. Managing risks is the fundamental basis of financial institutions, and in order to properly manage the risk, management must first develop policies and procedures to help mitigate the risk of model use. The policies and procedures should cover governance, controls, model due diligence, implementation, use, and validation. The risks will be relative to the institution’s complexity, business activities, corporate culture, and overall organizational structure. The Board of Governors of the Federal Reserve System Office of the Comptroller of Currency issued SR Letter 11-7 “Supervisory Guidance on Model Risk Management” as a prescriptive guide on how to properly manage model risk. This article will summarize the key points from this letter and provide important tips to ensure your institution will meet examiner expectations.

A model is defined by SR 11-7 as “a quantitative method, system, or approach that applies statistical, economic, financial, or mathematical theories, techniques, and assumptions to process input data into quantitative estimates. A model consists of three components: an information input component, which delivers assumptions and data to the model; a processing component, which transforms inputs into estimates; and a reporting component, which translates the estimates into useful business information.”

When relying on a model to make decisions, there is “potential for adverse consequences from decisions based on incorrect or misused model outputs and reports,” and this reliance results in model risk. The consequences of this risk could be financial loss, poor business and strategic decision making, or reputation damage to the institution.

Model risk occurs for two reasons according to SR 11-7:

• The model may have fundamental errors and may produce inaccurate outputs when viewed against the design objective and intended business uses.
• The model may be used incorrectly or inappropriately.

Important Principles for Managing Model Risk

Effective Challenge

One of the most important principles for managing model risk is an “effective challenge” of the model. The challenge is a critical analysis of model limitations and assumptions that are performed by objective, competent, and technical parties depending on the combination of incentives, competency and influence of the parties.

Model risk cannot be completely mitigated so additional controls should be considered regarding establishing limits on model use, monitoring model performance, adjusting or revising models over time, and supplementing model results with other analysis and information.

Development and Implementation Process

Another important principle of model risk management is a “disciplined and knowledgeable development and implementation process that is consistent with the situation and goals of the model user and institution policy.” Decision makers should understand that subjective judgement is utilized during the stages of model development, implementation, use, and validation, which increases the importance of model risk management processes. Institutions may also consider utilizing vendor or third-party products that meet the definition of a model. The vendor should be subject to the institution’s vendor management program in regards to due diligence and monitoring. To ensure that the model is appropriate, it is recommended that the institution:

• obtain information regarding product components, design, and intended use,
• understand the limitations, assumptions, and problematic use of the product, and
• obtain appropriate testing results that document the product works ask expected.

In the implementation process on internally-developed and purchased models, it is important to establish rules-based systems at a proper level to detect intended information and not create alarm fatigue due to a large volume of false positives.

Model Validation

Model validation is another key principle in model risk management and the main focus of examiners. Validation is an independent review of all model components to verify the models are operating as expected based on their design objectives and business uses. The scope and sophistication of validation should be commensurate with the institution’s overall use of models, the complexity and materiality of models, and the size and complexity of the institution’s operations. Institutions are expected to validate their own use of the vendor models. The validation should be performed by “people who are not responsible for model development or use and do not have a stake in whether a model is determined to be valid.” The validation should be performed by people with appropriate incentives, competence, and influence.

An effective validation framework should include three core elements:

• Evaluation of conceptual soundness, including developmental evidence
• Ongoing monitoring, including process verification and benchmarking
• Outcomes analysis, including back-testing

Regulatory examiners are not only asking for model validation reports that include the above elements, but they are also asking for the resume of the person who performed the work as well as the validation workpapers that support the validation report.

Common Types of Models

The two most common types of models currently being used by community financial institutions are Asset-Liability Management/Interest Rate Risk (“ALM/IRR”) Models and Bank Secrecy Act/Anti-Money Laundering (BSA/AML) Models.

• ALM/IRR Models accept input of current financial information data, transform the input data to develop estimates regarding impacts of interest rate changes on income and economic value of the institution, and develop reports of the estimates to help management achieve satisfactory and consistent profits, liquidity, and safety of the institution.
• BSA/AML Models accept input of transaction data from the institution’s products and services and analyze the transaction data to produce reports that identify reportable cash transactions and identify suspicious activities.

In Summary

Model risk management is a hot topic for all of the regulatory agencies and will continue to be one as community financial institutions continue to utilize models to make decisions in the operations of the institutions. It is important to establish strong policies and procedures to mitigate model risk. Model validation is another integral part of risk management and will be the focus of your regulatory examiners.

Key Points on Model Risk Management

• Institutions should maintain a comprehensive set of information for models being used and considered for use, or that have been recently retired.
• Consider all software being utilized to make decisions to assist in identifying models.
• Maintain inventory of models.
• Develop policies and procedures that cover governance, controls, model due diligence, implementation, use, and validation.
• Establish limits on model use, monitor model performance, adjust or revise models over time, and supplement model results with other analysis and information.
• Constantly monitor and adjust rules-based systems to prevent significant false positives and alarm fatigue.
• For internally-developed models, ensure that an “effective challenge” is being performed periodically.
• For purchased models, ensure proper due diligence and monitoring of vendor.
• Periodic independent model validation is required and expected by regulatory examiners.
• Institutions cannot rely on the vendor’s model validation and must perform an institution-specific independent validation.

If you would like to learn more about managing model risk or model validation please contact Larry Brown at lbrown@blueandco.com or 513-834-6903.