Model Validation

About CompatibL CompatibL Technologies was founded in 2003 to offer risk technology solutions to banks and asset managers. Since then CompatibL has established itself as the leading provider of quantitative software and services for derivatives analytics, limits, and regulatory capital.

CompatibL has over 200 experienced developers and financial engineers and boasts a client base of over 50 banks, central banks, supranationals and asset managers in the US, EMEA and Asia, including 4 out of 5 largest derivatives dealers. Over 140 major projects have been implemented across this client base. CompatibL is headquartered in Princeton (United States) with the main delivery centres located in London (UK), Warsaw (Poland) and Singapore.

www.compatibl.com info@compatibl.com

Princeton (USA)

London

CompatibL Technologies LLC 100 Overlook Center Second Floor Princeton, NJ 08540, USA

CompatibL Technologies Ltd First Floor 100 Pall Mall London, SW1Y 5NQ, UK

+1 (609) 919 0939

+44 (20) 3743 8800

Singapore

Warsaw

CompatibL Pte. Ltd. 30 Cecil Street 19-08 Prudential Tower Singapore, 049712, Singapore

CompatibL Sp. z o.o.

Prosta 32 Second Floor Warsaw, 00-838, Poland

+65 6813 2067

+48 (22) 110 8005

CompatibL Model Validation A new approach to model risk control

About CompatibL

Model Validation

CompatibL is a leading software vendor and consultancy that combines advanced application engineering with top quant expertise for the capital markets industry.

The company provides enterprise risk solutions and multi-asset analytics for front, middle and back office. CompatibL is also a leading provider of custom software development services and model validation consultancy for the financial industry.

CompatibL offers independent validation services across a wide range of valuation and risk models, including formal regulatory validation reports as well internal reviews with actionable insights to mitigate models’ weaknesses and limitations.

We have expertise in the latest quantitative research and model validation methodologies as well as implementation practices.

Our software is running in over 85 financial institutions worldwide, including 4 of the 5 largest dealers, 33 central banks and some of the world’s largest asset managers in the Americas, EMEA and APAC. Over the years, our products have won industry recognition for their quality, performance and flexibility.

CompatibL has teams located in strategic global locations across Europe, North America, Asia working in close contact with clients. We also have experience in long-term partnerships and have been working closely with the clients’ internal teams for many years.

15+

200+

years experience in software development for financial institutions

in-house experts, including quants, software engineers, solution architects, UI/UX designers and analysts in 4 branch offices

140

85

delivered enterprise software

clients worldwide, including 4

projects and hundreds of quantitative

of the 5 largest dealers, 33 central

consultancy and model validation

banks and some of the world’s largest

engagements

asset managers

2

/

CompatibL Model Validation

Market Risk Technology Vendor of the Year Award

CompatibL Risk Product of the Year Award

Alexander Sokol Quant of the Year Award

CompatibL Model Validation

/

3

Why Choose CompatibL for Model Validation?

Technology behind the Validation

Today, there is significant emphasis on model risk management and validation.

A transparent and independent model review process is an integral part of modern model vetting and model risk frameworks.

Our validation approach is based on the CompatibL Model Validation Platform, which integrates into a single environment all the code necessary to specify, automate, run and report model tests for third party and proprietary models. It can be used not only to produce a formal model validation report, but also as a benchmark or challenger model to assess the accuracy of the primary model.

As a vendor of market and credit risk software, CompatibL has hands-on experience and expert judgement in all aspects of model QA and review processes. Over the years, we validated hundreds of models and software solutions for banks and asset managers across multiple model types with applications to valuation and risk management.

This knowledge and experience enabled us to develop an innovative, rigorous and cost-effective approach to model validation.

CompatibL Model Validation Platform is focused on model validation, rather than model development, and provides a wide range of functionality out-of-the-box, saving the validation team the time and expense of creating these components during the project. It supports a variety of model input formats, including those of mainstream vendors and provides a set of common statistical acceptance tests, instrumentation for running regression tests, and a comprehensive set of tools to produce reports on individual test cases and consolidate them into a model validation report template for internal or regulatory use.

Advantages of Model Validation with CompatibL A Wide Variety of Models Cost savings Our approach reduces the costs compared to a traditional validation approach as we reuse standard validation code components gathered over time in our model validation platform.

Documentation The documentation delivered during the validation process is comprehensive and sufficient to independently reproduce the calculation results.

4

/

CompatibL Model Validation

Highly qualified expert team Benefit from access to a wide range of quantitative and technical resources and ensure that top experts work on validating models critical to your business.

Experience with in-house and vendor models CompatibL has experience with the leading quantitative model types published in academic literature, and with the implementation of these models by software vendors, leading to more effective and streamlined validation process.

Independence Rely on independent high-quality model validation performed by a team with extensive experience and a solid model validation code framework.

Deep analysis of model assumptions, limitations and output

The platform provides a rich spectrum of single-asset and multi-asset hybrid models to assess the impact of model choice on prices and risk measures, providing an additional dimension to model risk assessment.

CompatibL Model Validation Platform provides an open API and can be customized, permitting non-standard models and alternative benchmarks to be used.

Benchmarking across a range of market practice models is encouraged by many supervisory guidelines. It also helps to reduce the risk of implementation error and prevent mis-specification of the model.

We don’t stop at producing a model validation report and also help you to take a wider view on model risk, its origins and potential impact on your business.

CompatibL Model Validation

/

5

Model Coverage Our expertise spans real-world and risk-neutral multi-factor and multi-asset risk models for most asset classes including interest rates, foreign exchange, equity, inflation, credit and commodities.

Models supported by CompatibL Model Validation Platform out of the box:  Multi-curve real world and risk neutral valuation, capital and PFE models

 Capital and regulatory metrics: - - - - - -

 Advanced multi-currency CSA and collateral simulations  All major risk metrics and valuation adjustments, including IM and MVA  Capital optimization models

Basel II, II.5, III, early IV IRRBB SA-CCR Regulatory MPOR ISDA SIMM FRTB for market and credit risk, including the latest revisions and consultative documents

Global regulatory standard coverage

Operational Risk Market Risk

Basel II

Basel IV FRTB

Basel II.5

Credit Risk

IRRBB SA-CCR

Basel III

Systemic Risk

ISDA SIMM

Margin 2006

6

/

CompatibL Model Validation

2008

2010

2012

2014

2016

2018

2020

Timeline

CompatibL Model Validation

/

7

Structured Approach

Benefits of the Structured Approach

When dealing with complex models, we first identify key components in their calculation flow and consider them as independent unit models, each subject to rigorous individual testing.

For each component we follow the validation methodology and steps shown below.

Once each component has undergone independent testing and passed its strict acceptance criteria, we connect the components and run consolidated tests.

We then verify the integrity of the model overall and ensure that all input and output data are properly connected in the full model.

Once we have ensured that the calculation flow works as expected, we perform tests and analyze the entire model.

Traditional approach

Inputs

Monolith model

Outputs

Inputs

Monolith model

Outputs

Validation Validation

CompatibL approach Validation Validation

Composite Model Composite Model

Focus not only on the entire model but also on its individual components

Unit tests for model components lead to easier to understand, more reliable test criteria and implementation

Better isolation of model weaknesses and identification of error sources

More clarity to remediation actions, reducing the risk of delays to model use in production

Validation

Components

Validation Components

Components

Components

Inputs

Outputs

Inputs

Validation Validation

Components

Components

Components

Components Validation

Consolidation, Validation

Outputs

Validation

Consolidation, Validation

8

/

CompatibL Model Validation

CompatibL Model Validation

/

9

Automation

Validation Steps

Automation and open API of CompatibL Model Validation Platform provide the ability to continuously retest the model as improvements are made and remediation action is taken. Following validation project completion, it can be rerun for model updates and new releases without CompatibL’s assistance. This enables continuous assessment of the model risk on an ongoing basis with low recurring costs.

CompatibL’s approach to model validation and verification includes the following stages for a comprehensive and detailed review of the model and its components.

Model specification stage: • Analyze model documentation, verify its consistency, level of detail and correctness • Analyze the relevant regulatory guidance and market practice, model choice applicability and adequacy, and propose operationally feasible alternatives • Identify and assess key model inputs and outputs, verify their completeness and proper use of market data for a given model • Identify and assess key model assumptions and limitations, then compare to other model alternatives

Validation specification stage: • Identify and assess key validation assumptions and limitations, given input and output data and possible operational constraints • Review data scope, validation criteria and functional tests for model vetting based on the results from the previous stage

Testing stage: • Run validation tests • Assess the impact of model choice on material measures

Compared to the traditional approach of using manual tests on hardcoded inputs and direct integration with market data, our model replication approach enables a wider validation scope, not limited by simple cases and analytical examples. If desired, it can be scaled to a full live portfolio, providing ultimate confidence in the validation results.

Overall, our approach to the automation of model validation tests provides depth and scale advantages over the legacy validation methodologies.

Conclusion stage: • Analyze individual test results, provide recommendations and conclusions • Gather all findings into an executive summary and a highly detailed full validation report, conforming to the relevant regulatory validation template when applicable

Delivery stage: • Deliver validation documentation, reports, and supporting materials • Deliver test code artifacts and detailed documentation that allows this validation to be rerun in-house after project completion

10

/

CompatibL Model Validation

CompatibL Model Validation

/

11

Test Types The first category of tests includes tests that verify theoretical expectations and asymptotic conditions. This category is the starting point of any model validation process, covering the most basic form of validation. However, with advanced models such tests are insufficient.

Overconfidence in validation results obtained only from this first category of tests can lead to material errors

due to operational or functional issues in more complex real-world cases.

The second category includes independent replication of the model and its components from the first principles according to the model specification. Such replication should give identical or statistically close results, provided the same input data is used in both production and challenger environments.

Quantitative Test Approach

The validation process followed by CompatibL’s team embraces both categories of tests.

CompatibL Model Validation Platform provides base models and statistics functionality for the first category of tests, and many out-of-the-box components for use by a replicating model for the second category of tests.

be run as part of the validation project deliverables to provide assurance of validation code accuracy. We include standard algorithms, such as American Monte Carlo (AMC) and hybrid model calibration, which are applicable across all asset classes and risk factors and enable a wider portfolio scope. The platform provides lightweight and transparent API that reduces the implementation costs.

The components are verified by a comprehensive set of internal tests, which can

Challenger Platform Approach

Models Model

Actual vs Expected

Test case 1:

Analytic Semi-analytic

Asymptotic Approximate

Actual model

Actual outputs 1

Expected model 1

Expected outputs 1

Models Model

Test case N:

12

/

Expected model N

Expected outputs N

Conclusion Test case 1:

Expert judgement Actual outputs N

Independently replicated Alternative

Conclusion

Inputs 1

Actual model

Actual vs Expected

Inputs 1

+ Test case N: Inputs N

Consolidated inputs

Actual model Challenger open-source model

Consolidated actual outputs Consolidated expected outputs

Expert judgement

Inputs N

CompatibL Model Validation

CompatibL Model Validation

/

13

Transparency and Openness

Model Documentation

Unlike most model validation consultancies, CompatibL provides the source code for all model validation tests, including the code in the CompatibL Model Validation Platform.

This gives you additional confidence in validation results, and further reduces operational and model risk.

Because complete and accurate model documentation is critical to model risk management, we include the review of code structure and quality as part of the validation project.

We ensure that documentation is sufficient for users and decision makers to have complete confidence in how the model operates, including its key inputs, outputs, assumptions and limitations. When improvements are needed, we deliver enhanced documentation with the validation report.

How we do it?

We verify that the existing model documentation meets the regulatory guidelines.

We provide recommendations in case of any documentation gaps or inconsistencies.

We deliver enhanced documentation with the validation report.

Professional presentation of the report We use the professional LaTeX typesetting system used by most financial book publishers to provide test result presentation within the report. We automate the generation of test case documents to produce professional supporting documentation.

CompatibL contributes to the model validation community and supports ModVal.org, a free online resource for regulatory and internal model validation. ModVal.org provides detailed calculation walkthroughs and validation spreadsheets for some of the most complex regulatory calculations:

Internal Model Approach for market risk (IMA-TB)

Standardised Approach for market risk (SA-TB)

Standardised Approach for CVA (SA-CVA)

Basic Approach for CVA (BA-CVA

CompatibL has developed and presented several training courses to help model validation specialists to broaden their expertise in the latest regulatory frameworks. Our experts can customize the training program to suit the needs of your quants and model validation experts.

14

/

CompatibL Model Validation

In our test-case templates and methodology review, we follow a unified document structure, which covers the following items.

Model review:

Validation model review:

Analysis and summary:

- Background, market practice and methodology

- Detailed test descriptions and complete inventory of the required input data

- Detailed analysis of test results, providing the foundation for report conclusions

- Detailed instructions on how to perform tests and independently reproduce their results

- Model risk assessment and conclusions

- A thorough analysis of model specifications, assumptions and limitations

CompatibL Model Validation

/

15