AI and automated testing are changing our workplace

What we can learn from compliance testing and taking the customer view...

By Jeff Hulett, March 27, 2021

The following is an excerpt from our article Every bank needs a nudge....

This example is specific to obligation compliance testing. This example will be different from the other article examples for a couple of reasons.

1. It is newer. Traditionally, analytics has been more focused on credit risk management than compliance risk management organizations, and

2. While Randomized Control Trials (RCT) are certainly possible, this example is focused more on automating compliance testing.

In the main, control groups are not always necessary or even desired for compliance.

Please note: Conceptually, a control in compliance testing could include being out of compliance. It would likely not make sense for a bank to purposely be out of compliance for the sake of an RCT! However, a bank may want to test the effectiveness of certain compliance communications or disclosures. This could certainly be done via an RCT structure. Also, sometimes natural experiments may occur with multiple operating groups performing a similar function. This could create a valid RCT - like test environment.

By the way, this example is a composite of recent experiences with multiple banking organizations.

Automated testing can:

1. increase compliance testing coverage,

2. decrease testing costs, and

3. improve testing quality.

From a customer and regulator standpoint, the bank's customer communication and documents (letters, statements, emails, texts, promissory notes, disclosures, etc) are the customer's "system of record." That is, customer communication and documentation are the ultimate confirmation source that the bank has met various regulatory, investor, and other obligations. Because customer communication is often stored as unstructured data, it requires cost-effective automation capabilities to interpret documents, ingest data, and evaluate bank obligations. See the following graphic to compare the bank's and the customer's perspectives.

Also, an operational complication could arise if third parties are involved in the creation and transmission process of customer communication and documentation. Given this, the ability to structure data and apply obligation tests are critical for testing the “customer view” and is the essence of compliance automated testing.

In general, automated testing is an updating process as communication, documents, and algorithms are validated. Below are key automation outcome categories, resolutions, and success suggestions depending on the nature of the automated testing outcomes. (1)

For more information, please see our article Making the most of Statistics and Automation.

Notes

(1) Key automation outcome categories, defining False Positives and False Negatives -

False Positives: A false positive error, or false positive, is a result that indicates a given condition exists when it does not. For example, a cancer test indicates a person has cancer when they do not. A false positive error is a type I error where the test is checking a single condition and wrongly gives an affirmative (positive) decision. However, it is important to distinguish between the type 1 error rate and the probability of a positive result being false. The latter is known as the false-positive risk.

False Negatives: A false negative error, or false negative, is a test result that wrongly indicates that a condition does not hold. For example, when a cancer test indicates a person does not have cancer, but they do. The condition "the person has cancer" holds, but the test (the cancer test) fails to realize this condition, and wrongly decides that the person does not have cancer. A false negative error is a type II error occurring in a test where a single condition is checked for, and the result of the test is erroneous, that the condition is absent.

Implications

Depending on the test context, the error type has significantly different implications. The cancer example is closest to banking transactional testing. That is, a false positive can be annoying or provide patient/client unnecessary apprehension. A false negative can be deadly, that is, cancer remains and is undetected. In the case of bank risk testing, a false positive can create a customer service problem or a false risk signal. A false negative can enable the very risk it is trying to detect. That is, not identifying credit, compliance, or fraud risk when it exists. False negatives are often the basis for regulatory enforcement action.