QAwerks CEO warns regulators are coming away with questions about financial institutions’ AI processes that many will struggle to answer Konstantin Klyagin. More documented hallucinations, customer complaints about banks’ disaster cycles, and CFPBs Determining that incorrect AI chatbot responses may be federal violations is among the reasons regulators are monitoring.
When customers, regulators, or board members ask questions about AI representatives, what systems they can access, what actions they can take, and who is ultimately responsible, the organization needs to provide clear answers.
But most cannot do this. Klyagin said there are many reasons, starting with fundamental operational differences between standard deterministic software and AI models. In the first case, audit trail is simpler. The operator makes a decision and gets a result.
Why are AI systems more complex?
In contrast, an AI-powered system often pulls information from many different areas. More than one AI agent may be involved. The API’s connection of AI models to payment rails, fraud engines, and data is often not independently tested. As more processes are involved, the risk of error accumulation increases. When problems arise, institutions have difficulty explaining why.
Models must protect against data leakage, irrelevant data access, and improper permissions. They need to know when to pass a query on for human review. In conversational situations, systems need to provide accurate results even when people provide multiple different and, in some cases, incorrect statements. Validation at the semantic level is very important in this step.
“If your solution has proper audit trail implemented, you know exactly which agent did what, what input they had, what output they had, what was handled from one agent to the next.” Klyagin said. “You will also know what they are pulling with the data context and APIs, whether they are silently failing on some information or getting the wrong information.
“If you have documented all of this, you can recreate and correct the entire decision. Without an audit trail, it’s just guesswork. You don’t own the agent; the agent owns you.”
Properly designed, AI-powered logs can easily be more comprehensive than traditional software. This sophistication comes at a price, as AI generates solutions across multiple systems. As the complexity of the path increases, so do the explanations.
Klyagin used the example of a mortgage provider. Early in the application, the income statement is read incorrectly or the decimal point is placed in the wrong place. It gives the applicant a salary of $500,000 instead of $50,000. As more agents become involved, the problem becomes larger, leading to poor decision making.
