Citi^® Gen AI Summit 2024 Takeaways: Gen AI Policy, Regulation, Security, Data and Governance

Vibhor Rastogi

Head of AI Investing, Citi Ventures

Published on October 1, 2024
The opinions expressed in this blog are solely the authors' and do not reflect the views of Citi.

For this year’s Citi^® Gen AI Summit, I had the honor of hosting a panel delving into how AI can help enterprises with cybersecurity and regulatory compliance. Joining me were some of the top innovators in the space:

Ari Tuchman, Co-Founder and CEO of Quantifind (a Citi Ventures portfolio company), an AI platform that helps drive automation for AML and KYC;
Jack Berkowitz, Chief Data Officer at Securiti (a Citi Ventures portfolio company), a platform that ensures data and Gen AI security;
John Nay, CEO of Norm Ai (a Citi Ventures portfolio company), a platform that automates regulatory compliance workflows;
Krishna Gade, Founder and CEO of Fiddler AI, which focuses on observability, model monitoring and explainability of models.

Read on for key insights from our talk.

Prompt firewalls are becoming key to enterprise AI security

In light of the increase in bad actors attempting to exfiltrate enterprise data via malicious LLM prompts, prompt firewalls have emerged as a critical security tool. These contextually aware firewalls don’t just look for keywords, but understand semantics well enough to recognize and block so-called jailbreak prompts — prompts that hide their malicious intent in apparently benign contexts, such as hypothetical questions that can lead a model to give an output it has been designed to withhold.

Gen AI is helping firms untangle complicated regulatory regimes

As the global regulatory environment has grown more complex over the last 50 years, the amount of paperwork that companies — especially financial institutions — must do to remain compliant has increased as well. Under current manual workflows, for example, a human worker might have to pore through a 500-page PDF just to determine if a single marketing asset violates a particular regulation. Gen AI startups like Norm Ai are making this process far more efficient, leveraging LLMs’ ability to quickly parse and summarize vast quantities of data to provide employees with immediate feedback during the first pass of a compliance review.

LLMs have changed the game in AML/KYC, but humans still need to be in the loop

LLMs are transforming AML/KYC efforts across financial services by broadening the evidentiary landscape from sanctions lists and transaction data to include novel data sources that may indicate criminal activity. Properly fine-tuned models can even surface anomalous behaviors associated with less common financial crimes that don’t offer as much training data as crimes such as anti-money laundering and insider threats.

However, companies should leverage these capabilities with caution for the time being, as LLMs aren’t always as accurate as required for AML/KYC work. In cases where the stakes are incredibly high, such as filing a Suspicious Activity Report, it’s still better to rely on human judgement than on an automated system.

Tech companies are learning from the finance industry’s risk management practices

The finance industry employs a three-step risk management framework for models, statistical abstractions that represent real-world financial situations. Inspired by this well-studied framework, tech companies are starting to implement similar governance practices for Gen AI model risk management, i.e.:

Engineers build the model;
Model risk evaluators independently test the model’s outputs for explainability, the possibility of model error or wrongful model usage, and hallucinations, making sure to cover relevant error cases;
Other second-line functions such as fair lending, operational risk, and compliance further review models from their perspectives for suitability for deployment to production;
Internal auditors assess the effectiveness of the model risk management framework, including governance, policies, procedures and activities conducted to address the risk of model errors.

We need compliance for AI, and AI for compliance

It was an honor to moderate this conversation about how innovators are not only solving for the risks that AI poses to enterprise security and regulatory compliance, but are also considering how AI can actually help enterprises become more secure and compliant. I hope the hundreds of in-person and thousands of livestream attendees of the 2024 Citi Gen AI Summit came away feeling as inspired as I did about the incredible potential of this emerging field.