Understanding Hallucinations and Prompt Injections in AI: A TPRM Perspective
– Lisa-Mae Hill, CTPRP, CCRP
As Artificial Intelligence (AI) tools flood industries, Third-Party Risk Management (TPRM) professionals face a new dimension of exposure: the reliability and security of AI-powered tools. From productivity tools to analytics platforms and customer support automation, AI is embedded in many third-party solutions. And while AI has immense benefits in how we do TPRM, there are new risks unique to AI systems that we must consider. Two key concerns for TPRM teams evaluating AI-enabled tools are hallucinations and prompt injections. Understanding these risks is crucial for assessing vendor resilience, ensuring compliance, and maintaining trust across your enterprise.
1. Hallucinations in AI Aren't About Acid Trips
In AI, a hallucination occurs when a model generates information that is not based on fact, misleading, or entirely fictional, yet presented convincingly and authoritatively. For those relying on large language models (LLMs) like OpenAI’s ChatGPT or similar tools to generate content, reports, or responses, this poses a significant risk to accuracy and reliability.
Why This Matters in TPRM
The Compliance Implications here are staggering. If we, as third party practitioners, use the wrong AI tool to assess vendor documentation, hallucinated outputs could result in the generation of false confirmations of compliance, misleading assumptions, or incorrect control validation. You could easily assign the incorrect amount of risk to a Vendor.
TPRM Due Diligence Recommendations
- Evaluate whether the AI provider uses generative AI for decision-critical tasks such as vendor control documentation.
- Ask your AI provider how they prevent and address the risk of Hallucinations. Are they able to provide:
- An audit trail for every piece of evidence
- An independent model to evaluate the generative model’s inputs and outputs.
2. Prompt Injections: Undermining the Confidentiality, Integrity and the Availability of AI Through Manipulated Inputs
Prompt Injections are when an attacker deliberately creates and inputs deceptive prompts into an AI model in an effort to alter the behavior of the model and the answers it provides. This is especially dangerous in the case of TPRM when the inputs are not controlled, like data and documents received from a vendor.
Why This Matters in TPRM
If prompt injection vulnerabilities exist, an AI tool could be tricked into revealing sensitive internal instructions or proprietary data. When AI is used to interface with other systems (e.g., TPRM platforms, databases, APIs, or code execution environments), prompt injections can lead to unauthorized actions and access. AI-based content filters or access control systems could be circumvented via prompt injection, allowing prohibited content to pass undetected. The bottom line is that Prompt Injections can have a direct impact to the Confidentiality, Integrity and the Availability of your and your Vendor’s data.
TPRM Due Diligence Recommendations
- Inquire about the AI input sanitization methods used by the AI provider.
- Ask if the AI models have been tested against adversarial prompts or injection attempts.
- Ask the AI provider how they are preventing and defending against prompt injections
Strategic Takeaways for TPRM Professionals
Incorporating AI-related risks into your third-party risk assessment framework is no longer optional. As regulatory scrutiny increases and supply chains digitize, AI governance is fast becoming a core element of enterprise risk.
Key Questions to Ask a Prospective AI Provider
- Hallucinations: What controls are in place to verify AI-generated output before it is shared or published?
- Prompt Injections: How does the vendor defend against input manipulation in AI interfaces?
- Malicious Content: What technical and procedural safeguards are in place to prevent the generation of harmful or non-compliant content?
The Path Forward: Shared Responsibility for AI Risk
In TPRM, we are always looking to increase efficiency and find tools that can help us do more with less. AI tools are a way to do that. But AI-related risks, such as hallucinations, prompt injections, and malicious content, are not just technical issues. They represent real, material risks to your third-party ecosystem.
We, as TPRM practitioners, are uniquely positioned to mitigate these new risks through proactive evaluation, rigorous vendor governance, and strategic alignment with AI providers that can proactively address these risks and provide tools built on models that are secure and immune to these risks.
The more deeply AI becomes embedded in our world, the more essential it is to ask the right questions—not just about what AI does, but how it does it. By incorporating AI risk awareness into your third-party assessment of AI providers, you help safeguard your organization’s reputation, resilience, and regulatory posture for the future.
