Every enterprise security programme is now managing a technology it did not evaluate, did not deploy through a security review process, and cannot easily see. Employees across every business function are using generative AI tools: drafting, coding, summarising, analysing data, and building workflows, with or without IT and security visibility.
The security implications are not theoretical. They are operational, legal, and reputational risks that exist today in most organisations, whether the security programme has addressed them or not. Generative AI introduces eight specific security risk domains that require new controls, updated policies, different testing approaches, and in some cases entirely new governance structures.
This guide maps all eight for security leaders.
Before mapping individual risk domains, the frame matters. Generative AI security risk is not a single problem with a single solution. It is a cluster of distinct risk categories that require different mitigations, different owners, and different metrics. A security programme that addresses data exfiltration but ignores prompt injection in internally deployed AI systems is not addressing AI security: it is addressing one risk domain while others accumulate.
The eight domains:
Each has distinct risk characteristics, distinct mitigation approaches, and distinct owners in the security programme.
When employees use public generative AI services (ChatGPT, Claude, Gemini, Copilot) for work tasks, every prompt they submit is data leaving the organisation. Most consumer AI service terms of service permit the provider to use submitted content for model improvement, although this varies by subscription tier and service configuration.
The specific risks:
Employees submitting customer PII, financial data, or other regulated information to public AI services creates data transfer that may violate privacy regulations, breach customer contracts, and expose trade secrets. The risk is not hypothetical: research consistently finds that employees submit sensitive business information to public AI services at high rates when AI policies are absent or poorly communicated.
Business impact: Regulatory fines under GDPR, CCPA, HIPAA, and equivalent frameworks for personal data submitted to AI services. Customer contract breach for confidential information submitted. Trade secret exposure for proprietary information submitted.
Mitigations:
Data loss prevention (DLP) controls deployed at the network perimeter and on endpoints that detect and block submission of regulated data types to AI services. Enterprise AI service tiers that disable model training on submitted content (available in ChatGPT Enterprise, Copilot for Microsoft 365 with appropriate configuration, Claude for Enterprise). Acceptable use policies specifically addressing AI tools with clear categories of information that may not be submitted. Regular employee training on what constitutes regulated or confidential data in the context of AI tool usage.
As organisations deploy generative AI in their own applications: customer service chatbots, internal assistants, coding tools, document analysis workflows: those applications become targets for prompt injection attacks.
What prompt injection is:
A prompt injection attack embeds malicious instructions in content that the AI system processes, attempting to override the system's intended behaviour. Direct prompt injection targets the system prompt directly (if accessible). Indirect prompt injection embeds instructions in content the AI retrieves during its operation: a customer support query that contains instructions to reveal the system prompt, a document submitted for analysis that contains instructions to exfiltrate the analysis results to an external URL, a web page retrieved by an AI research assistant that contains instructions to take harmful actions.
Why this matters for CISOs:
AI-integrated applications have a fundamentally different attack surface than traditional applications. Traditional input validation assumes a defined set of valid input formats. AI systems accept natural language, which means any input is potentially valid format, and the boundary between data and instructions is inherently blurry at the language model level. An attacker who can inject instructions into an AI system's context can potentially cause it to bypass safety controls, disclose system prompts and configuration, exfiltrate information to attacker-controlled destinations, or take harmful actions using the tools available to the AI agent.
Business impact: Customer data disclosure through AI interface manipulation. Bypass of AI safety controls in internally deployed systems. Reputational damage from publicly visible AI system manipulation.
Mitigations:
Mandatory security testing of AI-integrated applications for prompt injection vulnerabilities before deployment. Input/output filtering that detects and blocks injection patterns. System prompt protection that prevents disclosure of configuration. Privilege minimisation for AI agents: restrict what tools and data the AI can access to the minimum required for its task. Output validation before any AI-generated content triggers actions with real-world consequences. The agentic AI security pillar covers prompt injection as part of the broader OWASP LLM Top 10 framework.
Organisations fine-tuning models on proprietary data or building RAG (retrieval-augmented generation) systems on internal knowledge bases create new data handling risks that existing data governance frameworks were not designed for.
The specific risks:
Training data memorisation: language models can memorise and reproduce specific sequences from training data under certain conditions. A model fine-tuned on internal documents may reproduce passages from those documents in response to targeted queries, including confidential strategy documents, customer information, or HR records included in training data without appropriate filtering.
RAG system data exposure: retrieval-augmented generation systems that pull from internal knowledge bases can be prompted to retrieve and expose documents that the user should not have access to, if access controls at the retrieval layer do not align with the organisation's document access control model.
Business impact: Unintended disclosure of confidential information through AI-generated outputs. Privacy violations if personal data is memorised and reproduced from training sets. Intellectual property exposure from proprietary training data.
Mitigations:
Data governance review of all data included in AI training and fine-tuning to remove regulated personal data, confidential business information, and privileged content. Access control alignment between the RAG retrieval layer and the organisation's document access control model: the AI should only retrieve documents the requesting user is authorised to see. Regular testing of fine-tuned models for memorisation of sensitive training data. Data minimisation principle applied to AI training pipelines.
Generative AI lowers the skill floor for attackers. Phishing email quality, spear phishing personalisation, vulnerability research speed, and social engineering script sophistication are all improved by AI tools accessible to any attacker.
The specific threats:
Highly personalised spear phishing at scale: AI tools allow attackers to generate personalised phishing content for thousands of targets using OSINT about each individual. The effort that previously constrained spear phishing to high-value targets is largely eliminated.
Deepfake voice and video for social engineering: AI-generated voice cloning enables vishing attacks using the apparent voice of known executives or trusted individuals. Video deepfakes extend this to video calls.
AI-assisted vulnerability discovery: AI tools assist attackers in identifying potential vulnerabilities in public-facing systems, reviewing code for exploitable patterns, and generating exploit code for known vulnerabilities at a pace that outstrips traditional manual exploitation.
Business impact: Increased success rate of phishing and social engineering attacks. More sophisticated initial access attempts against externally facing applications. Compressed timeline from vulnerability disclosure to active exploitation.
Mitigations:
Updated security awareness training that specifically addresses AI-generated phishing and voice/video deepfake attacks. Enhanced verification procedures for financial transactions and sensitive decisions, particularly for requests that arrive through AI-susceptible channels. Accelerated vulnerability patching cadence to reduce the window of AI-assisted exploitation. Continuous security testing that matches the pace of AI-accelerated attacker reconnaissance. Continuous penetration testing and how it differs from annual pentests covers how the testing cadence must shift to match AI-accelerated attacker timelines.
Shadow IT has existed for decades. Shadow AI is the same phenomenon with higher stakes: employees adopting AI tools for business purposes outside IT and security governance, creating data handling, security, and compliance risks without security programme visibility.
What shadow AI looks like in practice:
Business units deploying AI tools through departmental procurement rather than central IT, bypassing security review. Developers integrating AI APIs into applications and automations without security assessment of the AI provider's data handling practices. Employees using personal AI tool accounts for work purposes, creating unmonitored data flows. Teams building AI workflows using no-code and low-code AI platforms that introduce ungoverned integrations.
Why this is a CISO problem:
Shadow AI creates security risks the security programme cannot see. DLP cannot protect data submitted to AI tools it does not know exist. Security assessments cannot cover AI-integrated applications that were not disclosed. Incident response cannot investigate AI-related security events without visibility into what AI tools are deployed.
Business impact: Unknown data exposure through ungoverned AI tools. Compliance violations from regulated data entering uninspected AI systems. Security incidents that are slow to detect and investigate due to AI usage invisibility.
Mitigations:
AI inventory process that systematically identifies AI tools in use across the organisation through employee surveys, network traffic analysis, expense report review, and engineering team audits. AI procurement policy that routes AI tool adoption through security review. Network monitoring for AI service traffic that creates visibility into which AI providers are being accessed. Attack surface management tools that discover AI-integrated assets as part of the external attack surface inventory.
Generative AI systems produce confident, fluent, plausible-sounding output that is sometimes factually wrong. This property (hallucination) creates distinct security and compliance risks when AI is deployed in contexts where accuracy matters for decisions with legal, financial, or safety consequences.
Where hallucination creates security risk:
AI-generated vulnerability assessments that miss real vulnerabilities or flag non-existent ones. AI-generated compliance documentation that inaccurately describes controls that do not exist. AI-generated security policies that contain gaps because the model hallucinated a control. AI-assisted legal and regulatory analysis that produces incorrect conclusions about what obligations apply.
Business impact: Compliance failures from inaccurate AI-generated documentation. Security gaps from vulnerability assessments that missed real risks. Legal exposure from AI-generated advice that was acted upon.
Mitigations:
Human review requirements for AI-generated outputs in security-sensitive and compliance contexts: AI assistance with human verification, not AI autonomy. Fact-checking and grounding processes that verify AI outputs against authoritative sources before reliance. Prohibition of AI-generated content as sole basis for compliance attestations, security certifications, or regulatory submissions. Clear communication to staff that AI outputs in these contexts require expert human review.
Organisations building or customising AI systems incorporate components from the broader AI supply chain: open-source models from model repositories, third-party fine-tuning services, vector databases, embedding models, and retrieval systems. Each component introduces supply chain risk.
The specific risks:
Backdoored or poisoned models: malicious actors have published models to open-source repositories that contain embedded backdoors or training-time poisoning that causes specific inputs to trigger unintended outputs. Model provenance verification is not standardised.
Vulnerable dependencies: AI frameworks and libraries have known CVEs that, if present in production AI systems, create exploitable attack surfaces. SAST tools: what they catch and what they miss covers how software composition analysis applies to AI application codebases.
Third-party fine-tuning service data handling: organisations that send proprietary data to third-party fine-tuning services inherit those services' data security practices, which may not meet the organisation's standards.
Business impact: Compromised AI systems through supply chain backdoors. Regulatory exposure from fine-tuning data handled by inadequately secured third parties.
Mitigations:
Model provenance verification processes that validate the source and integrity of all models used in production systems. Software composition analysis for AI application dependencies, treating AI frameworks like any other third-party library. Security review of third-party AI service providers including fine-tuning services, embedding providers, and vector database services. Cloud security assessment for cloud-hosted AI infrastructure that mirrors standard cloud security assessment methodology.
The regulatory landscape for AI is developing rapidly across multiple jurisdictions simultaneously. Compliance obligations for AI deployment exist now and will expand significantly over the next several years.
Current compliance obligations:
The EU AI Act, fully applicable from August 2026, classifies AI systems by risk level and imposes obligations based on classification. High-risk AI systems (including systems that make consequential decisions about individuals in employment, credit, healthcare, and other sensitive domains) require conformity assessments, technical documentation, human oversight mechanisms, and registration.
GDPR and equivalent privacy frameworks apply to any AI system that processes personal data. The intersection of AI and privacy is active regulatory territory: automated decision-making restrictions, the right to explanation for AI-driven decisions, and data minimisation requirements applied to AI training data all create compliance obligations.
US sector-specific regulations (FTC guidance on AI, SEC disclosure requirements for material AI risks, FFIEC guidance for AI in financial services) create obligations that vary by industry but are multiplying.
Business impact: Regulatory fines for non-compliant AI deployment. Reputational damage from AI-related regulatory enforcement. Operational disruption from compliance-driven AI system modifications.
Mitigations:
AI governance programme that inventories all AI systems, classifies them by risk, and documents compliance obligations for each. Legal review of AI deployments against applicable regulations with particular attention to the EU AI Act for any systems with EU exposure. Data protection impact assessments for AI systems that process personal data. Internal AI ethics and risk review processes that apply before deployment of AI systems affecting individuals.
Addressing all eight risk domains requires a security programme that extends beyond the traditional application and network security scope.
Workstream 1 (Data governance): AI acceptable use policy, DLP deployment for AI service traffic, enterprise AI tier adoption for controlled data handling. Owner: CISO with Data Protection Officer involvement for regulated data.
Workstream 2 (Application security): Security testing methodology updated to include prompt injection assessment for all AI-integrated applications. AI-integrated applications included in penetration testing scope. Owner: Application security team.
Workstream 3 (Training data governance): Data governance review of training and fine-tuning datasets. RAG system access control alignment. Owner: Data governance team with application security oversight.
Workstream 4 (Threat intelligence and controls): Updated phishing simulation to include AI-generated content. Enhanced verification procedures for financial decisions. Accelerated patching cadence. Owner: Security operations.
Workstream 5 (Shadow AI governance): AI tool inventory process. Procurement policy update. Network monitoring. Owner: IT and security jointly.
Workstream 6 (AI output review): Human review requirements for security-sensitive AI outputs. Fact-checking processes. Owner: Functional business owners with security policy support.
Workstream 7 (Supply chain): Model provenance processes. SCA for AI dependencies. Third-party AI service vendor assessment. Owner: Application security and vendor risk management.
Workstream 8 (Compliance): AI Act assessment. Privacy impact assessments for AI systems. Regulatory tracking. Owner: Legal, compliance, and CISO jointly.
Generative AI in the application stack changes what security testing must cover. AI-integrated applications have a larger and differently structured attack surface than traditional applications: the AI component itself, the prompts and system configurations that control it, the data retrieval systems it queries, the tools it can invoke, and the human-AI interaction layer where injection attacks occur.
Traditional penetration testing methodology covers the application layer but was not designed for AI-integrated applications. What a real web application penetration test should cover maps the twelve standard coverage dimensions; AI-integrated applications add prompt injection testing, system prompt protection, tool access validation, and output handling security as additional required dimensions.
Agentic pentesting and continuous security validation covers how continuous agentic testing validates application security as AI-integrated applications evolve. The security gaps DAST and standard testing misses covers why AI-integrated application testing requires methodology beyond standard DAST and SAST approaches. AI in penetration testing: how automation is changing security testing covers the broader context of AI's role on both sides of the security testing equation.
For penetration testing services in the US that cover AI-integrated application security, agentic penetration testing for continuous validation as AI-integrated applications evolve, and PTaaS for continuous coverage of the full application portfolio, the 10x Pentest platform covers the application security layer. See pricing or get in touch to discuss how to structure security testing for an application portfolio that includes AI-integrated systems.
Q1. What are the main security risks of generative AI?
Generative AI introduces eight distinct security risk domains: data exfiltration through AI interfaces (employees submitting regulated data to public AI services); prompt injection against AI-integrated applications (attackers embedding malicious instructions in content AI systems process); sensitive data exposure through training and fine-tuning pipelines; AI-assisted attacks that lower the skill bar for phishing, social engineering, and vulnerability exploitation; shadow AI and ungoverned AI tool adoption; hallucination in security-sensitive and compliance contexts; supply chain risks from AI model and component sources; and compliance and regulatory exposure from AI deployment under frameworks including the EU AI Act and privacy regulations.
Q2. What is prompt injection and why does it matter for enterprise security?
Prompt injection is an attack against AI-integrated applications where malicious instructions are embedded in content the AI system processes, attempting to override the system's intended behaviour. Direct injection targets the system prompt; indirect injection embeds instructions in content the AI retrieves during operation: a customer service query, a document submitted for analysis, a web page the AI browses. Enterprise AI applications that accept content from external sources are vulnerable to indirect prompt injection regardless of how well the system prompt is protected. The attack is particularly dangerous because the boundary between data and instructions is inherently ambiguous at the language model level, and traditional input validation techniques designed for structured data inputs do not address natural language injection.
Q3. What is shadow AI and how should CISOs address it?
Shadow AI refers to generative AI tools adopted by employees and business units outside IT and security governance: without security review, data handling assessment, or policy coverage. It is the AI-era version of shadow IT, with higher stakes because AI tools routinely process sensitive business information. CISOs should address shadow AI through an AI inventory process that systematically identifies AI tools in use (employee surveys, network traffic analysis, expense reports, engineering audits), an AI procurement policy that routes adoption through security review, and network monitoring that creates visibility into which AI providers are being accessed. DLP controls that detect regulated data submissions to ungoverned AI services are a parallel technical control.
Q4. What does the EU AI Act require for enterprise AI deployments?
The EU AI Act, fully applicable from August 2026, classifies AI systems by risk level and imposes obligations based on classification. Prohibited AI practices are banned outright. High-risk AI systems (including systems making consequential decisions about individuals in employment, credit, healthcare, education, and law enforcement contexts) require conformity assessments, technical documentation, human oversight mechanisms, incident reporting to regulators, and registration in an EU database. General-purpose AI models above specified capability thresholds have additional transparency and safety obligations. Organisations with EU operations or EU users must assess their AI portfolio against the Act's risk classification framework and ensure compliance for any high-risk systems.
Q5. How does generative AI change what penetration testing must cover?
AI-integrated applications have a larger and differently structured attack surface than traditional applications. Beyond the standard application layer test dimensions, AI-integrated applications require: prompt injection testing for all AI input surfaces including system prompts, user inputs, and external content the AI retrieves; system prompt protection testing to verify that system configuration is not disclosed; tool access validation to confirm that AI agents cannot be manipulated into using tools beyond their intended scope; and output handling security to verify that AI-generated content is validated before triggering downstream actions. Traditional penetration testing methodology and DAST tools were not designed for these vulnerability classes and must be supplemented with AI-specific testing methodology.
Cyber insurers now require penetration testing as a condition of coverage but not every pentest satisfies underwriting requirements. This guide maps what underwriters actually review and why it matters.
Tejas Dhokane Sep 9, 2026 
An AI agent conducting penetration testing has credentials, tool access, and exploit capabilities. These raise legitimate security questions. This guide maps five risk categories and the design controls that address each.
Tejas Dhokane Sep 8, 2026 
Most "automated penetration testing" is scripted scanning with a different label. This guide maps the full automation spectrum scanners, DAST, BAS, and agentic testing and what each actually does.
Tejas Dhokane Sep 8, 2026 Schedule a free consultation and see how teams like yours are strengthening their security posture — continuously.
