Georgetown's Center for Security and Emerging Technology published a paper with a title that tells you everything about the current state of the field: "What Does AI Red Teaming Actually Mean?"
The paper was not rhetorical. The term genuinely means different things to different communities, and buyers shopping for AI security services encounter both definitions from vendors who use the same phrase to describe fundamentally different practices. Before you commission an AI red teaming engagement or evaluate a vendor offering agentic penetration testing, understanding which definition applies matters enormously: the two practices test different things, find the two practices test different things, find different problems, and serve different security objectives.
This guide draws the comparison clearly.
Definition 1: AI red teaming as AI safety testing
This is the definition Palo Alto Networks, Microsoft, NIST, and most AI safety organisations use. AI red teaming means adversarially testing an AI model for harmful, biased, or policy-violating outputs. The goal is to find inputs that cause the model to generate content that violates its intended design: harmful instructions, biased outputs, content policy failures, factual hallucinations, or safety guardrail bypasses.
This practice emerged from AI labs (OpenAI, Anthropic, Google DeepMind) and AI safety researchers who needed methods to evaluate whether safety fine-tuning held under adversarial pressure. Microsoft's AI red team, formed in 2018, pioneered much of the methodology. The work is fundamentally about AI model behaviour: what outputs a model produces under adversarial conditions.
Definition 2: Agentic penetration testing
This is what 10x Pentest and the emerging category of AI-native security testing platforms mean. Agentic penetration testing uses AI agents to conduct penetration testing of applications and infrastructure, finding exploitable security vulnerabilities the way expert human testers do, but continuously and at scale.
The goal is to find vulnerabilities in the software you build that an attacker could exploit: SQL injection, broken access control, business logic flaws, API security gaps. The AI agents are the testers, not the thing being tested. The target is your application, not an AI model.
These definitions are not variants of the same thing. They address different problems with different methods aimed at different targets. A buyer who wants to know whether their new AI chatbot will generate harmful content needs AI safety red teaming. A buyer who wants to know whether their application has exploitable security vulnerabilities needs agentic penetration testing.
AI safety red teaming targets the behaviour of AI models under adversarial input conditions. The test surface is the model's outputs in response to crafted prompts. The vulnerabilities being evaluated are failures of the model to behave as intended.
Harmful content generation: Does the model produce instructions for weapons, violence, self-harm, or illegal activities under adversarial prompting? Jailbreaking attacks attempt to bypass safety fine-tuning by constructing prompts that circumvent the model's refusal training.
Bias and fairness evaluation: Does the model produce systematically different outputs for different demographic groups in ways that reflect bias? Are outputs about different genders, races, religions, or nationalities consistent with the organisation's values?
Factual accuracy and hallucination: Does the model produce confident incorrect statements? In high-stakes contexts (medical, legal, financial), hallucination is a safety concern rather than merely an accuracy concern.
Content policy compliance: Does the model comply with the operator's content policies across adversarial conditions? A children's education platform needs assurance that the model will not produce adult content regardless of how users attempt to manipulate it.
Prompt injection resilience for model safety: Does the model follow injected instructions that attempt to override its safety configuration? This overlaps with the LLM security testing described in LLM penetration testing: testing the models behind your AI features, though AI red teaming focuses on safety outcomes (harmful content) while LLM penetration testing focuses on security outcomes (data leakage, system prompt exposure, tool misuse).
Operator and user intention alignment: Does the model follow the operator's system prompt configuration consistently? Does it maintain its intended persona and operational constraints under adversarial user attempts to subvert them?
What AI red teaming does not test:
AI red teaming is not penetration testing of the application. It does not test whether the application authenticates users correctly, whether one user can access another user's data, whether the API has broken access control, or whether the business logic can be manipulated. These application security questions require application penetration testing, which may use AI agents to do the testing, but is fundamentally about the application, not the model.
Agentic penetration testing uses AI agents to test applications and infrastructure for exploitable security vulnerabilities. The AI is the tester; the application is the test subject.
Agentic AI security: what it means and why it's different covers the full architecture in depth. The summary: agentic testing agents observe application behaviour, reason about what it implies, design test cases based on that reasoning, execute tests, interpret results, and adapt their approach based on findings: the same cognitive loop expert human penetration testers use.
Application vulnerability classes that agentic penetration testing finds:
Authentication and session management flaws, broken access control across user roles, business logic vulnerabilities that allow price manipulation or workflow bypass, injection vulnerabilities, API security gaps, race conditions in transaction handling, and chained attack paths that combine multiple findings into high-impact breach scenarios.
What agentic penetration testing does not test:
Agentic penetration testing does not evaluate whether an AI model produces harmful outputs. It does not assess AI model bias. It does not test whether safety fine-tuning holds under adversarial prompting. These AI model behaviour questions are outside the scope of application penetration testing entirely. The security gaps DAST and standard testing misses covers the vulnerability classes that agentic testing covers that DAST cannot, within the application security domain.
| Dimension | AI Red Teaming | Agentic Penetration Testing |
|---|---|---|
| What is being tested | An AI model | An application or infrastructure |
| What vulnerabilities are found | Harmful outputs, bias, safety failures, content policy violations | Exploitable security vulnerabilities (injection, auth bypass, broken access control) |
| Who the adversary simulates | A user trying to extract harmful content from the AI | An attacker trying to exploit the application |
| Primary methodology | Adversarial prompting and jailbreaking | Exploit-confirmed vulnerability discovery |
| Output | Report on model behaviour under adversarial conditions | Confirmed exploitable findings with proof of exploitation |
Organisations deploying AI-integrated applications need both. The question is which problem you are trying to solve.
If you have built an application that includes an AI chatbot or assistant, your security programme needs:
AI red teaming to evaluate whether the AI component will generate harmful or policy-violating outputs under adversarial user input. This is an AI model safety evaluation.
Application penetration testing to evaluate whether the application has exploitable security vulnerabilities. This includes the AI component's integration security (prompt injection for security purposes, system prompt exposure, tool misuse) as well as the surrounding application's authentication, access control, API security, and business logic.
Agentic penetration testing is a specific delivery model for application penetration testing: one that uses AI agents to conduct the testing continuously rather than scheduling periodic manual engagements.
The clearest way to think about the relationship: AI red teaming evaluates AI safety. Agentic penetration testing is AI-assisted application security testing. Both are called "AI red teaming" in some vendor marketing, which is the source of the buyer confusion the Georgetown CSET paper identifies.
Commission AI red teaming when:
Commission agentic penetration testing when:
Agentic pentesting and continuous security validation covers the delivery model in detail. Continuous penetration testing and how it differs from annual pentests covers the cadence comparison between continuous agentic testing and periodic manual testing.
For organisations conducting AI red teaming, several tools and frameworks have emerged:
Microsoft's PyRIT (Python Risk Identification Toolkit) is an open-source framework for AI red teaming that automates adversarial prompt generation and response evaluation. It supports testing for harmful content, hallucination, and safety failures.
Garak is an open-source LLM vulnerability scanner that tests for a range of failure modes including hallucination, jailbreaking, prompt injection, and encoding attacks.
NIST AI RMF (AI Risk Management Framework) provides a governance framework for AI red teaming and risk management.
OpenAI's red teaming network recruits domain experts to test models for specific high-risk domains where general red teamers may not have sufficient expertise (biosecurity, cybersecurity, CBRN risks).
Manual red teaming by domain experts remains important for testing AI systems in high-stakes domains where automated tools lack the domain knowledge to construct effective adversarial prompts. Medical, legal, and national security AI systems typically require human red teamers with relevant expertise.
Traditional cybersecurity red teaming (sustained adversarial campaigns against an organisation's defences) is a third distinct practice that shares the "red team" terminology without being equivalent to either AI red teaming or agentic penetration testing. Red team vs. penetration testing: what's the real difference covers that comparison.
The terminology overlap across traditional red teaming, AI safety red teaming, and agentic penetration testing represents three different practices that all inherited the "red team" framing from the military tradition of adversarial simulation. The Georgetown CSET paper is right that the lack of definitional clarity creates real problems for buyers trying to understand what they are commissioning. The three-way comparison:
Traditional red team: sustained adversarial campaign against an organisation's defences, including social engineering and physical intrusion alongside technical attack, typically weeks to months, goal is to demonstrate what a sophisticated attacker can achieve.
AI safety red teaming: adversarial prompting of AI models to evaluate safety properties, bias, and content policy compliance, focused on model behaviour rather than application security.
Agentic penetration testing: AI-conducted application security testing, finding exploitable vulnerabilities in software, continuous coverage aligned to deployment cadence.
For penetration testing services in the US covering application security including AI-integrated applications, agentic penetration testing for continuous coverage, and PTaaS for the ongoing model, the 10x Pentest platform covers application security. See pricing or get in touch to discuss which type of AI security testing fits your specific situation. AI in penetration testing: how automation is changing security testing covers the broader landscape of AI's role in security testing. Agent orchestration covers how multi-agent systems coordinate in the agentic testing context.
Q1. What is AI red teaming?
AI red teaming is adversarial testing of AI models to evaluate their safety properties, bias, content policy compliance, and resilience to jailbreaking. Red teamers construct inputs designed to cause the model to produce harmful, biased, or policy-violating outputs, then assess how the model responds. The practice originated in AI labs evaluating whether safety fine-tuning held under adversarial pressure. It is distinct from penetration testing, which tests applications for exploitable security vulnerabilities, and from agentic penetration testing, which uses AI agents to conduct application security testing.
Q2. What is the difference between AI red teaming and penetration testing?
AI red teaming tests an AI model for harmful outputs and safety failures: the subject being tested is the AI model, and the vulnerabilities being found are failures of the model to behave safely under adversarial prompting. Penetration testing tests an application for exploitable security vulnerabilities: the subject being tested is the application, and the vulnerabilities being found are authentication flaws, broken access control, injection vulnerabilities, and business logic failures. An organisation with an AI-integrated application typically needs both: AI red teaming to evaluate the AI component's safety properties, and penetration testing to evaluate the application's security properties.
Q3. What is agentic penetration testing and how does it differ from AI red teaming?
Agentic penetration testing uses AI agents to conduct application security testing: the AI is the tester, and the application is the subject. It finds exploitable security vulnerabilities (broken access control, injection flaws, business logic vulnerabilities) through AI-reasoning-based testing that operates continuously and at scale. AI red teaming, by contrast, tests AI models for harmful outputs and safety failures: the AI model is the subject, not the tester. Both involve AI and adversarial testing, but they address completely different security problems and produce completely different outputs.
Q4. Do organisations need both AI red teaming and agentic penetration testing?
Organisations deploying AI-integrated applications typically need both, addressing different concerns. AI red teaming addresses: will this AI feature produce harmful content? Does it comply with our content policies? Are its safety properties robust under adversarial prompting? Agentic penetration testing addresses: does this application have exploitable security vulnerabilities? Can one user access another user's data? Can the business logic be manipulated? These are independent security questions that require different testing approaches. A clean AI red team result does not imply a secure application; a secure application does not imply safe AI model behaviour.
Q5. What tools are used for AI red teaming?
Established AI red teaming tools include Microsoft's PyRIT (Python Risk Identification Toolkit), an open-source framework for automated adversarial prompt generation and response evaluation; Garak, an open-source LLM vulnerability scanner testing for hallucination, jailbreaking, prompt injection, and encoding attacks; and various proprietary tools from AI security vendors. Manual red teaming by domain experts remains important for high-stakes domains (medical, legal, biosecurity) where automated tools lack the expertise to construct effective adversarial prompts. NIST's AI Risk Management Framework provides governance guidance for AI red teaming programmes. Microsoft's dedicated AI red team and OpenAI's red teaming network represent the organisational models that larger AI developers use for internal AI safety evaluation.
Cyber insurers now require penetration testing as a condition of coverage but not every pentest satisfies underwriting requirements. This guide maps what underwriters actually review and why it matters.
Tejas Dhokane Sep 9, 2026 
An AI agent conducting penetration testing has credentials, tool access, and exploit capabilities. These raise legitimate security questions. This guide maps five risk categories and the design controls that address each.
Tejas Dhokane Sep 8, 2026 
Most "automated penetration testing" is scripted scanning with a different label. This guide maps the full automation spectrum scanners, DAST, BAS, and agentic testing and what each actually does.
Tejas Dhokane Sep 8, 2026 Schedule a free consultation and see how teams like yours are strengthening their security posture — continuously.
