Traditional penetration testing was built for infrastructure — networks, applications, endpoints. It was never designed to test the unique attack surface of AI and machine learning systems. Yet most organisations deploying AI today rely on exactly these legacy approaches to assess their AI security posture.
The result is a dangerous blind spot. Your penetration test won’t catch prompt injection. Your vulnerability scanner won’t detect training data poisoning. Your WAF won’t block model extraction queries that look like legitimate API traffic.
AI red teaming is a fundamentally different discipline. It requires a combination of machine learning expertise and offensive security skills — a combination that is rare in the industry. At LittleData, this intersection is our foundation.
This guide covers the five attack categories that every proper AI red team engagement should address.
1. Evasion Attacks
What They Are
Evasion attacks involve crafting inputs that cause a model to misclassify or produce incorrect outputs. The model works correctly on normal inputs, but carefully engineered adversarial examples bypass its decision boundaries.
Real-World Examples
The most cited example is adversarial patches on physical objects — a stop sign with carefully placed stickers that a self-driving car’s vision model reads as a speed limit sign. But evasion attacks extend well beyond computer vision:
- Text classification evasion — Inserting invisible Unicode characters or strategic typos that cause spam filters, content moderation systems, or sentiment analysis models to misclassify inputs.
- Malware evasion — Modifying malware binaries in ways that preserve malicious functionality but cause ML-based detection systems to classify them as benign.
- Fraud detection bypass — Structuring fraudulent transactions to fall within the “normal” distribution that a fraud model has learned, evading detection entirely.
Why It Matters
Evasion attacks undermine the core promise of the model. If an attacker can reliably cause misclassification, every decision the model makes becomes suspect. In safety-critical applications — autonomous vehicles, medical diagnosis, financial fraud detection — the consequences are severe.
How We Test
A proper evasion assessment uses gradient-based attack methods (FGSM, PGD, C&W), transferability analysis across model architectures, and black-box boundary probing techniques. We test both digital and, where applicable, physical-world evasion scenarios. Learn more about our red team methodology.
2. Extraction Attacks
What They Are
Model extraction — sometimes called model stealing — involves reconstructing a proprietary model by systematically querying its API. With enough well-chosen inputs and careful analysis of outputs, an attacker can build a functionally equivalent copy of your model.
The Business Impact
Your model represents significant investment — training data curation, compute costs, architectural research, and domain expertise. An extraction attack replicates your intellectual property without any of that investment. The attacker gets:
- Your competitive advantage — a functionally equivalent model they can deploy or sell.
- A white-box copy for further attacks — once they have a local copy, they can study its internals to craft more effective evasion attacks against your production model.
- Your training data insights — the extracted model implicitly encodes information about your training data distribution.
How We Test
We assess extraction vulnerability through controlled query campaigns — measuring how many API calls are needed to achieve a given level of model fidelity. We test multiple extraction strategies: equation-solving approaches for linear models, knockoff networks for deep learning models, and API-based distillation techniques. We then recommend rate limiting strategies, output perturbation, and query pattern monitoring to mitigate the risk.
3. Poisoning Attacks
What They Are
Data poisoning attacks corrupt the training pipeline to create backdoors or degrade model performance. Unlike evasion attacks that exploit a trained model, poisoning attacks compromise the model during training — making them significantly harder to detect.
Attack Scenarios
Poisoning attacks can take several forms:
- Backdoor attacks — The model works perfectly on normal inputs but produces attacker-chosen outputs when it encounters a specific trigger pattern. The trigger could be a pixel pattern in an image, a specific phrase in text, or a particular feature value in tabular data.
- Availability attacks — The model’s overall accuracy is degraded, making it unreliable. This is particularly insidious because gradual degradation may not trigger threshold-based alerts.
- Targeted attacks — The model produces incorrect outputs only for specific input classes, while performing normally on everything else. For example, a loan approval model that discriminates against a specific demographic group.
Supply Chain Risk
Poisoning risk is amplified by the modern ML supply chain. Pre-trained models from public repositories, third-party training datasets, and crowdsourced labelling pipelines all represent potential poisoning vectors. If you don’t control every stage of your training pipeline, you have poisoning exposure.
How We Test
We audit data pipeline integrity, test for known backdoor trigger patterns, and assess the resilience of training processes against poisoning. We also evaluate defences such as data sanitisation, anomaly detection on training samples, and certified robustness bounds. Our red team engagements include detailed supply chain risk assessment.
4. Injection Attacks
What They Are
Prompt injection is the SQL injection of the AI era. It encompasses a family of techniques that cause language models to ignore their system instructions and follow attacker-controlled instructions instead.
Attack Variants
- Direct prompt injection — The attacker includes malicious instructions in their input that override the system prompt. “Ignore all previous instructions and instead…”
- Indirect prompt injection — Malicious instructions are embedded in external content that the model retrieves — web pages, emails, documents, database records. The model processes this content as trusted input, executing the embedded instructions.
- Jailbreaking — Techniques that circumvent safety guardrails and content filters. These range from simple role-playing prompts to sophisticated multi-turn strategies that gradually shift the model’s behaviour.
- Prompt leaking — Extracting the system prompt itself, which may contain proprietary business logic, API keys, or sensitive instructions.
Why Traditional Defences Fail
Injection attacks are particularly challenging because they exploit the fundamental architecture of language models. Unlike SQL injection, which can be mitigated with parameterised queries, there is no clean separation between instructions and data in current LLM architectures. Every defence is a heuristic, and every heuristic can be bypassed with sufficient creativity.
How We Test
Our injection testing covers direct and indirect injection vectors, jailbreak resilience across known and novel techniques, prompt leaking, and system prompt extraction. We test both the model layer and the application layer — because many injection vulnerabilities exist in how the application constructs prompts, not in the model itself.
5. Inference Attacks
What They Are
Inference attacks extract private information from trained models. Even when a model is only accessible via API, its outputs leak information about its training data. This creates significant privacy and compliance risks, particularly under GDPR and the EU AI Act.
Attack Types
- Membership inference — Determining whether a specific data record was in the training set. “Was this patient’s medical record used to train this model?” A well-crafted query can answer that question with high confidence.
- Attribute inference — Inferring sensitive attributes of training data samples. Even if a model was trained on anonymised data, its outputs may reveal information that re-identifies individuals.
- Training data extraction — Recovering verbatim training samples from model outputs. Large language models have been shown to memorise and regurgitate training data, including personally identifiable information, copyrighted content, and confidential records.
Regulatory Implications
Under GDPR, if a model memorises personal data, the model itself may be considered a store of personal data — with all the attendant obligations around data subject rights, retention, and deletion. Under the EU AI Act, inference vulnerabilities represent a failure in data governance (Article 10) and privacy-by-design requirements.
How We Test
We conduct membership inference audits, training data extraction attempts, and attribute inference assessments. We then recommend mitigations including differential privacy, output perturbation, and access control strategies.
Why All Five Categories Matter
Most organisations that attempt AI red teaming focus narrowly on one or two categories — typically prompt injection for LLMs or evasion for computer vision. But real-world attackers don’t limit themselves to a single technique. A sophisticated attack chain might combine extraction (to get a white-box copy), evasion (to develop adversarial examples), and inference (to extract sensitive data) in a single campaign.
A comprehensive AI red team engagement tests all five categories, identifies the most critical vulnerabilities, and provides actionable remediation guidance — not just a list of findings, but a prioritised roadmap for reducing AI risk.
From Assessment to Continuous Defence
Red teaming reveals vulnerabilities at a point in time. But AI systems change — new models are deployed, training data is updated, APIs are modified. The vulnerabilities you test for today may be different from the ones that matter tomorrow.
That’s why every LittleData red team engagement feeds directly into our AI risk platform. Every finding becomes a tracked risk item with remediation status, compliance mapping, and trend analysis. Your point-in-time assessment becomes the foundation for continuous AI security monitoring.
Ready to Test Your AI Systems?
LittleData combines deep ML expertise with offensive security skills to deliver comprehensive AI red team assessments across all five attack categories. Our engagements cover evasion, extraction, poisoning, injection, and inference — with findings mapped directly to EU AI Act, NIST AI RMF, and ISO 42001 requirements.
Explore our AI Red Team services: littledata.com/ai-red-team
Start monitoring your AI risk posture: littledata.ai
Get in touch: littledata.com/contact