AI red teaming for developers: a practical introduction
Prompt injection, jailbreaks and extraction attacks are part of building with LLMs today. Here's how a red-team mindset fits into the engineering workflow — and what an attacker simulator automates for you.
Daniel Martínez Marqués
independent developer & software builder
When you ship an LLM-powered feature, you are shipping a program whose "code" is written at runtime by an external party: the user's prompt — and often other prompts carried by emails, websites or files. That's why the security model of an LLM application is different from a classic backend.
An attacker doesn't need a buffer overflow. They just need to make the model act outside the instructions you gave it.
What red teaming means for AI systems
Red teaming is adversarial testing with a clear scope and explicit authorization. You deliberately try to break what you are building — not for drama, but to learn how it breaks before someone else does.
For AI systems, that means testing the attack classes models are known to be sensitive to:
- Prompt injection — tricking the model into ignoring its instructions, directly in the prompt or indirectly through content it reads (a page, an email, a file).
- Jailbreaks — encoding or reframing a request so the model's refusal patterns don't trigger.
- System prompt extraction — getting the model to reveal the instructions it was given.
- Tool abuse — for agent applications, making the model call tools in ways the developer never intended.
Why deterministic checks come first
Automated red teaming usually starts with a technique library: known payloads (and variations) sent at an endpoint. Results are classified by rules first, for three practical reasons:
- Rules are instant and cheap — thousands of payloads can run inside a CI job.
- Rules are explainable — "the response contains the extraction marker" is a finding a developer can act on.
- Rules don't hallucinate — the verdict is reproducible from the response text.
An optional LLM judge can catch semantic escapes that rules miss, but it adds cost and latency. Most teams run rules in CI and save a judge for a slower nightly pass.
A practical workflow
- Define the target: an endpoint, a model, an agent configuration.
- Pick the techniques you care about — start with direct and indirect injection, plus extractions.
- Run the suite. Keep concurrency sane and rate limits set, so you're testing the model, not melting your own infrastructure.
- Read the findings by severity. Fix the cheap wins first: stronger system prompts, input/output guards, least-privilege tool configuration.
- Re-run. An attack suite is a regression test before it becomes a scoring exercise.
What this looks like in practice
Talking to an OpenAI-compatible endpoint, a tool like AMAS packages that loop: a technique library, rule-based detection, an optional LLM judge, severity-weighted scoring, and Markdown or JSON reports — including a mock target so the whole pipeline runs offline without spending tokens.
The point isn't reaching zero findings. It's knowing exactly which techniques were tested, when, and what happened.
What doesn't work
A few common beliefs need a reality check:
- One guardrail. A single input classifier, prompt wrapper or output filter will be bypassed by variations. Defense needs layers.
- Treating the model as trusted. The model is the untrusted surface; everything it can reach should be treated as remotely controllable.
- Promising "red teamed" without scope. A red teaming result is only meaningful with the technique list and date attached.
The honest baseline
No tool makes an LLM application unhackable. Red teaming is risk reduction: you test the known attack classes on a schedule, fix what you can reproduce, and stay honest about what remains. For authorized systems, that is a genuinely useful baseline — and it's a skill that transfers to every AI product you build.