hego.red - Practical AI/LLM Red Teaming Notes

Practical notes on AI/LLM red teaming

Start here

Welcome. These are practical, hands-on notes on red teaming LLMs, written from a pentester's point of view. Use the tabs above to move around: Foundation explains what you are really testing, Attacks and Methodology are the how, Bootcamp is a guided course and PortSwigger has worked labs, Scope and Attack Flow help you scope a target, and Terminology is the quick dictionary. New here? Just keep reading this tab from top to bottom. Press / any time to search everything.

Foundation - What Are We Actually Testing?

Read this first. In one minute it shows you what an "AI feature" really is, the words model / chatbot / agent, and how the AI makes an answer. Once you see the picture, the other tabs make sense.

1. You test the app, not the brain

An "AI feature" is just a normal app with an AI model plugged in. You test the app the client built. The model's brain is usually the vendor's (Claude, OpenAI) and out of scope.

You / the chat box
where you type your message
↓
The AppYOU TEST THIS
the client built this part:
  • adds hidden rules (the system prompt)
  • may read documents or a database (RAG)
  • may call tools (email, database, run code)
  • shows the answer back to the user
↓
The Model / "the brain"usually the vendor's
it just turns text into more text
Almost every bug lives in the green box (the app), not the brain. Getting the brain to say something rude is the vendor's problem, not a real finding.

2. Model vs Chatbot vs Agent

These three words confuse everyone. It is just a ladder - each step adds one thing.

Model (LLM)
the brain. Reads text, guesses the next word. That is all.
→
Chatbot
model + hidden rules + a chat window. It talks to you.
→
RAG app
chatbot + it can read documents and your data.
→
Agent
model + tools. It can DO things and take steps, not just talk.

The further right, the more it can do - and the more you can attack.

3. How it makes an answer

The model does not "think". It reads text and guesses the next word, again and again. Here is what happens when you hit send:

1
You send a message.
↓
2
The app glues things into ONE block of text: the hidden rules + your message + any documents + the past chat.
↓
3
The model reads it all as one block. It cannot tell the rules apart from your text.
↓
4
It writes the answer one word at a time.
↓
5
If it needs a tool, it asks the app to run it, gets the result, and keeps going.
↓
6
The app shows or uses the final answer.

4. The one flaw everything comes from

Look at steps 2 and 3 again. The rules and your input get mixed into one block, and the model treats it all the same.

Hidden rules+Your input+Documents
↓
The model sees ONE blob
so your input can act like a rule
This is the whole game: the model can't tell instructions from data, so your text can become an instruction. That is prompt injection, and almost every other attack builds on it.

5. So where are the bugs?

Each layer you saw above has its own bug. This is the OWASP LLM Top 10, in plain words:

LayerThe bug
Your inputPrompt injection - your text acts like a command.
The modelJailbreak (break its safety); if fine-tuned, leak its training data.
Documents / RAGPoison the documents so the model obeys them (indirect injection).
ToolsMake it use a tool it shouldn't, or attack the tool's input (SQLi, SSRF, run code).
The answer being shownInsecure output handling - the app runs the answer as HTML/SQL, so XSS and friends.

6. The risk depends on the app

The same answer can be fine in one app and a disaster in another. So before you test, ask: what is this app for, and what would count as "bad" here?

AppWhat "bad" looks like
Story / game generatorwants wild, creative output - almost anything goes.
Internal HR or support botmust stick to the facts - making up a policy is the bug.
Email writer for the companyshould be honest but on-brand - rude or dishonest text is the bug.

You usually want the model's intelligence (good language and reasoning), not its knowledge - it should answer from your data and say "I don't know" otherwise.

Two myths to drop. (1) "AI risk is just sci-fi robots taking over." No - the real risks are here now: your bot can leak data, give harmful answers, or get the company sued today. (2) "A bigger, smarter model is safer." Benchmark scores do not tell you how safe it is in YOUR app. Test your app, not the leaderboard.

Sources: OWASP Top 10 for LLM Apps, PortSwigger Web LLM attacks, MITRE ATLAS. Next: the Terminology tab for the words, then Methodology for the plan.