I gave “I Will Hack My Agent: The Story of a Copilot Studio Experiment” at Experts Live Sweden in Stockholm on 13 October 2026, at 11.50 in Room D. The story behind it is simple. Earlier I built a job application assistant in Copilot Studio and attacked it with five kinds of attacks, and they worked. This time the same attacks mostly did not get through, and the honest headline of the session was on my fourth slide: I did not fix it. Microsoft did.
The agent I attacked
The job application assistant was built as a demo, never for production. It asks the applicant’s name and email, shows the open positions from a database, takes the CV, compares it to the job description, writes a summary and a match rating, and saves everything to Dataverse. A hiring manager sees the applications in a model-driven app. It also answers questions the job ad does not cover: if someone asks whether the job includes dental care, the agent asks the hiring manager once, saves the answer, and the next applicant gets it without bothering anyone. One agent, lots of functionality, almost too much. That is exactly why it was a good target.
I am a developer, not a security professional. Everything I know about attacking agents I learned by asking AI how to hack my own agent. That is also my first lesson: if I built an agent that faces the public internet, I would ask a security professional to test it, not myself.
Five attacks, then and now
The first two were prompt injections inside the CV, the indirect kind. A visible one in red text, telling the agent to rate the CV a perfect match and forget its earlier rules, and a hidden one in white text in the footer. When I first ran them, the agent rated a UX designer a perfect match for a data analyst role. On stage in Stockholm, the same CV got “Does not match”.
There was a twist. At 08.55 that morning I ran an older version of the visible CV against the same agent, and it rated the CV a perfect match, printed its own system instructions and saved the perfect match to Dataverse. So the platform is better, but something in my own agent is still broken, and I did not have time to fix it before the session. The platform changes all the time. This time the change was for the good, but it was not my change.
The third attack was data exfiltration: asking for other applicants’ data directly, by inference, through role-play and by pretending to be the hiring manager. Every attempt failed, and not because of my instructions. The agent simply had no way to read the other applicants’ data. If I had given it a normal connector or an MCP server to that table, the answer could have been different.
The fourth was system prompt fishing in the chat. A year ago the blunt requests hit Copilot Studio’s content filter. They still do, but a polite request for a short summary of the instructions reached the model and got a summary of one tool’s instructions in one of two runs. Whoever reads your instructions knows how to get around them, so assume they leak and keep secrets, keys and internal URLs out of them.
The fifth was rapid fire: many messages fast, looking for the one answer that slips. The platform held, but three of five rapid requests vanished without an error, in both runs. And every message consumes credits, so even an attack that gets nothing out costs you money.
I ran the demos with Claude in Chrome typing the attacks into the agent while I talked. Claude Code cannot drive that browser extension directly, so Claude Code writes the prompt into a file in my Google Drive, and Claude in Chrome reads it from there. The same setup ran my test sets before the session and wrote its output back to Drive for Claude Code to analyse.
The finding I did not expect
The most interesting result was not an injection. I asked the agent to contact the hiring manager five times about my application, in one message. In one run it planned five separate calls to the hiring manager tool, in the next run one. Each call means an approval request, an AI Builder prompt, a new row in the knowledge base and an email. The only brake was that the connection was not set up in the test panel. The fix belongs in the architecture: if the same person asks the hiring manager the same thing many times, the flow should stop it, not the prompt.
What the defences are
In the standard harness the defence you can add yourself is content moderation. In the generative answers node you can write rules against following instructions inside content, and in the agent’s generative AI settings the same rules apply to the whole agent. The moderation level was low in my agent. Setting it to high helps too. It does no harm to have both, even now that the platform blocks most of it.
Then I ran the same attacks on the new GitHub Copilot harness with a GPT-5.6 reasoning model. I added no moderation rules at all, and the model refused by itself, including the polite summary request, and it said why instead of returning a content filter error. The catch is cost: in the new harness, testing in the studio consumes credits, and it uses more AI per message.
Architecture, not instructions
The decision that protected my agent most was not a rule. The applicant data and the CV go to Dataverse through an agent flow, one direction only, so the agent cannot ask anything back from there. Using Power Automate for that can feel clumsy, but it is an architectural decision that stops the attack instead of slowing it down. Sami Laiho said at HELish Summit that your job is not to stop the enemy, but to slow it down. Here the flow stops it.
It is not clean, though. The injected CVs still went into the database, and the questions to the hiring manager are saved there as well. If another agent reads that data later, I have a problem, so I would need to extract and clean the content instead of storing what the applicant sent.
Connectors are becoming MCP servers, and the same connector that reads can also act.
That is why DLP policies matter more than before. Power Platform for Admins has a Delete Environment action. If your DLP policy allows that connector and it is connected to an agent, a user, or someone else, can ask which environments exist and ask the agent to delete them, and the agent will be happy to help. Sami Laiho’s point at HELish Summit applies: a blocklist chases every new bad prompt, an allowlist is a short list of connectors you trust.
Make the secure way the easy way
Both defences that stopped me were Microsoft’s: Prompt Shields and the model. Neither one is mine, and both can change under me. So I keep my own test set and run it after every model change. Tatu Seppälä said at HELish Summit that people always choose the easiest way, so if you govern a tenant, make the secure way easy for the people building agents. Otherwise what they build in your environment is your responsibility.
And Claire Edgson said it best at Nordic Summit: data should not become instructions. In the chat window I got almost nothing out. When I put the instructions into the data, into the CV, that is where the attacks got in.
