I have been building quite a different kind of Copilot Studio agents, but this latest project was the most challenging one. The knowledge source list was very narrow, and the agent was not allowed to hallucinate or to say anything that is not true. Instead it should tell what is available, and if that information is not what the user needed, point them to the team that owns the process.
Another challenge was that I did not know the context well enough. Luckily I had a product owner. At first we developed the agent side by side, but the results scattered so badly that I had to ask for a rule: only I, working through Claude Code, would make changes. The product owner provided very good testing questions, and somewhere in the middle of that I found a completely new way of developing agents.
Instead of designing, I let the test results guide the development
There were gaps in the knowledge, unrelated test questions, questions without expected results but then test run by run there picture started to appear into my mind. I also thought that why I’m running these test in Evaluation tab manually and then dragging result csv’s to Claude – that could be automated as well but needed too much effort and we had only couple days before product owner’s holiday.
- A fifth of the test cases flipped between identical runs, so a single run can neither show a regression nor confirm a fix.
- Half the decisions ended in a change to the test set, not to the agent.
- Claude Code made every change to the agent. The test results decided which change to make.
An agent building an agent
Most writing about TDD and AI puts the agent on the writing side: it produces the tests, the implementation, the edge cases you forgot. This project had an agent on both sides. Claude Code made every change to the Copilot Studio agent, and the Copilot Studio agent was the thing under test.
The part that was new to me is what sat between them. I did not decide the next change and ask for it. The test results decided, and Claude Code carried it out. My job moved from designing the agent to reading the evidence and choosing which failures were worth acting on.
The tests were written first, by the person who knows the process, and the agent was changed until it passed them. That is red, green, refactor, except for the one property the whole discipline rests on. In normal TDD a passing test stays passing until you change the code. Here it does not.
One run tells you nothing, in either direction
On the last full day I published two fixes, ran ten test rounds and put 210 questions through the agent. One planned release I deliberately did not ship.
The morning round looked bad. Four targeted cases improved exactly as predicted, but five other cases regressed in areas I had not touched at all. The obvious move is to fix them.
Instead I ran the same three sets again with nothing changed in between. Same agent, same questions, same grader.
| Set | Round 1 | Round 2, nothing changed |
|---|---|---|
| Core process, 10 questions | 5 | 8 |
| Operational detail, 13 questions | 9 | 9 |
| Coverage sweep, 20 questions | 10 | 14 |
| Total | 24 / 43 | 31 / 43 |
Four of the five regressions repaired themselves.
Seven points of improvement, from changing nothing.
Without that control run I would have spent the afternoon fixing five things that were never broken, and reported four real fixes as a net loss.
The same mistake is available in the other direction. One fix targeted a case where the agent refused a question whose source it definitely had. After the first run it passed and I wrote down that the fix had worked. Second run, fail. Third run, fail. What I had seen the first time was noise wearing the costume of a result.
That case gave the sharpest finding of the project. Two other questions asked for the same thing in different words and both passed from the same corrected content. The third phrasing did not trigger the topic at all. The content was right, the routing was not, and adding trigger phrases closer to the failing wording changed nothing. That is the useful part: it told me the lever was wrong.
A fifth of the cases are non deterministic
By the end I ran the combined set three times without touching the agent. Thirty documentation questions:
| Count | |
|---|---|
| Stable pass | 20 |
| Stable fail | 4 |
| Flipped between runs | 6 |
The honest answer to “how good is the agent” is a range, 67 to 87 percent, and running more does not narrow it. It only samples the same distribution again.
So before a test suite can measure your agent, you have to measure the suite. Run it twice with nothing changed. If it moves more than your changes do, you are not measuring the agent yet. Two runs is the minimum to separate noise from regression, and it is not enough to call anything stable: the set that looked perfectly deterministic across two runs produced a flip on the third.
Not every failure is the agent’s fault
Once the numbers stopped lying to me, the failures sorted into four groups, and only the first is fixable in the agent.
The agent’s fault: the content is reachable and it still refused, or answered from the wrong place.
The measurement’s fault: one question asked for a deadline, the agent answered with a link to the calendar holding the deadlines, and the grader failed it for “only directing the user to another link”. The product owner later confirmed the link is the correct behaviour. The agent had been right all along and the test was wrong.
Structural: a deterministic response cannot report a specific date when the calendar contents are not a connected source. No instruction fixes that. It is a product decision.
A real content gap: some answers exist only in a training video, and a video cannot be searched as text without a transcript. A documentation backlog item, not an agent defect.
Sorting failures into those four buckets before touching anything was the highest leverage habit I picked up.
The test set is a product too
The questions came from the product owner, which is exactly right, but a test set written by a domain expert has its own failure modes.
Some questions were not about the documentation at all. They asked what the agent should do in a situation, which is a question about instructions, not content. No source will ever answer them, so they were split into a set of their own.
Worse, three cases had an expected answer containing a note addressed to a human. It read “NOTE FOR PO: please confirm the authoritative location for this”, and that text went to the grader as the standard the agent was measured against. Unpassable by construction. If a perfect agent cannot pass a case, it is not a test. Read the expected answers, not just the questions.
What I would automate next
I would automate the transport and leave the judgment alone. Every decision worth making in those three days came from looking at why a case failed, and half of them ended in a change to the test rather than to the agent. A loop that only optimises the pass rate would have driven straight past all of it.
That is the shape of the thing. A product owner who knows the process, a test set that measures it, and one agent making the changes to another. The build was fast because the agent did the work. It was correct because the test results, not my design instincts, decided what the work should be.