Our NLP API serves the conversational AI features of our Order-to-Cash platform, from email summaries to the guardrails that check every request. It covers 19 evaluated use cases, and each one runs on a system prompt. Every time we change the model underneath, those prompts have to be checked and often rewritten.
Today an engineer does that by hand, usually with Claude's help, and we estimate 2 to 4 days per use case. Across 19 use cases, a model swap is weeks of work.
For Hackathon 2026, three of us Zoya Qureshi, Oliver Agar, and Rodrigue Talla Kuate, handed that job to automatic prompt optimisers and pointed them at the evaluation suite we already had.
It worked where it mattered. The best optimiser, GEPA, took a failing email summarisation prompt from 0% to 91% of test cases passed in a single rewrite. It then matched our production prompt with 42% fewer tokens, in 26 minutes and with no human edits. On the prompt our engineers had already tuned, all three optimisers matched its quality and found almost nothing left to gain.
The evaluation suite we already had
An optimiser is only as good as the score it chases, so the starting point was the NLP API's evaluation suite. For email summarisation it runs 68 fixed test cases, and a judge model scores each answer on five dimensions, such as factual accuracy and output format, with a written reason for every score. The judge is Mistral Small 4, a different model from the one writing the summaries. A case passes only when every dimension clears its bar.
We tested three optimisers against that suite. GEPA comes from a 2025 research paper, and we ran the open-source gepa library inside the NLP API. MIPROv2 and BootstrapFewShot come from the DSPy library. MIPROv2 proposes candidate instructions and example sets, then searches over them. BootstrapFewShot collects examples the model already answers well and adds them to the prompt. All three used the same test set and the same judge, so their results compare directly.
How GEPA rewrites a prompt
GEPA (Genetic-Pareto) rewrites a prompt using the judge's written reasons, not just its scores. Each round, it takes a prompt from its pool and runs it on three training cases. Our judge scores each answer and explains what went wrong. A second model, sidetrade-standard, reads those reasons and rewrites the prompt.
If the rewrite does better, it is scored on the full validation set and joins the pool. Pareto means the pool keeps every prompt that is best on at least one case, so the search does not lock onto a single idea. When the budget runs out, the best prompt becomes a draft for an engineer to review.
flowchart LR
P["Prompt pool"] --> R["Run on three<br/>training cases"] --> J["Judge scores<br/>and explains"] --> W["sidetrade-standard<br/>rewrites the prompt"] --> C{"Better?"}
C -->|Yes| V["Score on the full<br/>validation set"] --> P
C -->|No| P
P -.->|Budget spent| D["Best prompt,<br/>draft for review"]
classDef model fill:#ffe9e0,stroke:#ff5a28,color:#0C1B45
classDef plain fill:#ffffff,stroke:#BFC7D5,color:#0C1B45
class J,W model
class P,R,C,V,D plainA tuned prompt that held its own
We started with the prompt we already run in production, optimising the email summarisation prompt on sidetrade-light (Glimmer 30B) with no model swap. All three optimisers matched its quality, which is a good sign for the prompts our engineers write by hand. Mean scores ranged from 0.978 to 0.985 against 0.983 for production, every confidence interval crossed zero, and none of the differences was significant (p between 0.36 and 0.83).
The production prompt simply left very little headroom. 53 of the 68 cases already scored a perfect 1.0, so no prompt could gain more than 0.017. What the optimisers did add was length: every optimised prompt raised the tokens per summary by 10% to 49%, and GEPA's took it from 1,383 to 2,063.
The run also gave us a clear picture of the evaluation suite itself. Repeat runs of one unchanged prompt passed between 61 and 64 of the 68 cases, so we treat a difference of one or two cases as noise. Five test cases with labels to correct accounted for 41 of the 55 failures we looked at, and correcting them is first on our list. The search was quick throughout: GEPA ran 14 iterations in 13 minutes, MIPROv2 13 trials in 60 minutes, and BootstrapFewShot one pass in 8 minutes.
A broken prompt, fixed in one rewrite
Next we gave the optimisers a prompt that needed fixing: a new email summary prompt that never says what format to answer in. On sidetrade-light it failed all 68 cases. Every failure was on output format, because the model wrote long markdown reports, while the other judged dimensions already scored 0.94 to 1.00.
GEPA needed a single reflection. It read the judge's comments, saw that every failure was about format, and rewrote the instruction to fix it. The result passed 62 of 68 cases, 91%, after grading 142 summaries during the search, and scored 0.147 higher than the same starting prompt on our previous internal model (95% interval 0.129 to 0.156).
MIPROv2 graded six times as many summaries, 867, and reached 49 of 68. BootstrapFewShot learns from examples that already pass, and there were none yet, so it returned the prompt unchanged. GEPA's fix also stayed lean: 971 tokens per summary against 693 before, about a third below the production prompt on the same model (1,445).
The same quality for 42% fewer tokens
For the third run we changed two things. GEPA could search three times longer, and its score penalised any summary over 624 tokens. The token limit is what made the difference: without one, a longer search only produced longer prompts and longer summaries.
Both prompts ran back to back on the same 68 cases. GEPA's 58-word prompt passed 65 against 63 for production. The quality difference was +0.002, with an interval of −0.004 to +0.011, so the two are level, and a second run reproduced the result. Every summary came out 42% shorter and faster to produce.
821, against 1,427 for the production prompt. 42% fewer.
2.7 seconds, against 4.2 for the production prompt.
26 minutes with no human edits, against an estimated 2 to 4 engineer-days by hand.
Where review stays in charge
We also optimised the guardrail prompts that check what goes into and comes out of the model. The test set was 30 OWASP-style attacks, plus 30 harmless requests to check that normal use still gets through.
Our live guardrails stopped all 30 attacks. GEPA's version blocked none of the harmless requests, against 2 for the live prompts, but our review caught one attack it let through, and it used 12% more tokens (5,597 against 4,982). MIPROv2 and BootstrapFewShot found nothing to change. A single missed attack is enough for us to keep the live prompts, and catching it is exactly what the review step is for.
An optimised prompt is a draft, and it gets the same review as one a person wrote.
What we would tell another team
Use an optimiser to repair, not to polish. It fixed a broken prompt in minutes, while our tuned prompt left it little to gain. The obvious use is restoring prompts after a model swap, which is exactly the job that costs us weeks today.
Invest in the evaluation suite first. An optimiser is only as good as the judge it learns from, and a handful of test cases with labels to correct explained most of the failures in our first run. Know the noise too: repeat runs moved by three cases out of 68, so we read results as paired comparisons with confidence intervals, not raw pass counts.
Feedback beats search volume. GEPA reads the judge's written reasons, not only the score, which is likely why it fixed the format problem in one rewrite while MIPROv2 graded six times as many summaries. GEPA also rewrote with our stronger sidetrade-standard model, which may account for part of the gap.
Put cost in the score, or prompts grow. Every optimiser without a token limit made summaries 10% to 49% longer. One limit in the score turned that into a 42% saving.
Next, we plan to run GEPA with a token limit on all 19 use cases after the next model swap, with every result reviewed before it ships. The principle carries beyond prompts. Let people decide what good looks like and write it down as an evaluation, then let the optimiser do the rewriting. The judgement stays with us, and the days of rewriting do not.
