I'd tell Claude "make the interview questions feel more natural." It would hand back a diff, and the new prompt would read great. More thoughtful. More rules. More polish. I'd approve it and ship. Did it actually help? I had no idea. I was approving prompt diffs on vibes.
The stakes on those diffs are higher than they look. The app I've been building, Xtraktr, runs its AI entirely on-device. Apple's Foundation Models is the default, since it works out of the box on most devices. There's also an optional heavier model, a roughly 2.5 GB Gemma download the app calls Premium Intelligence, that asks better questions at a real resource cost and covers some devices Foundation Models can't run on at all. The quality of an interview question or a summary bullet is the product, so for me prompts are load-bearing code. And since I don't hand-write code anymore (I've been 100% agentic for months), my prompts were being edited by an agent too.
With prompts, vibes are worse than useless, because a prompt that reads better to a human very often performs worse on a small on-device model. I know that now because I built the thing that told me.
The Decision: A Tiny Harness, No Framework
When I decided to get serious about prompt quality, the obvious move was reaching for an eval framework. I didn't. I had Claude build a small macOS command-line harness that lives in the repo at Evals/, and the whole thing is maybe a dozen Swift files.
The one decision that makes it work: the harness depends on the same XtraktrAI package the app ships. There's no copy of the prompt pasted into a test file that drifts out of sync. The harness calls generateFollowUp and generateSeedQuestions through the exact code path the app uses, against the exact models the app uses, on the machine sitting on my desk.
That's the best practice worth naming first: eval the shipped path, not a replica. If you have ever watched your code sail through the simulator and then fall apart on a real device, you already have the intuition for this. Most eval setups I see score a prompt string in a notebook while the app assembles and runs something slightly different. That notebook is the simulator version of a test: close enough to feel real, different enough to lie to you. My harness is the physical-device version. It runs the real code against the real on-device models, so when a number moves here, it moved for the thing that actually ships. The moment your test and your app drift apart, your numbers are about nothing.
Task selection is an environment variable. EVAL_TASK=action-items scores the action-items artifact, EVAL_TASK=seed-questions scores interview seed questions, no variable runs the default follow-up-question matrix. Each task runs a grid of variants (backend times prompt) against fixed fixtures (conversations, imported-content briefs, or transcripts, depending on the task) and writes markdown plus JSON to ~/xtraktr-evals/. Adding a task means cloning one Swift file. That's the entire architecture.
Two Layers: Deterministic First, Judge Second
The scoring has two layers, and the order matters.
Layer one is a deterministic rubric. No LLM involved, so no flakiness. For follow-up questions it directly encodes the prompt's contract:
/// Passes the structural contract the app's prompt asks for.
var passesFormat: Bool {
sentenceCount <= 1 && endsWithQuestion && !startsWithPreamble
}Plus word count, a readability score, and a banned-opener check. A prompt change either moves these numbers or it doesn't. Cheap, reproducible, runs on every variant.
Layer two is LLM judges, but only for the dimension the rubric can't see: is this actually a good question? Set EXTERNAL_JUDGE=1 and the harness skips its internal judge and emits a candidates JSON file for external judges to score. No API key in the harness. I hand the candidates to a panel of three Claude judges, and a small Python script (seed_judge.py) builds blind A/B pairs first: old prompt's output and new prompt's output, shuffled into slots A and B by index parity so the judges never know which is which.
Blind, and a panel of three, are both non-negotiable. A single judge that knows which output is "new" will flatter the new one. I learned to trust the panel when all three agreed and to get suspicious when they split.
The Time More Rules Made Everything Worse
Here's the finding that converted me for good.
While tuning the seed-question prompt, the judges suggested improvements, and the natural agentic move is to stack them all into the prompt. So I had Claude build a hardened nine-rule variant and ran it against the plain baseline. It read like a better prompt. Numbered rules, explicit constraints, no ambiguity.
It regressed format pass from 100% to 47%.
The small on-device model loses format discipline when you overload the instruction with written-out rules. One whole lens collapsed to a handful of questions when the prompt asked for a full set. Without the harness, I would have shipped that prompt, because the diff looked like pure improvement. This is the exact failure mode of agent-tuned prompts: agents love adding rules, and on a small phone-sized model, rules are not free.
The rewrite that eventually won was the opposite of that instinct: short, research-backed, with a small cleanup function doing the format enforcement the prompt used to beg for. Through the same blind panel, the judges preferred it over the old version almost every single time. Shorter won, and I only believe that because I measured it instead of eyeballing the diff.
Proving Temperature Was Hurting
At one point I had a theory that raising sampling temperature would buy more variety across question sets. It's a reasonable theory. Apple's GenerationOptions supports it, and an audit of the app's Foundation Models usage had flagged that it was never set.
So instead of shipping the theory, I had Claude add a SEED_TEMP sweep to the harness:
let temps: [(String, GenerationOptions)] = [
("t-default", GenerationOptions()),
("t10", GenerationOptions(temperature: 1.0)),
("t15", GenerationOptions(temperature: 1.5)),
]Result: net negative. Format compliance slipped, the questions read a notch more scattered, and the extra variety I was chasing never showed up. At the highest setting the model actually did worse than leaving temperature alone, and it started leaking bits of the prompt's own scaffolding into the output. So temperature stays at the model default in the app, and now there's a measurement behind that decision instead of a hunch I would have shipped.
The Eval That Told Me to Stop
The most valuable thing the harness ever did was end an argument I was having with myself.
Xtraktr has steering modes (expand, deepen, clarify, synthesize) that shape the follow-up question. I wanted to know if they actually produce different behavior, so I ran a blind mode-label panel: judges see the outputs from all four modes, shuffled, and try to assign each to its mode. If the modes are real, judges should recover the labels.
The judges could barely do it. Their guesses landed just above the one-in-four you'd expect from picking at random, which is another way of saying the modes were there in the code but had nearly vanished from the actual questions. Then the uncomfortable part. I ran an older, pre-rewrite prompt through the same panel and it did noticeably better. The cleaner follow-up rewrite I'd been so happy to ship had quietly smeared the four modes together on its way in, and deepen had taken the worst of it. Nothing in the quality rubric measures whether the modes stay distinct, so nothing had flagged the regression until I went looking.
My first instinct was that the mode instructions were just weak, so I sharpened them: crisper contrasts, explicit do-not rules. That exact fix had rescued a related feature earlier, where the same kind of blind test went from the model treating everything identically to the judges reliably telling the outputs apart. For the steering modes it barely moved. More prohibitions just herded the model toward safe, generic questions.
That's a model-capability ceiling. Four distinguishable steering modes may be more than this on-device model can honor, and no amount of prompt iteration was going to fix it. Without the eval I'd have burned a week having an agent rewrite those fragments, and every rewrite would have looked plausible. The harness turned "keep trying" into "stop, this is a product decision now."
One more lesson from that experiment. Before the blind panel, I was measuring variety the lazy way, by counting how many questions were literally unique strings, and that number sat at a reassuring 99% while the questions had actually collapsed into near-identical phrasing with a word swapped here and there. String metrics lie. If you want to know whether two prompts produce genuinely different behavior, make a blind judge label the behavior.
What I Tell the Agent Now
If you direct agents rather than hand-code, the harness changes what you put in the prompt. Mine now amounts to this, and most of it lives in project docs the agent reads:
Never ship a prompt change without an eval run. The agent writes the variant, runs the harness, and reports the aggregate numbers against baseline before I even look at the diff.
Aggregates only. The model does not give you the same output twice, and single-run rubric numbers on this harness carry roughly 15 to 20% noise. An agent will happily cherry-pick one good run and declare victory. Make it run enough fixtures that the rates are stable.
Rules are suspect. When the agent proposes adding constraints to a prompt for a small model, the prior should be that format compliance drops. Test the stacked version against the short version every time.
Blind everything that involves a judge. And know the bias you can't fully remove: the harness's internal fallback judge is Foundation Models scoring Foundation Models output, and the code comments openly flag the self-preference problem. Prefer a stronger external judge, and read same-model scores with a grain of salt.
What Still Sucks
The harness is a second codebase to maintain, and every prompt seam needs its rubric kept in sync by hand. The judge-panel flow is semi-manual: build pairs, dispatch three judges, aggregate. It works, but it's not a button. And deterministic rubrics can validate a contract that's wrong; the harness once had an ends-with-question-mark check that failed "Tell me about..." questions the research actually favors.
But the core trade holds up. Agentic development gave me speed and took away my feel for whether changes helped. The eval harness gave the feel back as numbers. The agent steers the code. The evals are how I steer the agent.