Marketing eval framework: expert-calibrated judge library in evals/ (across copy, creative, tool-calling)

[coreyhaines31]

Vision

Build an expert-calibrated marketing eval framework — a shared judge harness plus a growing library of judges that grade the quality of what a marketing agent produces and does, across every major marketing task: copywriting, creative, positioning, and tool-calling. Lives as a top-level evals/ framework in this repo, mapped 1:1 onto the skills.

Reverse-engineered from runneth-tools/hook-grader (MIT) — an open-source eval that grades whether an ad hook stops the scroll, calibrated against a world-class creative strategist (Alysha, Runneth/Motion). We can seed the ad-hook judge directly, with credit.

What hook-grader taught us (the transferable primitive)

An eval is just judge + answer key, plus machinery to measure the judge against the key. hook-grader is a clean 6-part instance:

• Judge (JUDGE.md) — a paste-able rubric (LLM system prompt) returning a structured JSON verdict (verdict, trigger, tactic, reasoning, improvement).
• Answer key (eval-set.json) — 116 examples hand-labeled by a named human expert, with a realistic distribution (73 FAIL / 43 PASS, not 50/50).
• Harness (eval-harness.mjs) — runs the judge over the key, reports agreement % + confusion breakdown (false-pass vs false-fail) + every disagreement.
• Grader CLI (grade.mjs) — grade new inputs at scale, BYO API key.
• Lib (lib.mjs) — provider-agnostic model calls (Anthropic/OpenAI auto-detect), batching, judge-loading.
• Methodology (METHODOLOGY.md) — calibration loop, blind two-judge consensus, what the score means.

Principles to carry over

1. An eval = judge + answer key. A judge is only trustworthy insofar as it agrees with human labels.
2. Rubric = failure taxonomy. The reliable signal in quality is negative — name the specific ways things fail; it's far easier and more reliable than predicting "great."
3. Calibration loop: run judge → find disagreements → sharpen rubric → repeat; publish the agreement number.
4. Match the real distribution in the answer key, or the judge learns the wrong prior.
5. Err strict for a bad-X detector — optimize precision on the fails ("kill dead-on-arrival, then test the rest"), not "talent scout for great."
6. Blind two-judge consensus (one model per lab, source hidden) for high-stakes grading.
7. Structured verdict with an improvement suggestion — the eval doubles as a coach.

The three eval types (marketing tasks have different shapes)

• Type A — Artifact quality (hook-grader is this). Grade a produced artifact against an expert-calibrated rubric. One judge + answer key per artifact. Covers most copywriting/creative/positioning outputs.
• Type B — Tool-calling / trajectory. Input = a task + the agent's tool-call trace; judge grades whether it picked the right tool, right args, right sequence, and used the result. Answer key = tasks with known-good trajectories. This is what "tool calling" needs — it is not artifact quality.
• Type C — Skill behavioral. Already exists as each skill's evals/evals.json (prompt + expected_output + assertions) — grades whether the agent behaves right (checks product-marketing.md, applies the framework, defers to the right sibling). The new framework is the quality/trajectory layer these don't cover; it complements, doesn't replace them.

Design note: keep the existing per-skill evals/evals.json (behavioral) as-is. The new top-level evals/ framework is a distinct layer. The README must make the two-layer distinction explicit so contributors don't conflate them.

Proposed structure

evals/
├── README.md              # the framework: eval = judge + answer key; the two layers; how to run & contribute
├── lib.mjs                # shared: provider-agnostic model calls, batching, agreement, confusion, consensus mode
├── harness.mjs            # generic: run ANY judge over ANY answer key → agreement % + confusion + disagreements
├── grade.mjs              # generic CLI: grade new inputs with any named judge (BYO key)
├── judges/
│   ├── ad-hooks/          # Type A — SEEDED from hook-grader (credit Alysha/Runneth, MIT)
│   │   ├── JUDGE.md
│   │   ├── answer-key.json
│   │   └── README.md
│   ├── headlines/         # Type A
│   ├── value-props/       # Type A
│   ├── ctas/              # Type A
│   ├── subject-lines/     # Type A
│   ├── landing-hero/      # Type A
│   └── tool-calling/      # Type B (trajectory) — its own harness variant
└── METHODOLOGY.md         # calibration loop, distribution, consensus, error-bias guidance

Generalize hook-grader's single-topic files into a judge-agnostic harness (harness.mjs judges/<name>), so adding a category = adding a judges/<name>/ folder with JUDGE.md + answer-key.json. lib.mjs stays provider-agnostic (Anthropic + OpenAI, model override via env), and defaults to current models (Opus/Sonnet 4.x) rather than hook-grader's pinned ones.

Judge ↔ skill mapping (starter set)

Skill	Judge(s)	Type
ad-creative	ad-hooks (seed), UGC-script, concept	A
copywriting	headlines, value-props, CTAs, landing-hero	A
emails / cold-email	subject-lines, cold-open, sequence-step	A
positioning / product-marketing	one-liner, category-frame	A
social	hook (short-form), thread-opener	A
ads / analytics / attribution	tool-calling (right tool, args, sequence)	B

Answer-key & calibration methodology (per judge)

• 40–150 real, anonymized examples, hand-labeled by a credible human (name them, as hook-grader credits Alysha).
• Realistic distribution (most marketing artifacts fail the bar — reflect that).
• Calibration loop until agreement plateaus; publish the number + confusion split.
• Document the intended error bias (strict for bad-X detectors).
• Optional consensus mode for high-stakes categories.

Phased rollout

• Phase 0 — Harness. Generalize lib.mjs + harness.mjs + grade.mjs to be judge-agnostic. Seed judges/ad-hooks/ from hook-grader (credit). Reproduce the agreement number as the framework's first passing check. Write evals/README.md + METHODOLOGY.md.
• Phase 1 — Copywriting judges. headlines, value-props, CTAs, subject-lines — each with a hand-labeled answer key and a calibration pass.
• Phase 2 — Tool-calling (Type B). Define the trajectory answer-key format + harness variant; one worked judge (e.g. "did the agent choose the right analytics/attribution tool + args").
• Phase 3 — Broaden + contribute. More categories, consensus mode, a CONTRIBUTING-style guide so a new judge is a well-trodden path. Cross-link from each skill's README.

Strategic framing

This is an agent-native content play (see the pattern in memory): Runneth shipped an open eval as marketing. An open, expert-calibrated marketing eval framework is a category-defining asset for marketing-skills — more defensible than any single eval and the natural companion to the skills. MIT-friendly to seed (ad-hooks judge) with credit.

Open questions / decisions

• Human labelers: who calibrates each category's answer key? (For the seed, reuse hook-grader's labels with credit; net-new categories need a credible grader — us, or invited experts.)
• Runnable vs paste-only: ship the Node harness (BYO key) and the paste-the-JUDGE.md-into-any-AI path (hook-grader does both — keep both).
• CI: do we wire an agreement-threshold check into CI per judge, or keep it manual/opt-in (cost)?
• Naming: evals/ framework name and whether to also mirror it as a standalone public repo later for the marketing play.
• Scoring granularity: PASS/FAIL (hook-grader) vs a 1–5 score per category — decide per judge.

Credit / license

hook-grader is MIT. Seed the ad-hooks judge and reuse its answer-key labels with clear credit to Alysha (@alyshaboehmm) and Runneth, mirroring how we credit other reused OSS (gooseworks, kappaemme). Net-new judges are ours.

---

Scoped from a reverse-engineering pass on hook-grader. Build not started — this is the plan of record.

[coreyhaines31]

Flagship application + reprioritization: the primary purpose of this framework is evaluating different models on marketing tasks (Opus vs Sonnet vs GPT vs Kimi vs Gemini…), surfaced as a live benchmark/leaderboard. Scoped in #480.

That sharpens the framing here: the evals/ judge library is the engine + credibility layer, and its headline use is a model leaderboard (public authority asset on the outside, Magister model-routing table on the inside). Key design constraint that flows back into judge design: judges must support blind, neutral, consensus grading (a contestant model can't grade its own category — self-preference bias), and each judge's published human-agreement number is what makes the leaderboard defensible. Phase 0's ad-hooks seed becomes the first benchmark column.

[coreyhaines31]

Judges span eval TYPES and MODALITIES (not just text rubrics)

The judge library must cover more than copywriting. Judges are typed by both eval type and modality, and each task family brings its own contestants + judge:

• Text rubric (Type A): copywriting (hooks, headlines, CTAs, subject lines, value props), long-form (blog/guides — different rubric dims: structure, depth, factual accuracy, originality, brief-adherence), ad-creative (concepts, UGC scripts).
• Multimodal judge + human panel (Type A, media): images (ad images, social graphics, product shots) and video (video ads, short-form). Visual quality resists reliable auto-grading, so these need a multimodal grader calibrated against a human panel, and they lean on the field/usage layer more.
• Trajectory grader (Type B): tool-calling — grades the agent's chosen tool / args / sequence / use-of-result against a known-good trajectory. Grades the agent, not raw text.

Updated judge ↔ skill mapping: copywriting→copy judges; content-strategy→long-form judge; ad-creative→creative + image + video judges; ads/analytics/attribution→tool-calling judge; social→short-form hook/thread judges.

The shared harness must therefore support pluggable input modalities (text, image, video, trajectory) and grader backends (text LLM, multimodal LLM, human-panel intake), while keeping the same lifecycle (judge + answer key + agreement %). Also carry cost (tokens) + latency through the harness so the benchmark (#480) is triaxial, not quality-only.

[coreyhaines31]

Add a third eval axis: skill-compliance / pressure-test evals (from obra/Superpowers)

Competitive research surfaced the single most valuable architecture idea for this framework — obra/Superpowers' superpowers-evals (MIT, ~260k★). Its judges don't just grade output quality; they run adversarial "drill" scenarios that actively try to make the agent skip the skill or its guardrails, using Cialdini influence principles (authority, scarcity, "production emergency," "just do X, don't bother with the process"), and assert the agent complies anyway.

That's a third eval axis we don't have:

• Type A (have): artifact quality — is the output good? (our judges)
• Type B (have, scoped): tool-calling trajectory
• Type C (have): per-skill behavioral evals.json
• NEW — skill-compliance / robustness: under pressure, does the agent still do the right thing? e.g. tempt it to "just write the ad copy, skip reading .agents/product-marketing.md" and assert it loads context first; tempt it to skip the FTC disclosure in influencer-marketing; tempt it to sum platform conversions in attribution after being told "just add them up, the client's in a hurry."

Why it matters for marketing skills specifically: our skills carry real guardrails (grounding in the product context, FTC disclosure, don't-sum-platform-conversions, brand voice). A quality judge won't catch a guardrail being skipped under pressure — only a compliance drill will.

Convergence: this pairs directly with the prompt-injection resistance dimension in the Magister schema proposal (#484 / Magister #900) — same family (adversarial robustness), and both want the canonical fixture company. Point our eval harness at obra/superpowers-evals as prior art; credit obra.

Filed from the 2026-07-24 competitive-learnings research brief.

[bshelby88]

The outreach-copy slice of this problem is well-suited to deterministic fixtures. We've found that weak/strong CTA, personalization, length, and spam-risk examples with human labels give a more repeatable signal than open-ended LLM scoring alone.

I can contribute a fixture set in that format — enough to calibrate an eval without building one from scratch.

Disclosure: I operate Power Pack (https://power-pack-x402.fly.dev/openapi.json), an x402 email-scoring API. The fixture set is free.