GEIS: A Generation-Evaluation-Improvement Loop of Agent Skills for Long-Form Article Generation
Summary
GEIS proposes a skill-based generation–evaluation–improvement loop for long-form article generation, implemented in Tasi Harness. It decomposes writing into six explicit stages, delegates retrieval and diagramming to separate skills, uses pairwise PDF-aware evaluation, and applies permanent patches to the writing skill based on recurrent findings across 20 Wikipedia topics.
Mathematical/empirical assessment
The claimed +4.05-point gain in the 20-topic improvement experiment (Table 5) rests entirely on within-system comparisons: pre- vs. post-patch outputs evaluated by the same LLM-as-a-judge (Qwen 3.5 Plus) using the same pairwise rubric. No control for judge drift, no inter-annotator agreement, no human validation, and no ablation of the evaluation skill’s influence on the reported improvement. The paper provides no evidence that the observed score shifts reflect objective quality gains rather than rubric overfitting or judge bias amplification.
Strengths
Clear architectural separation of skills (Table 1, Table 7); explicit audit/refine gates; deterministic image/diagram anchoring; systematic patching of SKILL.md rules; reproducible 20-topic setup (Table 2); transparent reporting of regressions (Table 6).
Concerns
The decisive flaw is circularity: improvement is measured and guided by the same evaluation skill (pce) whose own reliability is unvalidated. The paper omits the essential test—human evaluation or blinded expert scoring—on a held-out subset to confirm that the +4.05-point gain corresponds to actual content or structural improvement. Without it, the central claim—that GEIS enables reliable evaluation-guided skill evolution—lacks empirical grounding. To meet standard practice in LLM evaluation (e.g., as in FActScore or MT-Bench), the paper would need human-rated deltas on ≥5 topics, with inter-rater agreement reported, and demonstration that Qwen 3.5 Plus scores correlate significantly with human judgments on those topics.
Final decision
Strong reject