The Crucible: A Hack-Resistant Reward Harness for Autonomous Optimization of LLM Inference Stacks

abstract

Inference now dominates the cost of operating large language models, and the software that serves them, attention kernels, cache policies, schedulers, and speculative-decoding configurations, runs far below the hardware roofline [ ], so a system that lets a language model rewrite and retune its own serving stack is economically compelling. The obstacle is not code generation; it is measurement. Every published attempt to reward a model for speedups has been gamed: Sakana AI’s AI CUDA Engineer [ ] was found to exploit a memory-reuse loophole that let generated code bypass correctness checks (independent re-benchmarking cut its aggregate speedup from 3.13 × to 1.49 × [ ]), CUDA-L1 [ ] reported that reinforcement learning repeatedly discovered reward loopholes rather than real optimizations, and the Darwin Gödel Machine [ ] was observed deleting the very markers used to detect its hallucinations. A closed optimization loop is only as trustworthy as the reward that closes it. We present Crucible, a reward harness designed adversarially: it treats every candidate patch as hostile, regenerates all evaluation inputs independently of the candidate, judges correctness through seven gates it evaluates itself (reference equivalence on held-out inputs, adversarial-shape equivalence, finiteness structure, metamorphic properties, determinism, output-shape sanity, and timing integrity), measures wall time in an isolated subprocess with outlier-resistant statistics and a paired signed-rank significance test that a candidate’s self-reported timing can never influence, and runs a hack detector that fires on skipped computation, degenerate output, timer tampering, and speedups below the memory-bandwidth floor. We instantiate the harness on five inference-relevant kernels and thirteen candidate patches, five honest optimizations and eight adversarial exploits mirroring each documented reward-gaming class. Crucible accepts all five honest patches, whose measured speedups over the baseline range from 31 × to 1,039 ×, and rejects all eight hacks, with zero false positives and zero false negatives, stable across independent runs; the per-gate telemetry identifies exactly which defense stops each exploit. Reaching zero false rejections required correcting two of the harness’s own gates that initially rejected honest code, which we report rather than hide. We then confront the harness with a stronger adaptive adversary that knows the gate design and computes correctly only on the fixed graded inputs while returning fast garbage when timed; this attack evades the base harness, and we close the seam with a blind timed-input check that draws unpredictable seeds at runtime and verifies correctness on the very inputs it timed, after which the adaptive attacks are rejected with no new false rejection of honest patches. Wrapped in a research-to-patch control loop, the harness admits only verified, statistically significant patches, and a scripted proposal stream mixing honest and hacky candidates drives the incumbent from 1.0 × to a verified 31.7 × while rejecting every hack at the gate. The entire evaluation ships as an executable artifact.