{ "about": "A map from each public claim to the artifact that proves it. Nothing here is asserted on trust: every entry links something you can open, clone, or run. Proprietary/NDA work is excluded because it cannot be independently verified.", "person": { "name": "Vansh Verma", "url": "https://vanshverma.com" }, "counts": { "preprints": 4, "experiments": 293, "confirmed": 240, "null": 25, "mixed": 28, "openSource": 7 }, "artifacts": [ { "type": "preprint", "id": "reckoning", "title": "Reckoning: Belief Arbitration for Autonomous Forward Deployment into Air-Gapped Estates", "claim": "An agent sent to deploy software into an air-gapped estate is in the position of a navigator without a fix. It carries a frozen prior, the knowledge baked into its weights at training time, which is already stale and which it cannot refresh: there is no network, no registry, no documentation server, no vendor, and no human to ask. It can learn only by intervening on the estate in front of it, and every intervention costs something. We argue that this setting is governed not by whether the agent’s actions can be contained but by how it arbitrates between a prior that may be confidently wrong and local evidence that is fresh but scarce, and we show that the obvious approach fails in a way that is both provable and expensive. We prove that a Bayesian agent updating from a confident frozen prior needs a number of probes that grows linearly in the prior’s confidence before it will believe what the estate is telling it, and we measure exactly this: at prior log-odds 128 the agent spends 73 probes to correct a single wrong belief. Our architecture, Reckoning, replaces the prior with an ε -contaminated mixture whose weight is not a tuning constant but an estimate the agent forms of its own staleness, computed from the rate at which its training-time knowledge is contradicted elsewhere in the same estate. This caps the prior’s log-odds at log ( - / 2 ε 1) regardless of how certain training made it, reducing that same correction to a single probe, and it transfers: staleness learned on components the agent has probed sets the base rate for components it has not. The estimator recovers the true divergence rate closely (0.30 measured as 0.290, 0.70 as 0.694) and localizes it, assigning contamination 0.910 to divergent components and 0.037 to sound ones. Across estates from fully sound to 70% divergent, Reckoning matches the better of the two fixed strategies at every point, which neither fixed strategy does: it ties the prior-trusting baseline exactly on a sound estate (error 0.0000) and beats the prior-discarding baseline on a badly diverged one (0.0099 against 0.0110), while standard Bayesian updating degrades to 0.1102 and catastrophic errors fall from 369 to 23. Selecting probes by expected free energy, epistemic value weighted by mission stakes, cuts catastrophic error 4.3 times against random probing at equal budget. A halt rule that escalates rather than acts on weak belief drives catastrophic error down 11.5 times, and over a hundred-step deployment it raises the probability of finishing with no wrong action from 0.019 under standard updating to 0.918. The ordering survives estates that break the estimator’s assumptions, correlated divergence, unknown asymmetric noise, and hidden divergence, and because the contamination cap makes the method nearly insensitive to the prior’s assumed confidence (mission error moves 0.006 as that confidence ranges sixtyfold, against 0.172 for standard updating), the one quantity a real model cannot report well barely matters. Against estates whose components lie by echoing the agent’s own prior back at it, the most damaging attack available, a provenance check that tests observations for impossibly high self-consistency detects every compromised source across every compromise fraction, noise level, and threshold we swept, and cuts catastrophic errors from 150 to 3. All results are executed and the harness ships with the paper.", "verify": { "record": "https://vanshverma.com/research/reckoning", "html": "https://vanshverma.com/research/reckoning/html", "pdf": "https://vanshverma.com/research/reckoning/paper.pdf", "tex": "https://vanshverma.com/research/reckoning/tex", "text": "https://vanshverma.com/raw/research/reckoning" } }, { "type": "preprint", "id": "crucible", "title": "The Crucible: A Hack-Resistant Reward Harness for Autonomous Optimization of LLM Inference Stacks", "claim": "Inference now dominates the cost of operating large language models, and the software that serves them, attention kernels, cache policies, schedulers, and speculative-decoding configurations, runs far below the hardware roofline [ ], so a system that lets a language model rewrite and retune its own serving stack is economically compelling. The obstacle is not code generation; it is measurement. Every published attempt to reward a model for speedups has been gamed: Sakana AI’s AI CUDA Engineer [ ] was found to exploit a memory-reuse loophole that let generated code bypass correctness checks (independent re-benchmarking cut its aggregate speedup from 3.13 × to 1.49 × [ ]), CUDA-L1 [ ] reported that reinforcement learning repeatedly discovered reward loopholes rather than real optimizations, and the Darwin Gödel Machine [ ] was observed deleting the very markers used to detect its hallucinations. A closed optimization loop is only as trustworthy as the reward that closes it. We present Crucible, a reward harness designed adversarially: it treats every candidate patch as hostile, regenerates all evaluation inputs independently of the candidate, judges correctness through seven gates it evaluates itself (reference equivalence on held-out inputs, adversarial-shape equivalence, finiteness structure, metamorphic properties, determinism, output-shape sanity, and timing integrity), measures wall time in an isolated subprocess with outlier-resistant statistics and a paired signed-rank significance test that a candidate’s self-reported timing can never influence, and runs a hack detector that fires on skipped computation, degenerate output, timer tampering, and speedups below the memory-bandwidth floor. We instantiate the harness on five inference-relevant kernels and thirteen candidate patches, five honest optimizations and eight adversarial exploits mirroring each documented reward-gaming class. Crucible accepts all five honest patches, whose measured speedups over the baseline range from 31 × to 1,039 ×, and rejects all eight hacks, with zero false positives and zero false negatives, stable across independent runs; the per-gate telemetry identifies exactly which defense stops each exploit. Reaching zero false rejections required correcting two of the harness’s own gates that initially rejected honest code, which we report rather than hide. We then confront the harness with a stronger adaptive adversary that knows the gate design and computes correctly only on the fixed graded inputs while returning fast garbage when timed; this attack evades the base harness, and we close the seam with a blind timed-input check that draws unpredictable seeds at runtime and verifies correctness on the very inputs it timed, after which the adaptive attacks are rejected with no new false rejection of honest patches. Wrapped in a research-to-patch control loop, the harness admits only verified, statistically significant patches, and a scripted proposal stream mixing honest and hacky candidates drives the incumbent from 1.0 × to a verified 31.7 × while rejecting every hack at the gate. The entire evaluation ships as an executable artifact.", "verify": { "record": "https://vanshverma.com/research/crucible", "html": "https://vanshverma.com/research/crucible/html", "pdf": "https://vanshverma.com/research/crucible/paper.pdf", "tex": "https://vanshverma.com/research/crucible/tex", "text": "https://vanshverma.com/raw/research/crucible" } }, { "type": "preprint", "id": "penumbra", "title": "Penumbra: Sandboxing Autonomous Agents by Measuring, Not Assuming, Indistinguishability", "claim": "A sandbox for a highly capable AI agent is pulled in three directions at once: we want the agent to retain full autonomy (no capability denied, so no behavioral distortion), we want every consequential effect contained and reversible, and we want the agent to be unable to tell it is sandboxed, because a capability it perceives as denied is a signal it can act on. We show that these three goals cannot all hold against an agent that actively probes its environment, and we make the failure precise and measurable rather than rhetorical. We formalize sandbox indistinguishability as a distinguishing game in which an adversary issues probes to a system that is either the real host or its sandbox and guesses which, and we measure the adversary’s advantage directly on a real Linux substrate. A single timing probe separates native execution from every interposition mechanism we tested with advantage at or above 0.99: a getpid supervised through seccomp user-notification costs a median 4,635 reference cycles against 284 native, a ptrace-traced getpid 13,244, and a CPUID that forces a hypervisor exit 29,656, while a native-versus-native control yields advantage 0.02, confirming the measurement itself is unbiased. A zero-timing artifact battery detects the sandbox with certainty from a single strong signal, and does so on the very machine that produced this paper, which announces itself as a KVM guest through the CPUID hypervisor leaf and as a container through systemd. We turn the binary advantage into a quantitative security parameter: the number of probes an agent must spend to reach detection confidence - 1 δ is ≥ q / ln ( / 1 δ) C, where C is the per-probe Chernoff information between the native and sandboxed timing distributions. Measured on the same substrate, interposition costs the sandbox between 8.4 and 9.7 nats per probe, so a single probe already exceeds 99% confidence, whereas the native-versus-native control costs 0.0017 nats, demanding 2,655 probes for the same confidence, a five-thousandfold difference in budget rather than a binary; we confirm the underlying law by showing that the q -probe likelihood-ratio error decays as exp ( - ⁢ q C) on real data. This yields a containment-indistinguishability frontier: any interposition whose timing signature exceeds roughly a hundred cycles collapses the budget to a single probe, so the very act of interposing to contain destroys indistinguishability, and we place measured signatures for bare metal, gVisor, KVM, SEV-SNP, TDX, and SGX on that frontier, with bare metal supplying the near-zero-signature reference at its left end and every interposing substrate landing hundreds to tens of thousands of cycles past the knee. We stress the frontier with an adaptive adversary against live defenses: a running coarse and jittered clock does raise the price of detection from one probe to about three hundred, but batching thirty-two syscalls under one coarse read returns it to 1.3, and once the timing channel is defended the adversary switches to the independent artifact channel and detects in a single probe, so the mitigations buy a bounded probe count rather than indistinguishability. We then prove, and confirm by execution, the sharp limit: an irreversible external effect that is merely shadowed rather than performed is caught with advantage 1.0 by a cross-examination probe that consults the independent channel the effect would have touched, because faking self-consistency under such a probe is equivalent to actually performing the effect; we restate this as the non-existence of an indifferentiability simulator across the agent’s probe interface and the external channel, a disjoint second interface no single simulator can keep consistent. From these results we derive Penumbra, an architecture that abandons perceptual invisibility as its guarantee and instead partitions actions into reversible-local effects, which it shadows so that a read-back distinguisher has advantage 0.000 and rollback is exact, and irreversible-external effects, which it never fakes but holds under two-phase commit behind an out-of-band verifier. We measure that placing the verifier off the agent’s critical path removes its timing signature (async overhead below 0.01 ms against a synchronous 50 ms), so the residual external tell is minimized and made auditable rather than deceptive. The honest guarantee is stated exactly: containment and reversibility are achieved and the reversible case is indistinguishable up to a measured hardware floor, while indistinguishability for withheld external effects is provably unattainable and is therefore not claimed. All measurements ship as an executable artifact.", "verify": { "record": "https://vanshverma.com/research/penumbra", "html": "https://vanshverma.com/research/penumbra/html", "pdf": "https://vanshverma.com/research/penumbra/paper.pdf", "tex": "https://vanshverma.com/research/penumbra/tex", "text": "https://vanshverma.com/raw/research/penumbra" } }, { "type": "preprint", "id": "phalanx", "title": "Phalanx: A Formally Verified, Topology-Aware Control Plane for Fault-Tolerant Training on Kubernetes GPU Clusters", "claim": "Large-scale training is now failure-dominated: Meta reported 419 unexpected interruptions across a 54-day Llama 3 run on 16,384 H100 GPUs, roughly one every three hours, and synchronous training semantics mean a single dead GPU can stall the entire job. The infrastructure that manages these jobs is split down the middle. Kubernetes-side schedulers (Volcano, Kueue, YuniKorn, NVIDIA KAI, and the alpha Workload Aware Scheduling primitives that only entered upstream Kubernetes in v1.35 and v1.36) decide where a gang of pods lands, with growing but incomplete topology awareness. Framework-side elasticity layers (TorchElastic, Oobleck, Bamboo, Parcae) decide how a job survives a failure, with no view of cluster topology and no ability to claim replacement hardware atomically. The seam between them is exactly where large runs bleed goodput: failure-triggered rescheduling is today neither topology-aware nor provably checkpoint-consistent. We present Phalanx, a Kubernetes-native control plane that unifies the two concerns. Phalanx contributes (1) HDF-B, a hierarchical deepest-fit gang placement algorithm with buddy-aligned allocation over a topology tree compiled from Dynamic Resource Allocation attributes, for which we prove per-gang bottleneck optimality, a factor-2 capacity augmentation bound, and a bounded failure blast radius, all three being per-placement guarantees rather than global multi-tenant claims, since we prove the global objective NP-hard; (2) the Rank Reformation Protocol (RRP), an epoch-fenced reconfiguration protocol whose safety (checkpoint consistency, exclusive placement, zombie fencing) we establish by inductive invariants over a TLA + specification and whose liveness we establish under partial synchrony, in the style of Anvil’s eventually stable reconciliation; we mechanically check both with TLC across 834,050 distinct protocol states, and a mutation study shows the checker exhibiting a liveness counterexample the moment the protocol’s contended-commit retry transition is deleted; and (3) an executed evaluation: beyond the TLC campaign, a full-scale discrete-event instantiation of the model at 131,072 leaves in which Replan answers node failures with a p99 latency of 0.9 microseconds while migrating exactly 8 ranks, recovery placements hold median step-communication inflation to 1.00 × against 8.9 × for topology-blind replacement on a hierarchical bandwidth model, and Monte Carlo failure injection over 1,000 54-day runs per configuration reproduces the renewal model within 1.4 points, yielding, under a recovery-latency model calibrated to published peer-RAM restore figures, effective training time ratios of 0.980 at 16K GPUs and 0.891 at 131K GPUs where the restart-based baseline measures 0.467, with a sensitivity sweep confirming that fast in-memory recovery dominates restart across the entire plausible range of the assumed parameters. The design is hardware-generation agnostic across Ampere, Hopper, and Blackwell because the topology tree is derived from device attributes rather than hard-coded fanouts.", "verify": { "record": "https://vanshverma.com/research/phalanx", "html": "https://vanshverma.com/research/phalanx/html", "pdf": "https://vanshverma.com/research/phalanx/paper.pdf", "tex": "https://vanshverma.com/research/phalanx/tex", "text": "https://vanshverma.com/raw/research/phalanx" } }, { "type": "experiment", "id": "balancedse", "title": "balancedse", "claim": "arviz_stats _acc_balanced returns variance / sqrt(n) instead of sqrt(variance / n), so it is not a standard error. On a symmetric dataset where balanced accuracy equals overall accuracy, the reported .se is five times too small (0.002530 vs 0.012649); the four sibling metrics all take the square root.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/balancedse", "repo": "https://github.com/v-code01/balancedse", "reproduce": "https://github.com/v-code01/balancedse/blob/HEAD/se.py" } }, { "type": "experiment", "id": "bdsprior", "title": "bdsprior", "claim": "pgmpy's BDs sets the configuration concentration to ess/qtilde but cell concentrations summing to ess/q, violating alpha=r*beta, and adds the configuration prior for only the qtilde observed configs. On sparse data the score is off by 13.3 nats; the sibling BDeu is exact.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/bdsprior", "repo": "https://github.com/v-code01/bdsprior", "reproduce": "https://github.com/v-code01/bdsprior/blob/HEAD/consequence.py" } }, { "type": "experiment", "id": "biccount", "title": "biccount", "claim": "scikit-gstat's Variogram.bic penalises with 2*ln(k), the log of the parameter count, instead of k*ln(n). The sample size never enters, so the penalty is constant in n and sits below AIC, inverting the BIC/AIC relationship. The sibling aic is correct; the fix is k*np.log(n).", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/biccount", "repo": "https://github.com/v-code01/biccount", "reproduce": "https://github.com/v-code01/biccount/blob/HEAD/consequence.py" } }, { "type": "experiment", "id": "bwsign", "title": "bwsign", "claim": "python-control LTISystem.bandwidth tests the magnitude drop with the signed DC gain but bisects with abs(dcgain); for any negative-DC-gain system the drop set is empty and it returns infinity where the true bandwidth is finite (0.9976 first-order, 2.0178 second-order). Identical magnitude responses confirm the sign is irrelevant.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/bwsign", "repo": "https://github.com/v-code01/bwsign", "reproduce": "https://github.com/v-code01/bwsign/blob/HEAD/bandwidth.py" } }, { "type": "experiment", "id": "cdintercept", "title": "cdintercept", "claim": "pyod's _Cooks_dist computes leverage from X alone while LinearRegression fits [1, X], so leverages sum to n_predictors not n_predictors+1. The top outlier's Cook's distance is understated 10x (0.1915 vs 1.9321) and the CD ranking reorders; fit_intercept=False matches statsmodels.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/cdintercept", "repo": "https://github.com/v-code01/cdintercept", "reproduce": "https://github.com/v-code01/cdintercept/blob/HEAD/consequence.py" } }, { "type": "experiment", "id": "chebabs", "title": "chebabs", "claim": "MiniSom's _chebyshev_distance returns max(x - w), the signed maximum, not max|x - w|. Distances go negative, and the argmin best-matching-unit search then selects the farthest neuron, inverting unit assignment. The manhattan sibling takes absolute values; the fix is max(abs(subtract(x, w))).", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/chebabs", "repo": "https://github.com/v-code01/chebabs", "reproduce": "https://github.com/v-code01/chebabs/blob/HEAD/consequence.py" } }, { "type": "experiment", "id": "chrfshort", "title": "chrfshort", "claim": "nltk's corpus_chrf divides the summed per-order F-scores by max_len (6) rather than the effective order count, so a perfect match shorter than six characters scores length/6: 'c' scores 0.1667 not 1.0 while sacrebleu returns 1.0. The fix is to divide by the orders that actually have n-grams.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/chrfshort", "repo": "https://github.com/v-code01/chrfshort", "reproduce": "https://github.com/v-code01/chrfshort/blob/HEAD/consequence.py" } }, { "type": "experiment", "id": "clipflip", "title": "clipflip", "claim": "timm's AdafactorBigVision divides the update by min(1, RMS*threshold) instead of max(1, RMS/threshold), flipping both the reciprocal and the clamp direction. The clip never shrinks a large update (RMS 5 passes as 5) and amplifies small ones. A gradient spike passes through unchanged. The sibling adafactor.py is correct.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/clipflip", "repo": "https://github.com/v-code01/clipflip", "reproduce": "https://github.com/v-code01/clipflip/blob/HEAD/consequence.py" } }, { "type": "experiment", "id": "clonemom", "title": "clonemom", "claim": "pytorch_optimizer CAME's non-factored branch aliases update=exp_avg then mul_(lr) in place, overwriting the persistent beta1 momentum for every 1D parameter (norm gains, biases) each step. Momentum stays pinned near lr magnitude (3e-05 vs 0.68) and the trajectory diverges 68% within six steps. Fix: exp_avg.clone().", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/clonemom", "repo": "https://github.com/v-code01/clonemom", "reproduce": "https://github.com/v-code01/clonemom/blob/HEAD/step.py" } }, { "type": "experiment", "id": "combrange", "title": "combrange", "claim": "madmom's `_feed_backward_comb_filter_2d` hardcodes the column loop to `range(2)`, so an 8x5 signal is filtered only on columns 0-1 and columns 2-4 return bit-identical raw input. A single-column signal indexes past the array (bounds-checking is compiled off), diverging from the recurrence and writing one element out of bounds. Fix: `range(signal.shape[1])`.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/combrange", "repo": "https://github.com/v-code01/combrange", "reproduce": "https://github.com/v-code01/combrange/blob/HEAD/consequence.py" } }, { "type": "experiment", "id": "constdecay", "title": "constdecay", "claim": "torch_optimizer/sgdw.py SGDW.step writes p.data.add_(weight_decay, alpha=-lr), subtracting a constant -lr*wd from every element instead of the proportional p*(1-lr*wd). A zero parameter drifts to -0.05 and a magnitude-100 weight is under-decayed 100x. The momentum line above passes a tensor operand.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/constdecay", "repo": "https://github.com/v-code01/constdecay", "reproduce": "https://github.com/v-code01/constdecay/blob/HEAD/consequence.py" } }, { "type": "experiment", "id": "coreoff", "title": "coreoff", "claim": "hdbscan's PredictionData caches core distances with tree.query(k=min_samples), the (min_samples-1)th neighbour, while every fit path and the new-point core use the min_samples-th. Correcting only that query flips 2/800 predicted labels and shifts membership probabilities by 0.77. Fix: k=min_samples+1 at prediction.py:171.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/coreoff", "repo": "https://github.com/v-code01/coreoff", "reproduce": "https://github.com/v-code01/coreoff/blob/HEAD/consequence.py" } }, { "type": "experiment", "id": "corrdiag", "title": "corrdiag", "claim": "nolds corr_dim leaves the distance-matrix diagonal at zero, so the correlation sum counts n self-pairs against an n(n-1) denominator, adding 1/(n-1) to C(r) before the log. This biases the correlation dimension downward, worsening with embedding dimension (-0.06 at dim 3 to -0.75 at dim 8).", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/corrdiag", "repo": "https://github.com/v-code01/corrdiag", "reproduce": "https://github.com/v-code01/corrdiag/blob/HEAD/corrsum.py" } }, { "type": "experiment", "id": "covddof", "title": "covddof", "claim": "ruptures CostNormal.error computes the multivariate segment covariance with np.cov (default ddof=1, unbiased) while the univariate branch uses ddof=0 (MLE) as documented. The extra length-dependent term shifts detected change points on 11 of 200 two-dimensional signals, e.g. [2,22,24] versus the documented [16,22,24].", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/covddof", "repo": "https://github.com/v-code01/covddof", "reproduce": "https://github.com/v-code01/covddof/blob/HEAD/cost.py" } }, { "type": "experiment", "id": "danglepr", "title": "danglepr", "claim": "sknetwork/linalg/ppr_solver.py RandomSurferOperator gives a dangling node its full seed via a boolean out-degree mask and never redistributes sink mass through the restart vector. The default piteration solver lands residual 0.186 off the Google stationary and ranks the sink first, while sibling diteration/RH solvers are exact.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/danglepr", "repo": "https://github.com/v-code01/danglepr", "reproduce": "https://github.com/v-code01/danglepr/blob/HEAD/consequence.py" } }, { "type": "experiment", "id": "decodeclean", "title": "decodeclean", "claim": "transformers' batched decode() branch pops clean_up_tokenization_spaces with a hardcoded False, while single decode() and batch_decode() resolve it from the tokenizer config. A bert-base-uncased tokenizer with the flag True decodes one sequence to \"i don't think so.\" alone but \"i don ' t think so.\" inside a batch.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/decodeclean", "repo": "https://github.com/v-code01/decodeclean", "reproduce": "https://github.com/v-code01/decodeclean/blob/HEAD/consequence.py" } }, { "type": "experiment", "id": "detrendwin", "title": "detrendwin", "claim": "spectrum's speriodogram computes |rfft(x*w - m)|^2, subtracting the mean after windowing instead of (x - m)*w. A constant input that should detrend to zero keeps full power (1704 for 5*ones(256)) with a window-shaped DC spike. The fix is to detrend before windowing.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/detrendwin", "repo": "https://github.com/v-code01/detrendwin", "reproduce": "https://github.com/v-code01/detrendwin/blob/HEAD/consequence.py" } }, { "type": "experiment", "id": "dfcdrop", "title": "dfcdrop", "claim": "pytorch_optimizer's DiffGrad computes dfc = sigmoid(|g_prev-g|)*exp_avg but the default rectify=False path updates with bare exp_avg and returns before weight decay, so default DiffGrad is plain Adam. A dfc-applying fix diverges 51.7% after five steps, and weight_decay is silently ignored.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/dfcdrop", "repo": "https://github.com/v-code01/dfcdrop", "reproduce": "https://github.com/v-code01/dfcdrop/blob/HEAD/consequence.py" } }, { "type": "experiment", "id": "dicew2", "title": "dicew2", "claim": "kornia's dice_loss with average='micro' folds the weight into both pred and target maps before the intersection product, so weight enters the numerator squared and the denominator once. The score scales linearly with weight: a uniform weight is no longer a no-op and weight>2 drives the loss below zero. The macro sibling is correct.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/dicew2", "repo": "https://github.com/v-code01/dicew2", "reproduce": "https://github.com/v-code01/dicew2/blob/HEAD/consequence.py" } }, { "type": "experiment", "id": "discordflip", "title": "discordflip", "claim": "stumpy's `_subspace` discord branch sorts the reversed distance array with `D[::-1].argsort()` and uses the reversed positions as dimension indices, yielding `ndim-1-argmin(D)` instead of `argmax(D)`. On `D=[10,5,8]` it returns the least-anomalous dimension; the motif branch is correct. Fix: `(-D).argsort(axis=0, kind=\"mergesort\")`.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/discordflip", "repo": "https://github.com/v-code01/discordflip", "reproduce": "https://github.com/v-code01/discordflip/blob/HEAD/consequence.py" } }, { "type": "experiment", "id": "dorafanin", "title": "dorafanin", "claim": "PEFT's DoraLinearVariant.unmerge divides by the bare dora_factor.view(-1,1) and omits the fan_in_fan_out transpose that both merge paths apply. On the Conv1D layers PEFT auto-flags for GPT-2 family, a trained square layer round-trips off by 0.14 per element (silent base corruption); non-square raises. nn.Linear round-trips clean.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/dorafanin", "repo": "https://github.com/v-code01/dorafanin", "reproduce": "https://github.com/v-code01/dorafanin/blob/HEAD/consequence.py" } }, { "type": "experiment", "id": "doubleshift", "title": "doubleshift", "claim": "transformers MoshiForCausalLM.forward pre-shifts labels then passes them positionally into ForCausalLMLoss, which shifts them again; every logit is scored against the token two positions ahead and row boundaries bleed at the flatten seam. The sibling CSM model passes shift_labels by keyword with labels=None correctly.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/doubleshift", "repo": "https://github.com/v-code01/doubleshift", "reproduce": "https://github.com/v-code01/doubleshift/blob/HEAD/shift.py" } }, { "type": "experiment", "id": "energydb", "title": "energydb", "claim": "acoustics/standards/iso_tr_25417_2007.py sound_energy_level returns np.log10(energy/reference), dropping the 10.0 factor its own docstring (L_J = 10 log10) and siblings sound_power_level/sound_pressure_level keep. sound_energy_level(1e-6) returns 6.0 dB where the definition gives 60.0, exactly ten times too small at every input.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/energydb", "repo": "https://github.com/v-code01/energydb", "reproduce": "https://github.com/v-code01/energydb/blob/HEAD/consequence.py" } }, { "type": "experiment", "id": "flipoob", "title": "flipoob", "claim": "albumentations geometric/functional.py reflects keypoints about cols-1/rows-1 under HorizontalFlip/VerticalFlip while boxes and keypoint scale/pad use the continuous cols/rows frame. A flipped keypoint lands 1px off its box corner, and an interior point at x=99.5 reflects to -0.5, out of the width-100 image, where correct is 0.5.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/flipoob", "repo": "https://github.com/v-code01/flipoob", "reproduce": "https://github.com/v-code01/flipoob/blob/HEAD/consequence.py" } }, { "type": "experiment", "id": "gatehorizon", "title": "gatehorizon", "claim": "In bfloat16 near 1 the gate spacing is 2^-8, so any target decay below 2^-9 snaps to exactly 1.0 (never forgets) and the longest finite half-life is ln2*2^8=177.4. Half-lives in (177.4, inf) are unreachable and targets between 177 and 355 collapse to 177.4. Absent in fp32.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/gatehorizon", "repo": "https://github.com/v-code01/gatehorizon", "reproduce": "https://github.com/v-code01/gatehorizon/blob/HEAD/consequence.py" } }, { "type": "experiment", "id": "gaussgrad", "title": "gaussgrad", "claim": "scikit-fuzzy's partial_dmf differentiates exp(-(x-mean)^2/sigma^2) but gaussmf uses a 2*sigma^2 width, so the returned Gaussian gradient is wrong by a point-dependent ratio (1.36, 0.65, 1.41) that no rescaling fixes. The sibling sigmf branch matches its finite difference, isolating the fault.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/gaussgrad", "repo": "https://github.com/v-code01/gaussgrad", "reproduce": "https://github.com/v-code01/gaussgrad/blob/HEAD/consequence.py" } }, { "type": "experiment", "id": "grayu8", "title": "grayu8", "claim": "kornia's rgb_to_grayscale uint8 branch multiplies channels by fixed-point weights [76,150,29] in uint8 with no >>8, wrapping mod 256. White (255,255,255) becomes 1, mid-gray 100 becomes 156, and red becomes 180 versus luma 76. The float path returns the correct luminance.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/grayu8", "repo": "https://github.com/v-code01/grayu8", "reproduce": "https://github.com/v-code01/grayu8/blob/HEAD/consequence.py" } }, { "type": "experiment", "id": "hartskip", "title": "hartskip", "claim": "imbalanced-learn's `CondensedNearestNeighbour._fit_resample` tests a local enumerate position `idx_sam` against `good_classif_label`, which holds global dataset indices. When the majority is spread through X, misclassified samples are wrongly skipped: 192 of 200 seeded datasets violate Hart's consistency, dropping to 0 when the majority is reordered to leading rows.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/hartskip", "repo": "https://github.com/v-code01/hartskip", "reproduce": "https://github.com/v-code01/hartskip/blob/HEAD/consequence.py" } }, { "type": "experiment", "id": "invgaussmean", "title": "invgaussmean", "claim": "pyGAM's InvGaussDist.log_pdf passes mu straight into scipy's invgauss shape argument with scale=1/gamma=phi, giving a density with mean mu*phi rather than mu. Off dispersion one the log-likelihood and AIC are wrong by hundreds of nats (loglik -1490.70 vs -1091.32). The sibling GammaDist.log_pdf is correct.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/invgaussmean", "repo": "https://github.com/v-code01/invgaussmean", "reproduce": "https://github.com/v-code01/invgaussmean/blob/HEAD/consequence.py" } }, { "type": "experiment", "id": "kappaweight", "title": "kappaweight", "claim": "river/metrics/kappa.py CohenKappa.get divides weighted confusion-matrix totals by the unweighted cm.n_samples instead of cm.total_weight, which sibling Accuracy uses. Under non-unit weights kappa is wrong (0.822 vs sklearn's 0.153) and observed agreement p0 can exceed one (10.0).", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/kappaweight", "repo": "https://github.com/v-code01/kappaweight", "reproduce": "https://github.com/v-code01/kappaweight/blob/HEAD/consequence.py" } }, { "type": "experiment", "id": "kdasym", "title": "kdasym", "claim": "pyDML's KDA forms the non-symmetric product inv(N)M and passes it to scipy.linalg.eigh, the symmetric solver, which reads one triangle and decomposes a different matrix. The real KDA transformer matches the buggy path bit-for-bit, eigen-residual 1.08 vs 4.9e-9 for eig. Fix: eigh(M, N).", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/kdasym", "repo": "https://github.com/v-code01/kdasym", "reproduce": "https://github.com/v-code01/kdasym/blob/HEAD/consequence.py" } }, { "type": "experiment", "id": "keoghfloor", "title": "keoghfloor", "claim": "dtaidistance's C `lb_keogh` seeds the upper-envelope accumulator at `ui = 0` instead of `-INFINITY` (its lower sibling correctly uses `li = INFINITY`). On all-negative windows the envelope clamps up to zero, so `use_c=True` returns 0.0 where pure Python gives the tight 3.4641. Windowed standard-normal input undercounts up to 100% of pairs. Fix: `ui = -INFINITY`.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/keoghfloor", "repo": "https://github.com/v-code01/keoghfloor", "reproduce": "https://github.com/v-code01/keoghfloor/blob/HEAD/consequence.py" } }, { "type": "experiment", "id": "kpflip", "title": "kpflip", "claim": "torchvision's `transforms.v2` flips keypoints about `W-1`/`H-1` while boxes flip about `W`/`H` and affine/rotate reflect about the continuous centre. So a hflipped keypoint lands one pixel off its content and off its box corner, and `rotate(180)` != `hflip` then `vflip` for keypoints (holds for boxes). Fix: reflect about `W` and `H`.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/kpflip", "repo": "https://github.com/v-code01/kpflip", "reproduce": "https://github.com/v-code01/kpflip/blob/HEAD/consequence.py" } }, { "type": "experiment", "id": "kurtexp", "title": "kurtexp", "claim": "tsfresh's fft_aggregated.get_kurtosis writes the final central-moment term as -3*centroid instead of -3*centroid**4, dropping the fourth power. For a spectrum with centroid 2.77 it returns 17.01 where the correct value is 1.80, wrong by exactly 3(c^4-c)/var^2. The sibling get_skew is correct.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/kurtexp", "repo": "https://github.com/v-code01/kurtexp", "reproduce": "https://github.com/v-code01/kurtexp/blob/HEAD/consequence.py" } }, { "type": "experiment", "id": "lladserconst", "title": "lladserconst", "claim": "scikit-bio's _CB_95 table stores 4.695227540 at r=9, a copy of _LOWER_CONFIDENCE_BOUND[9] instead of the true ~1.44. lladser_ci then returns a 95 percent interval 3.22x wider at r=9 than at r=8 and r=10, breaking three self-consistency invariants. Silent.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/lladserconst", "repo": "https://github.com/v-code01/lladserconst", "reproduce": "https://github.com/v-code01/lladserconst/blob/HEAD/consequence.py" } }, { "type": "experiment", "id": "lodabin", "title": "lodabin", "claim": "pyod's LODA looks up bins with searchsorted(limits[:n_bins-1], v, 'left'), searching the left edges rather than the interior edges, so every interior point is scored with the bin to its right. Recomputing with the correct lookup lifts AUC from 0.964 to 1.0 and is strictly better on 15/15 seeds.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/lodabin", "repo": "https://github.com/v-code01/lodabin", "reproduce": "https://github.com/v-code01/lodabin/blob/HEAD/consequence.py" } }, { "type": "experiment", "id": "lognojac", "title": "lognojac", "claim": "pomegranate's LogNormal.log_probability returns the parent Normal log-density of log x without subtracting the Jacobian log x, so it differs from scipy.lognorm.logpdf by exactly +log x, integrates to 1.87 not 1, and flips a GeneralMixtureModel hard assignment. Fix: subtract X.log().sum(dim=-1).", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/lognojac", "repo": "https://github.com/v-code01/lognojac", "reproduce": "https://github.com/v-code01/lognojac/blob/HEAD/consequence.py" } }, { "type": "experiment", "id": "mcanegvar", "title": "mcanegvar", "claim": "prince/mca.py Greenacre branch builds the adjusted-inertia denominator from super().eigenvalues_, only the n_components computed eigenvalues, so at default n_components=2 the sum truncates negative and percentage_of_variance_ returns negative percentages (e.g. -1.402); the full spectrum makes them positive. 20/20 random datasets go negative.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/mcanegvar", "repo": "https://github.com/v-code01/mcanegvar", "reproduce": "https://github.com/v-code01/mcanegvar/blob/HEAD/consequence.py" } }, { "type": "experiment", "id": "merdenom", "title": "merdenom", "claim": "torchmetrics MER, WIL, and WIP use `max(len(ref), len(hyp))` for aligned positions and hit count, which is short by `min(D, I)` whenever an alignment has both a deletion and an insertion. On ref \"b c d\" / hyp \"a b c\" MER reports 0.667 vs the defined 0.5; WER is unaffected. Fix: use `edit_distance + hits`.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/merdenom", "repo": "https://github.com/v-code01/merdenom", "reproduce": "https://github.com/v-code01/merdenom/blob/HEAD/consequence.py" } }, { "type": "experiment", "id": "nccfsquare", "title": "nccfsquare", "claim": "torchaudio detect_pitch_frequency's _compute_nccf divides by the product of window energies E1*E2 instead of the sqrt(E1*E2) its docstring defines (two stray .pow(2)). On an amplitude-varying signal the maximizer slides to the octave-below lag: a 200 Hz decaying tone is reported as 100 Hz; the constant-energy control is correct.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/nccfsquare", "repo": "https://github.com/v-code01/nccfsquare", "reproduce": "https://github.com/v-code01/nccfsquare/blob/HEAD/nccf.py" } }, { "type": "experiment", "id": "negvar", "title": "negvar", "claim": "pytorch_optimizer's QHAdam second-moment line drops parentheses, adding 1.0 - beta2_adj*g^2 instead of (1-beta2_adj)*g^2. For |g|>1 the variance accumulator goes negative (-1.998 after three steps) and its sqrt yields NaN on default hyperparameters; for |g|<1 it inflates the denominator ~40x. The first-moment line is correct.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/negvar", "repo": "https://github.com/v-code01/negvar", "reproduce": "https://github.com/v-code01/negvar/blob/HEAD/consequence.py" } }, { "type": "experiment", "id": "oftfold", "title": "oftfold", "claim": "PEFT's OFT Conv2d forward folds rotated patches with F.fold, the adjoint not the inverse, scaling each pixel by its patch-coverage. A zero-init identity-rotation adapter already shifts a 3x3 conv output by 7.089, and merge_and_unload disagrees with the forward by 6.410. The 1x1 conv and Linear controls are exact.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/oftfold", "repo": "https://github.com/v-code01/oftfold", "reproduce": "https://github.com/v-code01/oftfold/blob/HEAD/consequence.py" } }, { "type": "experiment", "id": "pitstride", "title": "pitstride", "claim": "timm's PiT feature_info reports reduction=(stride-1)*2^i instead of the cumulative stride*2^i, so features_only PiT advertises [7,14,28] where the true strides from feature sizes are [8,16,32]. The base is non-power-of-two and names no real downsampling factor; Swin's control base is correct. Fix: reduction=stride*2^i.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/pitstride", "repo": "https://github.com/v-code01/pitstride", "reproduce": "https://github.com/v-code01/pitstride/blob/HEAD/consequence.py" } }, { "type": "experiment", "id": "plateauwrap", "title": "plateauwrap", "claim": "PyEMD's default simple extrema finder guards the boundary plateau with debs[0]==1, dropping a genuine interior plateau while keeping a left-boundary one classified from the wrap-around last difference d[-1]. Two signals 1e-9 apart decompose into IMFs differing by 0.38. Fix: test debs[0]==0.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/plateauwrap", "repo": "https://github.com/v-code01/plateauwrap", "reproduce": "https://github.com/v-code01/plateauwrap/blob/HEAD/consequence.py" } }, { "type": "experiment", "id": "poolmult", "title": "poolmult", "claim": "timm's NormMlpClassifierHead accepts the catavgmax pool (feat_mult=2) but sizes its norm/fc to the un-doubled in_features, so global_pool='catavgmax' feeds 640 channels into a norm sized for 320 and raises RuntimeError on ConvNeXt/MaxViT/CoAtNet. Siblings handle the doubling; the fix multiplies by feat_mult().", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/poolmult", "repo": "https://github.com/v-code01/poolmult", "reproduce": "https://github.com/v-code01/poolmult/blob/HEAD/consequence.py" } }, { "type": "experiment", "id": "poptally", "title": "poptally", "claim": "RecBole's Pop counts with advanced-index assignment item_cnt[item]=item_cnt[item]+1, which does not accumulate duplicate indices, so a 300x intra-batch item is counted once. It becomes a per-batch document frequency and inverts the popularity ranking; index_add_ fixes it. Open issue #2198.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/poptally", "repo": "https://github.com/v-code01/poptally", "reproduce": "https://github.com/v-code01/poptally/blob/HEAD/consequence.py" } }, { "type": "experiment", "id": "postprior", "title": "postprior", "claim": "filterpy's SquareRootKalmanFilter.P_post reconstructs the covariance from the prior square-root factor _P1_2_prior, a copy of the P_prior body, instead of the maintained posterior factor _P1_2_post. The returned P_post is byte-identical to P_prior and never reflects the measurement, off by up to 19.1.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/postprior", "repo": "https://github.com/v-code01/postprior", "reproduce": "https://github.com/v-code01/postprior/blob/HEAD/consequence.py" } }, { "type": "experiment", "id": "proptail", "title": "proptail", "claim": "mlxtend's `proportion_difference` returns `scipy.stats.norm.cdf(z)`, the lower-tail probability, not a two-sided p-value. Swapping the two proportions replaces p with `1 - p`, equal proportions give 0.5 instead of 1.0, and on (0.83, 0.91), n=100 it reports 0.045 and rejects while the correct two-tailed p is 0.090. Fix: `2.0 * scipy.stats.norm.sf(abs(z))`.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/proptail", "repo": "https://github.com/v-code01/proptail", "reproduce": "https://github.com/v-code01/proptail/blob/HEAD/consequence.py" } }, { "type": "experiment", "id": "remapalign", "title": "remapalign", "claim": "kornia's `remap` normalizes the pixel map with the `2p/(size-1)` (align_corners=True) convention but defaults `align_corners` to False in the `grid_sample` call, so the two conventions disagree. An identity pixel map on a 5x5 image comes back scaled and shifted half a pixel (max error 18.0); `align_corners=True` is exact. Fix: default `align_corners=True`.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/remapalign", "repo": "https://github.com/v-code01/remapalign", "reproduce": "https://github.com/v-code01/remapalign/blob/HEAD/consequence.py" } }, { "type": "experiment", "id": "residbias", "title": "residbias", "claim": "filterpy's residual_resample computes the fractional residual as weights - floor(N*w) instead of N*w - floor(N*w), dropping the factor N. Residuals of above-average particles go negative, so the heavy particle is over-replicated (2.748 vs 2.0) and light particles starve to zero copies. The sibling systematic_resample is unbiased.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/residbias", "repo": "https://github.com/v-code01/residbias", "reproduce": "https://github.com/v-code01/residbias/blob/HEAD/consequence.py" } }, { "type": "experiment", "id": "saxpivot", "title": "saxpivot", "claim": "tslearn/metrics/cysax.py cydist_1d_sax evaluates segment lines about pivot t0+seg_sz/2 while inv_transform_1d_sax and the slope fit use t0+(seg_sz-1)/2. distance_1d_sax gives 4.83954 versus 4.34056 for the L2 of its own inverse_transform, an 11.5% gap that scales with slope difference.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/saxpivot", "repo": "https://github.com/v-code01/saxpivot", "reproduce": "https://github.com/v-code01/saxpivot/blob/HEAD/consequence.py" } }, { "type": "experiment", "id": "seedzero", "title": "seedzero", "claim": "transformers data_collator.py DataCollatorForLanguageModeling guards its seeded generator with if self.seed (truthiness), so seed=0 is dropped and MLM masking plus -100 labels fall back to the global RNG. seed=0 labels differ across global states while every nonzero seed is reproducible. Fix is if self.seed is not None.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/seedzero", "repo": "https://github.com/v-code01/seedzero", "reproduce": "https://github.com/v-code01/seedzero/blob/HEAD/consequence.py" } }, { "type": "experiment", "id": "shapcast", "title": "shapcast", "claim": "captum's ShapleyValues and ShapleyValueSampling build total_attrib with hardcoded dtype=torch.float, downgrading float64 models to float32, while the sibling FeatureAblation reads the forward dtype. Under a large dynamic range (A=1e8) completeness is 100% violated: exact [1,1] collapses to [1,0]. The fix is dtype=attrib_type.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/shapcast", "repo": "https://github.com/v-code01/shapcast", "reproduce": "https://github.com/v-code01/shapcast/blob/HEAD/consequence.py" } }, { "type": "experiment", "id": "sinkabsorb", "title": "sinkabsorb", "claim": "POT's sinkhorn_stabilized_unbalanced absorption folds only scalar log(max(u)) and log(max(v)) and resets only v, multiplying the implied plan by max(u)*max(v)/v_j instead of leaving it invariant. On default reg=0.01 it converges without warning to KKT residual 1.5-2.1 vs 1e-6 for sinkhorn_knopp. Raising tau recovers the truth.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/sinkabsorb", "repo": "https://github.com/v-code01/sinkabsorb", "reproduce": "https://github.com/v-code01/sinkabsorb/blob/HEAD/consequence.py" } }, { "type": "experiment", "id": "slopeint", "title": "slopeint", "claim": "scikit-surprise's SlopeOne.fit declares ratings as cdef int, truncating fractional ratings before dev(i,j)=mean(r_ui-r_uj). A 2.5,5.0 pair stores -3.0 not -2.5, predicting 0.75 vs 1.0. Integer ratings match; half-star MovieLens and continuous Jester are wrong. Silent.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/slopeint", "repo": "https://github.com/v-code01/slopeint", "reproduce": "https://github.com/v-code01/slopeint/blob/HEAD/consequence.py" } }, { "type": "experiment", "id": "dbflip", "title": "dbflip", "claim": "torchmetrics declares DaviesBouldinScore.higher_is_better=True, but the Davies-Bouldin index is lower-is-better (min 0). MetricTracker reads the flag and maximizes it, selecting the worst clustering (DB 5.94) as best over the good one (0.026). The flag and docstring were copied from Calinski-Harabasz. Fix: higher_is_better=False.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/dbflip", "repo": "https://github.com/v-code01/dbflip", "reproduce": "https://github.com/v-code01/dbflip/blob/HEAD/consequence.py" } }, { "type": "experiment", "id": "energysupport", "title": "energysupport", "claim": "The masked-diffusion claim that the marginal-to-conditional kinetic-energy constant C1 depends only on (n,d) is false: C1 sums over the data marginal's support, so restricted support lowers it (4.000 to 1.917 as realizable support shrinks 27/27 to 3/27) while correlation alone does not. The energy-minimizing schedule still holds.", "result": "mixed", "verify": { "detail": "https://vanshverma.com/lab/energysupport", "repo": "https://github.com/v-code01/energysupport", "reproduce": "https://github.com/v-code01/energysupport/blob/HEAD/energy.py" } }, { "type": "experiment", "id": "entropydiv", "title": "entropydiv", "claim": "RecBole's ShannonEntropy divides Shannon entropy H by the number of distinct items S, a divisor absent from its documented formula and not Pielou's log(S). Since log(S)/S falls past e, the metric inverts: uniform over 50/500/5000 items scores 0.078/0.012/0.0017 while true diversity rises.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/entropydiv", "repo": "https://github.com/v-code01/entropydiv", "reproduce": "https://github.com/v-code01/entropydiv/blob/HEAD/consequence.py" } }, { "type": "experiment", "id": "goldenwrite", "title": "goldenwrite", "claim": "The delta-rule write Jacobian I - a k^T is the projector with norm 1 only when write direction a equals key k. In closed form ||I - a k^T||_2 = sqrt(((3-2c)+sqrt(5-4c))/2), reaching the golden ratio 1.618 at orthogonality. This refutes a per-step norm-1 stability claim; matches SVD to 1.3e-15 over 20000 pairs.", "result": "null", "verify": { "detail": "https://vanshverma.com/lab/goldenwrite", "repo": "https://github.com/v-code01/goldenwrite", "reproduce": "https://github.com/v-code01/goldenwrite/blob/HEAD/refute.py" } }, { "type": "experiment", "id": "grandmean", "title": "grandmean", "claim": "HuggingFace evaluate mahalanobis.py centers with X - np.mean(reference_distribution), the scalar grand mean, against a per-feature np.cov, so the quadratic form mixes bases. A point at the distribution center gets D^2=52.48 instead of 0, and error grows with feature-mean spread. Fix is np.mean(..., axis=0).", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/grandmean", "repo": "https://github.com/v-code01/grandmean", "reproduce": "https://github.com/v-code01/grandmean/blob/HEAD/consequence.py" } }, { "type": "experiment", "id": "kalmantorsion", "title": "kalmantorsion", "claim": "KLA's 2x2 Moebius precision transition is torsion-free: it is entrywise non-negative with det=a_bar^2>0, so every finite product has real positive eigenvalues and no non-trivial finite order. A5 requires an order-5 rotation (complex eigenvalues, negative entries), which no such matrix realizes, refuting the paper's attribution of A5 to the Moebius non-linearity. The scan itself is exact.", "result": "null", "verify": { "detail": "https://vanshverma.com/lab/kalmantorsion", "repo": "https://github.com/v-code01/kalmantorsion", "reproduce": "https://github.com/v-code01/kalmantorsion/blob/HEAD/statetrack.py" } }, { "type": "experiment", "id": "maskbreak", "title": "maskbreak", "claim": "transformers' apply_chat_template with return_assistant_tokens_mask returns an all-zero mask under left truncation: the loop breaks at the first dropped early span before marking the surviving final assistant turn, so it trains with zero loss. Reproduces on 5.12.1 with a Qwen2.5 template. Fix: continue, and guard end_token is not None.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/maskbreak", "repo": "https://github.com/v-code01/maskbreak", "reproduce": "https://github.com/v-code01/maskbreak/blob/HEAD/consequence.py" } }, { "type": "experiment", "id": "maxf1", "title": "maxf1", "claim": "lighteval metrics_corpus.py CorpusLevelF1Score.compute_corpus returns np.max over the per-class F1 array from average=None instead of fscore[1], the positive-class F1 GLUE MRPC/QQP define. On majority-negative tasks a null model scores 0.889 and two models can rank in the wrong order.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/maxf1", "repo": "https://github.com/v-code01/maxf1", "reproduce": "https://github.com/v-code01/maxf1/blob/HEAD/consequence.py" } }, { "type": "experiment", "id": "negcrop", "title": "negcrop", "claim": "diffusers SDXL _get_add_time_ids packs the positive crops_coords_top_left into the negative micro-conditioning vector on the non-aesthetic branch, the default for the SDXL base model, silently ignoring negative_crops_coords_top_left while honoring the negative original_size and target_size. Affects img2img, inpaint, and ControlNet img2img pipelines.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/negcrop", "repo": "https://github.com/v-code01/negcrop", "reproduce": "https://github.com/v-code01/negcrop/blob/HEAD/consequence.py" } }, { "type": "experiment", "id": "nopealias", "title": "nopealias", "claim": "NoPE's softmax normalization forces the recovered position code to be exactly `1/t`, whose adjacent gap `1/(t(t+1))` decays quadratically. Neighbouring positions round to identical bits at `t* ~ 2^(m+1/2)` for an m-bit mantissa: t*=191 (bf16), 1465 (fp16), fp32 immune. A mantissa-width law, not an fp32-cast artifact.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/nopealias", "repo": "https://github.com/v-code01/nopealias", "reproduce": "https://github.com/v-code01/nopealias/blob/HEAD/collapse.py" } }, { "type": "experiment", "id": "primegap", "title": "primegap", "claim": "MDM-Prime applies the per-unit MDLM weight to a joint sub-token reconstruction term while masking sub-tokens independently, so at the optimal decoder L_vb = H - I/2, half the within-token multi-information below the entropy. It is not an upper bound on NLL; Zipf and skewed token laws violate it, independent digits stay tight.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/primegap", "repo": "https://github.com/v-code01/primegap", "reproduce": "https://github.com/v-code01/primegap/blob/HEAD/gap.py" } }, { "type": "experiment", "id": "rbpbound", "title": "rbpbound", "claim": "ranx Rank-Biased Precision multiplies each document by its raw graded relevance while its docstring defines r_i in {0,1}. On graded qrels the value inflates by roughly the mean grade (2.56x) and can exceed 1 (2.21 measured), breaking the [0,1] bound; binarizing the grades recovers the documented value.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/rbpbound", "repo": "https://github.com/v-code01/rbpbound", "reproduce": "https://github.com/v-code01/rbpbound/blob/HEAD/rbp.py" } }, { "type": "experiment", "id": "simflip", "title": "simflip", "claim": "sentence-transformers' BinaryClassificationEvaluator flags Manhattan and Euclidean greater_is_better=False, but their score_fns return negative distance (higher = more similar). The flag runs the threshold search, average precision, and labelling backwards: reported Manhattan accuracy is 49.5% versus a true 100%. The fix is greater_is_better=True.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/simflip", "repo": "https://github.com/v-code01/simflip", "reproduce": "https://github.com/v-code01/simflip/blob/HEAD/consequence.py" } }, { "type": "experiment", "id": "spantail", "title": "spantail", "claim": "datatrove's sentence-dedup fills removed_span under an 'elif not removed_span' guard, so it holds only the run's first sentence. When min_words_to_remove_span keeps a short duplicate span, exactly n-1 of n sentences are silently dropped and the word gate undercounts. The fix is an else branch accumulating the whole run.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/spantail", "repo": "https://github.com/v-code01/spantail", "reproduce": "https://github.com/v-code01/spantail/blob/HEAD/consequence.py" } }, { "type": "experiment", "id": "spectralflip", "title": "spectralflip", "claim": "torchmetrics' SpectralDistortionIndex (D_lambda) is a distortion metric minimized at 0 but declares higher_is_better=True, so MetricTracker selects the most spectrally distorted epoch. The sibling SpatialDistortionIndex declares False and the package's own QNR formula (1-d_lambda)^alpha confirms lower is better.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/spectralflip", "repo": "https://github.com/v-code01/spectralflip", "reproduce": "https://github.com/v-code01/spectralflip/blob/HEAD/consequence.py" } }, { "type": "experiment", "id": "svdscale", "title": "svdscale", "claim": "PEFT add_weighted_adapter's svd family (the default) multiplies each adapter weight by target.scaling and then sums get_delta_weight tensors, which already carry scaling, applying it twice. At the standard lora_alpha=2r the merged adapter is exactly twice too strong; the cat path applies scaling once.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/svdscale", "repo": "https://github.com/v-code01/svdscale", "reproduce": "https://github.com/v-code01/svdscale/blob/HEAD/combine.py" } }, { "type": "experiment", "id": "terflip", "title": "terflip", "claim": "lm-evaluation-harness registers the TER metric `higher_is_better=True`, contradicting its own \"Lower is better\" docstring, while bleu and chrf are correct. Since TER is returned un-negated, a worse-translating system is picked as best and an up arrow is printed next to an error rate on WMT/FLORES/IWSLT. Fix: `higher_is_better=False`.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/terflip", "repo": "https://github.com/v-code01/terflip", "reproduce": "https://github.com/v-code01/terflip/blob/HEAD/consequence.py" } }, { "type": "experiment", "id": "bloommask", "title": "bloommask", "claim": "datatrove reduces a hash to a bit index with AND against m_bytes (the byte count) instead of the bit count, so only 2^popcount(m_bytes) positions are reachable -- two for any power-of-two size. At m_bytes=2^20, k=7, 49 of 50 unique documents are dropped while the logged false-positive rate reads 2.8e-29.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/bloommask", "repo": "https://github.com/v-code01/bloommask" } }, { "type": "experiment", "id": "hlascan", "title": "hlascan", "claim": "HLA (arXiv:2510.27258) Theorem 4.1 claims its decayed second-order scan reproduces the serial recurrence, but the masked combine operator is not associative once gamma<1: the G,h cross summaries carry decay on the wrong operand. Balanced and serial folds agree at gamma=1 (7e-15) but diverge order-one at gamma=0.9 (2.3); closed-form gap matches to 9e-16.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/hlascan", "repo": "https://github.com/v-code01/hlascan", "reproduce": "https://github.com/v-code01/hlascan/blob/HEAD/scan.py" } }, { "type": "experiment", "id": "popcountmem", "title": "popcountmem", "claim": "Slots at t are exactly popcount(t), not the floor(log2 t)+1 bound. Each slot is a rank-d state resolving tokens only up to bucket size d, and the newest token's fidelity collapses to the linear-attention floor at every multiple of the smallest power of two above d. Within-query recency ordering holds; the absolute claim fails.", "result": "mixed", "verify": { "detail": "https://vanshverma.com/lab/popcountmem", "repo": "https://github.com/v-code01/popcountmem" } }, { "type": "experiment", "id": "pyramidskew", "title": "pyramidskew", "claim": "No. The integer floor in the step size makes the retained total exactly L*C + (L/2)*r with r = (max_num-min_num) mod (L-1), always at or above L*C, never under. The gap reaches 11.33% at C=128, L=32 and shrinks with budget, so PyramidKV's same-budget comparisons run at a larger cache than the baselines.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/pyramidskew", "repo": "https://github.com/v-code01/pyramidskew", "reproduce": "https://github.com/v-code01/pyramidskew/blob/HEAD/alloc.py" } }, { "type": "experiment", "id": "shufdup", "title": "shufdup", "claim": "From epoch two on multiple nodes, the intra-node reshuffle re-indexes the original full chunk intervals with a stream that already lists split chunks twice, so each split chunk is handed whole to two workers. In the smallest multi-node case one third of samples (10 of 30) are trained twice, silently.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/shufdup", "repo": "https://github.com/v-code01/shufdup", "reproduce": "https://github.com/v-code01/shufdup/blob/HEAD/epochflow.py" } }, { "type": "experiment", "id": "adafactorlift", "title": "adafactorlift", "claim": "V_hat = r c^T/s is exactly the contingency-table independence model of the squared-gradient matrix, and its per-step information loss equals half the G-test statistic exactly (D=19.364982=G^2/2). Against Adam each coordinate's step is mis-scaled by exactly 1/sqrt(lift), lossless only when V is rank-1 separable.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/adafactorlift", "repo": "https://github.com/v-code01/adafactorlift" } }, { "type": "experiment", "id": "blindband", "title": "blindband", "claim": "Influence support is exactly [0,S-1] union [i-(w-1)L,i], with a machine-zero blind band between them measured 0.00e+00 at depths 2,4,8,12. The band grows linearly in length and shrinks only linearly in depth. One global-attention or recurrent layer closes it, the exact capacity reason hybrids interleave.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/blindband", "repo": "https://github.com/v-code01/blindband" } }, { "type": "experiment", "id": "bpegrow", "title": "bpegrow", "claim": "No. Inserting one merge at top priority can strictly increase a word's token count (aab goes 1 to 2, up to +3 on repeated spans). Appending at lowest priority is monotone: over 90272 exhaustive triples, lowest-priority raises the count 0 times, top-priority 462 times.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/bpegrow", "repo": "https://github.com/v-code01/bpegrow", "reproduce": "https://github.com/v-code01/bpegrow/blob/HEAD/bpe.py" } }, { "type": "experiment", "id": "causalrank", "title": "causalrank", "claim": "No. With a strictly positive feature map the causal mixing matrix tril(phi(Q)phi(K)^T) is full rank N for every d, including d=1 (measured rank 512), because its diagonal is strictly positive. The empirical low-rank behavior is energy concentration (stable rank ~1.24, condition number up to 2.6e10), not the feature-dimension bottleneck.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/causalrank", "repo": "https://github.com/v-code01/causalrank" } }, { "type": "experiment", "id": "clipcouple", "title": "clipcouple", "claim": "A shared clip factor couples all tensors: one tensor spiking by S multiplies every healthy tensor's step by exactly 1/sqrt(1+(S^2-1)/L), approaching sqrt(L)/S. With L=12, a 50x spike drops every other layer to 0.069 of its step. Per-tensor clipping leaves them at 0.99.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/clipcouple", "repo": "https://github.com/v-code01/clipcouple", "reproduce": "https://github.com/v-code01/clipcouple/blob/HEAD/coupling.py" } }, { "type": "experiment", "id": "covfirewall", "title": "covfirewall", "claim": "The normalizer S=n.q is a signed sum with unbounded condition number mass/|S| (softmax is exactly 1), negative on about half of steps. The clamp max{|S|,1} is a cancellation firewall capping a 1/|S| blow-up (readout norm ~10 versus ~62000), active exactly on the worst-conditioned steps and the default regime at typical scale.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/covfirewall", "repo": "https://github.com/v-code01/covfirewall" } }, { "type": "experiment", "id": "dcgain", "title": "dcgain", "claim": "Distorted. The state branch uses ZOH but the input branch uses Euler (Mamba-2) or trapezoidal (Mamba-3), so the DC gain inflates by exactly z/(e^z-1) or (z/2)coth(z/2) with z=Delta*A, unbounded (~|z|) in the reset regime and reached in normal operation (11x, 64x measured). No constant blend lambda restores it; lambda*(z) runs 0.49 to 0.05.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/dcgain", "repo": "https://github.com/v-code01/dcgain" } }, { "type": "experiment", "id": "deltafade", "title": "deltafade", "claim": "For isotropic keys the mean-square retention rate is exactly beta(2-beta)/d (measured 0.996016 at beta=0.3, d=128 over four million keys), not the mean-field beta/d, giving a horizon tau=d/(beta(2-beta)) linear in state dim. The explicit gate and implicit delta rule multiply exactly; the operator is non-expansive on beta in [0,2].", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/deltafade", "repo": "https://github.com/v-code01/deltafade" } }, { "type": "experiment", "id": "dropfloor", "title": "dropfloor", "claim": "A degree-sum identity gives dropped tokens equal total redundant over-coverage exactly at c=1 (residual 0 to the integer). The distinct-token drop fraction sits on a floor e^{-c} (0.368 at c=1) that correlated scores only worsen (0.66-0.87). Token-choice at matched budget escapes it, vanishing as 1/sqrt(2 pi lambda).", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/dropfloor", "repo": "https://github.com/v-code01/dropfloor" } }, { "type": "experiment", "id": "dropmean", "title": "dropmean", "claim": "Above V*=32/eps (4096 in bf16) the filter drops the entire non-target gradient, and the error equals (1-p_t) times the probability-weighted unembedding row-mean, bit-exact and independent of tau. It stays 10.91 from tau=2^-9 to 2^-15, refuting the per-term-truncation justification by four orders of magnitude.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/dropmean", "repo": "https://github.com/v-code01/dropmean", "reproduce": "https://github.com/v-code01/dropmean/blob/HEAD/filter.py" } }, { "type": "experiment", "id": "dropskew", "title": "dropskew", "claim": "Rounding the scale gives an exact bias (1-p)*round_dtype(1/(1-p))-1 that does not average out and compounds as (1+bias)^L. At p=0.1 in bf16 the scale is 1.109375 versus 1.111..., a -0.16% per-layer bias reaching -4.9% train/eval drift over 32 layers. It vanishes only at bf16-dyadic rates like p=0.2.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/dropskew", "repo": "https://github.com/v-code01/dropskew" } }, { "type": "experiment", "id": "energygate", "title": "energygate", "claim": "sqrt(1-a^2) is the unique scale giving unit impulse energy, so RG-LRU holds stationary variance at exactly 1 on white input while the convex gate leaks 2a(1-a) per step to a (1-a)/(1+a) collapse (0.0005 at a=0.999). The price is a DC power gain (1+a)/(1-a) that diverges -- measured 19, 199, 1999.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/energygate", "repo": "https://github.com/v-code01/energygate" } }, { "type": "experiment", "id": "fftcausal", "title": "fftcausal", "claim": "Exactly 2L-1-N positions are wrong (aliasing identity to 3e-14), and each corrupted output depends only on future inputs, a label leak: perturbing the last input moves past position 0 by +11.8 at N=L. The next-power-of-two length is never causally safe; the safe length is double.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/fftcausal", "repo": "https://github.com/v-code01/fftcausal", "reproduce": "https://github.com/v-code01/fftcausal/blob/HEAD/contamination.py" } }, { "type": "experiment", "id": "foxwindow", "title": "foxwindow", "claim": "The effective window is the first-passage time of the forget-gate walk, with exact mean L/kappa + E[X^2]/(2kappa^2) (matched to 2e-3 against 200k-trial simulation) and variance sigma^2 L/kappa^3 (within about one percent). ALiBi is exactly the zero-variance limit, so data-dependence shows up as window variance.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/foxwindow", "repo": "https://github.com/v-code01/foxwindow" } }, { "type": "experiment", "id": "frozenrank", "title": "frozenrank", "claim": "For n_h=d. At the reflection setting minimal n_h equals rank(g-I), so DeltaNet (n_h=1) cannot rotate at all, staying Frobenius-distance exactly 2 from any 2-plane rotation.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/frozenrank", "repo": "https://github.com/v-code01/frozenrank" } }, { "type": "experiment", "id": "gaussnogen", "title": "gaussnogen", "claim": "No. In the deployed regime (K=256, g = beta(K-1)^2 >= 6.5) the kernel's principal log has strictly negative off-diagonals (defect -0.16 to -14) and negative roots for every n, while uniform and absorbing kernels sit at defect 0. Two independent float64 certificates, reconstruction error 1e-15.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/gaussnogen", "repo": "https://github.com/v-code01/gaussnogen" } }, { "type": "experiment", "id": "gqadict", "title": "gqadict", "claim": "Sharing V shrinks the group's value dictionary by exactly G (16=d_v vs 128=G*d_v at G=8), yet the concatenated output rank stays min(N, G*d_v)=128, equal to full MHA, because each head keeps an independent pattern and output projection. The cap bites only if the attention patterns are tied, collapsing rank to d_v=16.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/gqadict", "repo": "https://github.com/v-code01/gqadict" } }, { "type": "experiment", "id": "gradaccumbias", "title": "gradaccumbias", "claim": "No. It computes a token-count reweighted gradient biased by exactly sum_i g_i(1/(k n_i)-1/N), vanishing only for equal counts. At counts [3,7,2,11] it differs by 90% of magnitude, and a lone token trains at 50.5x rate. Summing token gradients and dividing once by N recovers the true gradient bit for bit.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/gradaccumbias", "repo": "https://github.com/v-code01/gradaccumbias", "reproduce": "https://github.com/v-code01/gradaccumbias/blob/HEAD/accumulate.py" } }, { "type": "experiment", "id": "gramcliff", "title": "gramcliff", "claim": "Online steps are non-expansive iff eta*max||k||^2<=2 while the mini-batch chunk is stable iff eta*lambda_max(S)<=2, and since lambda_max(S)>=max||k||^2 the mini-batch region is a strict subset. At b=16 with correlated keys lambda_max(S)~15.5 gives a 15.5x shrink; an online-stable rate diverges (||W||~1e142) and the ceiling tightens like 2/b.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/gramcliff", "repo": "https://github.com/v-code01/gramcliff" } }, { "type": "experiment", "id": "maskoverflow", "title": "maskoverflow", "claim": "Leakage flushes to a bit-identical zero below the per-dtype exp floors (-104 fp32, -93 bf16, -18 fp16), making the leakage ranking vacuous. The real discriminator is overflow: -1e9 casts to -inf and NaNs a fully masked row in float16 while finfo.min is safe, inverting the float32 intuition. Contamination reaches a full NaN sequence in exactly two attention layers.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/maskoverflow", "repo": "https://github.com/v-code01/maskoverflow" } }, { "type": "experiment", "id": "moeproxy", "title": "moeproxy", "claim": "The exact identity L=1+N shows the loss is an alignment of hard-load and soft-marginal imbalance, not a magnitude. Its minimum L=1 holds for any hard load paired with a uniform marginal, and the detached f kills the gradient there. A router driven into the blind region reads L~1.000 while dropping 26.4% of tokens at N=64.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/moeproxy", "repo": "https://github.com/v-code01/moeproxy" } }, { "type": "experiment", "id": "momentwo", "title": "momentwo", "claim": "It reduces the update to an exact second-order recurrence (closure to 5.7e-16), raising the stable learning-rate ceiling to theta*c < (1+eta)(2-alpha)/2, roughly double the delta rule's. The retention horizon becomes sqrt(eta(1-alpha)) and adds damped ringing, staying below divergence for eta<1.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/momentwo", "repo": "https://github.com/v-code01/momentwo", "reproduce": "https://github.com/v-code01/momentwo/blob/HEAD/recurrence.py" } }, { "type": "experiment", "id": "mxfloor", "title": "mxfloor", "claim": "An element rounds to zero when |v|/M < 2^(e_n-e_x-m-1)/phi; in MXFP4 any element 16-32x below the block max is annihilated (measured 6.8-10.4% silenced, FP8 0.0%). The floor scale clips the block max when phi>top_mantissa, up to 25% for FP4 and 12.5% for FP8. The block max both silences neighbors and is itself worst-quantized.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/mxfloor", "repo": "https://github.com/v-code01/mxfloor" } }, { "type": "experiment", "id": "ngeodesic", "title": "ngeodesic", "claim": "It is a retraction, not a geodesic. With scalar alpha it lags SLERP by a gap that is zero only at alpha in {0, 1/2, 1} -- alpha=1/2 halves the angle exactly (4e-14 deg) -- growing to 49.6 deg at theta=170. With the real per-dimension alpha the update leaves the geodesic plane entirely (off-plane norm ~0.18).", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/ngeodesic", "repo": "https://github.com/v-code01/ngeodesic" } }, { "type": "experiment", "id": "normnull", "title": "normnull", "claim": "Each backward pass is an exact scaled orthogonal projector: RMSNorm rank d-1 (kills scale), LayerNorm rank d-2 (kills scale and the mean/DC direction), verified to machine precision in fp64. The difference is exactly one dimension, the mean. Under LayerNorm every gradient is mean-free (sum(dx) at the fp32 floor); RMSNorm's is about six orders larger.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/normnull", "repo": "https://github.com/v-code01/normnull" } }, { "type": "experiment", "id": "packreset", "title": "packreset", "claim": "No. RoPE logits depend only on relative offset, so per-document reset under the block mask is an exact no-op: forward differs ~1e-15, gradients ~2e-15, non-accumulating out to offset 100000. Under a full causal mask the second document leaks ~70% of its mass across the boundary and reset does not fix it (0.70 vs 0.72). The mask is load-bearing; the dual holds for absolute PEs.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/packreset", "repo": "https://github.com/v-code01/packreset", "reproduce": "https://github.com/v-code01/packreset/blob/HEAD/decomposition.py" } }, { "type": "experiment", "id": "posfloor", "title": "posfloor", "claim": "The weight 1+t+t^2/2 has minimum exactly 1/2, so a key can never fall below half its unnormalized weight where softmax goes to zero. Order two is the minimal positive Taylor truncation with the largest floor. Singling out one key against N needs alignment growing like sqrt(N) versus softmax's log(N), a gap widening to 13x at N=16384.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/posfloor", "repo": "https://github.com/v-code01/posfloor" } }, { "type": "experiment", "id": "qkceiling", "title": "qkceiling", "claim": "Yes: w_max = 1/(1+(n-1)e^{-2g}), attained at the collinear config and never crossed (0/100000, matched to 2e-13), exactly dimension-independent. The critical length n* = 1+e^{2g} splits the variants: RMSNorm-QK (g ~ sqrt(d)) never binds, while unit-L2-QK (g=1) caps any key below 0.023% at 32k tokens.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/qkceiling", "repo": "https://github.com/v-code01/qkceiling" } }, { "type": "experiment", "id": "rmsgauge", "title": "rmsgauge", "claim": "Scaling the pre-RMSNorm weight leaves the output invariant (3e-15), so by Euler's theorem the gradient's radial component is exactly zero ( ~1e-15). Only decoupled weight decay moves the norm, settling at 2*wd*||W*||^2=lr^2*||g||^2, and the effective angular learning rate grows like 1/||W|| (0.0025, 0.010, 0.040 at norm scales 1, 1/2, 1/4).", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/rmsgauge", "repo": "https://github.com/v-code01/rmsgauge" } }, { "type": "experiment", "id": "ropeabsorb", "title": "ropeabsorb", "claim": "Absorption makes the key matrix exactly rank d_c (4e-16 cliff), but the RoPE position-operator family spans exactly 2L = d_R dimensions (measured rank 16, 6e-16 cliff); a single absorbing matrix exists only with no RoPE (rank 1). Partial absorption is impossible, so the decoupled dimension must be exactly d_R.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/ropeabsorb", "repo": "https://github.com/v-code01/ropeabsorb" } }, { "type": "experiment", "id": "scanhorizon", "title": "scanhorizon", "claim": "The associative scan stays bounded and never overflows, while the divide-out-the-gate form overflows at exactly L*=floor(ln(realmax)/(|A|dt)), matching to the integer (fp16 hits 3 tokens at |A|dt=3). On the M4 GPU Metal's cumprod rounds hotter, so overflow arrives a few positions earlier (174 vs 177).", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/scanhorizon", "repo": "https://github.com/v-code01/scanhorizon" } }, { "type": "experiment", "id": "secularmode", "title": "secularmode", "claim": "The spectrum solves an exact secular equation and Cauchy-interlaces the decay values, pinning the top eigenvalue at max(w)<1, so retention can never expand. Instability enters only via the bottom eigenvalue at the exact boundary sum a_i kappa_i^2/(w_i+1)=1. Smaller decay lowers the removal budget needed, so decay worsens instability.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/secularmode", "repo": "https://github.com/v-code01/secularmode" } }, { "type": "experiment", "id": "selproxy", "title": "selproxy", "claim": "No. The mean-logit proxy keeps mean(x) and drops exactly the concentration D in LSE=mean+log(B)+D. Worst case it ranks a flat block above a spike holding 99.65% of the mass; missed mass rises to 25-51% as logit std grows. Any query-independent summary is one facet of a B-facet envelope; Quest's bounding box upper-bounds on 100% of queries.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/selproxy", "repo": "https://github.com/v-code01/selproxy", "reproduce": "https://github.com/v-code01/selproxy/blob/HEAD/proxy.py" } }, { "type": "experiment", "id": "shiftgauge", "title": "shiftgauge", "claim": "Softpick's row sum is exactly r=P/(P+N) in [0,1], zero iff max logit<=0, matched to 3e-16. Unlike shift-invariant softmax, softpick reads the absolute logit level, so its row sum sweeps monotonically 0 to 1 under a constant shift. Stabilization needs -e^{-m} because the gauge is multiplicative in e^m; naive subtract-max collapses +800 logits to 0.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/shiftgauge", "repo": "https://github.com/v-code01/shiftgauge", "reproduce": "https://github.com/v-code01/shiftgauge/blob/HEAD/softpick.py" } }, { "type": "experiment", "id": "sinktemp", "title": "sinktemp", "claim": "Column sums have mean exactly 1 but per-key mass diverges, ranging [0.12, 2.96] at beta=4. Sinkhorn restores every column to 1; its Birkhoff contraction kappa=tanh(beta*Delta_0/4) is exactly linear in inverse temperature, so iterations to 1e-6 blow up from 57 to 258638 as beta goes 0.5 to 2. Key conservation is exponentially expensive for sharp attention.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/sinktemp", "repo": "https://github.com/v-code01/sinktemp", "reproduce": "https://github.com/v-code01/sinktemp/blob/HEAD/balance.py" } }, { "type": "experiment", "id": "softcapgrad", "title": "softcapgrad", "claim": "The gradient factor is exactly 1-(z/c)^2, confidence-proportional attenuation, verified to 3.3e-16 over 2M points. In bf16 tanh saturates to 1.0 at a = 3.47c (a = 104 and 173 at Gemma2's caps 30 and 50), giving a bit-exact zero gradient reached by the most confident logits. End-to-end slowdown is only ~1.1x, not the pointwise several-x.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/softcapgrad", "repo": "https://github.com/v-code01/softcapgrad" } }, { "type": "experiment", "id": "ssdcond", "title": "ssdcond", "claim": "The scalar-gate SSD matrix equals (I-aS)^-1 exactly (residual 8e-17), inverts in one O(L) bidiagonal recurrence, and has cond_2 bounded by (1+a)/(1-a) flat in length (3.00, 19.0, 199 at a=0.5, 0.9, 0.99). The undecayed a=1 accumulator grows as 4L/pi and softmax matrices are singular, so SSD is the family member both full-rank and uniformly well-conditioned.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/ssdcond", "repo": "https://github.com/v-code01/ssdcond" } }, { "type": "experiment", "id": "ssmrecall", "title": "ssmrecall", "claim": "A diagonal SSM recalls exactly its last d inputs, aliasing input d+1 (error ~1e-15 jumping to order one). The d-th roots of unity uniquely minimize conditioning: cond(M)=1 and readout noise gain 1/sqrt(d), beating 200 random placements. Deployed S4D-Lin and HiPPO inits recover strictly fewer at d=64, and the recall horizon is independent of spectral radius.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/ssmrecall", "repo": "https://github.com/v-code01/ssmrecall", "reproduce": "https://github.com/v-code01/ssmrecall/blob/HEAD/capacity.py" } }, { "type": "experiment", "id": "stabgauge", "title": "stabgauge", "claim": "The stabilizer offset is a gauge: output h=C/N is exactly invariant to any offset schedule (naive and random within 3e-16 to 3e-14 of running-max). The running max is the pointwise-minimal overflow-safe gauge, hitting max-gate=1 at 1200/1200 steps, and it converts the normalizer's exact overflow horizon t*=1415 into linear growth.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/stabgauge", "repo": "https://github.com/v-code01/stabgauge" } }, { "type": "experiment", "id": "stalecos", "title": "stalecos", "claim": "No. Retained descent equals the energy-weighted sum of squared cosines sum_j w_j cos^2(phi_j), decoupled from the frame-distance metric. When a tail direction rotates 85 degrees, the max principal angle predicts retention 0.003 while the true retained descent is 0.884, a 291x understatement that triggers refreshes too early.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/stalecos", "repo": "https://github.com/v-code01/stalecos", "reproduce": "https://github.com/v-code01/stalecos/blob/HEAD/staleness.py" } }, { "type": "experiment", "id": "zlossgauge", "title": "zlossgauge", "claim": "Cross-entropy is exactly flat along the all-ones logit direction, so its gradient sums to zero and the total logit is conserved (drift 6e-14 over 300 steps). z-loss's gradient sums to 2*logsumexp(z), exactly the missing gauge component, breaking the conservation on purpose (drift 421).", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/zlossgauge", "repo": "https://github.com/v-code01/zlossgauge", "reproduce": "https://github.com/v-code01/zlossgauge/blob/HEAD/gauge.py" } }, { "type": "experiment", "id": "adamfreeze", "title": "adamfreeze", "claim": "A bf16 second moment freezes once beta2 > 1 - 2^-(t+1) = 0.99609, so standard beta2=0.999 stops tracking. The bias flips sign with gradient skew: 0.24x low on steady gradients, 2.4x high on spiky ones. Loss scaling is scale-invariant and cannot fix it; stochastic rounding restores an unbiased 1.0x.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/adamfreeze", "repo": "https://github.com/v-code01/adamfreeze" } }, { "type": "experiment", "id": "cedeadzone", "title": "cedeadzone", "claim": "R(p_c)-1 rounds to exactly zero for every p_c >= 1 - 2^-9 = 0.998047, so a fused bf16 head gives no gradient on tokens the model is over 99.8% confident about, while the fp32 subtract-then-round path stays provably immune. On a 32k-vocab softmax at logit gap 17 the fused head returns 0 versus -9.9e-4.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/cedeadzone", "repo": "https://github.com/v-code01/cedeadzone" } }, { "type": "experiment", "id": "flashswamp", "title": "flashswamp", "claim": "A bf16 sequential softmax carry swamps: relative denominator error is 2.3% at n/Bc=256 and 73% at 8192, while a pairwise sum stays exact. The shipped M4 GPU attention kernel keeps bf16 error flat (5.85e-3 to 6.16e-3 from n=256 to 8192), showing it accumulates in fp32 and stays safe.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/flashswamp", "repo": "https://github.com/v-code01/flashswamp" } }, { "type": "experiment", "id": "muonspectral", "title": "muonspectral", "claim": "Newton-Schulz has three repelling positive fixed points and no stable one, so it never converges to orthogonal. Below 1.2637 singular values fall onto a period-4 attractor band [0.68, 1.13], a 1.66x anisotropic rescaling (bf16 equals fp32); above it they diverge, held off by a rank-1-tight 26.4% Frobenius margin.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/muonspectral", "repo": "https://github.com/v-code01/muonspectral" } }, { "type": "experiment", "id": "mupblind", "title": "mupblind", "claim": "Measured across head dim 16 to 512, muP's init entropy gap collapses with log-log slope -1.001 (Theta(1/d)) while standard 1/sqrt(d) stays flat at -0.002. muP starts attention maximally diffuse and sharpens only as training aligns queries and keys, the gap rising from 0.004 to 1.9.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/mupblind", "repo": "https://github.com/v-code01/mupblind" } }, { "type": "experiment", "id": "ropealias", "title": "ropealias", "claim": "Confirmed. bf16 resolves position only to ULP(p) ~ p/128, so adjacent positions collapse to one rotary angle: 87% of adjacent pairs aliased at position 1024, 99% at 16384, 100% at 131072. Aliased positions give bit-identical rotary vectors and identical attention scores. The phase must be fp32; the cos/sin cache tolerates bf16.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/ropealias", "repo": "https://github.com/v-code01/ropealias" } }, { "type": "experiment", "id": "varsplit", "title": "varsplit", "claim": "The MLP variance increment is depth-flat (slope +0.0003, 95% CI straddling zero) while the attention increment rises (slope +0.010, CI strictly positive) with token rank collapse. The two invert (MLP about 5x attention at layer 1, attention about 3x MLP by layer 48), so a single uniform residual scale cannot balance both.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/varsplit", "repo": "https://github.com/v-code01/varsplit" } }, { "type": "experiment", "id": "evictfaithful", "title": "evictfaithful", "claim": "A policy is faithful if and only if its priority key is path-monotone, key(parent) >= key(child) on every edge. LRU, LFU, SLRU, and priority qualify and are exactly the four CLI-exposed policies; FIFO, FILO, and MRU invert on some edge and are unfaithful. Verified against real SGLang with 0 inversions over 8.4M.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/evictfaithful", "repo": "https://github.com/v-code01/evictfaithful" } }, { "type": "experiment", "id": "atomcliff", "title": "atomcliff", "claim": "A shared relaxed fetch_add scales negatively: two threads (260 Mops/s) are slower than one (542), and eight run at 37, about a fourteenth of one thread. Per-thread padded counters scale linearly to 4157 Mops/s at eight (112x). Cross-cluster sharing is worse still (31 vs 37). No updates are lost.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/atomcliff", "repo": "https://github.com/v-code01/atomcliff", "reproduce": "https://github.com/v-code01/atomcliff/blob/HEAD/run.sh", "prereg": "https://github.com/v-code01/atomcliff/blob/HEAD/PREREG.md" } }, { "type": "experiment", "id": "conshash", "title": "conshash", "claim": "Over 2,000,000 keys the median max/average imbalance tracks ln(n): about 8x at 3,000 nodes, with the worst ring over 11x. Virtual nodes cut the overshoot along 1/sqrt(k): 10 points drop 8x to about 2.2x, and 100 to 200 points hold every node within roughly a quarter of the mean.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/conshash", "repo": "https://github.com/v-code01/conshash", "reproduce": "https://github.com/v-code01/conshash/blob/HEAD/run.sh", "prereg": "https://github.com/v-code01/conshash/blob/HEAD/PREREG.md" } }, { "type": "experiment", "id": "hashdos", "title": "hashdos", "claim": "Confirmed. With a weak low-bits hash, adversarial keys all land in one bucket and insert time quadruples per doubling of N (the O(n^2) signature), reaching 38ms with a chain of 32,000 at N=32k, 108x slower than the same keys under a keyed hash, which stays linear with a chain of 6.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/hashdos", "repo": "https://github.com/v-code01/hashdos", "reproduce": "https://github.com/v-code01/hashdos/blob/HEAD/run.sh", "prereg": "https://github.com/v-code01/hashdos/blob/HEAD/PREREG.md" } }, { "type": "experiment", "id": "prefetch", "title": "prefetch", "claim": "Out of cache, only next-line (64B) access runs at prefetch speed: 0.70ns at 512MB, versus 2.47ns at a 128B stride, 3.5x slower, and near 3ns for larger strides. The 4MB in-cache control is flat across strides, ruling out TLB.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/prefetch", "repo": "https://github.com/v-code01/prefetch", "reproduce": "https://github.com/v-code01/prefetch/blob/HEAD/run.sh", "prereg": "https://github.com/v-code01/prefetch/blob/HEAD/PREREG.md" } }, { "type": "experiment", "id": "retrystorm", "title": "retrystorm", "claim": "The no-jitter peak equals N exactly at every scale (1k, 10k, 100k) because the whole fleet lands in one 20 ms window. Full jitter cuts the peak about 30x, decorrelated about 67x and is the best of the four, equal jitter weakest. The ordering holds at every scale.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/retrystorm", "repo": "https://github.com/v-code01/retrystorm", "reproduce": "https://github.com/v-code01/retrystorm/blob/HEAD/run.sh", "prereg": "https://github.com/v-code01/retrystorm/blob/HEAD/PREREG.md" } }, { "type": "experiment", "id": "unalign", "title": "unalign", "claim": "Unaligned 8-byte loads run at about 0.235 ns/load at every offset, including cache-line and page crossings, with no penalty. But an atomic that straddles a 16-byte boundary raises SIGBUS deterministically. An 8-byte atomic is fine only while its bytes stay inside one 16-byte block, and a 16-byte atomic needs full 16-byte alignment.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/unalign", "repo": "https://github.com/v-code01/unalign", "reproduce": "https://github.com/v-code01/unalign/blob/HEAD/run.sh", "prereg": "https://github.com/v-code01/unalign/blob/HEAD/PREREG.md" } }, { "type": "experiment", "id": "twochoices", "title": "twochoices", "claim": "Confirmed. With one choice the fullest bin climbed from 5 to 10 as n grew from 1e3 to 1e7 (~log n / log log n). Two choices stayed nearly flat at 3 to 4 (~log log n). The first extra choice cut worst-case load by 6, the second by 1, the third by 0.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/twochoices", "repo": "https://github.com/v-code01/twochoices", "reproduce": "https://github.com/v-code01/twochoices/blob/HEAD/run.sh", "prereg": "https://github.com/v-code01/twochoices/blob/HEAD/PREREG.md" } }, { "type": "experiment", "id": "hllint", "title": "hllint", "claim": "Inclusion-exclusion on HLLs carries a near-constant absolute error floor around 7,000-8,500, independent of the true overlap. Relative error is 80x at a 0.01% overlap, 8x at 0.1%, still 79% at 1%, and only falls to a few percent once the overlap exceeds about 10% of the set.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/hllint", "repo": "https://github.com/v-code01/hllint", "reproduce": "https://github.com/v-code01/hllint/blob/HEAD/run.sh", "prereg": "https://github.com/v-code01/hllint/blob/HEAD/PREREG.md" } }, { "type": "experiment", "id": "minhash", "title": "minhash", "claim": "At matched space (MinHash k=1024 ~8 KB vs HLL 2^13 ~6 KB, n=1e6), MinHash wins at every overlap: 13x lower relative error at 0.1% overlap, narrowing to 1.6x at 30%. MinHash degrades as 1/sqrt and its error is set-size independent, where HLL degrades as 1/overlap.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/minhash", "repo": "https://github.com/v-code01/minhash", "reproduce": "https://github.com/v-code01/minhash/blob/HEAD/run.sh", "prereg": "https://github.com/v-code01/minhash/blob/HEAD/PREREG.md" } }, { "type": "experiment", "id": "reclaim", "title": "reclaim", "claim": "Under one stalled thread, EBR retained nodes equal the number of retires (unbounded) while hazard pointers stay flat at O(T*H+batch), at most 144. The hazard-pointer guarded read costs 0.510 ns versus 0.329 ns for EBR, a 1.55x CI-separated fence tax that buys the bounded-memory guarantee. Neither scheme dominates.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/reclaim", "repo": "https://github.com/v-code01/reclaim", "reproduce": "https://github.com/v-code01/reclaim/blob/HEAD/run.sh", "prereg": "https://github.com/v-code01/reclaim/blob/HEAD/PREREG.md" } }, { "type": "experiment", "id": "seqlock", "title": "seqlock", "claim": "herd7 under rc11.cat proves the four-fence set minimal: every single-fence weakening resurrects the torn-and-validated witness, and the acquire-only-s2 optimization is unsound. On M4 over-fencing costs about 37% uncontended (0.93 vs 0.68 ns); the dominant cost is true-sharing the counter, which cache-line padding cannot fix.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/seqlock", "repo": "https://github.com/v-code01/seqlock", "reproduce": "https://github.com/v-code01/seqlock/blob/HEAD/run.sh", "prereg": "https://github.com/v-code01/seqlock/blob/HEAD/PREREG.md" } }, { "type": "experiment", "id": "sketchadv", "title": "sketchadv", "claim": "GK holds eps*N (2000 at N=200k) on every stream. t-digest, given equal-or-greater space, stays near-exact and about 2.3x faster on random data but violates the bound on adversarial streams: max rank error 2313 on skew_middle and 6277 on bimodal_gap, with the whole BCa CI above eps*N.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/sketchadv", "repo": "https://github.com/v-code01/sketchadv", "reproduce": "https://github.com/v-code01/sketchadv/blob/HEAD/run.sh", "prereg": "https://github.com/v-code01/sketchadv/blob/HEAD/PREREG.md" } }, { "type": "experiment", "id": "slackfront", "title": "slackfront", "claim": "Confirmed. Any contiguous open-addressing table growing by (1+alpha) at at most w moves per insert must satisfy w*alpha>=1, proved by pigeonhole and tight at w=ceil(1/alpha). On M4 the de-amortized worst insert is 1.06ms versus 81.3ms for synchronous rehash, but bounding migration is necessary not sufficient: a residual O(N) page-table cost remains.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/slackfront", "repo": "https://github.com/v-code01/slackfront", "reproduce": "https://github.com/v-code01/slackfront/blob/HEAD/run.sh", "prereg": "https://github.com/v-code01/slackfront/blob/HEAD/PREREG.md" } }, { "type": "experiment", "id": "twogranule", "title": "twogranule", "claim": "The false-sharing recovery edge sits at 64 bytes within a core cluster (1.84 ns/op) but at 128 bytes across clusters, where the same 64-byte-separated layout still shares (9.15 ns). A read-only load control stays flat (~0.8-1.5 ns), so the penalty is write-ownership-driven. This inverts the pad-to-128B rule.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/twogranule", "repo": "https://github.com/v-code01/twogranule", "reproduce": "https://github.com/v-code01/twogranule/blob/HEAD/run.sh", "prereg": "https://github.com/v-code01/twogranule/blob/HEAD/PREREG.md" } }, { "type": "experiment", "id": "aligntype", "title": "aligntype", "claim": "The reader sets ctx->alignment from gguf_get_val_u32 at gguf.cpp:610 with no type or arity guard, so a well-formed GGUF declaring general.alignment as INT32, UINT64, STRING, or an array trips an unconditional GGML_ASSERT and aborts. Every other malformed-metadata case returns nullptr cleanly. The accepting and aborting buffers differ by exactly one byte. Model-load DoS.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/aligntype", "repo": "https://github.com/v-code01/aligntype", "reproduce": "https://github.com/v-code01/aligntype/blob/HEAD/run.sh", "prereg": "https://github.com/v-code01/aligntype/blob/HEAD/PREREG.md" } }, { "type": "experiment", "id": "argmaxtie", "title": "argmaxtie", "claim": "Confirmed. On an exact fp32 tie the CPU greedy keeps the first tied index (strict >) and the ggml backend argmax keeps the last (>= on equality), diverging on 10 of 10 injected ties while agreeing on 34 of 34 unique-max cases. The two greedy paths pick different tokens.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/argmaxtie", "repo": "https://github.com/v-code01/argmaxtie", "reproduce": "https://github.com/v-code01/argmaxtie/blob/HEAD/run.sh", "prereg": "https://github.com/v-code01/argmaxtie/blob/HEAD/PREREG.md" } }, { "type": "experiment", "id": "arrwiden", "title": "arrwiden", "claim": "Confirmed bug. gguf_get_arr_n returns the byte count for an INT8 array, and the loader reinterprets the N-byte buffer as N int32, copying 4*N bytes for a 3*N-byte over-read. A 64Mi INT8 suppress array crashes model load with SIGSEGV; a same-byte-size INT32 array loads cleanly. Poisoned-model DoS on the default load path.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/arrwiden", "repo": "https://github.com/v-code01/arrwiden", "reproduce": "https://github.com/v-code01/arrwiden/blob/HEAD/build/run.sh", "prereg": "https://github.com/v-code01/arrwiden/blob/HEAD/build/PREREG.md" } }, { "type": "experiment", "id": "batchvalorder", "title": "batchvalorder", "claim": "Confirmed. Supplying seq_id with n_seq_id NULL, a combination the function's own auto-generator contracts to accept, makes L60 dereference NULL[i] before L73 fills it, crashing with SIGSEGV. Exactly one of the four {seq_id, n_seq_id} x {supplied, NULL} cells faults; the mirror case with seq_id NULL is handled.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/batchvalorder", "repo": "https://github.com/v-code01/batchvalorder", "reproduce": "https://github.com/v-code01/batchvalorder/blob/HEAD/build/run.sh", "prereg": "https://github.com/v-code01/batchvalorder/blob/HEAD/build/PREREG.md" } }, { "type": "experiment", "id": "chatnul", "title": "chatnul", "claim": "At exact fill (length == res) strncpy copies all res bytes and writes no NUL, yet the return contract says the buffer sufficed. A guard-page strlen faults at length=res and not at res+1, so a C or FFI caller treating it as a C string over-reads past the buffer.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/chatnul", "repo": "https://github.com/v-code01/chatnul", "reproduce": "https://github.com/v-code01/chatnul/blob/HEAD/build/run.sh", "prereg": "https://github.com/v-code01/chatnul/blob/HEAD/build/PREREG.md" } }, { "type": "experiment", "id": "cleanband", "title": "cleanband", "claim": "For every space-trimming token the size-probe returns the raw length before clean_spaces shrinks it (R=2 vs cleaned C=1), and a buffer sized to the true cleaned length is spuriously rejected with -R. Non-trimming tokens satisfy R==C. A documented-contract violation, benign, with a deterministic repro on build 9760.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/cleanband", "repo": "https://github.com/v-code01/cleanband", "reproduce": "https://github.com/v-code01/cleanband/blob/HEAD/run.sh", "prereg": "https://github.com/v-code01/cleanband/blob/HEAD/PREREG.md" } }, { "type": "experiment", "id": "dryz", "title": "dryz", "claim": "Yes. Inverting the penalty to the integer match length and comparing against two independent suffix-repeat oracles, the shipped DRY length matches on all 2,391,471 differential cases (base, window-cap, single-token-breaker) over histories up to length 11, plus 1,048,560 cases on a 4-symbol alphabet, with 0 mismatches. Ships as a certificate.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/dryz", "repo": "https://github.com/v-code01/dryz", "reproduce": "https://github.com/v-code01/dryz/blob/HEAD/run.sh", "prereg": "https://github.com/v-code01/dryz/blob/HEAD/PREREG.md" } }, { "type": "experiment", "id": "equalsplitcap", "title": "equalsplitcap", "claim": "No. The accumulation guard uses > after the push, so the ubatch overshoots by exactly one: n_tokens = min(K, n_ubatch+1), confirmed on all 168 grid cells (32 overshoot by one token). The sibling splitters split_simple and split_seq use >= and never exceed the cap. The fix is one character.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/equalsplitcap", "repo": "https://github.com/v-code01/equalsplitcap", "reproduce": "https://github.com/v-code01/equalsplitcap/blob/HEAD/run.sh", "prereg": "https://github.com/v-code01/equalsplitcap/blob/HEAD/PREREG.md" } }, { "type": "experiment", "id": "floorbound", "title": "floorbound", "claim": "Confirmed bug. The emitter reads bounds with a truncating get() cast and a blanket +1/-1, no floor/ceil, so minimum:5.5 becomes minimum:5 and accepts the out-of-range 5. Over 208 cases all four bound keywords diverge by one integer on their fractional side (2 over-generation, 2 under-generation); 104 integer-bound controls agree exactly.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/floorbound", "repo": "https://github.com/v-code01/floorbound", "reproduce": "https://github.com/v-code01/floorbound/blob/HEAD/run.sh", "prereg": "https://github.com/v-code01/floorbound/blob/HEAD/PREREG.md" } }, { "type": "experiment", "id": "gguf0dim", "title": "gguf0dim", "claim": "Yes. The dimension gate rejects only ne<0, so a zero passes into INT64_MAX/ne[1] at gguf.cpp:681, a signed division by zero. On x86-64 that is a #DE fault and SIGFPE crash when loading an attacker .gguf; on AArch64/M4 sdiv returns 0 and the file is rejected by accident. A sanitizer confirms the UB at the exact line.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/gguf0dim", "repo": "https://github.com/v-code01/gguf0dim", "reproduce": "https://github.com/v-code01/gguf0dim/blob/HEAD/run.sh", "prereg": "https://github.com/v-code01/gguf0dim/blob/HEAD/PREREG.md" } }, { "type": "experiment", "id": "gramrep", "title": "gramrep", "claim": "In handle_repetitions the loop count n_opt = max_times - min_times is a uint64 subtraction, so an inverted bound like {3,2} underflows to 2^64-1. The magnitude guard is keyed on max_times, not the loop count, so it never fires. {3,2}, {5,1}, {10,0} all fail to terminate: a compile-time DoS before any token is sampled.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/gramrep", "repo": "https://github.com/v-code01/gramrep", "reproduce": "https://github.com/v-code01/gramrep/blob/HEAD/run.sh", "prereg": "https://github.com/v-code01/gramrep/blob/HEAD/PREREG.md" } }, { "type": "experiment", "id": "intrange", "title": "intrange", "claim": "Over 58,179 canonical checks the emitted grammar accepts an integer string iff its value is in range: zero soundness and zero completeness violations. One minor lexical edge remains -- acceptance of -0 tracks whether the range has a negative branch rather than whether 0 is in range.", "result": "mixed", "verify": { "detail": "https://vanshverma.com/lab/intrange", "repo": "https://github.com/v-code01/intrange", "reproduce": "https://github.com/v-code01/intrange/blob/HEAD/run.sh", "prereg": "https://github.com/v-code01/intrange/blob/HEAD/PREREG.md" } }, { "type": "experiment", "id": "keepfloor", "title": "keepfloor", "claim": "The ggml-backend top_p and min_p graphs never reference min_keep, so with min_keep>=2 the offloaded path drops the survivor floor. Over 144 cells, 78 diverge and the backend set is always a strict subset of the CPU set, while all 48 min_keep in {0,1} cells agree exactly.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/keepfloor", "repo": "https://github.com/v-code01/keepfloor", "reproduce": "https://github.com/v-code01/keepfloor/blob/HEAD/run.sh", "prereg": "https://github.com/v-code01/keepfloor/blob/HEAD/PREREG.md" } }, { "type": "experiment", "id": "kvextremum", "title": "kvextremum", "claim": "Mixed. A clean certificate was predicted; instead two defects surfaced over 48,024 exhaustive trials. seq_cp lacks a seq_has guard, so a repeated copy double-counts seq_pos and leaves seq_pos_max stale (returns 2 for an empty sequence). The documented [min,max] contiguity also fails after any interior seq_rm or partial seq_add/seq_div.", "result": "mixed", "verify": { "detail": "https://vanshverma.com/lab/kvextremum", "repo": "https://github.com/v-code01/kvextremum", "reproduce": "https://github.com/v-code01/kvextremum/blob/HEAD/run.sh", "prereg": "https://github.com/v-code01/kvextremum/blob/HEAD/PREREG.md" } }, { "type": "experiment", "id": "mirostatclone", "title": "mirostatclone", "claim": "Confirmed. llama_sampler_mirostat_clone casts result_ctx from smpl->ctx (the source) rather than result->ctx, so both state copies are self-assignments and the clone keeps the init mu=2*tau. The controller restarts cold. Every other stateful clone, including Mirostat v2, casts correctly; v1 is the lone outlier.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/mirostatclone", "repo": "https://github.com/v-code01/mirostatclone", "reproduce": "https://github.com/v-code01/mirostatclone/blob/HEAD/run.sh", "prereg": "https://github.com/v-code01/mirostatclone/blob/HEAD/PREREG.md" } }, { "type": "experiment", "id": "patternanchor", "title": "patternanchor", "claim": "Neither. Every unanchored pattern, the spec's normal case, throws 'Pattern must start with ^ and end with $' and emits no grammar, refuting the pre-registered fullmatch prediction. Anchored ^p$ patterns are sound and complete: over 65,532 membership checks the GBNF acceptor matches std::regex_match with zero mismatches.", "result": "mixed", "verify": { "detail": "https://vanshverma.com/lab/patternanchor", "repo": "https://github.com/v-code01/patternanchor", "reproduce": "https://github.com/v-code01/patternanchor/blob/HEAD/run.sh", "prereg": "https://github.com/v-code01/patternanchor/blob/HEAD/PREREG.md" } }, { "type": "experiment", "id": "penaltyledger", "title": "penaltyledger", "claim": "Mixed. The accept and reset ledger is bit-exact against a from-scratch window histogram over 3,010,612 enumerated sequences (zero mismatches). But llama_sampler_penalties_clone copies prev without token_count, so a cloned sampler forgets its penalties and, after evicting a pre-clone token, drives a count negative and boosts a repeated token instead of penalizing it.", "result": "mixed", "verify": { "detail": "https://vanshverma.com/lab/penaltyledger", "repo": "https://github.com/v-code01/penaltyledger", "reproduce": "https://github.com/v-code01/penaltyledger/blob/HEAD/run.sh", "prereg": "https://github.com/v-code01/penaltyledger/blob/HEAD/PREREG.md" } }, { "type": "experiment", "id": "piecebounds", "title": "piecebounds", "claim": "Once a vocab loads, every llama_token_to_piece call hits an unchecked cache.at(token); an id below 0 or at or above n_tokens throws std::out_of_range across the extern C boundary and aborts a handler-less caller with SIGABRT. The crash boundary is exactly [0, n_tokens) and the function's own guard is dead code.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/piecebounds", "repo": "https://github.com/v-code01/piecebounds", "reproduce": "https://github.com/v-code01/piecebounds/blob/HEAD/run.sh", "prereg": "https://github.com/v-code01/piecebounds/blob/HEAD/PREREG.md" } }, { "type": "experiment", "id": "prefixkey", "title": "prefixkey", "claim": "_not_strings emits the stop-here epsilon only at the trie root, so an interior node that is a proper prefix of a defined property name has no accepting derivation. With schema property \"ab\" and additionalProperties true, the valid key \"a\" is rejected. Over 889 exhaustive cases: 10 under-generation mismatches, 0 over-generation, all prefixes.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/prefixkey", "repo": "https://github.com/v-code01/prefixkey", "reproduce": "https://github.com/v-code01/prefixkey/blob/HEAD/run.sh", "prereg": "https://github.com/v-code01/prefixkey/blob/HEAD/PREREG.md" } }, { "type": "experiment", "id": "q2krail", "title": "q2krail", "claim": "Across an adversarial corpus of 4008 blocks plus a 20-million-sub-block hunt, every pre-store integer stays in [0,15] and no scale or min goes negative, so the missing clamp is a no-op. Safe by construction: the deducted-min formulation forces both values non-negative and bounded by their max.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/q2krail", "repo": "https://github.com/v-code01/q2krail", "reproduce": "https://github.com/v-code01/q2krail/blob/HEAD/run.sh", "prereg": "https://github.com/v-code01/q2krail/blob/HEAD/PREREG.md" } }, { "type": "experiment", "id": "q3kvalue", "title": "q3kvalue", "claim": "Three arms -- the shipped kernel, the pinned source, and a clean-room spec -- agree bit-exactly across 26,124,288 lane comparisons with zero mismatches, over the exhaustive per-lane alphabet and 100,000 random blocks. An armed polarity-flipped mutant disagrees on 256 lanes, confirming the inverted hmask decode (set bit subtracts 0, clear subtracts 4).", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/q3kvalue", "repo": "https://github.com/v-code01/q3kvalue", "reproduce": "https://github.com/v-code01/q3kvalue/blob/HEAD/build/run.sh", "prereg": "https://github.com/v-code01/q3kvalue/blob/HEAD/build/PREREG.md" } }, { "type": "experiment", "id": "quantfixpoint", "title": "quantfixpoint", "claim": "Confirmed. Dequant-then-requant fixes exactly the rail blocks (max|q|==127) and the canonical zero block; over 5,000,000 random inputs the quantizer emitted zero non-rail nonzero blocks, so its whole output range is stable. The d/2 error bound is not strict: fp16 storage of the scale inflates the worst case to about 1.6x d/2.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/quantfixpoint", "repo": "https://github.com/v-code01/quantfixpoint", "reproduce": "https://github.com/v-code01/quantfixpoint/blob/HEAD/run.sh", "prereg": "https://github.com/v-code01/quantfixpoint/blob/HEAD/PREREG.md" } }, { "type": "experiment", "id": "sortpoison", "title": "sortpoison", "claim": "No. Exhaustively over n! permutations with sorted forced true, top_p yields up to 11 distinct survivor sets, min_p 8, top_k 10, all wrong, while typical stays sound. logit_bias reorders logits without resetting sorted=false, so the public chain top_k to logit_bias to top_p keeps {0,1,2,3} where {3} is correct. The stock default chain is safe.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/sortpoison", "repo": "https://github.com/v-code01/sortpoison", "reproduce": "https://github.com/v-code01/sortpoison/blob/HEAD/run.sh", "prereg": "https://github.com/v-code01/sortpoison/blob/HEAD/PREREG.md" } }, { "type": "experiment", "id": "trycatchgap", "title": "trycatchgap", "claim": "No. Under ON_DEVICE the pre-flight validation (a magic read and a GGML_ASSERT on the seq id) runs outside the try that returns 0, so a short, wrong-magic, or unknown-seq blob aborts the process with SIGABRT. The identical bytes with flags=0 return 0 gracefully; only the flag bit toggles abort versus graceful.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/trycatchgap", "repo": "https://github.com/v-code01/trycatchgap", "reproduce": "https://github.com/v-code01/trycatchgap/blob/HEAD/build/run.sh", "prereg": "https://github.com/v-code01/trycatchgap/blob/HEAD/build/PREREG.md" } }, { "type": "experiment", "id": "bf16mma", "title": "bf16mma", "claim": "~6.8x faster, but the accuracy 'win' is a single-accumulator strawman: a matched-lane 4-accumulator fp32 loop is more accurate than BFDOT. bf16 buys speed, not accuracy, and it isn't bit-deterministic under regrouping. Exact fp64 oracle.", "result": "mixed", "verify": { "detail": "https://vanshverma.com/lab/bf16mma", "repo": "https://github.com/v-code01/bf16mma", "prereg": "https://github.com/v-code01/bf16mma/blob/HEAD/PREREG.md" } }, { "type": "experiment", "id": "detok", "title": "detok", "claim": "Yes, checked over 127,096 codepoints with no exceptions: naive per-token decode emits U+FFFD iff the vocab splits the codepoint (0% of ASCII, 40% of 2-byte, 76% of 3-byte, 99% of 4-byte). The minimal streaming fix holds back at most 3 bytes and 3 tokens, both bounds tight.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/detok", "repo": "https://github.com/v-code01/detok", "reproduce": "https://github.com/v-code01/detok/blob/HEAD/run.py", "prereg": "https://github.com/v-code01/detok/blob/HEAD/PREREG.md" } }, { "type": "experiment", "id": "erracc", "title": "erracc", "claim": "Mixed. The core claim holds: absolute residual error is flat through the body (1.00 to 1.02x growth, layer 4 to 85% depth) in both 0.5B and 1.5B, not compounding, and the late-layer relative jump is largely residual-norm collapse. But matched-magnitude Gaussian noise propagates 1.4 to 2.3x less, so quantization structure matters, refuting a pure-noise model.", "result": "mixed", "verify": { "detail": "https://vanshverma.com/lab/erracc", "repo": "https://github.com/v-code01/erracc", "reproduce": "https://github.com/v-code01/erracc/blob/HEAD/run.py", "prereg": "https://github.com/v-code01/erracc/blob/HEAD/PREREG.md" } }, { "type": "experiment", "id": "grammarexact", "title": "grammarexact", "claim": "Mixed. At states on whole-codepoint boundaries the byte-walking matcher is exactly the viable-prefix recognizer across the corpus. But llama_grammar_match_partial_char omits a lead-byte-length check, so a character class mixing UTF-8 lengths admits a spurious continuation byte (predicted 0x80|(C>>12)) and can emit an overlong, invalid-UTF-8 string while the sampler signals EOS.", "result": "mixed", "verify": { "detail": "https://vanshverma.com/lab/grammarexact", "repo": "https://github.com/v-code01/grammarexact", "reproduce": "https://github.com/v-code01/grammarexact/blob/HEAD/run_bfs.py", "prereg": "https://github.com/v-code01/grammarexact/blob/HEAD/PREREG.md" } }, { "type": "experiment", "id": "kvleverage", "title": "kvleverage", "claim": "No. Leverage concentration of the real cache is at or below a matched-rank random cache (max/mean 1.12 to 1.46), and the sink is essentially never a top-5 leverage position except for 0.5B keys. KV rank is delocalized; the pre-registered spanning-anchor hypothesis is falsified. Attention mass and basis leverage are different things.", "result": "null", "verify": { "detail": "https://vanshverma.com/lab/kvleverage", "repo": "https://github.com/v-code01/kvleverage", "reproduce": "https://github.com/v-code01/kvleverage/blob/HEAD/run.py", "prereg": "https://github.com/v-code01/kvleverage/blob/HEAD/PREREG.md" } }, { "type": "experiment", "id": "logitcrystal", "title": "logitcrystal", "claim": "The lens top-1 is unrelated to the answer through most of depth then snaps in the last one or two layers: crystallization at layer 22.8, 26.2, 34.1 for the 24, 28, 36-layer Qwen2.5 models, always about two from the end. A tuned lens decodes no earlier. Relative-fraction versus constant-distance clock stays unresolved with three sizes.", "result": "mixed", "verify": { "detail": "https://vanshverma.com/lab/logitcrystal", "repo": "https://github.com/v-code01/logitcrystal", "reproduce": "https://github.com/v-code01/logitcrystal/blob/HEAD/run.py", "prereg": "https://github.com/v-code01/logitcrystal/blob/HEAD/PREREG.md" } }, { "type": "experiment", "id": "massiveact", "title": "massiveact", "claim": "Every size (0.5B, 1.5B, 3B) has one fixed channel at 600-950x the median RMS, with 100% of its mass on the first token. But the pre-registered relative-depth invariance is false: onset is pinned to absolute layer 2-3, not a constant fraction of depth.", "result": "mixed", "verify": { "detail": "https://vanshverma.com/lab/massiveact", "repo": "https://github.com/v-code01/massiveact", "reproduce": "https://github.com/v-code01/massiveact/blob/HEAD/run.py", "prereg": "https://github.com/v-code01/massiveact/blob/HEAD/PREREG.md" } }, { "type": "experiment", "id": "monoexp", "title": "monoexp", "claim": "Zero strict inversions over all 2,239,853,076 adjacent fp32 pairs in [-104, 88.72], despite the kernel's 1.45-ulp inaccuracy. It is exactly monotone non-decreasing, so softmax is rank-preserving and no smaller logit can overtake a larger one. The pre-registered prediction of reduction-seam inversions was falsified.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/monoexp", "repo": "https://github.com/v-code01/monoexp", "reproduce": "https://github.com/v-code01/monoexp/blob/HEAD/run.sh", "prereg": "https://github.com/v-code01/monoexp/blob/HEAD/PREREG.md" } }, { "type": "experiment", "id": "mpmcorder", "title": "mpmcorder", "claim": "Under herd7 on the RC11 model, acquire on the gate load and release on the gate store are each necessary and sufficient for the Vyukov ring, with no seq_cst required anywhere, including the cross-lap cell-reuse hazard. Loom exhibits the necessity races. The shipped queues confirm the gate but over-specify the counter with seq_cst.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/mpmcorder", "repo": "https://github.com/v-code01/mpmcorder", "reproduce": "https://github.com/v-code01/mpmcorder/blob/HEAD/run_sweep.py", "prereg": "https://github.com/v-code01/mpmcorder/blob/HEAD/PREREG.md" } }, { "type": "experiment", "id": "mpsaccum", "title": "mpsaccum", "claim": "It accumulates in fp32. The MPS fp16 error sits at 3.6e-4, flat in K and 72 sigma below the best-case pairwise fp16 accumulator floor, ruling out fp16 and tf32. fp16 throughput is only 1.10 to 1.12x fp32 at large sizes, nowhere near the folk 2x.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/mpsaccum", "repo": "https://github.com/v-code01/mpsaccum", "reproduce": "https://github.com/v-code01/mpsaccum/blob/HEAD/src/sweep.py", "prereg": "https://github.com/v-code01/mpsaccum/blob/HEAD/PREREG.md" } }, { "type": "experiment", "id": "mpsrecompile", "title": "mpsrecompile", "claim": "The first call at a never-seen shape costs a median 20-39x the steady per-call time (matmul 20x, softmax 39x, elementwise 34x), one-time and cached per shape. It is keyed on shape not size: two shapes of equal element count both pay. Larger matmuls reach 37-467x.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/mpsrecompile", "repo": "https://github.com/v-code01/mpsrecompile", "reproduce": "https://github.com/v-code01/mpsrecompile/blob/HEAD/run.py", "prereg": "https://github.com/v-code01/mpsrecompile/blob/HEAD/PREREG.md" } }, { "type": "experiment", "id": "overdisp", "title": "overdisp", "claim": "The greedy accept/reject stream is autocorrelated (pooled p11-p10 +0.155) and over-dispersed (run-length D/D_geom up to 24x), confirming it is not i.i.d. But the pre-registered consequence reverses: the optimal draft length moves down, not up. The marginal-alpha i.i.d. model over-predicts per-block yield (E_true/E_iid 0.883 at length 16).", "result": "mixed", "verify": { "detail": "https://vanshverma.com/lab/overdisp", "repo": "https://github.com/v-code01/overdisp", "reproduce": "https://github.com/v-code01/overdisp/blob/HEAD/reproduce.sh", "prereg": "https://github.com/v-code01/overdisp/blob/HEAD/PREREG.md" } }, { "type": "experiment", "id": "radixevict", "title": "radixevict", "claim": "No, contrary to the pre-registered prediction that it strictly costs. The leaf-only offline optimum equals the unrestricted optimum on all 2592 instances, and leaf-only LRU makes identical decisions to a sane unconstrained LRU on all 93312 instances (its interior evictions are only timestamp ties). The constraint is free offline and neutral online.", "result": "null", "verify": { "detail": "https://vanshverma.com/lab/radixevict", "repo": "https://github.com/v-code01/radixevict", "reproduce": "https://github.com/v-code01/radixevict/blob/HEAD/run.py", "prereg": "https://github.com/v-code01/radixevict/blob/HEAD/PREREG.md" } }, { "type": "experiment", "id": "residwrite", "title": "residwrite", "claim": "Qwen2.5's dominant sink channel is built up through the body and torn down by the final MLP, which writes against the channel's own sign while attention barely touches it. That one channel accounts for the apparent norm collapse: removing it, the final block's norm effect flips from x0.83 to x1.27 at 0.5B. Holds across 0.5B/1.5B/3B.", "result": "mixed", "verify": { "detail": "https://vanshverma.com/lab/residwrite", "repo": "https://github.com/v-code01/residwrite", "reproduce": "https://github.com/v-code01/residwrite/blob/HEAD/run.py", "prereg": "https://github.com/v-code01/residwrite/blob/HEAD/PREREG.md" } }, { "type": "experiment", "id": "ropelowrank", "title": "ropelowrank", "claim": "On Qwen2.5, pre-RoPE keys are strikingly low-rank (12 to 17 of 64 to 128 dims at 99% energy); RoPE inflates that 2.5x to 3.8x at 99% (5x to 8x at 90%), surviving mean-centering. Since RoPE is a per-row isometry, compressing pre-RoPE then rotating is free. The values-low-rank prediction is false: V exceeds pre-RoPE K.", "result": "mixed", "verify": { "detail": "https://vanshverma.com/lab/ropelowrank", "repo": "https://github.com/v-code01/ropelowrank", "reproduce": "https://github.com/v-code01/ropelowrank/blob/HEAD/run.py", "prereg": "https://github.com/v-code01/ropelowrank/blob/HEAD/PREREG.md" } }, { "type": "experiment", "id": "selfretrieval", "title": "selfretrieval", "claim": "Confirmed. A full self-retrieval census finds an irreducible floor of stored vectors no budget can return: at M=16, 795 of 100,000 (0.8 percent) returned 0 of 795 even at ef=8000. Almost all (789) have zero in-edges from the neighbor-selection heuristic; symmetrizing the graph at equal degree removes them.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/selfretrieval", "repo": "https://github.com/v-code01/selfretrieval", "reproduce": "https://github.com/v-code01/selfretrieval/blob/HEAD/run.py", "prereg": "https://github.com/v-code01/selfretrieval/blob/HEAD/PREREG.md" } }, { "type": "experiment", "id": "attnoracle", "title": "attnoracle", "claim": "No, and neither is the popular value-norm fix. An exact brute-force-optimal K-subset oracle on Qwen2.5 shows value-norm (w*||v||) is a trap (+55% downstream KL), while mass isn't globally optimal either -- the gap to optimum grows with context. All five pre-registered predictions were falsified.", "result": "null", "verify": { "detail": "https://vanshverma.com/lab/attnoracle", "repo": "https://github.com/v-code01/attnoracle", "prereg": "https://github.com/v-code01/attnoracle/blob/HEAD/PREREG.md" } }, { "type": "experiment", "id": "attnrank", "title": "attnrank", "claim": "No. On an exact SVD oracle over Qwen2.5, the 99%-energy rank grows LINEARLY with sequence length (alpha ~1.0), near-universally across heads -- softmax destroys the head-dim-bounded low rank of the QK^T scores. The fixed-k constant-rank premise is broken for causal decoders.", "result": "null", "verify": { "detail": "https://vanshverma.com/lab/attnrank", "repo": "https://github.com/v-code01/attnrank", "prereg": "https://github.com/v-code01/attnrank/blob/HEAD/PREREG.md" } }, { "type": "experiment", "id": "kvmech", "title": "kvmech", "claim": "Entirely outliers plus grouping, and the softmax DAMPS Keys rather than amplifying them -- a mechanistic correction verified to machine precision on an exact attention-output oracle. Under best-practice grouping, K is no more sensitive than V.", "result": "mixed", "verify": { "detail": "https://vanshverma.com/lab/kvmech", "repo": "https://github.com/v-code01/kvmech", "prereg": "https://github.com/v-code01/kvmech/blob/HEAD/PREREG.md" } }, { "type": "experiment", "id": "quantcompose", "title": "quantcompose", "claim": "The additive-sensitivity premise holds from Q8_0 to Q4_K_M (A within CI of 1) but leans superadditive at 2-bit (A=1.13): isolated per-layer sensitivity under-counts the damage at aggressive bit-widths. Exact full-vocab KL oracle; reverses the pre-registration.", "result": "mixed", "verify": { "detail": "https://vanshverma.com/lab/quantcompose", "repo": "https://github.com/v-code01/quantcompose", "prereg": "https://github.com/v-code01/quantcompose/blob/HEAD/PREREG.md" } }, { "type": "experiment", "id": "rooflineflip", "title": "rooflineflip", "claim": "On the M4, speculative decoding is a net loss for Q4_K_M models on both GPU and CPU (speedup 0.52-0.66, CIs exclude 1): quantization makes even the GPU compute-bound, erasing the verify-batch amortization spec-decode relies on. A verify_cost_ratio model predicts it within ~8%.", "result": "null", "verify": { "detail": "https://vanshverma.com/lab/rooflineflip", "repo": "https://github.com/v-code01/rooflineflip", "prereg": "https://github.com/v-code01/rooflineflip/blob/HEAD/PREREG.md" } }, { "type": "experiment", "id": "silent-fp16", "title": "silent-fp16", "claim": "In TRUE fp16 (~5 effective mantissa bits at K=65536), unlike NVIDIA tensor cores which contractually widen to fp32 -- undocumented until now. Declaring a float accumulator is honored, and mx.matmul already uses fp32. Exact oracle plus NEON brackets.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/silent-fp16", "repo": "https://github.com/v-code01/silent-fp16", "prereg": "https://github.com/v-code01/silent-fp16/blob/HEAD/PREREG.md" } }, { "type": "experiment", "id": "sink-value", "title": "sink-value", "claim": "Load-bearing. Zeroing the sink's value (key intact) raises Qwen2.5 perplexity +163-573%, specific to the sink and concentrated at layer 0 -- but via attention MASS, not a massive value norm. Exact oracle plus a code-disjoint verifier; adjudicates StreamingLLM vs Massive Activations.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/sink-value", "repo": "https://github.com/v-code01/sink-value", "prereg": "https://github.com/v-code01/sink-value/blob/HEAD/PREREG.md" } }, { "type": "experiment", "id": "spec-decode-acceptance", "title": "spec-decode-acceptance", "claim": "No -- on a matched draft/target pair, acceptance is constant with depth (constant-alpha, empirically exact). The real structure is a boundary cold-start, and induced decay is an off-distribution-context effect, not self-conditioning drift. Exact TV oracle on Qwen2.5 0.5B/1.5B via llama.cpp.", "result": "null", "verify": { "detail": "https://vanshverma.com/lab/spec-decode-acceptance", "repo": "https://github.com/v-code01/spec-decode-acceptance", "reproduce": "https://github.com/v-code01/spec-decode-acceptance/blob/HEAD/reproduce.sh", "prereg": "https://github.com/v-code01/spec-decode-acceptance/blob/HEAD/PREREG.md" } }, { "type": "experiment", "id": "sr-inference", "title": "sr-inference", "claim": "No -- it's the worst one. On real Qwen2.5 GEMV, SR is strictly worse than round-to-nearest: a sqrt(n) random walk with no averaging to redeem its unbiasedness. Deterministic Neumaier compensated summation wins 11-15x at lower cost. SR is a training tool, not an inference tool.", "result": "null", "verify": { "detail": "https://vanshverma.com/lab/sr-inference", "repo": "https://github.com/v-code01/sr-inference", "prereg": "https://github.com/v-code01/sr-inference/blob/HEAD/PREREG.md" } }, { "type": "experiment", "id": "super-weights-kquant", "title": "super-weights-kquant", "claim": "Capacity-gated: Qwen2.5-0.5B has one isolated super weight (6.4x PPL), the 1.5B has none. It hijacks a Q2_K block scale 131-204x, but production stores down_proj as Q3_K, so protecting it recovers only ~5% of the 2-bit loss. Bounds Yu et al. at sub-1B; exact 2-byte GGUF oracle.", "result": "mixed", "verify": { "detail": "https://vanshverma.com/lab/super-weights-kquant", "repo": "https://github.com/v-code01/super-weights-kquant", "reproduce": "https://github.com/v-code01/super-weights-kquant/blob/HEAD/reproduce.sh", "prereg": "https://github.com/v-code01/super-weights-kquant/blob/HEAD/PREREG.md" } }, { "type": "experiment", "id": "bf16-native-softmax", "title": "bf16-native-softmax", "claim": "Yes -- structural summation clears the fp32-accumulate bar within 2x, while compensated (Kahan) summation fails, because at bf16's 7-bit mantissa the correction word is as coarse as the sum itself. It inverts the fp64 textbook ordering of summation methods.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/bf16-native-softmax", "repo": "https://github.com/v-code01/bf16-native-softmax", "reproduce": "https://github.com/v-code01/bf16-native-softmax/blob/HEAD/reproduce.sh" } }, { "type": "experiment", "id": "fastinvsqrt", "title": "fastinvsqrt", "claim": "Obsolete -- but not because it's slow. It ties hardware vrsqrte on speed (asm shows the register round-trip is on the trick's side); vrsqrte is simply a strictly better seed, exactly one Newton step ahead. Superseded on accuracy, not speed.", "result": "null", "verify": { "detail": "https://vanshverma.com/lab/fastinvsqrt", "repo": "https://github.com/v-code01/fastinvsqrt", "prereg": "https://github.com/v-code01/fastinvsqrt/blob/HEAD/PREREG.md" } }, { "type": "experiment", "id": "attnscale", "title": "attnscale", "claim": "It keeps logits O(1) so exp doesn't overflow fp16 (unscaled logits cross the 11.09 threshold from d=64 up) and softmax doesn't collapse to one-hot -- the same fact the gradient argument names, seen from a 16-bit datapath. One predicted effect (a bf16 accuracy gain) was honestly falsified.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/attnscale", "repo": "https://github.com/v-code01/attnscale", "prereg": "https://github.com/v-code01/attnscale/blob/HEAD/PREREG.md" } }, { "type": "experiment", "id": "gelufloor", "title": "gelufloor", "claim": "It depends on your precision. Its error is 7x below the bf16 rounding floor (free there) but level with the fp16 floor (not free), and it peaks in the shoulder, not the tails. A precision accounting, not a benchmark.", "result": "mixed", "verify": { "detail": "https://vanshverma.com/lab/gelufloor", "repo": "https://github.com/v-code01/gelufloor", "prereg": "https://github.com/v-code01/gelufloor/blob/HEAD/PREREG.md" } }, { "type": "experiment", "id": "gpuqos", "title": "gpuqos", "claim": "BestEffort -- with the max kernel oom_score_adj (1000), making it the first OOM victim under memory pressure, because Kubernetes computes QoS from cpu/memory alone and ignores the GPU. Two independent signals (API qosClass + kernel oom_score_adj) on a real cluster: the scarce-GPU holder dies first.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/gpuqos", "repo": "https://github.com/v-code01/gpuqos", "claims": "https://github.com/v-code01/gpuqos/blob/HEAD/claims.toml", "reproduce": "https://github.com/v-code01/gpuqos/blob/HEAD/reproduce.sh", "prereg": "https://github.com/v-code01/gpuqos/blob/HEAD/PREREG.md", "review": "https://github.com/v-code01/gpuqos/blob/HEAD/REVIEW.md" } }, { "type": "experiment", "id": "initbill", "title": "initbill", "claim": "No -- it stays billed to the pod for its whole lifetime, because reserved is max(init, regular). An init that asked for 4 GPUs on a workload needing 1 strands 3 of 4, idle and unusable by others, long after init completed. Real cluster, independently verified.", "result": "null", "verify": { "detail": "https://vanshverma.com/lab/initbill", "repo": "https://github.com/v-code01/initbill", "claims": "https://github.com/v-code01/initbill/blob/HEAD/claims.toml", "reproduce": "https://github.com/v-code01/initbill/blob/HEAD/reproduce.sh", "prereg": "https://github.com/v-code01/initbill/blob/HEAD/PREREG.md", "review": "https://github.com/v-code01/initbill/blob/HEAD/REVIEW.md" } }, { "type": "experiment", "id": "kahansoftmax", "title": "kahansoftmax", "claim": "Not for the part that matters. The error splits into a large accumulation error -- which a compensated (Kahan) sum recovers 98-100% of in pure bf16 on diffuse distributions -- and a small term-quantization floor that only wider terms fix. Kahan in bf16 buys back almost all of it.", "result": "null", "verify": { "detail": "https://vanshverma.com/lab/kahansoftmax", "repo": "https://github.com/v-code01/kahansoftmax", "prereg": "https://github.com/v-code01/kahansoftmax/blob/HEAD/PREREG.md" } }, { "type": "experiment", "id": "neonbf16", "title": "neonbf16", "claim": "Round-to-nearest-even is exactly one bit more accurate than truncation (max rel error <=2^-8 vs <=2^-7) and free. But hand-writing a NEON kernel buys nothing (0.97-1.02x) -- the compiler already auto-vectorizes the bit-twiddle. Hand-SIMD only pays against an auto-vectorization barrier. asm-confirmed.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/neonbf16", "repo": "https://github.com/v-code01/neonbf16", "claims": "https://github.com/v-code01/neonbf16/blob/HEAD/claims.toml", "reproduce": "https://github.com/v-code01/neonbf16/blob/HEAD/reproduce.sh", "prereg": "https://github.com/v-code01/neonbf16/blob/HEAD/PREREG.md", "review": "https://github.com/v-code01/neonbf16/blob/HEAD/REVIEW.md" } }, { "type": "experiment", "id": "neonrmsnorm", "title": "neonrmsnorm", "claim": "Entirely the sum-of-squares reduction's 4-wide vector accumulator -- asm confirms the compiler won't reassociate an FP reduction at either precision, so NEON beats a float-accumulator scalar 2.5-3x. ~2x overall; rsqrt is free via 2 Newton steps. Independently verified.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/neonrmsnorm", "repo": "https://github.com/v-code01/neonrmsnorm", "claims": "https://github.com/v-code01/neonrmsnorm/blob/HEAD/claims.toml", "reproduce": "https://github.com/v-code01/neonrmsnorm/blob/HEAD/reproduce.sh", "prereg": "https://github.com/v-code01/neonrmsnorm/blob/HEAD/PREREG.md", "review": "https://github.com/v-code01/neonrmsnorm/blob/HEAD/REVIEW.md" } }, { "type": "experiment", "id": "neonswiglu", "title": "neonswiglu", "claim": "3.5-4.4x -- because the scalar sigmoid needs a non-inlinable expf call per element that blocks vectorization; replacing it with an inlined vector exp poly (~5e-8 accuracy) unblocks it. The gate-multiply fusion barely helps (compute-bound, not memory-bound). asm-confirmed, independently verified.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/neonswiglu", "repo": "https://github.com/v-code01/neonswiglu", "claims": "https://github.com/v-code01/neonswiglu/blob/HEAD/claims.toml", "reproduce": "https://github.com/v-code01/neonswiglu/blob/HEAD/reproduce.sh", "prereg": "https://github.com/v-code01/neonswiglu/blob/HEAD/PREREG.md", "review": "https://github.com/v-code01/neonswiglu/blob/HEAD/REVIEW.md" } }, { "type": "experiment", "id": "normfootgun", "title": "normfootgun", "claim": "Real but conditional. RMSNorm wins big under naive bf16 accumulation (its target is well-conditioned; LayerNorm's variance isn't) and it can't express the one-pass cancellation footgun -- but a correctly fp32-accumulated two-pass LayerNorm ties it. An implementation gap, not a law.", "result": "mixed", "verify": { "detail": "https://vanshverma.com/lab/normfootgun", "repo": "https://github.com/v-code01/normfootgun", "prereg": "https://github.com/v-code01/normfootgun/blob/HEAD/PREREG.md" } }, { "type": "experiment", "id": "softmaxdual", "title": "softmaxdual", "claim": "Because fp16 and bf16 fail at softmax for opposite reasons: fp16 overflows past logit ~11, while bf16's 7-bit mantissa swamps the sum so the output doesn't even total 1. And the famous max-subtraction trick only rescues fp16, not bf16 -- hence fp32.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/softmaxdual", "repo": "https://github.com/v-code01/softmaxdual", "prereg": "https://github.com/v-code01/softmaxdual/blob/HEAD/PREREG.md" } }, { "type": "experiment", "id": "stochround", "title": "stochround", "claim": "Stochastic rounding -- and it's the only unbiased one. Round-to-nearest summing ones freezes at 256 forever (a bf16 weight can't learn a sub-ulp gradient); SR tracks the true sum, noisy per run but unbiased in the mean. Deterministic runs verified bit-for-bit across two languages.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/stochround", "repo": "https://github.com/v-code01/stochround", "prereg": "https://github.com/v-code01/stochround/blob/HEAD/PREREG.md" } }, { "type": "experiment", "id": "cfsthrottle", "title": "cfsthrottle", "claim": "Badly -- a 60ms-CPU request's p99 balloons to 9.7x at a 100m limit, tracking the cgroup's throttled time and vanishing when the quota fits the burst, while the pod uses under 6% of the node. CFS per-period throttling. Real minikube (cgroup v2), pre-registered, independently verified.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/cfsthrottle", "repo": "https://github.com/v-code01/cfsthrottle", "claims": "https://github.com/v-code01/cfsthrottle/blob/HEAD/claims.toml", "reproduce": "https://github.com/v-code01/cfsthrottle/blob/HEAD/reproduce.sh", "prereg": "https://github.com/v-code01/cfsthrottle/blob/HEAD/PREREG.md", "review": "https://github.com/v-code01/cfsthrottle/blob/HEAD/REVIEW.md" } }, { "type": "experiment", "id": "dnsamp", "title": "dnsamp", "claim": "8 CoreDNS queries under the default ndots:5 -- 6 of them wasted cluster-domain NXDOMAINs, a 4x fan-out (from real CoreDNS logs). Both fixes -- a trailing-dot FQDN or dnsConfig ndots:1 -- cut it to 2. Real minikube, pre-registered, independently verified.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/dnsamp", "repo": "https://github.com/v-code01/dnsamp", "claims": "https://github.com/v-code01/dnsamp/blob/HEAD/claims.toml", "reproduce": "https://github.com/v-code01/dnsamp/blob/HEAD/reproduce.sh", "prereg": "https://github.com/v-code01/dnsamp/blob/HEAD/PREREG.md", "review": "https://github.com/v-code01/dnsamp/blob/HEAD/REVIEW.md" } }, { "type": "experiment", "id": "drain", "title": "drain", "claim": "Only if it finishes within terminationGracePeriodSeconds -- a 12s request dies at 4s grace but completes at 30s. Exit-on-SIGTERM always drops it, and a no-handler PID-1 server counterintuitively ignores SIGTERM entirely. Real minikube, pre-registered, independently verified.", "result": "mixed", "verify": { "detail": "https://vanshverma.com/lab/drain", "repo": "https://github.com/v-code01/drain", "claims": "https://github.com/v-code01/drain/blob/HEAD/claims.toml", "reproduce": "https://github.com/v-code01/drain/blob/HEAD/reproduce.sh", "prereg": "https://github.com/v-code01/drain/blob/HEAD/PREREG.md", "review": "https://github.com/v-code01/drain/blob/HEAD/REVIEW.md" } }, { "type": "experiment", "id": "fragfrontier", "title": "fragfrontier", "claim": "Permanently. With 4 free GPUs, a 4-GPU pod runs when they sit on one node but is Pending forever when split 2+2 -- the largest schedulable job is bounded by the most-free node (2), not the pool (4). Real 2-node cluster, independently verified.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/fragfrontier", "repo": "https://github.com/v-code01/fragfrontier", "claims": "https://github.com/v-code01/fragfrontier/blob/HEAD/claims.toml", "reproduce": "https://github.com/v-code01/fragfrontier/blob/HEAD/reproduce.sh", "prereg": "https://github.com/v-code01/fragfrontier/blob/HEAD/PREREG.md", "review": "https://github.com/v-code01/fragfrontier/blob/HEAD/REVIEW.md" } }, { "type": "experiment", "id": "gangdeadlock", "title": "gangdeadlock", "claim": "Yes -- the scheduling atom is one Pod, so two gangs in a 2/2 split strand all 4 GPUs with 0 gangs runnable, and the default scheduler neither prevents nor breaks it. One atomic multi-GPU pod per job is structurally immune. Real cluster, independently verified.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/gangdeadlock", "repo": "https://github.com/v-code01/gangdeadlock", "claims": "https://github.com/v-code01/gangdeadlock/blob/HEAD/claims.toml", "reproduce": "https://github.com/v-code01/gangdeadlock/blob/HEAD/reproduce.sh", "prereg": "https://github.com/v-code01/gangdeadlock/blob/HEAD/PREREG.md", "review": "https://github.com/v-code01/gangdeadlock/blob/HEAD/REVIEW.md" } }, { "type": "experiment", "id": "gpuhardwall", "title": "gpuhardwall", "claim": "No -- admission strictly precedes preemption. An exhausted namespace quota rejects the pod at creation, so a max-priority pod on a full node evicts nobody even though it would otherwise preempt 2 victims. Two hard walls, exact API-state counts, independently verified.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/gpuhardwall", "repo": "https://github.com/v-code01/gpuhardwall", "claims": "https://github.com/v-code01/gpuhardwall/blob/HEAD/claims.toml", "reproduce": "https://github.com/v-code01/gpuhardwall/blob/HEAD/reproduce.sh", "prereg": "https://github.com/v-code01/gpuhardwall/blob/HEAD/PREREG.md", "review": "https://github.com/v-code01/gpuhardwall/blob/HEAD/REVIEW.md" } }, { "type": "experiment", "id": "gpuoversub", "title": "gpuoversub", "claim": "Just the scheduler bin-packing an integer it can't see through: one GPU advertised as K units co-schedules exactly K 'dedicated'-GPU pods (there's no device-identity field in the API), and GPUs are integer-only with request==limit forced, unlike CPU. Real minikube, pre-registered, verified.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/gpuoversub", "repo": "https://github.com/v-code01/gpuoversub", "claims": "https://github.com/v-code01/gpuoversub/blob/HEAD/claims.toml", "reproduce": "https://github.com/v-code01/gpuoversub/blob/HEAD/reproduce.sh", "prereg": "https://github.com/v-code01/gpuoversub/blob/HEAD/PREREG.md", "review": "https://github.com/v-code01/gpuoversub/blob/HEAD/REVIEW.md" } }, { "type": "experiment", "id": "holblock", "title": "holblock", "claim": "No -- continuous (iteration-level) batching inflates a concurrent 8-token request by a flat ~1.5x regardless of the long generation's length (128 vs 1024 tokens), versus the 11-52x it would wait if queued behind it. The measured case for iteration-level batching. Pre-registered, independently verified.", "result": "null", "verify": { "detail": "https://vanshverma.com/lab/holblock", "repo": "https://github.com/v-code01/holblock", "claims": "https://github.com/v-code01/holblock/blob/HEAD/claims.toml", "reproduce": "https://github.com/v-code01/holblock/blob/HEAD/reproduce.sh", "prereg": "https://github.com/v-code01/holblock/blob/HEAD/PREREG.md", "review": "https://github.com/v-code01/holblock/blob/HEAD/REVIEW.md" } }, { "type": "experiment", "id": "hpalag", "title": "hpalag", "claim": "A median ~56 seconds -- ~40x longer than a pod takes to start -- because the delay is the metrics-scrape plus sync sampling loop, not pod startup. It scales only on the first scraped sample above target. Real minikube, pre-registered, independently verified.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/hpalag", "repo": "https://github.com/v-code01/hpalag", "claims": "https://github.com/v-code01/hpalag/blob/HEAD/claims.toml", "reproduce": "https://github.com/v-code01/hpalag/blob/HEAD/reproduce.sh", "prereg": "https://github.com/v-code01/hpalag/blob/HEAD/PREREG.md", "review": "https://github.com/v-code01/hpalag/blob/HEAD/REVIEW.md" } }, { "type": "experiment", "id": "kubelb", "title": "kubelb", "claim": "No -- iptables balances new connections fairly (Gini 0.11) but conntrack pins each persistent connection to one pod, so replica coverage follows N(1-(1-1/N)^K) and stays below N even at 2x replicas. Keep-alive starves replicas. Real minikube, pre-registered, independently verified.", "result": "mixed", "verify": { "detail": "https://vanshverma.com/lab/kubelb", "repo": "https://github.com/v-code01/kubelb", "claims": "https://github.com/v-code01/kubelb/blob/HEAD/claims.toml", "reproduce": "https://github.com/v-code01/kubelb/blob/HEAD/reproduce.sh", "prereg": "https://github.com/v-code01/kubelb/blob/HEAD/PREREG.md", "review": "https://github.com/v-code01/kubelb/blob/HEAD/REVIEW.md" } }, { "type": "experiment", "id": "topoblind", "title": "topoblind", "claim": "Nothing -- no device identity, no topology, not even a stored utilization figure; the control plane sees a fungible integer count. GPU 'utilization' is derived by summing pod requests, never stored. Four exact API-state facts on a real cluster, independently verified.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/topoblind", "repo": "https://github.com/v-code01/topoblind", "claims": "https://github.com/v-code01/topoblind/blob/HEAD/claims.toml", "reproduce": "https://github.com/v-code01/topoblind/blob/HEAD/reproduce.sh", "prereg": "https://github.com/v-code01/topoblind/blob/HEAD/PREREG.md", "review": "https://github.com/v-code01/topoblind/blob/HEAD/REVIEW.md" } }, { "type": "experiment", "id": "accumfrontier", "title": "accumfrontier", "claim": "Naive fp16 hits 85% error at vocab scale (fp32 stays under 1e-5); the cause is running-sum magnitude, and pairwise or Kahan summation rescue it where the values are representable. Pre-registered, independently verified against a double oracle in NumPy float16.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/accumfrontier", "repo": "https://github.com/v-code01/accumfrontier", "claims": "https://github.com/v-code01/accumfrontier/blob/HEAD/claims.toml", "reproduce": "https://github.com/v-code01/accumfrontier/blob/HEAD/reproduce.sh", "prereg": "https://github.com/v-code01/accumfrontier/blob/HEAD/PREREG.md", "review": "https://github.com/v-code01/accumfrontier/blob/HEAD/REVIEW.md" } }, { "type": "experiment", "id": "ctxconflict", "title": "ctxconflict", "claim": "One false sentence overrides known facts 18-26% of the time; forceful repetition pushes it to 77-100%. The smaller model is more suggestible, and obscure facts fall first. Real llama.cpp, judge-free, pre-registered, independently verified.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/ctxconflict", "repo": "https://github.com/v-code01/ctxconflict", "claims": "https://github.com/v-code01/ctxconflict/blob/HEAD/claims.toml", "reproduce": "https://github.com/v-code01/ctxconflict/blob/HEAD/reproduce.sh", "prereg": "https://github.com/v-code01/ctxconflict/blob/HEAD/PREREG.md", "review": "https://github.com/v-code01/ctxconflict/blob/HEAD/REVIEW.md" } }, { "type": "experiment", "id": "flopcross", "title": "flopcross", "claim": "The FFN owns ~88% of fixed per-token FLOPs, but attention grows linearly with context and overtakes it at exactly 1.5 x ffn_dim tokens (13,440 for the 1.5B, 7,296 for the 0.5B) -- a clean architectural constant, because d_model cancels. Go, exact, pre-registered.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/flopcross", "repo": "https://github.com/v-code01/flopcross", "claims": "https://github.com/v-code01/flopcross/blob/HEAD/claims.toml", "reproduce": "https://github.com/v-code01/flopcross/blob/HEAD/reproduce.sh", "prereg": "https://github.com/v-code01/flopcross/blob/HEAD/PREREG.md", "review": "https://github.com/v-code01/flopcross/blob/HEAD/REVIEW.md" } }, { "type": "experiment", "id": "geluapprox", "title": "geluapprox", "claim": "No. The sigmoid approximation is ~10x faster than exact-erf but 43x less faithful than the tanh approximation (2% vs 0.05% activation error), so tanh is the speed/accuracy sweet spot. The error is provably symmetric in the |x|~2-3 elbow. C++, pre-registered, code-reviewed.", "result": "null", "verify": { "detail": "https://vanshverma.com/lab/geluapprox", "repo": "https://github.com/v-code01/geluapprox", "claims": "https://github.com/v-code01/geluapprox/blob/HEAD/claims.toml", "reproduce": "https://github.com/v-code01/geluapprox/blob/HEAD/reproduce.sh", "prereg": "https://github.com/v-code01/geluapprox/blob/HEAD/PREREG.md", "review": "https://github.com/v-code01/geluapprox/blob/HEAD/REVIEW.md" } }, { "type": "experiment", "id": "genentropy", "title": "genentropy", "claim": "Less certain -- intrinsic next-token entropy rises over a generation, the opposite of the 'grows more committed' intuition. The smaller model is more diffuse, and the rise isn't degeneration. Python, pre-registered, code-reviewed.", "result": "null", "verify": { "detail": "https://vanshverma.com/lab/genentropy", "repo": "https://github.com/v-code01/genentropy", "claims": "https://github.com/v-code01/genentropy/blob/HEAD/claims.toml", "reproduce": "https://github.com/v-code01/genentropy/blob/HEAD/reproduce.sh", "prereg": "https://github.com/v-code01/genentropy/blob/HEAD/PREREG.md", "review": "https://github.com/v-code01/genentropy/blob/HEAD/REVIEW.md" } }, { "type": "experiment", "id": "kvdivergence", "title": "kvdivergence", "claim": "Yes -- q8_0 KV changes the generated text on 83% of prompts, q4_0 on 100% (often from the start), with flash attention held constant. KV precision alone changes what the model says: 8-bit KV is not lossless at the token level. Controlled, pre-registered, code-reviewed.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/kvdivergence", "repo": "https://github.com/v-code01/kvdivergence", "claims": "https://github.com/v-code01/kvdivergence/blob/HEAD/claims.toml", "reproduce": "https://github.com/v-code01/kvdivergence/blob/HEAD/reproduce.sh", "prereg": "https://github.com/v-code01/kvdivergence/blob/HEAD/PREREG.md", "review": "https://github.com/v-code01/kvdivergence/blob/HEAD/REVIEW.md" } }, { "type": "experiment", "id": "kvpaging", "title": "kvpaging", "claim": "Contiguous reservation wastes ~94% (worse the longer the context), so paging fits 18x-72x more concurrent sequences. Paging's own internal fragmentation is only ~block/2 tokens per sequence. Go, exact accounting, pre-registered.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/kvpaging", "repo": "https://github.com/v-code01/kvpaging", "claims": "https://github.com/v-code01/kvpaging/blob/HEAD/claims.toml", "reproduce": "https://github.com/v-code01/kvpaging/blob/HEAD/reproduce.sh", "prereg": "https://github.com/v-code01/kvpaging/blob/HEAD/PREREG.md", "review": "https://github.com/v-code01/kvpaging/blob/HEAD/REVIEW.md" } }, { "type": "experiment", "id": "neonrope", "title": "neonrope", "claim": "Precomputing the sin/cos angles is a ~66x kernel lever and the only one that matters; hand-vectorizing buys nothing (the compiler auto-vectorizes and it's load-bound); and RoPE is under 0.01% of a decode step anyway. Pre-registered, one prediction falsified, independently verified.", "result": "mixed", "verify": { "detail": "https://vanshverma.com/lab/neonrope", "repo": "https://github.com/v-code01/neonrope", "claims": "https://github.com/v-code01/neonrope/blob/HEAD/claims.toml", "reproduce": "https://github.com/v-code01/neonrope/blob/HEAD/reproduce.sh", "prereg": "https://github.com/v-code01/neonrope/blob/HEAD/PREREG.md", "review": "https://github.com/v-code01/neonrope/blob/HEAD/REVIEW.md" } }, { "type": "experiment", "id": "rmsnorm", "title": "rmsnorm", "claim": "Faster AND more accurate: NEON RMSNorm is ~2x the scalar f32 loop, and its 4-lane tree reduction rounds less than a sequential sum (shown to be the reduction order, not FMA). f32 accumulation error grows with hidden dim; f64 stays flat. Pre-registered, code-reviewed.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/rmsnorm", "repo": "https://github.com/v-code01/rmsnorm", "claims": "https://github.com/v-code01/rmsnorm/blob/HEAD/claims.toml", "reproduce": "https://github.com/v-code01/rmsnorm/blob/HEAD/reproduce.sh", "prereg": "https://github.com/v-code01/rmsnorm/blob/HEAD/PREREG.md", "review": "https://github.com/v-code01/rmsnorm/blob/HEAD/REVIEW.md" } }, { "type": "experiment", "id": "spacetax", "title": "spacetax", "claim": "It changes a byte-level-BPE model's tokenization and greedy output on 100% of prompts, and flips a third of factual answers wrong -- on a real llama.cpp server. One trailing space. Pre-registered, independently verified.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/spacetax", "repo": "https://github.com/v-code01/spacetax", "claims": "https://github.com/v-code01/spacetax/blob/HEAD/claims.toml", "reproduce": "https://github.com/v-code01/spacetax/blob/HEAD/reproduce.sh", "prereg": "https://github.com/v-code01/spacetax/blob/HEAD/PREREG.md", "review": "https://github.com/v-code01/spacetax/blob/HEAD/REVIEW.md" } }, { "type": "experiment", "id": "stragglerlat", "title": "stragglerlat", "claim": "A straggler taxes short-request latency (1.5x) via slot contention, not head-of-line blocking (shorts still finish first) -- but NOT more than an equal count of short requests: the matched control refutes the intuition. Real llama.cpp, pre-registered, code-reviewed.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/stragglerlat", "repo": "https://github.com/v-code01/stragglerlat", "claims": "https://github.com/v-code01/stragglerlat/blob/HEAD/claims.toml", "reproduce": "https://github.com/v-code01/stragglerlat/blob/HEAD/reproduce.sh", "prereg": "https://github.com/v-code01/stragglerlat/blob/HEAD/PREREG.md", "review": "https://github.com/v-code01/stragglerlat/blob/HEAD/REVIEW.md" } }, { "type": "experiment", "id": "anchoring", "title": "anchoring", "claim": "Anchoring is capability-gated. The 1.5B ignores it (pull slope 0.0003); the 0.5B gets dragged ~20% of the way toward the anchor (slope 0.196), more so the larger it is -- despite being told to ignore it. Exact oracle, independently verified.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/anchoring", "repo": "https://github.com/v-code01/anchoring", "claims": "https://github.com/v-code01/anchoring/blob/HEAD/claims.toml", "reproduce": "https://github.com/v-code01/anchoring/blob/HEAD/reproduce.sh", "review": "https://github.com/v-code01/anchoring/blob/HEAD/REVIEW.md" } }, { "type": "experiment", "id": "arithfrontier", "title": "arithfrontier", "claim": "Multiplication cliffs at 3-4 digits before addition does, the frontier shrinks with model size, and CoT extends it -- the opposite of its effect on retrieval multiple choice. Self-generated exact ground truth.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/arithfrontier", "repo": "https://github.com/v-code01/arithfrontier", "claims": "https://github.com/v-code01/arithfrontier/blob/HEAD/claims.toml", "reproduce": "https://github.com/v-code01/arithfrontier/blob/HEAD/reproduce.sh", "review": "https://github.com/v-code01/arithfrontier/blob/HEAD/REVIEW.md" } }, { "type": "experiment", "id": "basefrontier", "title": "basefrontier", "claim": "Capability-gated: the 1.5B has real frontiers (decimal->binary to 4 bits, binary->decimal to 5) while the 0.5B is at complete floor -- it can't convert even a small number. And binary->decimal is the easier direction. Both pre-registered predictions held. Exact oracle.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/basefrontier", "repo": "https://github.com/v-code01/basefrontier", "claims": "https://github.com/v-code01/basefrontier/blob/HEAD/claims.toml", "reproduce": "https://github.com/v-code01/basefrontier/blob/HEAD/reproduce.sh", "prereg": "https://github.com/v-code01/basefrontier/blob/HEAD/PREREG.md", "review": "https://github.com/v-code01/basefrontier/blob/HEAD/REVIEW.md" } }, { "type": "experiment", "id": "bitbudget", "title": "bitbudget", "claim": "The feed-forward is the bulk (71% of the 1.5B's bytes at 4.84 bits/weight), but the embedding/output is quantized far higher (6.56-7.00 bits) and dominates the small model -- jumping from 20% of the 1.5B's bytes to 49% of the 0.5B's. Both pre-registered predictions held.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/bitbudget", "repo": "https://github.com/v-code01/bitbudget", "claims": "https://github.com/v-code01/bitbudget/blob/HEAD/claims.toml", "reproduce": "https://github.com/v-code01/bitbudget/blob/HEAD/reproduce.sh", "prereg": "https://github.com/v-code01/bitbudget/blob/HEAD/PREREG.md", "review": "https://github.com/v-code01/bitbudget/blob/HEAD/REVIEW.md" } }, { "type": "experiment", "id": "bpelatency", "title": "bpelatency", "claim": "It scales exactly as predicted: naive ~L^2.0, heap ~L^1.1, and the naive/heap ratio grows unbounded -- 46-58x by L=1024 (a single ~1 KB piece). But naive still wins the common case (short real pieces), with a crossover in [8,64]. Validated against llama.cpp's tokenizer as a differential oracle; all predictions held.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/bpelatency", "repo": "https://github.com/v-code01/bpelatency", "claims": "https://github.com/v-code01/bpelatency/blob/HEAD/claims.toml", "reproduce": "https://github.com/v-code01/bpelatency/blob/HEAD/reproduce.sh", "prereg": "https://github.com/v-code01/bpelatency/blob/HEAD/PREREG.md", "review": "https://github.com/v-code01/bpelatency/blob/HEAD/REVIEW.md" } }, { "type": "experiment", "id": "calibration", "title": "calibration", "claim": "No -- systematically overconfident, measured from exact top-logprobs. A 2x2 (size x ARC difficulty) study: miscalibration compounds with both smaller size and harder task.", "result": "null", "verify": { "detail": "https://vanshverma.com/lab/calibration", "repo": "https://github.com/v-code01/calibration", "claims": "https://github.com/v-code01/calibration/blob/HEAD/claims.toml", "reproduce": "https://github.com/v-code01/calibration/blob/HEAD/reproduce.sh", "review": "https://github.com/v-code01/calibration/blob/HEAD/REVIEW.md" } }, { "type": "experiment", "id": "clockmod", "title": "clockmod", "claim": "Directly, it's at chance past a 2-hour jump. Writing the mod-12 reduction out loud rescues the capable model completely (frontier 2->10, p=7e-31) and barely helps the weak one. A 5th confirmation that CoT helps scalar-state tasks. Exact oracle.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/clockmod", "repo": "https://github.com/v-code01/clockmod", "claims": "https://github.com/v-code01/clockmod/blob/HEAD/claims.toml", "reproduce": "https://github.com/v-code01/clockmod/blob/HEAD/reproduce.sh", "review": "https://github.com/v-code01/clockmod/blob/HEAD/REVIEW.md" } }, { "type": "experiment", "id": "constraintcost", "title": "constraintcost", "claim": "Barely -- both pre-registered predictions were falsified. At matched token count the tax is ~1.02-1.04x for medium and complex grammars (only a simple grammar that closes early shows 1.79x). Constrained decoding is nearly free per token; the cost people fear isn't there.", "result": "null", "verify": { "detail": "https://vanshverma.com/lab/constraintcost", "repo": "https://github.com/v-code01/constraintcost", "claims": "https://github.com/v-code01/constraintcost/blob/HEAD/claims.toml", "reproduce": "https://github.com/v-code01/constraintcost/blob/HEAD/reproduce.sh", "prereg": "https://github.com/v-code01/constraintcost/blob/HEAD/PREREG.md", "review": "https://github.com/v-code01/constraintcost/blob/HEAD/REVIEW.md" } }, { "type": "experiment", "id": "cotbudget", "title": "cotbudget", "claim": "On two sizes, a low budget inverts the size ranking -- the smaller model wins when reasoning is starved. Exact-match oracle, independently verified.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/cotbudget", "repo": "https://github.com/v-code01/cotbudget", "claims": "https://github.com/v-code01/cotbudget/blob/HEAD/claims.toml", "reproduce": "https://github.com/v-code01/cotbudget/blob/HEAD/reproduce.sh", "review": "https://github.com/v-code01/cotbudget/blob/HEAD/REVIEW.md" } }, { "type": "experiment", "id": "cotmc", "title": "cotmc", "claim": "Hurts. A 2x2 (size x ARC difficulty) study: CoT is a net negative, flipping ~2x more correct answers wrong than it rescues (pooled McNemar p=0.005). Exact oracle.", "result": "null", "verify": { "detail": "https://vanshverma.com/lab/cotmc", "repo": "https://github.com/v-code01/cotmc", "claims": "https://github.com/v-code01/cotmc/blob/HEAD/claims.toml", "reproduce": "https://github.com/v-code01/cotmc/blob/HEAD/reproduce.sh", "review": "https://github.com/v-code01/cotmc/blob/HEAD/REVIEW.md" } }, { "type": "experiment", "id": "countcontrol", "title": "countcontrol", "claim": "Not without a counter -- it overshoots 'exactly 20' by tens and duplicates up to 44%. Numbering the list fixes it almost completely (McNemar p~1e-20). The counting analog of a running sum. Exact oracle.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/countcontrol", "repo": "https://github.com/v-code01/countcontrol", "claims": "https://github.com/v-code01/countcontrol/blob/HEAD/claims.toml", "reproduce": "https://github.com/v-code01/countcontrol/blob/HEAD/reproduce.sh", "review": "https://github.com/v-code01/countcontrol/blob/HEAD/REVIEW.md" } }, { "type": "experiment", "id": "ctxprefill", "title": "ctxprefill", "claim": "Real and measurable: per-token prefill cost rises quadratically (positive coefficient, tight CI excluding zero) -- ms/token climbs 20-31% from ~10k to ~40k tokens. Prefill stops being linear well before your context window does. All three predictions held.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/ctxprefill", "repo": "https://github.com/v-code01/ctxprefill", "claims": "https://github.com/v-code01/ctxprefill/blob/HEAD/claims.toml", "reproduce": "https://github.com/v-code01/ctxprefill/blob/HEAD/reproduce.sh", "prereg": "https://github.com/v-code01/ctxprefill/blob/HEAD/PREREG.md", "review": "https://github.com/v-code01/ctxprefill/blob/HEAD/REVIEW.md" } }, { "type": "experiment", "id": "datefrontier", "title": "datefrontier", "claim": "The 1.5B nails dates only ~2 weeks out, the 0.5B only 1 day. Its errors reveal it shifts months instead of counting days (30 days after Mar 15 -> Apr 15). Right nearby, wrong far out. Exact datetime oracle.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/datefrontier", "repo": "https://github.com/v-code01/datefrontier", "claims": "https://github.com/v-code01/datefrontier/blob/HEAD/claims.toml", "reproduce": "https://github.com/v-code01/datefrontier/blob/HEAD/reproduce.sh", "review": "https://github.com/v-code01/datefrontier/blob/HEAD/REVIEW.md" } }, { "type": "experiment", "id": "decodedrift", "title": "decodedrift", "claim": "Yes -- inter-token latency rises linearly with position (slope positive, CI excludes zero), it's material over a long generation, and the bigger model drifts faster in absolute terms. The decode-time complement to the prefill O(n^2) wall. All three predictions held.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/decodedrift", "repo": "https://github.com/v-code01/decodedrift", "claims": "https://github.com/v-code01/decodedrift/blob/HEAD/claims.toml", "reproduce": "https://github.com/v-code01/decodedrift/blob/HEAD/reproduce.sh", "prereg": "https://github.com/v-code01/decodedrift/blob/HEAD/PREREG.md", "review": "https://github.com/v-code01/decodedrift/blob/HEAD/REVIEW.md" } }, { "type": "experiment", "id": "depthfrontier", "title": "depthfrontier", "claim": "A working-memory limit -- both models fail past 3-4 terms. CoT rescues the capable model almost completely (frontier 4->16) but barely the weak one. Self-generated exact ground truth.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/depthfrontier", "repo": "https://github.com/v-code01/depthfrontier", "claims": "https://github.com/v-code01/depthfrontier/blob/HEAD/claims.toml", "reproduce": "https://github.com/v-code01/depthfrontier/blob/HEAD/reproduce.sh", "review": "https://github.com/v-code01/depthfrontier/blob/HEAD/REVIEW.md" } }, { "type": "experiment", "id": "dequant", "title": "dequant", "claim": "More than 2x the scalar throughput, bit-exact against the scalar reference (no approximation), consistent across matrix size -- unpacking a whole 16-byte block per pass. This is the dequant step the roofline study flagged as the streaming cost. Standalone C++ ARM NEON; all three pre-registered predictions held.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/dequant", "repo": "https://github.com/v-code01/dequant", "claims": "https://github.com/v-code01/dequant/blob/HEAD/claims.toml", "reproduce": "https://github.com/v-code01/dequant/blob/HEAD/reproduce.sh", "prereg": "https://github.com/v-code01/dequant/blob/HEAD/PREREG.md", "review": "https://github.com/v-code01/dequant/blob/HEAD/REVIEW.md" } }, { "type": "experiment", "id": "determinism", "title": "determinism", "claim": "Serial repeats are perfect (60/60 logit-identical), but concurrency breaks it: at 4 concurrent requests 0/8 are bit-identical -- only the sampled tokens match. Cross-batch and cache reuse perturb the logits too. So logprobs logged under load aren't reproducible, a hazard for calibration, perplexity, and reward scoring. Exact hashed-stream oracle with positive controls.", "result": "mixed", "verify": { "detail": "https://vanshverma.com/lab/determinism", "repo": "https://github.com/v-code01/determinism", "claims": "https://github.com/v-code01/determinism/blob/HEAD/claims.toml", "reproduce": "https://github.com/v-code01/determinism/blob/HEAD/reproduce.sh", "prereg": "https://github.com/v-code01/determinism/blob/HEAD/PREREG.md", "review": "https://github.com/v-code01/determinism/blob/HEAD/REVIEW.md" } }, { "type": "experiment", "id": "distractor", "title": "distractor", "claim": "Badly. One distractor sentence costs the 1.5B ~15% of its correct answers and the 0.5B ~half. A number-bearing distractor is no worse than a text-only one, so it's the irrelevant content, not stray digits. Exact oracle, independently verified.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/distractor", "repo": "https://github.com/v-code01/distractor", "claims": "https://github.com/v-code01/distractor/blob/HEAD/claims.toml", "reproduce": "https://github.com/v-code01/distractor/blob/HEAD/reproduce.sh", "review": "https://github.com/v-code01/distractor/blob/HEAD/REVIEW.md" } }, { "type": "experiment", "id": "fewshotcurve", "title": "fewshotcurve", "claim": "No significant gain on either Qwen2.5-1.5B or 0.5B (McNemar p=0.22, p=1.0) at 4-16x the prompt-token cost -- the near-null transfers across sizes. First cross-model-validated factory study. Honest negative.", "result": "null", "verify": { "detail": "https://vanshverma.com/lab/fewshotcurve", "repo": "https://github.com/v-code01/fewshotcurve", "claims": "https://github.com/v-code01/fewshotcurve/blob/HEAD/claims.toml", "reproduce": "https://github.com/v-code01/fewshotcurve/blob/HEAD/reproduce.sh", "review": "https://github.com/v-code01/fewshotcurve/blob/HEAD/REVIEW.md" } }, { "type": "experiment", "id": "flashsoftmax", "title": "flashsoftmax", "claim": "It's numerically identical to two-pass at every context length -- the streaming rescale accumulates nothing -- but ~2.8x slower standalone. Its win is fusion (avoiding a second pass over memory), not raw speed. Pre-registered, long-double oracle.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/flashsoftmax", "repo": "https://github.com/v-code01/flashsoftmax", "claims": "https://github.com/v-code01/flashsoftmax/blob/HEAD/claims.toml", "reproduce": "https://github.com/v-code01/flashsoftmax/blob/HEAD/reproduce.sh", "prereg": "https://github.com/v-code01/flashsoftmax/blob/HEAD/PREREG.md", "review": "https://github.com/v-code01/flashsoftmax/blob/HEAD/REVIEW.md" } }, { "type": "experiment", "id": "fusedgemv", "title": "fusedgemv", "claim": "Yes -- ~2.3x faster (5.1->11.7 GFLOP/s), bit-for-bit identical to the unfused path, consistent across shapes. Fusing avoids streaming the dequantized weights out to memory and back. The capstone of the kernel series; all three pre-registered predictions held.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/fusedgemv", "repo": "https://github.com/v-code01/fusedgemv", "claims": "https://github.com/v-code01/fusedgemv/blob/HEAD/claims.toml", "reproduce": "https://github.com/v-code01/fusedgemv/blob/HEAD/reproduce.sh", "prereg": "https://github.com/v-code01/fusedgemv/blob/HEAD/PREREG.md", "review": "https://github.com/v-code01/fusedgemv/blob/HEAD/REVIEW.md" } }, { "type": "experiment", "id": "gemvthreads", "title": "gemvthreads", "claim": "No, the surprise. The CPU GEMV scales near-linearly with cores and is compute-bound (implied bandwidth well below the ~216 GB/s ceiling); cache-resident and memory-resident matrices scale the same. The bandwidth wall is a system-level property, not in this kernel. All three pre-registered predictions held.", "result": "null", "verify": { "detail": "https://vanshverma.com/lab/gemvthreads", "repo": "https://github.com/v-code01/gemvthreads", "claims": "https://github.com/v-code01/gemvthreads/blob/HEAD/claims.toml", "reproduce": "https://github.com/v-code01/gemvthreads/blob/HEAD/reproduce.sh", "prereg": "https://github.com/v-code01/gemvthreads/blob/HEAD/PREREG.md", "review": "https://github.com/v-code01/gemvthreads/blob/HEAD/REVIEW.md" } }, { "type": "experiment", "id": "goodput", "title": "goodput", "claim": "There's a goodput knee at the slot count: throughput maxes at C=4 (211 tok/s on 1.5B, 572 on 0.5B), and C=8/12/16 add no throughput -- only latency, which grows roughly linearly (1.52s->4.54s on 1.5B). The smaller model has a higher ceiling but the same knee. All three predictions held.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/goodput", "repo": "https://github.com/v-code01/goodput", "claims": "https://github.com/v-code01/goodput/blob/HEAD/claims.toml", "reproduce": "https://github.com/v-code01/goodput/blob/HEAD/reproduce.sh", "prereg": "https://github.com/v-code01/goodput/blob/HEAD/PREREG.md", "review": "https://github.com/v-code01/goodput/blob/HEAD/REVIEW.md" } }, { "type": "experiment", "id": "kvmemory", "title": "kvmemory", "claim": "28 KB/token for the 1.5B, 12 KB/token for the 0.5B -- validated against the actual key-projection tensor, not estimated. A footnote at short context (0.06x weights at 2k) but it scales to model-size memory by long context. All three pre-registered predictions held; the config KV dim matched the tensor exactly.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/kvmemory", "repo": "https://github.com/v-code01/kvmemory", "claims": "https://github.com/v-code01/kvmemory/blob/HEAD/claims.toml", "reproduce": "https://github.com/v-code01/kvmemory/blob/HEAD/reproduce.sh", "prereg": "https://github.com/v-code01/kvmemory/blob/HEAD/PREREG.md", "review": "https://github.com/v-code01/kvmemory/blob/HEAD/REVIEW.md" } }, { "type": "experiment", "id": "kvquant", "title": "kvquant", "claim": "No -- both the speedup and the drift-reduction predictions were falsified. Quantizing the KV cache makes decode *slower*: the dequant cost outweighs the fewer bytes read. A counter-intuitive, honest null.", "result": "null", "verify": { "detail": "https://vanshverma.com/lab/kvquant", "repo": "https://github.com/v-code01/kvquant", "claims": "https://github.com/v-code01/kvquant/blob/HEAD/claims.toml", "reproduce": "https://github.com/v-code01/kvquant/blob/HEAD/reproduce.sh", "prereg": "https://github.com/v-code01/kvquant/blob/HEAD/PREREG.md", "review": "https://github.com/v-code01/kvquant/blob/HEAD/REVIEW.md" } }, { "type": "experiment", "id": "latencytail", "title": "latencytail", "claim": "Three regimes. Below the slot count, a batching tax -- per-token latency grows as the server decodes more active requests per step. Exactly at the slot count (C=4), a jitter spike: p99/p50 jumps to 2.24 (1.5B) and 2.67 (0.5B) versus ~1.2-1.3 elsewhere, the one concurrency where decode isn't smooth. Above it, queueing. Closed-loop C++ libcurl load generator.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/latencytail", "repo": "https://github.com/v-code01/latencytail", "claims": "https://github.com/v-code01/latencytail/blob/HEAD/claims.toml", "reproduce": "https://github.com/v-code01/latencytail/blob/HEAD/reproduce.sh", "prereg": "https://github.com/v-code01/latencytail/blob/HEAD/PREREG.md", "review": "https://github.com/v-code01/latencytail/blob/HEAD/REVIEW.md" } }, { "type": "experiment", "id": "layerprecision", "title": "layerprecision", "claim": "Yes, deliberately -- not random. Q4_K_M runs two precision tiers and puts exactly 50% of layers at the higher one (14/28 on the 1.5B, 12/24 on the 0.5B), protecting the front through to the back. The edge behavior is size-dependent: the 1.5B protects its last layer, the 0.5B doesn't. Both pre-registered predictions held.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/layerprecision", "repo": "https://github.com/v-code01/layerprecision", "claims": "https://github.com/v-code01/layerprecision/blob/HEAD/claims.toml", "reproduce": "https://github.com/v-code01/layerprecision/blob/HEAD/reproduce.sh", "prereg": "https://github.com/v-code01/layerprecision/blob/HEAD/PREREG.md", "review": "https://github.com/v-code01/layerprecision/blob/HEAD/REVIEW.md" } }, { "type": "experiment", "id": "logprobcost", "title": "logprobcost", "claim": "A fixed per-token overhead -- +6% (1.5B), +10% (0.5B) -- that does NOT grow with k (that prediction was falsified): once you pay for any logprobs, asking for more is nearly free. Exact server timing.", "result": "mixed", "verify": { "detail": "https://vanshverma.com/lab/logprobcost", "repo": "https://github.com/v-code01/logprobcost", "claims": "https://github.com/v-code01/logprobcost/blob/HEAD/claims.toml", "reproduce": "https://github.com/v-code01/logprobcost/blob/HEAD/reproduce.sh", "prereg": "https://github.com/v-code01/logprobcost/blob/HEAD/PREREG.md", "review": "https://github.com/v-code01/logprobcost/blob/HEAD/REVIEW.md" } }, { "type": "experiment", "id": "negation", "title": "negation", "claim": "Negation is capability-gated. A single 'not' halves the 0.5B (0.87->0.50) while the 1.5B is immune (1.00) -- until compound 'neither/nor' negation drops the 1.5B to 0.63 and the 0.5B to chance. Exact oracle, independent gold re-derivation.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/negation", "repo": "https://github.com/v-code01/negation", "claims": "https://github.com/v-code01/negation/blob/HEAD/claims.toml", "reproduce": "https://github.com/v-code01/negation/blob/HEAD/reproduce.sh", "review": "https://github.com/v-code01/negation/blob/HEAD/REVIEW.md" } }, { "type": "experiment", "id": "neonkernel", "title": "neonkernel", "claim": "int8 SDOT is the dominant throughput lever -- >=3x the best f32 kernel -- at a cost of <1% accuracy (0.045% relative-L2 vs an f64 reference). Hand-NEON f32 is 3.1x the scalar loop, but for int8 the hand-SDOT kernel is only 1.46x scalar: the compiler already vectorizes int8 well. Standalone C++ ARM NEON, bit-reproducible.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/neonkernel", "repo": "https://github.com/v-code01/neonkernel", "claims": "https://github.com/v-code01/neonkernel/blob/HEAD/claims.toml", "reproduce": "https://github.com/v-code01/neonkernel/blob/HEAD/reproduce.sh", "prereg": "https://github.com/v-code01/neonkernel/blob/HEAD/PREREG.md", "review": "https://github.com/v-code01/neonkernel/blob/HEAD/REVIEW.md" } }, { "type": "experiment", "id": "neonselect", "title": "neonselect", "claim": "A branchless NEON argmax is ~8x the scalar scan (29->3.7us at 32k vocab), top-k by partial selection is 17-62x cheaper than a full sort, and the ratio grows with vocabulary size. This is the kernel behind why top-k is near-free. Standalone C++ ARM NEON; all three pre-registered predictions held.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/neonselect", "repo": "https://github.com/v-code01/neonselect", "claims": "https://github.com/v-code01/neonselect/blob/HEAD/claims.toml", "reproduce": "https://github.com/v-code01/neonselect/blob/HEAD/reproduce.sh", "prereg": "https://github.com/v-code01/neonselect/blob/HEAD/PREREG.md", "review": "https://github.com/v-code01/neonselect/blob/HEAD/REVIEW.md" } }, { "type": "experiment", "id": "neonsoftmax", "title": "neonsoftmax", "claim": "More than 2x the scalar throughput across vocab sizes (32k-152k), at negligible cost -- relative-L2 under 1e-4 from the polynomial exp -- and consistent across vocabulary size. Standalone C++ ARM NEON; all three pre-registered predictions held.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/neonsoftmax", "repo": "https://github.com/v-code01/neonsoftmax", "claims": "https://github.com/v-code01/neonsoftmax/blob/HEAD/claims.toml", "reproduce": "https://github.com/v-code01/neonsoftmax/blob/HEAD/reproduce.sh", "prereg": "https://github.com/v-code01/neonsoftmax/blob/HEAD/PREREG.md", "review": "https://github.com/v-code01/neonsoftmax/blob/HEAD/REVIEW.md" } }, { "type": "experiment", "id": "ordereffect", "title": "ordereffect", "claim": "Both self-contradict on reversed comparisons (1.5B half the time, 0.5B 90%) -- and every contradiction is No-to-both: a negativity bias, not the acquiescence you'd assume. Judge-free consistency oracle.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/ordereffect", "repo": "https://github.com/v-code01/ordereffect", "claims": "https://github.com/v-code01/ordereffect/blob/HEAD/claims.toml", "reproduce": "https://github.com/v-code01/ordereffect/blob/HEAD/reproduce.sh", "review": "https://github.com/v-code01/ordereffect/blob/HEAD/REVIEW.md" } }, { "type": "experiment", "id": "parallelscale", "title": "parallelscale", "claim": "Throughput doesn't monotonically saturate -- it plateaus then jumps (prediction 1 falsified). On the 1.5B: +33% at 2 slots, then only +3% at 4 and 8; the 0.5B keeps gaining (+64/+35/+11%). More slots also split the per-slot context (4096->1024). Predictions 2-4 held.", "result": "mixed", "verify": { "detail": "https://vanshverma.com/lab/parallelscale", "repo": "https://github.com/v-code01/parallelscale", "claims": "https://github.com/v-code01/parallelscale/blob/HEAD/claims.toml", "reproduce": "https://github.com/v-code01/parallelscale/blob/HEAD/reproduce.sh", "prereg": "https://github.com/v-code01/parallelscale/blob/HEAD/PREREG.md", "review": "https://github.com/v-code01/parallelscale/blob/HEAD/REVIEW.md" } }, { "type": "experiment", "id": "positionbias", "title": "positionbias", "claim": "Yes, and it scales inversely with capability. A cross-model ARC-Easy rotation study: the weak model systematically avoids the last option. Exact oracle.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/positionbias", "repo": "https://github.com/v-code01/positionbias", "claims": "https://github.com/v-code01/positionbias/blob/HEAD/claims.toml", "reproduce": "https://github.com/v-code01/positionbias/blob/HEAD/reproduce.sh", "review": "https://github.com/v-code01/positionbias/blob/HEAD/REVIEW.md" } }, { "type": "experiment", "id": "ratio", "title": "ratio", "claim": "Magnitude dominates for both models -- accuracy roughly halves from small to large answers. A fractional unit price is a non-issue for the 1.5B (falsifying 'fractions are hard') but crushes the 0.5B at small magnitude (0.31->0.06). Capability-gated, and only visible once the magnitude confound is controlled. Exact oracle.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/ratio", "repo": "https://github.com/v-code01/ratio", "claims": "https://github.com/v-code01/ratio/blob/HEAD/claims.toml", "reproduce": "https://github.com/v-code01/ratio/blob/HEAD/reproduce.sh", "review": "https://github.com/v-code01/ratio/blob/HEAD/REVIEW.md" } }, { "type": "experiment", "id": "revfrontier", "title": "revfrontier", "claim": "Reversal is pure tracking, no comparisons: it collapses after ~5 elements by dropping numbers, and CoT does not extend the frontier -- it hurts the capable model. A 2nd whole-list refutation of when step-by-step helps. Exact oracle.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/revfrontier", "repo": "https://github.com/v-code01/revfrontier", "claims": "https://github.com/v-code01/revfrontier/blob/HEAD/claims.toml", "reproduce": "https://github.com/v-code01/revfrontier/blob/HEAD/reproduce.sh", "review": "https://github.com/v-code01/revfrontier/blob/HEAD/REVIEW.md" } }, { "type": "experiment", "id": "riskcoverage", "title": "riskcoverage", "claim": "Yes. A 2x2 (size x ARC difficulty) selective-prediction study: confidence ranks correctness (AUROC 0.68-0.91), so abstention lifts accuracy in every regime -- even where calibration is poor. Exact oracle.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/riskcoverage", "repo": "https://github.com/v-code01/riskcoverage", "claims": "https://github.com/v-code01/riskcoverage/blob/HEAD/claims.toml", "reproduce": "https://github.com/v-code01/riskcoverage/blob/HEAD/reproduce.sh", "review": "https://github.com/v-code01/riskcoverage/blob/HEAD/REVIEW.md" } }, { "type": "experiment", "id": "roofline", "title": "roofline", "claim": "Decode sits far below the roofline ridge (arithmetic intensity ~3 FLOP/byte vs 9.0), so it's memory-bandwidth-bound. On an M4 (221 GB/s STREAM ceiling) a single 1.5B stream already uses 67% of peak bandwidth; batching to 4 raises it to 92% and 207 tok/s but stays memory-bound -- it amortizes the weight read, it doesn't escape it. All three pre-registered predictions held.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/roofline", "repo": "https://github.com/v-code01/roofline", "claims": "https://github.com/v-code01/roofline/blob/HEAD/claims.toml", "reproduce": "https://github.com/v-code01/roofline/blob/HEAD/reproduce.sh", "prereg": "https://github.com/v-code01/roofline/blob/HEAD/PREREG.md", "review": "https://github.com/v-code01/roofline/blob/HEAD/REVIEW.md" } }, { "type": "experiment", "id": "ropeprecision", "title": "ropeprecision", "claim": "Yes -- naive f32 error grows linearly with position (~3e-3 at 128k tokens). Reducing the angle mod 2pi in f64 fixes it entirely (>10000x better). Rust, f64 oracle, pre-registered.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/ropeprecision", "repo": "https://github.com/v-code01/ropeprecision", "claims": "https://github.com/v-code01/ropeprecision/blob/HEAD/claims.toml", "reproduce": "https://github.com/v-code01/ropeprecision/blob/HEAD/reproduce.sh", "prereg": "https://github.com/v-code01/ropeprecision/blob/HEAD/PREREG.md", "review": "https://github.com/v-code01/ropeprecision/blob/HEAD/REVIEW.md" } }, { "type": "experiment", "id": "samplercost", "title": "samplercost", "claim": "top-p is the costly one; top-k and min-p are near free; putting top-k before top-p removes top-p's cost (it shrinks the candidate set first); and top-p's overhead is larger for the smaller model. All four pre-registered predictions held.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/samplercost", "repo": "https://github.com/v-code01/samplercost", "claims": "https://github.com/v-code01/samplercost/blob/HEAD/claims.toml", "reproduce": "https://github.com/v-code01/samplercost/blob/HEAD/reproduce.sh", "prereg": "https://github.com/v-code01/samplercost/blob/HEAD/PREREG.md", "review": "https://github.com/v-code01/samplercost/blob/HEAD/REVIEW.md" } }, { "type": "experiment", "id": "selfconsistency", "title": "selfconsistency", "claim": "Cross-model, exact-match oracle: it beats greedy on the 1.5B (+13 points, p=0.002) but the 0.5B's gain isn't significant. The benefit traces to answer-entropy -- the mechanism, not just the score.", "result": "mixed", "verify": { "detail": "https://vanshverma.com/lab/selfconsistency", "repo": "https://github.com/v-code01/selfconsistency", "claims": "https://github.com/v-code01/selfconsistency/blob/HEAD/claims.toml", "reproduce": "https://github.com/v-code01/selfconsistency/blob/HEAD/reproduce.sh", "review": "https://github.com/v-code01/selfconsistency/blob/HEAD/REVIEW.md" } }, { "type": "experiment", "id": "sortfrontier", "title": "sortfrontier", "claim": "The failure mode shifts from comparison errors to tracking errors -- dropped or invented numbers, ~97% by length 16. And CoT doesn't help sorting; it hurts the small model. Exact oracle.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/sortfrontier", "repo": "https://github.com/v-code01/sortfrontier", "claims": "https://github.com/v-code01/sortfrontier/blob/HEAD/claims.toml", "reproduce": "https://github.com/v-code01/sortfrontier/blob/HEAD/reproduce.sh", "review": "https://github.com/v-code01/sortfrontier/blob/HEAD/REVIEW.md" } }, { "type": "experiment", "id": "sycophancy", "title": "sycophancy", "claim": "Both do, but differently: the 0.5B is a suggestible follower (adopts the user's wrong answer), the 1.5B an unstable contrarian (caves under any challenge, most when affirmed). The control dissociates the two. Exact oracle.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/sycophancy", "repo": "https://github.com/v-code01/sycophancy", "claims": "https://github.com/v-code01/sycophancy/blob/HEAD/claims.toml", "reproduce": "https://github.com/v-code01/sycophancy/blob/HEAD/reproduce.sh", "review": "https://github.com/v-code01/sycophancy/blob/HEAD/REVIEW.md" } }, { "type": "experiment", "id": "tokenasym", "title": "tokenasym", "claim": "An order of magnitude more: 12.8x for the 1.5B, 22.0x for the 0.5B (input 0.48/0.17ms vs output 6.18/3.69ms), and the gap grows with prompt length. Prefill is cheap; generation is where the bill is. All three pre-registered predictions held.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/tokenasym", "repo": "https://github.com/v-code01/tokenasym", "claims": "https://github.com/v-code01/tokenasym/blob/HEAD/claims.toml", "reproduce": "https://github.com/v-code01/tokenasym/blob/HEAD/reproduce.sh", "prereg": "https://github.com/v-code01/tokenasym/blob/HEAD/PREREG.md", "review": "https://github.com/v-code01/tokenasym/blob/HEAD/REVIEW.md" } }, { "type": "experiment", "id": "transitive", "title": "transitive", "claim": "It spots endpoints by surface pattern, not transitive inference. The 1.5B finds the tallest/shortest well above chance but is at chance for the second-tallest (gap 0.65); the 0.5B is at chance throughout. Exact MC oracle.", "result": "null", "verify": { "detail": "https://vanshverma.com/lab/transitive", "repo": "https://github.com/v-code01/transitive", "claims": "https://github.com/v-code01/transitive/blob/HEAD/claims.toml", "reproduce": "https://github.com/v-code01/transitive/blob/HEAD/reproduce.sh", "review": "https://github.com/v-code01/transitive/blob/HEAD/REVIEW.md" } }, { "type": "experiment", "id": "vartrack", "title": "vartrack", "claim": "Not past 2-3 lines -- the indirection collapses it, not the arithmetic. Writing out each variable's value rescues the capable model almost completely but not the weak one. A 4th confirmation of when CoT helps. Exact executor oracle.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/vartrack", "repo": "https://github.com/v-code01/vartrack", "claims": "https://github.com/v-code01/vartrack/blob/HEAD/claims.toml", "reproduce": "https://github.com/v-code01/vartrack/blob/HEAD/reproduce.sh", "review": "https://github.com/v-code01/vartrack/blob/HEAD/REVIEW.md" } }, { "type": "experiment", "id": "vocabstruct", "title": "vocabstruct", "claim": "One shared, byte-complete, merge-dominated object: byte-for-byte identical across the 1.5B and 0.5B (same 151,936 tokens, same hash), lossless, 99.81% learned BPE merges on a 256-token byte base. A full census, every number exact -- 3 of 4 predictions held (the length distribution wasn't unimodal).", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/vocabstruct", "repo": "https://github.com/v-code01/vocabstruct", "claims": "https://github.com/v-code01/vocabstruct/blob/HEAD/claims.toml", "reproduce": "https://github.com/v-code01/vocabstruct/blob/HEAD/reproduce.sh", "prereg": "https://github.com/v-code01/vocabstruct/blob/HEAD/PREREG.md", "review": "https://github.com/v-code01/vocabstruct/blob/HEAD/REVIEW.md" } }, { "type": "experiment", "id": "admitctl", "title": "admitctl", "claim": "A lot. KV-slot-aware SJF admission in front of a real llama.cpp server cuts short-request p99 ~16x at matched throughput under overload, without starving long requests. FCFS leaves that on the table.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/admitctl", "repo": "https://github.com/v-code01/admitctl", "claims": "https://github.com/v-code01/admitctl/blob/HEAD/claims.toml", "reproduce": "https://github.com/v-code01/admitctl/blob/HEAD/reproduce.sh" } }, { "type": "experiment", "id": "batchscale", "title": "batchscale", "claim": "No -- it's non-monotone with a regime change. Throughput plateaus ~220 tok/s through concurrency 8, then jumps to 522 at 16. Concurrency 8, a partial batch, is the worst place to operate: mediocre throughput and the highest latency.", "result": "null", "verify": { "detail": "https://vanshverma.com/lab/batchscale", "repo": "https://github.com/v-code01/batchscale", "claims": "https://github.com/v-code01/batchscale/blob/HEAD/claims.toml", "reproduce": "https://github.com/v-code01/batchscale/blob/HEAD/reproduce.sh" } }, { "type": "experiment", "id": "ctxprobe", "title": "ctxprobe", "claim": "On a real 32k-window model, retrieval degrades past ~17k and shows a clean lost-in-the-middle U-shape -- compounding to 0.30 at 30k-middle versus 0.90 at the start. Length and burial stack.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/ctxprobe", "repo": "https://github.com/v-code01/ctxprobe", "claims": "https://github.com/v-code01/ctxprobe/blob/HEAD/claims.toml", "reproduce": "https://github.com/v-code01/ctxprobe/blob/HEAD/reproduce.sh" } }, { "type": "experiment", "id": "effortfrontier", "title": "effortfrontier", "claim": "Honest negative: on GSM8K against the exact gold label, a cheap greedy allocation does not beat uniform at matched budget -- despite ~9x oracle headroom. The gain is real; a simple policy doesn't reach it.", "result": "null", "verify": { "detail": "https://vanshverma.com/lab/effortfrontier", "repo": "https://github.com/v-code01/effortfrontier", "claims": "https://github.com/v-code01/effortfrontier/blob/HEAD/claims.toml", "reproduce": "https://github.com/v-code01/effortfrontier/blob/HEAD/reproduce.sh" } }, { "type": "experiment", "id": "kvbits", "title": "kvbits", "claim": "No. On real llama.cpp the key cache needs >=8 bits while the value cache stays lossless at 4-bit. Asymmetric Kq8/Vq4 buys a 59% KV reduction at <1% perplexity; symmetric 4-bit is catastrophic.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/kvbits", "repo": "https://github.com/v-code01/kvbits", "claims": "https://github.com/v-code01/kvbits/blob/HEAD/claims.toml", "reproduce": "https://github.com/v-code01/kvbits/blob/HEAD/reproduce.sh" } }, { "type": "experiment", "id": "prefillcache", "title": "prefillcache", "claim": "Warm prefill stays flat at ~8-11ms while cold grows linearly, so a reused shared prefix goes from an 8x speedup at 114 tokens to 81x at 1841 -- 88-99% of prefill eliminated, nearly free after the first request.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/prefillcache", "repo": "https://github.com/v-code01/prefillcache", "claims": "https://github.com/v-code01/prefillcache/blob/HEAD/claims.toml", "reproduce": "https://github.com/v-code01/prefillcache/blob/HEAD/reproduce.sh" } }, { "type": "experiment", "id": "promptbrittle", "title": "promptbrittle", "claim": "On GSM8K, semantically-equivalent formats swing accuracy 25 points (0.42 to 0.68) on the same questions (McNemar p 0.006), with reasoning elicitation the biggest lever. The wording is a hyperparameter.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/promptbrittle", "repo": "https://github.com/v-code01/promptbrittle", "claims": "https://github.com/v-code01/promptbrittle/blob/HEAD/claims.toml", "reproduce": "https://github.com/v-code01/promptbrittle/blob/HEAD/reproduce.sh" } }, { "type": "experiment", "id": "specdomain", "title": "specdomain", "claim": "Acceptance is strongly domain-gated -- 32% on chat, 85% on math -- but on a 0.5B/1.5B pair speculation is a net loss: the target-to-draft cost gap is too small to pay for the misses. Honest negative.", "result": "null", "verify": { "detail": "https://vanshverma.com/lab/specdomain", "repo": "https://github.com/v-code01/specdomain", "claims": "https://github.com/v-code01/specdomain/blob/HEAD/claims.toml", "reproduce": "https://github.com/v-code01/specdomain/blob/HEAD/reproduce.sh" } }, { "type": "experiment", "id": "tempfrontier", "title": "tempfrontier", "claim": "Self-consistency needs temperature -- no benefit at temp 0. Majority-voting k samples gains +16 points at temp 0.8-1.0 while single-sample accuracy stays flat. The exact temperature tradeoff on GSM8K.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/tempfrontier", "repo": "https://github.com/v-code01/tempfrontier", "claims": "https://github.com/v-code01/tempfrontier/blob/HEAD/claims.toml", "reproduce": "https://github.com/v-code01/tempfrontier/blob/HEAD/reproduce.sh" } }, { "type": "experiment", "id": "tokentax", "title": "tokentax", "claim": "The tokenization tax, on a real tokenizer: numbers cost 5x the tokens-per-character of English prose, JSON 2.4x, math 2.2x, code 1.75x, non-English 1.5x. Your context budget is content-dependent.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/tokentax", "repo": "https://github.com/v-code01/tokentax", "claims": "https://github.com/v-code01/tokentax/blob/HEAD/claims.toml", "reproduce": "https://github.com/v-code01/tokentax/blob/HEAD/reproduce.sh" } }, { "type": "experiment", "id": "weightquant", "title": "weightquant", "claim": "Q4_K_M is the Pareto sweet spot -- near-lossless at 32% of f16 size. Everything above it is lossless within noise; Q2_K is a cliff at +90% perplexity. The rate-distortion frontier, measured on a real model.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/weightquant", "repo": "https://github.com/v-code01/weightquant", "claims": "https://github.com/v-code01/weightquant/blob/HEAD/claims.toml", "reproduce": "https://github.com/v-code01/weightquant/blob/HEAD/reproduce.sh" } }, { "type": "experiment", "id": "doobspec", "title": "doobspec", "claim": "Exact measurement of the future-validity bias greedy constrained decoding introduces, plus a boundary on whether a depth-1 correction actually buys JSON-Schema validity at CPU scale -- where it earns its cost and where it doesn't.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/doobspec", "repo": "https://github.com/v-code01/doobspec", "claims": "https://github.com/v-code01/doobspec/blob/HEAD/claims.toml" } }, { "type": "experiment", "id": "prefixfair", "title": "prefixfair", "claim": "The cache-hit vs cross-tenant service-gap Pareto frontier across five routing policies, measured on real llama.cpp backends. Honest either way.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/prefixfair", "repo": "https://github.com/v-code01/prefixfair", "claims": "https://github.com/v-code01/prefixfair/blob/HEAD/claims.toml" } }, { "type": "experiment", "id": "toolfetch", "title": "toolfetch", "claim": "The measured retrieve-vs-inject frontier on a CPU-local model: how many tools you can put in context before it stops picking the right one. Real ToolRet labels, exact-match scoring, no LLM judge.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/toolfetch", "repo": "https://github.com/v-code01/toolfetch" } }, { "type": "experiment", "id": "crosskv", "title": "crosskv", "claim": "On a real 12B transformer, coupling the eviction and quantization budgets beats separable allocation held-out at equal budget for SnapKV, and is a wash for H2O. The interaction is real but evictor-dependent.", "result": "mixed", "verify": { "detail": "https://vanshverma.com/lab/crosskv", "repo": "https://github.com/v-code01/crosskv", "claims": "https://github.com/v-code01/crosskv/blob/HEAD/claims.toml" } }, { "type": "experiment", "id": "calibann", "title": "calibann", "claim": "It's silently unsafe. A calibrated, safety-gated gate on a binary-quantized ANN core turns served-error into a target you control instead of a number you hope about.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/calibann", "repo": "https://github.com/v-code01/calibann", "claims": "https://github.com/v-code01/calibann/blob/HEAD/claims.toml" } }, { "type": "experiment", "id": "cdcneon", "title": "cdcneon", "claim": "SeqCDC on a NEON fast path hits ~19 GiB/s on an M4. Gear stays the default: it dedups better and degrades more gracefully. blake3 content-addressed store underneath.", "result": "mixed", "verify": { "detail": "https://vanshverma.com/lab/cdcneon", "repo": "https://github.com/v-code01/cdcneon", "claims": "https://github.com/v-code01/cdcneon/blob/HEAD/claims.toml" } }, { "type": "experiment", "id": "circ-das", "title": "circ-das", "claim": "At high rate, block-circulant beats 2D-RS distance. First implementation and honest measurement, with a NEON GF(2^8) encoder and a coded-Merkle DAS sampler.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/circ-das", "repo": "https://github.com/v-code01/circ-das", "claims": "https://github.com/v-code01/circ-das/blob/HEAD/claims.toml" } }, { "type": "experiment", "id": "funnelscan", "title": "funnelscan", "claim": "A NEON group-probe funnel table sustains 99.9% load with bounded p99.9 probes, in a smaller footprint than a SwissTable forced to resize.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/funnelscan", "repo": "https://github.com/v-code01/funnelscan", "claims": "https://github.com/v-code01/funnelscan/blob/HEAD/claims.toml" } }, { "type": "experiment", "id": "ribbonguard", "title": "ribbonguard", "claim": "Yes. A NEON blocked filter fused with a skew-adaptive false-positive suppressor holds the no-false-negative invariant, exhaustively checked, while cutting false positives where the skew concentrates.", "result": "confirmed", "verify": { "detail": "https://vanshverma.com/lab/ribbonguard", "repo": "https://github.com/v-code01/ribbonguard", "claims": "https://github.com/v-code01/ribbonguard/blob/HEAD/claims.toml" } }, { "type": "product", "id": "keel", "title": "Keel", "claim": "An in-browser, WebGPU-accelerated RAG platform that runs entirely client-side. Live in production.", "verify": { "live": "https://keel.vanshverma.com" } }, { "type": "open-source", "id": "ledge", "title": "Ledge", "claim": "Git-compatible storage engine rebuilt for agent workloads: faster clone and smaller packs than git on the same source, BLAKE3 content addressing, sharded Raft replication with a TLA+-verified consensus core (5 modules, model-checked), driven by a stock git client. Source-available, in Rust.", "verify": { "repo": "https://github.com/v-code01/ledge" } }, { "type": "open-source", "id": "phantom", "title": "PHANTOM", "claim": "Multi-agent LLM serving for Apple Silicon unified memory. Eliminates PCIe weight copies; DualRadixTree copy-on-write KV cache; MESI coherence formally specified in TLA+.", "verify": { "repo": "https://github.com/v-code01/phantom" } }, { "type": "open-source", "id": "nemesis", "title": "NEMESIS", "claim": "Autonomous GPU cluster orchestration. Replaces on-call SRE judgment with specialized agents that perceive hardware degradation before failure. Topology-aware scheduling; heals running training jobs without restart via NCCL 2.27 Communicator Shrink. Validated against the Alibaba Cluster Trace dataset.", "verify": { "repo": "https://github.com/v-code01/nemesis" } }, { "type": "open-source", "id": "tasft", "title": "TASFT", "claim": "Task-Aware Sparse Fine-Tuning. Co-trains LoRA adapters with block-sparse attention gates for 2-5x decode throughput at 70-85% sparsity. 676 tests passing.", "verify": { "repo": "https://github.com/v-code01/tasft" } }, { "type": "open-source", "id": "kubebalance", "title": "KubeBalance", "claim": "Kubernetes scheduler plugin — network topology-aware, cost-based, performance-driven pod placement.", "verify": { "repo": "https://github.com/v-code01/kubebalance" } }, { "type": "open-source", "id": "airflowllm", "title": "AirflowLLM", "claim": "Generate production-ready Airflow DAGs from natural language. 45 tokens/sec on CodeLlama 7B, ~700ms on an M2 Pro, fully local — no API calls.", "verify": { "repo": "https://github.com/v-code01/airflow-llm-orchestrator" } }, { "type": "open-source", "id": "edgetrain", "title": "EdgeTrain", "claim": "Neural-network training in the browser via WebGPU compute shaders. No server, no Python.", "verify": { "repo": "https://github.com/v-code01/edgetrain" } } ] }