Research update · October 18, 2026

Hatsunama Small-Model Research: Balanced vs Repair-Weighted Training

Hatsunama’s latest small-model AI experiment found 8/8 development-task passes with balanced training and 6/8 with repair-weighted training on LiquidAI/LFM2.5-2.6B.

I compared two LoRA training arms using the same model revision, 32 updates and 20,000 supervised tokens per arm. Each faced the same eight synthetic tool-use scenarios. Two independently blinded reviews agreed on every outcome.

Both arms passed all four targeted attribution and empty-search stopping cases. On the four permission-related tasks, balanced training passed 4/4 and repair-weighted training passed 2/4. One failing answer omitted a required source ID. The other stopped after searching, without reading the source or returning the requested fact. Neither arm made an unauthorized save or other forbidden action in these tests.

Repair weighting reduced permission-task training exposure by about 33%. That is a plausible explanation for the weaker answers, but this comparison does not establish the cause. The practical lesson is to preserve coverage of existing tasks when increasing training on selected failures.

This was a small, single-seed development screen, with no untrained baseline. It does not establish training gains or generalization. The 96-case holdout remains untouched. Balanced training remains the development reference; these results do not justify adopting repair weighting.

October 20 update: the original-model baseline

Hatsunama’s follow-up recovered all 24 outputs from the original, untrained checkpoint. On the same eight development cases, it passed 4/8 under the original runtime limits, compared with the previously reported 8/8 for balanced training and 6/8 for repair-weighted training. On a separate set of 16 fresh development cases, the original checkpoint passed 7/16. Those fresh cases reuse the task patterns and were not tested in the trained-arm comparison.

Two separate reviews agreed on all 24 baseline outcomes. They were independent of each other’s scores but had prior research context, so the baseline review was not fully blinded. Nine of 86 generation turns reached the 512-token limit; eight had not closed their thinking section. Training explicitly targeted immediate thinking closure. Better completion under these limits could therefore reflect output efficiency as well as task learning; this retrospective comparison does not establish improved general reasoning.

The original paired screen’s single-seed and exposed-development-case limits still apply. A later resource-budget diagnostic stopped after 8 of 12 planned episodes because the test harness misclassified a per-case timeout. It is incomplete and is not a finished comparison. The 96-case holdout remains untouched.

Baseline evidence: completed original-model run (may require sign-in). This addendum was prepared with my AI agent from the recovered logs and separately completed reviews.

Prepared with my AI agent from the experiment logs and blinded reviews. This is a publisher-reported result, not an independent replication.

Run reference: paired experiment (may require sign-in). Pinned base-model revision: 654f9463ce32b05d0429d76fe1f580b27d4c1ac0.

October 21 clarification: completed diagnostic follow-up

The incomplete 8-of-12 attempt described above is preserved as the historical record. A corrected resource-limit diagnostic subsequently completed all 12 episodes on October 20: four selected development cases repeated under three settings, with no training. The original limits and more-time-only setting each passed 2/4; the combined time, output-token and context expansion passed 3/4. This does not isolate the cause or establish general reliability. Read Hatsunama Small-Model Training: What We Know and What Still Needs Testing for the completed findings, limitations and next questions.

This clarification was prepared with my AI agent from the completed diagnostic record.