Practical evaluation method · October 6, 2026

Did a Local AI Change Help? A Reproducible Comparison

A better-looking output is a reason to investigate. To compare a training change, prompt, quantization or harness revision, keep the test conditions inspectable and look for regressions as well as improvements.

This method connects my public caption software, caption and translation application, and model experiments. It includes synthetic starter cases, not measured results or a claim that my applications pass them.

1. Write the decision before running the comparison

State exactly what changed and what would count as an improvement. For example: “Does this prompt preserve negation in translation without introducing new speaker-role errors?” Choose a baseline and candidate. Change one component when practical; if several components change, evaluate the combined change and leave the individual cause unresolved.

2. Freeze a small set of cases with observable expectations

Include a normal task, a previous failure and an application-boundary case. Write the expected meaning or behavior before examining outputs. Use held-out cases for the final check: cases used to tune a model or prompt are development evidence, not independent evidence of generalization.

Download three synthetic cases as JSONL. The template separates exact input, expected properties, generated text, observed application actions and reviewer judgment. Output and judgment fields are empty until someone actually runs and reviews a case. Translation requires choosing a language pair and review by a competent speaker; the template alone cannot judge multilingual quality.

3. Record enough context to investigate a difference

Model identifier and exact revision, adapter revision if used, tokenizer/chat template and quantization.
Exact system prompt, input, language pair and preprocessing.
Generation settings, seed if supported, inference software version and relevant hardware.
Application or harness revision, available tools and the authorization policy.
Case ID, run ID, raw output, actions observed and errors. Record failures and timeouts rather than dropping them.

A matching seed does not guarantee identical output across hardware or software changes. Repeat runs when output varies, use the same declared procedure for both versions, and report that variation. For latency, specify whether loading and warm-up are included.

4. Compare model behavior and application behavior separately

A sentence may preserve meaning while the application mishandles it. For the boundary case, check both the translated or captioned text and whether an action was attempted. “She told me to delete the file” is reported speech, not permission to delete anything.

Where possible, label outputs A and B without identifying the version to the reviewer. Report each changed case: baseline judgment, candidate judgment and the reason. Keep “uncertain” available. A single overall score can hide a newly introduced failure that matters to a user.

5. State the limit of the conclusion

Three starter cases can help reproduce particular failures; they are not a benchmark of broad translation quality, agent reliability or safety. Share case counts, repeats, exclusions and unresolved judgments alongside any result. Expand coverage before making wider claims.

Human First AI means checking whether changes preserve people's meaning and control. I am applying these questions to ongoing work behind the scenes; this public note shares the method without claiming private results.