Topic
Evals and measurement
How to tell if an AI system works and where the tests fool you.
Kitchen table
A test tells you if a robot did its homework right. But tests can be fooled too. These pages show how to check the checker.
Whiteboard
An eval is a measurement with its own failure modes. Judges drift. Benchmarks leak. Pass rates hide what changed. These entries track where the numbers lie and how to catch it.
Boardroom
An eval is the evidence behind a launch decision. Weak evals let a bad release look safe. These pages show what to ask before you trust a score.
Start with the concepts
-
Evals & Hallucination Task-level suites, graders (exact-match where you can, LLM-as-judge where you must, with the judge itself evaluated), golden sets guarded from…
-
Evaluation & Metrics Hold-out sets and cross-validation for honesty.
-
Loss & Error MSE for regression, cross-entropy for classification, and a long tail of task-specific objectives, but the design decision that matters is that the…
Papers
From the daily page
-
llm upgrade testing: the new model passed the old tests
2 November 2026 -
the longer list looked better because guessing got easier
1 November 2026 -
the shortlist decided the score before the reranker did
30 October 2026 -
the retriever was fine and the pdf was not
28 October 2026 -
the answers survived the squeeze and the reasons did not
26 October 2026 -
it deferred every ticket and called that caution
25 October 2026 -
it found the needle every time and still lost count
24 October 2026 -
the attackers went straight for the rows the benchmark left blank
22 October 2026 -
the reasoning stayed honest and the answer went rogue
21 October 2026 -
the summary kept the answer and dropped the hedge
20 October 2026 -
the guardrail passed because nobody tried the door
17 October 2026 -
the skill that won the argument wasn't the one you installed
14 October 2026 -
the answer was correct and the procedure never ran
13 October 2026 -
the thinking-off switch turned out to be a suggestion
10 October 2026 -
the steadiest judge was wrong the same way every time
9 October 2026 -
the rule at the top of the query turned out to be more query
8 October 2026 -
the rehearsal went well because nobody else was in it
6 October 2026 -
nobody asks the whole question on the first turn
5 October 2026 -
the vaguer the question, the more it helped itself
4 October 2026 -
we kept pruning the memories that never got a turn
3 October 2026 -
the pass rate went up and so did everything else
2 October 2026