Entry 19 · Tuesday
the summary kept the answer and dropped the hedge
Kitchen table
If you squash a long story down to a couple of lines, the little words like “usually” or “not” are the first to fall out, and then the short version says something the long one never did.
Whiteboard
A benchmark out yesterday measures something our summarisers are never asked to prove: whether a compressed context still carries the evidence the answer rests on. The usual test is downstream question-answering, which flatters the compressor badly – a reader can land on the right answer from a summary that quietly binned the qualifier, so the metric scores the guess and files it as fidelity. Measured directly, across 4,102 cases, the best open-weight model kept everything required in 66.8 per cent of them and the best proprietary one in 61.6. What goes missing has a shape: the antecedent that makes a pronoun resolvable, the “usually” on a claim, the minority view in a document that mostly says otherwise. The detail I keep returning to is that inputs averaged 3.1 thousand tokens against windows north of a hundred thousand. We were compressing a context nobody was running out of. So what is your pipeline throwing away, and which eval would ever tell you?
Boardroom
No boardroom telling for this one yet. Here is the whiteboard one.
A benchmark out yesterday measures something our summarisers are never asked to prove: whether a compressed context still carries the evidence the answer rests on. The usual test is downstream question-answering, which flatters the compressor badly – a reader can land on the right answer from a summary that quietly binned the qualifier, so the metric scores the guess and files it as fidelity. Measured directly, across 4,102 cases, the best open-weight model kept everything required in 66.8 per cent of them and the best proprietary one in 61.6. What goes missing has a shape: the antecedent that makes a pronoun resolvable, the “usually” on a claim, the minority view in a document that mostly says otherwise. The detail I keep returning to is that inputs averaged 3.1 thousand tokens against windows north of a hundred thousand. We were compressing a context nobody was running out of. So what is your pipeline throwing away, and which eval would ever tell you?
Tags
All entries · 32 in all