the shortlist decided the score before the reranker did

I spent a week tuning a reranker that could not have helped. The first stage returned a shortlist of candidate documents. The reranker only reordered that shortlist. If the right document never entered it, no ordering can win. That is a ceiling, not a tuning problem. A recent paper, The Recall Ceiling of LLM Recommendation Reranking, makes this formal. It shows that oracle evaluation quietly hides the ceiling from you. The correct item gets injected into the candidate set before any scoring. Scores then measure the reranker against a set it cannot possibly miss. Under honest retrieval the clever strategies collapse toward the simple baseline. The authors propose classifying the recall regime before comparing rerankers at all. So measure recall at the candidate window before anything else. That one number bounds every downstream metric you report. If recall is low, widen or hybridise the first stage instead. Tuning the second stage is theatre until the first stage improves.