Appendix

Supplementary material for Amortized Reasoning Index: Scaling the Judgment of Reasoning Models Beyond Their Context Window.

Contents

Appendix A: Inference Implementation and Profiling

We measure the cost of serving the trained ARI index on one NVIDIA H200; reasoning-model (teacher) calls and gradient updates during adaptation are not included. The prototype is the evaluator used during adaptation, which scores every document in the corpus with fp32 weights under bf16 autocast. The optimized version of ARI’s retriever (the student) stores bf16 weights, uses PyTorch scaled dot-product attention and torch.compile, tokenizes the corpus once, and batches documents of similar length. A filtered configuration has the optimized student score only a cheap first stage’s top K candidates: Qwen3-Embedding-4B for Twitter-Conflict and BM25 for Writing. Latency is the median time from submitting a query until its top 100 results reach CPU memory, including first-stage retrieval, with requests served one at a time on fixed samples of 50 held-out Twitter-Conflict queries and 30 Writing queries; it excludes model loading, compilation, corpus embedding and index construction. Quality uses all held-out queries (151 and 348), with Twitter-Conflict’s pooled labels (Appendix B.1.1) and Writing’s original labels.

To test whether a configuration preserves quality, we compute each query’s NDCG@10 difference from the prototype’s full-corpus ranking and a 95% confidence interval from a paired bootstrap over queries (10,000 resamples). A configuration passes if the interval’s lower bound is at least −0.01, a tolerance fixed in advance; recall differences are reported without intervals.

Table 1: Twitter-Conflict latency and quality versus how many documents the student scores, from the unoptimized full scan (61.8 s p50) down to the top 1,000 Qwen3-Embedding-4B candidates (276 ms p50), with NDCG@10 deltas, confidence intervals and Recall@100 on pooled and original labels.
Table 1. Twitter-Conflict serving latency and retrieval quality on one NVIDIA H200. Latency is measured on 50 fixed held-out queries; quality uses all 151 held-out queries. Pooled-label values restrict the prototype’s saved full-corpus scores to each configuration’s candidates. Filtered configurations were not pooled, so tweets they bring into the top 100 from deeper in the ranking may be unjudged, and their pooled Recall@100 is a slight underestimate. NDCG@10 differences and confidence intervals are relative to the prototype’s full-corpus ranking. ✓ marks a configuration that passes the non-inferiority test.
Table 2: Writing latency and quality versus how many documents the student scores, from the unoptimized full scan (103 s p50) down to the top 250 BM25 candidates (1.13 s p50), with NDCG@10, its change and confidence interval, and Recall@100.
Table 2. Writing serving latency and retrieval quality on one NVIDIA H200. Latency is measured on 30 fixed held-out queries; quality uses all 348 held-out queries and original labels. NDCG@10 differences and confidence intervals are relative to the prototype’s full-corpus ranking. ✓ marks a configuration that passes the non-inferiority test.

Appendix B: Implementation Details and Per-Task Results

We initialize the student from the pretrained Ettin-0.4B reranker [8], retaining its original scoring head, and use Kimi-K3 as the teacher. Each batch of 20 queries undergoes three rounds of retrieval, judgment, and training.

In each round, the current student scores the corpus and retains its top \(K = 100\) documents per query. We shuffle each candidate set three times and divide each shuffle into groups of 20 for the teacher to rank. This gives 15 groups per query and exposes each document to several comparison contexts.

We trust only the top of each ranking. In a group \(g\), each of the teacher’s top four documents is taken to beat each document it ranks fifth or lower, giving a set \(\mathcal{O}_g\) of \(n_g\) observed wins (\(n_g = 4 \times 16 = 64\) when all 20 are ranked). A Bradley–Terry model [11] gives each candidate \(d\) a strength \(u_d\), with \(P(a \succ b) = \sigma(u_a - u_b)\), where \(\sigma\) is the logistic function. We fit the strengths of a query’s candidates jointly over all of its groups, by ridge-regularized maximum likelihood with every group weighted equally:

\[ \hat{u} = \arg\min_{u} \; \sum_{g} \frac{1}{n_g} \sum_{(w,\ell) \in \mathcal{O}_g} \log\!\left(1 + e^{\,u_\ell - u_w}\right) + \frac{\lambda}{2}\,\lVert u \rVert_2^2, \qquad \lambda = 0.1. \]

We then form training pairs among the documents that appear in all three shuffles. Each pair \((a, b)\), ordered so that \(\hat{u}_a > \hat{u}_b\), gets the sampling weight

\[ \omega(a, b) = \exp\!\left(-\frac{r(a) - 1}{20}\right) \cdot \max\!\left(0,\; 2\,\sigma(\hat{u}_a - \hat{u}_b) - 1\right), \]

where \(r(a)\) is \(a\)’s rank by fitted strength. The first factor favors pairs whose winner sits near the top. The second is zero for a coin flip and approaches one for a confident comparison. We draw \(K \ln K \approx 460\) pairs \(\mathcal{P}_q\) per query without replacement, with probability proportional to \(\omega\), and update the student with the RankNet loss [12] on its raw scores \(s_d\):

\[ \mathcal{L}_q = \frac{1}{|\mathcal{P}_q|} \sum_{(a,b) \in \mathcal{P}_q} \log\!\left(1 + e^{-(s_a - s_b)}\right). \]

This is the same logistic form as the Bradley–Terry fit, applied now to the student’s scores: it pushes the student to score \(a\) above \(b\), and pushes harder the more the student currently disagrees.

Alongside each incoming batch, we revisit historical queries. Some receive fresh retrievals and teacher judgments; rotating replay distributes training across the query history.

B.1 Twitter-Conflict

The corpus contains 72,122 tweets about the Iran–Israel–US confrontation. Queries ask for tweets that express an implicit stance, so topical similarity offers little guidance: tweets about the same events can take opposing positions. We adapt on 100 queries and evaluate on 151 held-out queries.

B.1.1 Expanded relevance judgments on Twitter-Conflict

Twitter-Conflict uses one expanded label set. Each contributing system supplies its top 100 results per held-out query; Opus 5.5 judges relevance blind to source and rank; proposed positives receive a second verification pass. The judge differs from the adaptation teacher, Kimi-K3. The contributing systems are ARI after 0, 20, 40 and 100 queries, and all other systems we report.

We suggest pooled labels as the default evaluation protocol for OBLIQ going forward, provided the pool covers every system being compared, all are judged under one protocol before any is scored, and that protocol (judge model, depth, verification) is reported. Because pooled scores shift as systems are added, we release our judgments for others to extend.

B.1.2 Performance at Inference

ARI on Twitter-Conflict: NDCG@10, Recall@10 and @100, and geometric-mean rank versus production queries processed, under pooled labels and original gold labels.
Table of NDCG@10, R@10 and R@100 under gold and pooled labels for pretrained Ettin-0.4B, ARI after 100 queries, Gemini, and Qwen3-Embedding-8B and 0.6B on Twitter-Conflict.
Figure 5. Twitter-Conflict performance on 151 held-out queries as production queries are processed. Blue uses original gold labels; orange uses the expanded pool that includes candidates from ARI’s checkpoints at 0, 20, 40 and 100 queries and from all other retrieval systems.

B.2 Writing

The corpus contains 10,388 passages, and relevance depends on writing style. We adapt on 100 queries and evaluate on 348 held-out queries over the same corpus, applying the task’s query-specific exclusions before evaluation. Held-out queries include authors represented during adaptation, so the split tests whether learned stylistic distinctions transfer to new passages, not recognition of unseen authors.

B.2.1 Performance at Inference

ARI on Writing: NDCG@10, Recall@10 and @100, and geometric-mean rank versus production queries processed.
Table of NDCG@10, Recall@10 and Recall@100 for pretrained Ettin-0.4B, ARI after 100 queries, Gemini, and Qwen3-Embedding-8B and 0.6B on Writing.
Figure 6. Writing performance on 348 held-out queries after learning from up to 100 production queries. The split tests new queries over the same corpus, including authors represented during adaptation.

B.3 BRIGHT AoPS

For this task, a useful candidate applies the same theorem or idea as the query’s solution, even when the problems look different. We carry over the Writing configuration and change only the teacher’s instruction, to judge technique similarity. Each run starts from pretrained Ettin; across four random splits of the 111 queries, we adapt on 60 queries in three batches of 20 and evaluate on the remaining 51.

B.3.1 BRIGHT AoPS reduced pool

For quick experimentation, every AoPS system ranks the same pool of 49,987 documents. We built it from Qwen retrievals (Qwen3-Embedding-0.6B top 1,572) for all 111 queries, then added three gold documents those retrievals missed. Held-out queries, and the three gold documents added from their labels, therefore influence pool construction, but their relevance labels never enter teacher judgments or student training.

These are reduced-pool results and are not directly comparable with published full-corpus BRIGHT scores. We use four random splits (seeds 42–45), each with 60 adaptation and 51 held-out queries; reported values average the four. Per-split curves and Qwen3-Embedding-0.6B and Qwen3-Embedding-8B results appear in Figure 7, and ARI exceeds both Qwen models on both NDCG@10 and Recall@100 in every split.

B.3.2 Performance at Inference

ARI on BRIGHT AoPS for four seeds: NDCG@10, Recall@10 and @100, and geometric-mean rank versus production queries processed, with Qwen3 stars and Gemini-Embedding-2 reference lines.
Figure 7. Learning from a weak retriever on BRIGHT AoPS. Each color shows one run, starting from pretrained Ettin and adapting on 60 queries while evaluating on 51 held-out queries that receive no teacher judgments. Panels show NDCG@10, Recall@10/100, and geometric mean rank (lower is better). Stars mark the corresponding Qwen3-Embedding-0.6B and Qwen3-Embedding-8B values; their horizontal positions are illustrative, not training checkpoints. Dash-dot lines mark Gemini-Embedding-2. All systems rank the same 49,987-document pool.

References

  1. [1] OBLIQ-Bench: Exposing Overlooked Bottlenecks in Modern Retrievers with Latent and Implicit Queries
    Tchuindjo, D., Shah, D. and Khattab, O., 2026. arXiv:2605.06235.
  2. [2] Relevance Feedback in Information Retrieval
    Rocchio, J.J., 1971. In G. Salton (ed.), The SMART Retrieval System: Experiments in Automatic Document Processing, pp. 313–323. Prentice-Hall.
  3. [3] Relevance-Based Language Models
    Lavrenko, V. and Croft, W.B., 2001. Proceedings of the 24th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 120–127.
  4. [4] Optimizing Test-Time Query Representations for Dense Retrieval
    Sung, M., Park, J., Kang, J., Chen, D. and Lee, J., 2023. Findings of the Association for Computational Linguistics: ACL 2023, pp. 5731–5746.
  5. [5] ReFIT: Relevance Feedback from a Reranker during Inference
    Reddy, R.G., Dasigi, P., Sultan, M.A., Cohan, A., Sil, A., Ji, H. and Hajishirzi, H., 2023. arXiv:2305.11744.
  6. [6] Test-Time Optimization of Query Embeddings with Ranking Aware Reward Maximization
    Chen, T. and Wu, J., 2026. arXiv:2608.12569.
  7. [7] Q-REACT: Full-Index Test-Time Adaptation with Query-Dependent Residuals for Visual Document Retrieval
    Li, Z., Xing, X., Guo, K. and Xu, X., 2026. arXiv:2609.27688.
  8. [8] Introducing the Ettin Reranker Family
    Aarsen, T., 2026. Hugging Face blog.
  9. [9] On Strengths and Limitations of Single-Vector Embeddings
    S, A., Agarwal, M., Garg, A., Kayal, N. and Shiragur, K., 2026. arXiv:2603.29519.
  10. [10] Is ChatGPT Good at Search? Investigating Large Language Models as Re-Ranking Agents
    Sun, W., Yan, L., Ma, X., Wang, S., Ren, P., Chen, Z., Yin, D. and Ren, Z., 2023. Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 14918–14937.
  11. [11] Rank Analysis of Incomplete Block Designs: I. The Method of Paired Comparisons
    Bradley, R.A. and Terry, M.E., 1952. Biometrika, 39(3–4), pp. 324–345.
  12. [12] Learning to Rank Using Gradient Descent
    Burges, C.J.C., Shaked, T., Renshaw, E., Lazier, A., Deeds, M., Hamilton, N. and Hullender, G., 2005. Proceedings of the 22nd International Conference on Machine Learning, pp. 89–96.
  13. [13] Interactively Optimizing Information Retrieval Systems as a Dueling Bandits Problem
    Yue, Y. and Joachims, T., 2009. Proceedings of the 26th International Conference on Machine Learning, pp. 1201–1208.
  14. [14] Test-Time Training with Self-Supervision for Generalization under Distribution Shifts
    Sun, Y., Wang, X., Liu, Z., Miller, J., Efros, A. and Hardt, M., 2020. Proceedings of the 37th International Conference on Machine Learning, PMLR 119, pp. 9229–9248.
  15. [15] BRIGHT: A Realistic and Challenging Benchmark for Reasoning-Intensive Retrieval
    Su, H., Yen, H., Xia, M., Shi, W., Muennighoff, N., Wang, H.-y., Liu, H., Shi, Q., Siegel, Z.S., Tang, M., Sun, R., Yoon, J., Arik, S.Ö., Chen, D. and Yu, T., 2025. Thirteenth International Conference on Learning Representations.
  16. [16] Mice to Meet You: From Re-ranking Only to End-to-End Retrieval
    Morand, V., Zong, Y., Vast, M., Mothe, J. and Piwowarski, B., 2026. arXiv:2610.31932.
  17. [17] Amortized Inference in Probabilistic Reasoning
    Gershman, S.J. and Goodman, N.D., 2014. Proceedings of the 36th Annual Conference of the Cognitive Science Society.
  18. [18] The Retriever Should Remember: Experience-Amortized Reranking for Long-Term Agent Memory
    Feng, Q., Ding, C. and Fan, J., 2026. arXiv:2608.22767.
  19. [19] Librarian Agents: Training LLMs to Curate Long-Context Workspaces that Anticipate Future Queries
    Singh, A., Zhang, T., Lee, Y., Nair, R.S., Kumar, A. and Finn, C., 2026. arXiv:2610.22273.
  20. [20] Machine Studying
    Li, J.X., Battle, R. and Khattab, O., 2026. Blog post.