Online‐optimized RAG for tool use and function calling

Abstract

In many applications, retrieval‐augmented generation (RAG) drives tool use and function calling by embedding the (user) queries and matching them to prespecified tool/function descriptions. In this article, we address an embedding misalignment issue that often arises in practical applications due to imperfect embedding models or noisy descriptions; such misalignment may lead to incorrect retrieval and task failure. We introduce Online‐optimized RAG (ORAG), a deployment‐time framework that adapts item embeddings from binary feedback attributed specifically to retrieval correctness. ORAG applies lightweight online gradient updates with negligible per‐query latency and requires no changes to the underlying large language model (LLM). The method supports single‐ and multihop tool use, dynamic tool inventories, and ‐retrieval with reranking. We provide a problem‐dependent theoretical analysis that quantifies how the method's performance depends on the initialization quality of the embeddings and other related quantities. Across diverse tool‐use and document‐retrieval scenarios, ORAG generally improves retrieval and tool‐selection accuracy, providing a simple, practical path toward self‐improving RAG systems. We further evaluate the method under varying query recurrence, feedback‐error patterns, learning‐rate settings, and changing tool inventories to identify the conditions under which online optimization is most appropriate.