Optimizing information retrieval tasks with large language model for data enhancement

Abstract

The data science and artificial intelligence, optimizing information retrieval tasks has become crucial for extracting actionable insights from vast amounts of data. The problem is the need for precise query formulation to retrieve relevant data effectively, as LLMs can generate vast amounts of information that might include noise or irrelevant details. The objective of this study is to enhance the efficiency and accuracy of information retrieval tasks by leveraging large language models (LLMs) for data augmentation. Gathering a diverse dataset from various sources like online search engines, social media platforms, and online forums is crucial for meeting text and information needs effectively. The term frequency-inverse document frequency (TF-IDF) technique is applied to calculate the importance of each term in the dataset, allowing for the differentiation of significant words from common ones. This step is crucial in the data pre-processing phase to enhance the relevance and precision of information retrieval tasks. The goal is to optimize the normalized discounted cumulative gain (NDCG) metric, which measures the ranking quality of the retrieved documents. The framework integrates the fine-tuned LLM model with the chosen information retrieval system by incorporating the model’s outputs into ranking algorithms. The results show that the proposed method has the maximum accuracy, with an average accuracy of around 10 % when implemented using Python software. The future scope for optimizing information retrieval tasks with large language models (LLMs) for data enhancement is vast and promising.