Evaluating Large Language Models on the Certified Internal Auditor Exam

The internal audit function provides consulting services and recommendations for management on performance quality, business environment security, or risk management. Internal auditors are expected to monitor the senior managers’ behaviors and support them in operational and strategic decision making. They should recognize and reduce the possibility of an employee’s opportunistic behavior, improve internal control, and prevent financial fraud. Therefore, they should be highly qualified, and their competencies are to be constantly updated. Hence, they are encouraged to learn and participate in certification processes organized by various institutions, e.g., the Institute of Internal Auditors (IIA). Although there are many strategies of learning in the certification process, this study focuses on learning-by-testing approach. This method emphasizes the use of testing to acquire the internal audit knowledge needed to successfully pass the professional certified exams. Beyond that, the authors present the use of large language models (LLMs) in internal auditing. However, the research goal is to how the large language models (LLMs) can be used to answer questions included in the Certified Internal Auditor (CIA) exam directed by the Institute of Internal Auditors (IIA). In the research section, fourteen LLMs were evaluated using English and Polish-language CIA test sets comprising 184 and 183 multiple-choice questions, respectively, with accuracy as the primary performance metric. The best-performing individual model achieved approximately 91% accuracy, while the Friedman test identified statistically significant overall differences among the models for both datasets. The findings indicate that a multi-agent system can achieve better performance than an individual LLM. The learning-by-testing method can be supported not only by the use of professional association materials but also by the opportunity learners have to enrich their knowledge through the use of LLMs in the learning process.