DOEM: A Novel Development-Oriented Evaluation Metric Framework for Service Recommendation

Service recommendation systems are critical tools for facilitating the integration of Web services to meet diverse user requirements. However, classical evaluation metrics, which rely heavily on static ground-truth labels, are often insufficient to capture the dynamic, multidimensional nature of real-world development needs. To address these limitations, we propose the Development-Oriented Evaluation Metric Framework (DOEM), a novel approach designed to evaluate service recommendation systems based on practical development scenarios. DOEM introduces four key evaluation dimensions: i) functional coverage, to assess the breadth of services meeting user needs, ii) fault tolerance, to evaluate the system resilience, iii) efficiency, focusing on the computational performance, and iv) redundancy, to measure the overlap in recommended services. A hierarchical taxonomy is developed for annotating APIs and mashups, ensuring consistent tagging and a comprehensive view of functional relationships. We further introduce a Metric-specific Optimization Strategy inspired by LambdaRank, which uses metric-oriented swap gains to guide pairwise ranking optimization and align recommendation models with different development-oriented objectives. Through extensive experiments on real-world datasets, we demonstrate the effectiveness of DOEM in capturing diverse evaluation perspectives while maintaining a balance between cost-efficiency and performance. The results highlight that DOEM significantly outperforms classical metrics in reflecting the multidimensional requirements of service development. This work establishes a comprehensive framework for advancing service recommendation systems and addressing the dynamic and complex demands of modern software development.