Investigation of preference-based speech quality assessment by integrating adaptive pair selection and pseudo-labeling
ChengHung Hu, Yusuke Yasuda, Tomoki TodaPurpose
This paper aims to investigate the efficiency and viability of online pair selection methods for preference score collection in Speech Quality Assessment (SQA). It addresses the challenge of high annotation costs in traditional pairwise comparisons and explores whether preference data collected in a sparse manner can successfully produce accurate rankings and support robust SQA model training.
Design/methodology/approach
We propose an improved aggregation method to derive system-level scores from raw preference data. To equalize uncertainty across system pairs, we introduce a novel bias-based pair selection method. Furthermore, we develop a pseudo-labeling strategy that utilizes unlabeled utterances to infer missing preferences, thereby augmenting the training data and enhancing SQA model performance.
Findings
Experiments demonstrate that the proposed bias-based pair selection method achieves the most robust performance. Additionally, the proposed pseudo-labeling approach effectively compensates for data sparsity, significantly improving the SQA model's performance by leveraging unlabeled datasets.
Originality/value
This paper fulfills an identified need for cost-effective SQA data collection. It bridges the gap between active pair selection and deep learning-based SQA model training by introducing a novel bias-based selection mechanism and a pseudo-labeling strategy that leverages system-level scores to predict missing preferences.