Developing a Machine Learning–Based System for Nutrient Profiling of Pakistani Cuisine

Background: Chronic diseases such as diabetes and heart disease are rising in South Asia, with poor diet a major contributor. Most dietary apps are designed for Western foods and struggle with culturally diverse dishes such as Pakistani cuisine. Objective: This study aimed to build a machine learning-based system that can recognise Pakistani dishes, estimate portion sizes, and calculate their nutritional value. Methods: We developed a multi-stage automated dietary assessment pipeline for Pakistani cuisine using a multi-source dataset covering 83 dish classes across 10 categories. Following data cleaning, class balancing, and targeted augmentation, deep learning architectures were evaluated for food classification and segmentation. ResNet-50, MobileNetV2, and YOLOv8 were used for food recognition, while YOLOv8-Seg and Mask R-CNN enabled pixel-level segmentation for portion and nutrient estimation. All models were optimized using transfer learning and evaluated using standard classification and mask-quality metrics, followed by mobile application usability testing. Results: A multi-source dataset of ten categories and 83 Pakistani dishes was used to train and compare deep learning models for food classification and segmentation. Portion size was estimated using a segmentation- and depth-based approach without a reference object, while nutritional values were derived from standardised recipes and established food databases. The models were integrated into a mobile application for usability testing. YOLO-based models achieved the best balance of accuracy and speed for real-time classification and segmentation. Portion estimates were reasonably close to measured values, and usability testing showed the app was easy to use, with minor improvements suggested. Conclusions: Overall, this work demonstrates the feasibility of an AI-based dietary assessment system adapted to Pakistani cuisine, with future work focused on reducing processing demands and improving depth-based portion estimation.