DOI: 10.18848/2327-7920/cgp/a188 ISSN: 2327-8692

Explainable Automated Assessment of Pragmatic Quality in Vocational Negotiation Writing Using Rule-Based Natural Language Processing

Anggoro Abiyyu Ristio Cahyo, Mulyono Mulyono, Syamsul Sodiq
Show PDF Cite
Vocational negotiation writing requires students to integrate professional communication with workplace knowledge, while classroom assessment needs clear and traceable guidance for revision. Existing automated writing evaluation research provides limited evidence for task-specific Indonesian vocational writing that integrates negotiation structure, pragmatic strategies, and workplace-relevant technical language within an explainable assessment process. This study aims to develop the Vocational Negotiation E-Assessment System (VNES) as a rule-based automated assessment prototype for vocational negotiation writing. The study employed the Analysis, Design, Development, Implementation, and Evaluation (ADDIE) model and involved thirty-four Grade 10 Machining Engineering students in one 120-minute classroom session. Students produced thirty-four initial and thirty-four revised drafts. VNES assessed Input Validity, Negotiation Structure, Pragmatic Strategy, and Technical Literacy through sequential gating and displayed criterion-based categories, detected indicators, unmet criteria, and revision guidance. One content expert and one media expert appraised the prototype, and students completed a twelve-item perception questionnaire after revision. Content and media appraisal means were 4.75 and 4.77. After revision, twenty-seven drafts reached the Green category, compared with none initially, and the mean Pragmatic Quality Score increased descriptively from 35.29 to 93.38. Revised drafts contained more pragmatic and technical indicators, while perception means ranged from 4.65 to 4.88, and the complete questionnaire yielded a Cronbach’s alpha of 0.975. These findings indicate greater fulfillment of the operational criteria and positive initial user perceptions. VNES contributes an initial transparent, criterion-referenced assessment architecture with formative potential for teacher-supervised vocational writing revision.

More from our Archive