Benchmarking Large Language Models on Production BIM Workflows Using Agentic Automation

Large language models (LLMs) are increasingly proposed for Building Information Modeling (BIM) work, yet published evaluations test one or two models on a single task using simplified inputs, leaving practitioners without an evidence base for choosing between them. This paper’s contribution is a four-track benchmark of ten current models run inside one commercial Revit add-in suite on the production workloads of a working BIM department: requirement extraction from a live ISO 19650 BIM Execution Plan; rule extraction from an 88-page scanned Arabic building code; multimodal drawing review with live application programming interface (API) access, adjudicated and reconciled by two practitioners; and ten colloquial prompts executed in Revit, each carrying an adversarial probe. Scoring combines itemized rubrics, human verification, cross-model consensus and file forensics with telemetry from 218 logged calls. Three models form a leading group, Claude Opus 5 at 88.7 of 100, GPT 5.6 Sol at 86.2 and Claude Fable 5 at 84.7. Neither re-weighting the composite 20,000 times nor repeating the agentic track displaces that group or moves a composite by more than 1.8 points, but the separation inside it is smaller than that variation, so no ordering inside the group is claimed. On these prompts fuzzy name resolution, not code generation, accounts for 60 percent of agentic points lost, which argues for guardrails enforced in the tool layer rather than assumed from model behavior.