A Reproducible Benchmark-Validity Audit and Calibration Study for Cross-Home Fault Diagnosis in Smart-Home Sensor Systems

Diagnosing faults across different smart homes is hard: sensor names, layouts, and daily routines differ from home to home, so a model trained in one home rarely works in another. We study an ontology-guided framework for cross-home fault diagnosis, but our main contribution is a benchmark-validity audit—a systematic check of whether the datasets used to evaluate such systems actually measure fault detection. Using the public Center for Advanced Studies in Adaptive Systems (CASAS) smart-home datasets (homes hh101–hh110) and real household power data (HomeC, UMass Smart*), we show that much of the high cross-home accuracy reported on these benchmarks is an artifact of features that re-encode the labelling rules rather than evidence of transfer: when those features are removed, the macro-averaged F1 score (macro-F1) collapses toward the level obtained with randomly permuted labels. We therefore treat these datasets as semantic-transfer and benchmark-validity studies, not fault-detection results. The framework’s distinguishing component is a counterfactual calibration layer that returns a probability for its recommended intervention; on a controlled structural causal model with known interventions, it achieves a Brier skill score of 0.369 for intervention-success probabilities. Separately, on the simulation-derived LBNL Fan Coil Unit benchmark, a conventional gradient-boosted multiclass fault classifier achieves accuracy comparable to a random forest but about six times lower expected calibration error (0.026 vs. 0.159) under a scenario-matched split. This calibration advantage does not generalize to held-out simulation scenarios, where the calibration error rises to 0.372; we report this negative result as a limitation. We are explicit about scope: the ontology reasoner and the real-stream causal graph are only partially implemented, and the counterfactual recommendations are validated only under controlled or simulated conditions, not in deployed homes. The results are intended for researchers who build or benchmark sensor-based fault-diagnosis models, for dataset curators, and for practitioners who need calibrated rather than merely accurate outputs. All code, the proxy-label rules, and the leakage audit are released.