Bayesian-inspired verification of data quality rule candidates
Hui Liu, QiongYu Lei, Rui FengPurpose
Organisations rely on cataloged data quality rules for governance, yet profiler outputs still need screening before promotion to production monitors. This paper studies how semantic plausibility and SQL violation evidence can form an auditable verification score for candidate rules.
Design/methodology/approach
We present Route B, a Bayesian-inspired, prior-adjusted verification score fusing Laplace-smoothed SQL violation rates with hierarchical semantic prior templates. Evaluation uses TPC-H, Financial, and a desensitised Shenzhen open-data catalog (56 candidates). Shared candidate sets are compared with prior-only, SQL-only, confidence-interval, and beta-binomial baselines alongside Route B and optional closed-loop prior correction.
Findings
On the Catalog benchmark, prior-only yields F1 0.33 (36 false promotions) versus Route B F1 0.95 (2 false promotions), cutting proxy review time by about 510 min. Under ambiguous violation rates, hierarchical priors raise F1 from 0.539 to 0.800 versus likelihood-only filtering. On clean warehouses, SQL validation already separates most candidates, and Route B matches SQL-only at precision 1.000 on accepted rules.
Research limitations/implications
Evaluation uses synthetic warehouses and a desensitised catalog; candidates are scored independently; catalog labels follow a written steward protocol; mixed row noise up to 20% does not change warehouse F1.
Practical implications
Route B sits between rule discovery and monitors such as Great Expectations or Deequ as an audit-oriented acceptance layer. A reproduction package regenerates the reported tables without a live LLM endpoint.
Originality/value
An auditable prior-adjusted verification framework with closed-form accept boundaries; a warehouse-plus-catalog verification benchmark from SQL-profiler proposals; and evidence on when semantic–SQL fusion changes decisions relative to SQL-only or prior-only filtering.