Statistical Process Control and
its Relevance in Data Quality
Monitoring and Reporting
Laura Sebastian-Coleman, IQCP,
Christopher Heien, IQCP,
Raj Vadlamudi, and
CIGNA, Bloomfield, CT, USA
A core function of any data quality (DQ) program is measurement, monitoring, and ideally, certification of critical data in critical systems. Statistical Process Control (SPC) provides a scalable means of monitoring data quality levels and reporting on trends over time. In this paper we explain the basis of SPC and describe how to apply it in data quality management including determination of thresholds. We also present information from a case study to illustrate the application of the ideas. The primary audience for the paper is data quality practitioners who are looking to implement or improve automated data quality measurement.
Most approaches to data quality improvement start with a comparison between the production of data and the production of manufactured goods; and recognize the value of treating data like a product [Sebastian-Coleman, 2013]. Much thinking about data quality is directly rooted in methods for process quality, including SPC. Statistical process control methods have been successfully applied to the measurement of data quality both for initial analysis, which identifies causes of variation, and for ongoing measurement to confirm whether a process remains in control.
SPC has a distinct advantage over other forms of data quality measurement: Because SPC is mathematically-based, it can be used to automate the identification of anomalies. We donít need to rely on a personís gut reaction to data to determine whether there is likely a problem. We rely on the measurement numbers to detect significant changes that need to be investigated. This characteristic also means the approach is scalable. SPC can also be useful determining thresholds. These thresholds do not need to be maintained manually, and much more data can be measured at a time.
Read this article in the December 2015 IQ International Journal
Non members: Price US$25. Pay online and the journal will be delivered by email within the hour.