Performance of a quality assurance program for assessing dental health in methamphetamine users

Study setting

The study was conducted in Los Angeles County, one of the largest and most populous
urban areas in the USA and beset with high rates of MA use 10], 11]. Between February 9, 2011 and August 26, 2013, 574 MA users recruited from local
communities underwent comprehensive oral examinations and psychosocial assessments
at dental clinics associated with two large community health centers: a) the AIDS
Project, Los Angeles (APLA) center that primarily serves a sociodemographically diverse
group of individuals with HIV/AIDS, and b) the Mission Community Hospital (Mission)
in the San Fernando Valley that caters to a large, underserved migrant population.
The study sites were chosen to provide access to a diverse cohort of afflicted Angelenos
with a broad range of MA-use behaviors. For the main study, we used a case-control
study design that compared the MA users to a control group of propensity-score matched
samples from the National Health and Nutrition Examination Survey (NHANES) 12], 13], where propensity scores were used to identify a demographically comparable subset
of individuals from the population-based NHANES. Study findings presented in this
paper are based on a selected sub-sample of participants who underwent replicate examinations
for data quality assurance purposes. The study design and data collection protocol
was reviewed and approved by the UCLA Institutional Review Board. Trained dental examiners
conducted the dental examinations, with the data recorded by dental assistants. In
conjunction with the dental exam, an experienced bilingual interviewer conducted comprehensive
assessments of psychosocial, behavioral, and substance-use characteristics of study
participants.

Training and quality assurance

The study was launched with a 2-day project orientation for all persons involved in
the study, including the Principal Investigator, co-investigators, project manager,
reference examiners, dental examiners, dental assistants, research assistant, and
the statistical and data management team. At this initial meeting, the lead investigators
reviewed the study’s scientific background, objectives, inclusion and exclusion criteria,
evaluation criteria, key outcome variables and study enrollment targets. The practical
aspects of the study were discussed including recruitment and screening strategies,
management of any adverse events, technical considerations relating to data collection
and recording, as well as responsibilities with regard to protocol adherence, quality
assurance and logistical support.

Training and Quality Assurance (QA) were conducted under the leadership of BD (lead
reference examiner), who is the national trainer and reference examiner for NHANES.
He was assisted by a locally-based dental epidemiologist (local reference examiner)
who provided ongoing monitoring of the dental examiners, evaluating their assessments
and providing remediation when necessary. A dentist from each participating health
center was selected and trained to conduct a standardized dental examination. Both
dentists provided direct patient care at the participatory sites but had little prior
experience in clinical research. On the 1st day of the orientation meeting, the lead
reference examiner used lecture and slide presentations to familiarize the study dental
examiners with customary NHANES study protocols and assessment criteria followed by
explanation of the examination technique and equipment use. To initiate the standardization
process, demonstration examinations, conducted by the lead reference examiner, was
followed by the trainee dental examiners conducting practice examinations in volunteer
subjects. During the standardization process, the trainee examiners were encouraged
to ask questions regarding assessment criteria while conducting the study protocols.
In an effort to minimize differences in examination findings, data from each standardization
round were reviewed for inconsistencies and findings were discussed with the trainee
examiners led by the reference examiner. In a subsequent calibration phase, the reference
examiner and dental examiners repeated dental and periodontal assessments in the same
set of 12 volunteers.

Once the field study was initiated, the local reference examiner visited each site
monthly to observe data collection and to randomly replicate the dental examination
in 2–3 subjects. Data from these replicate exams were used to produce inter-rater
reliability statistics to evaluate examiner performance and to provide feedback. If
an examiner’s performance fell below an acceptable level, retraining was conducted
on site. About 1 year after study launch, the lead reference examiner returned to
observe field operations and, along with the local reference examiner, conducted another
round of calibration exercises with the dental examiners. The calibration visits took
place at each clinical site during normal field operations and involved 12 subjects
enrolled in the study. The monitoring visits helped determine whether the dental examiners
were conducting the oral health examinations within the parameters of the NHANES study
protocols and if the standards for examination between the dental examiners and reference
examiner had been maintained.

The project manager oversaw the data collection activities at each field site, providing
ongoing logistical support and supervision and ensuring that the each site followed
all appropriate procedures for subject recruitment and consenting. Dental examiners
were clinic-based for this study. At the APLA site, where the bulk of the assessments
took place, the study examiner remained for the duration of the study. The initial
examiner at Mission site departed towards the midpoint of the data collection phase,
and a replacement examiner was trained and calibrated to collect data. Overall, 51
enrolled subjects, representing 9 % of the study sample, received replicate exams.

Variables and their measurement

The main oral-health outcome variables were the rates and patterns of dental caries
and the periodontal disease status of the subjects. Assessments for dental caries
and periodontal status adhered to NHANES examination protocols, which have been described
in greater detail elsewhere 14]–16]. In brief, NHANES diagnostic criteria were used to assess for dental caries at the
surface-level and presence required manifest cavitation. Examinations were conducted
under artificial light with the study participant in the supine position using a standard
explorer and dental mirror without additional magnification. Radiographs were not
taken. A Hu-Friedy periodontal probe color-coded and graduated at 2–4–6–8–10–12 mm
was used to assess periodontal status. Dental caries experience was calculated as
the number of diseased (D), missing (M), and filled (F) teeth (T) and the number of
untreated dental caries was calculated as the number of diseased surfaces (DS). Periodontal
disease status was assessed using the case definitions recommended for periodontitis
surveillance by the CDC Periodontitis Workgroup (CDC/AAP) (17]. The CDC/AAP case definitions require information from two interproximal sites (DF,
MF, ML, and/or DL) and are not dependent upon the presence of an adjacent tooth. Gingival
recession and pocket depth measures were made at four sites per tooth (the disto-facial
(D), mid-facial (B), mesio-facial (M), and the disto-lingual (DL) sites). An algorithm
calculated loss of attachment from the information on gingival recession and pocket
depth. All four quadrants were examined and 3rd molars were excluded.

Data collection and management

To ensure standardization and quality assurance in data collection and processing,
all dental and psychosocial data were captured directly on a laptop computer using
a web-based data-management system developed and maintained by the UCLA-Semel Institute
Statistics Core (SIStat). Data collected through the user-friendly graphical interface
on the laptop was encrypted and transmitted to be stored centrally in a secure server
with firewall protection. Built-in logic and data-range checks allowed data verification
to prevent invalid data. Automated reports and dashboards allowed the investigators
and project manager to monitor the quality of the data collected at each clinical
site by generating a variety of summary reports on data completeness and questionable
values. The real-time input verification facilitated the timely identification and
resolution of any problems in data collection and processing.

Statistical methods

SAS software (Version 9.3; SAS Institute Inc., Cary, NC, USA) was used for statistical
analysis and data handling. Demographic information was tabulated for the full sample
(n?=?574) as well as within each of the replicate samples at APLA (n?=?33) and Mission (n?=?18). Participants who indicated that they had used methamphetamine for less than
10 of the last 30 days at the time of screening were classified as being “light” methamphetamine
users, while the study participants who used MA for more than 10 of the last 30 days
were classified as “moderate +” users. Education level was broken into three categories:
less than high school, high school completion, and more than high school. High school
completion was indicated by high school graduation or by obtaining a GED.

The reliability analysis was conducted using a set of both continuously-scaled and
dichotomous outcome measurements. Continuously-scaled measurements from the caries
examination included DMFT (the total number of decayed, missing, or filled teeth in
a participant’s mouth), DFT (number of decayed or filled teeth), DS (number of decayed
surfaces), and tooth retention (number of teeth present in the mouth). Continuously-scaled
outcomes from the periodontal examination included mean gum recession, pocket depth,
and calculated attachment loss per participant. Average attachment loss and pocket
depth were additionally stratified by the four periodontal sites (mid-facial, mesial-facial,
disto-facial, disto-lingual), recognizing that different site types could have differing
variability. Means and standard errors were produced for each rater on these outcomes,
as well as the bias, defined as the difference between the examination rater measurement
and the reference measurement. Intra-class correlation (ICC) coefficients were estimated
as the primary reliability statistic for the continuously-scaled measurements. The
intra-class correlation was defined as the ratio of variance between subjects to the
total variance between and within subjects. ICCs closer to 100 % indicate greater
inter-rater reliability.

Discrete outcomes from the caries examination contained the caries experience (defined
above), having at least one surface of untreated decay, having at least one restoration
or surface restoration, tooth retention (all teeth present), and having at least five
anterior surfaces with untreated decay. From the periodontal examination, the dichotomous
outcomes included having at least one site with attachment loss greater than or equal
to a given threshold (taken to be either 3, 4, or 6 mm); having at least one site
with pocket depth greater than or equal to a given threshold (either 4, 5, or 7 mm);
and indicators of periodontitis, moderate periodontitis, and severe periodontitis.
Percent agreement between the examination and reference raters was computed as one
metric of reliability. Cohen’s kappa statistic and a corresponding asymptotic standard
error (SE) were also produced. Examiner strength of agreement using the Kappa coefficient
was not evaluated by hypothesis (statistical) testing and instead was evaluated by
applying commonly used guidelines 18].