Machine learning identification of EEG features predicting working memory performance in schizophrenia and healthy adults

Demographics

Groups did not differ in basic demographic composition (Table 1). Three (15 %) participants in the HC sample were over age 55 (aged 63, 59, 56),
while 8 (20 %) in the SZ sample were over 55 (aged 70, 66, 63, 62, 60, 59, 58, 58).
Statistical analysis were unaffected by entering age as a covariate and reported results
were not significantly changed by restricting the sample to those aged 55 or younger
(N?=?41).

Behavioral data

Group comparison on SWMT behavioral data revealed overall lower accuracy in SZ than
HC (Table 1). As a group, working memory performance was impaired in SZ participants based on
MCCB WM composite score, but visual attention was within normal range based on the
CPT-IP. SZ participants were estimated to have average range of IQ but, overall, scored
lower than HC.

Model 1: Classification of SWMT performance accuracy

Healthy normal sample

SVM Model 1 identified frontal (Fz) gamma activity during encoding and occipital (Oz)
theta 2 during retrieval as the primary EEG features associated with SWMT accuracy
in the HC sample (Table 2). Additional features retained in the model had weightings of .10 or less and were
not regarded as meaningful for further analysis. The negative valence of feature weights
indicated that higher values for each preceded incorrect behavioral responses. Model
classification accuracy was 84 % and all additional performance statistics (F1 score?=?0.96,
precision?=?0.92, recall?=?1.0, estimated AUC of ROC?=?0.98), suggested excellent
model fit and stability. Cross-validation of this model applied to SZ data yielded
lower, yet acceptable model, classification accuracy (74 %) and performance statistics
(F1 score?=?0.77, precision?=?0.68, recall?=?0.9, estimated AUC of ROC?=?0.84). Accordingly,
primary features determining SWMT performance in HC also applied to SZ; however, an
overall decrease in model performance suggested that other or additional features
were explanatory for SZ.

Table 2. EEG Features Predicting Trial Accuracy in HC

To further assess the stability of SVM Model 1 based on HC data, the analysis was
repeated with features entered separately by stage of WM processing (i.e., baseline,
encoding, retention, retrieval). This analysis was conducted to determine whether
experimenter decisions regarding method of feature entry (i.e., 60 features entered
simultaneously vs. 15 features entered into 4 separate models) would substantially
influence the outcome of feature selection. Overall, the two approaches converged
on the same primary features (Table 3). As observed with simultaneous entry of 60 EEG features, frontal gamma during encoding
was the feature most highly weighted in predicting SWMT accuracy. Notably, the only
two features identified at the encoding stage with non-zero weightings both involved
gamma activity, the second feature being centrally distributed gamma, and together
predicted SWMT trial performance with 96 % accuracy. Retrieval stage features also
predicted SWMT with high accuracy (88 %) based primarily on occipital activity in
gamma and theta 2 ranges (Table 3). In this case, the ordering of features differed slightly from the model constructed
by simultaneous entry in that theta 2, rather than gamma, was most highly weighted.
Furthermore, modeling data independently according to WM stage identified features
that were evidently suppressed by the primary features of the original model. No feature
representing the pre-trial baseline stage entered the original model when applied
to HC data; however, a contribution of baseline activity accounted for almost entirely
by central theta (feature weight?=??1.13), in association with inaccurate performance,
was identified when modeled independently. Finally, the contribution of retention
stage activity to performance was best characterized by central theta 1, both when
features were modeled simultaneously (Table 2, 3
^rd
ranked feature) and independently by WM stage.

Table 3. SVM Model 1 Coefficients Extracted by Stage of Working Memory

Schizophrenia sample

Many more features entered the model when constructed using SZ sample data (Table 4), with central and frontal gamma during encoding identified as the primary classifiers
of SWMT accuracy. As observed in the HC sample data, the valence of coefficients indicated
that higher values for these features preceded incorrect behavioral responses. Interestingly,
beta activity during retrieval was also identified as a predictor of trial accuracy
but with a positive coefficient, indicating that higher activity preceded correct
behavioral responses. Theta 1 during retention and theta 1 and gamma activity during
retrieval entered as negative predictors of trial accuracy with weightings above .5.
Overall classification accuracy was 80 % and model performance statistics (F1 score?=?0.80,
precision?=?0.78, recall?=?0.83, estimated AUC of ROC?=?0.88) suggested good fit and
stability. Importantly, although SVM modeled directly on SZ data performed slightly
better than when parameters extracted from HC Model 1 were applied to SZ data (i.e.,
F1 scores of 0.80 and 0.77, respectively), gamma activity at encoding received the
highest weightings in both cases.

Table 4. EEG Features Predicting Trial Accuracy in SZ

Model 2: Classification of diagnostic status

Features selected by SVM models used to classify diagnostic status (SZ labeled +1
and HC labeled ?1) based on correct and incorrect behavioral responses are presented
in Tables 5 and 6, respectively. Overall classification accuracy of 79 % was achieved by EEG features
selected from correct response trials, with higher values of frontal and central theta
at baseline associated with SZ group membership (Table 5). Gamma band activity during retrieval and encoding stages also entered the model
but with relatively low weightings. Performance statistics of this diagnostic classification
model were acceptable (F1 score?=?0.87, precision?=?0.77, recall?=?1, estimated AUC
of ROC?=?0.77). SVM modeled on incorrect trials (Table 6) identified frontal alpha at retrieval as the highest weighted feature, with a near-zero
contribution of central gamma during encoding. The valence of coefficients indicated
that higher values were associated with HC group membership. Performance statistics
of this diagnostic classification model using incorrect trial data were exactly identical
to those of the other model using correct trial data. Taken together, these findings
are interpreted to suggest that SZ is generally distinguished from HC by higher levels
of low-frequency (theta 1) spectral power at pre-trial baseline, and lower levels
of alpha band power during retrieval than HC, particularly when WM load exceeds capacity
(i.e., incorrect responses).

Table 5. EEG Features Predicting Diagnostic Group Based on Correct Trials

Table 6. EEG Features Predicting Diagnostic Group Based on Incorrect Trials

Concurrent and external validity

SVM Model 1

As a test of concurrent validity based on classification method, EEG features selected
by SVM Model 1 in HC data (Table 2) were submitted to stepwise linear regression as predictors of SWMT total score in
the full sample of HC and SZ participants (N?=?52). The model was highly statistically significant (F_{(4, 47)}
?=?37.67, p??0.0005, R?=?0.87) and explained 76 % of the variance in SWMT score (Fig. 2). Central theta 1 during retention in correct trials entered as the first step, frontal
gamma during encoding in correct trials as the second step, frontal gamma during encoding
in incorrect trials as the third step, and central theta 1 during retention in incorrect
trials as the fourth and final step (Table 7). Beta and partial correlation coefficients suggested that when participants answered
incorrectly, presumably challenged by higher WM load, performance was associated with
higher levels of frontal gamma during encoding and central theta 1 power during retention
(beta?=?0.38 and 0.39, partial r?=?0.51 and 0.48, respectively), while lower levels were associated with correct responses
(beta?=??0.44 and ?0.55, partial r?=??0.62 and ?0.61, respectively). The same EEG features were retained, with exactly
the same model coefficients, when the regression analysis was repeated by replacing
the predictors with the 1
^st
ranked feature of each WM stage (Table 3).

Fig. 2. Scatterplot of Sternberg Working Memory Task (SWMT) performance (out of 90 trials
possible) as predicted by SVM Model 1 across the full study sample (N = 52). Multiple regression explained 76 % of the variance in SWMT performance based
on frontal gamma activity during encoding and central theta 1 activity during retention.
Both correct and incorrect trials entered the model for each feature. SVM Model 1
score (x-axis) represents the residual difference between predicted (trend line) and
observed value for SWMT performance

Table 7. Multiple Linear Regression Model Predicting SWMT Performance by SVM Model 1 Features

To examine external validity of the EEG features derived by SVM, the same regression
model was repeated to predict MCCB WM Composite (Fig. 3) and CPT-IP (Fig. 4) scores in separate analysis. MCCB WM Composite score was predicted (F_{(2, 49)}
?=?17.39, p??0.0005, R?=?0.64) with 42 % of variance explained by two features, i.e., frontal gamma during
encoding in correct trials (R²
?=?0.31, F change
_{(1, 50)}
?=?23.92, significant F change??0.0005) and central theta 1 during retention in correct
trials (R²
?=?0.42, R²
change?=?0.09, F change
_{(1, 49)}
?=?7.67, significant F change?=?0.008). The direction of association was consistent
with previous models, with frontal gamma at encoding (beta?=??0.59, partial r?=??0.61) and central theta 1 at retention (beta?=??0.30, partial r?=??0.37) associated negatively with working memory test performance. The CPT-IP was
selected as an additional cross-validation measure due to dependence of this task
on visual encoding and retrieval processes similar to the SWMT. CPT-IP performance
was predicted with 39 % of variance explained (F_{(3, 48)}
?=?10.15, p??0.0005, R?=?0.62) based on three features: frontal gamma activity at encoding in correct trials
as the first step (R²
?=?0.19, F change
_{(1, 50)}
?=?11.54, significant F change?=?0.001), central theta 1 activity at retention in
correct trials as the second step (R²
?=?30, R²
change?=?0.11, F change
_{(1, 49)}
?=?7.90, significant F change?=?0.007), and occipital gamma activity at retrieval
in incorrect trials as the third step (R²
?=?39, R²
change?=?0.09, F change
_{(1, 48)}
?=?6.89, significant F change?=?0.012). Consistent with prior models, beta and partial
correlations for frontal gamma during encoding and central delta during retention
in correct trials were negatively associated with CPT-IP AGT (beta?=??0.45 and ?0.32,
partial r?=??0.50 and ?0.37, respectively) while occipital gamma during retrieval in incorrect
trials entered with positive coefficients (beta?=?0.30, partial correlation?=?0.35).
These results confirmed that EEG features modeled on SWMT performance are generalizable
with respect to neuropsychological measures of working memory and visual attention.

Fig. 3. Scatterplot of MCCB Working Memory (WM) Composite score (standardized; t-score) as
predicted by SVM Model 1 across the full study sample (N = 52). Multiple regression explained 42 % of the variance in MCCB WM score based
on frontal gamma activity during encoding and central theta 1 activity during retention,
with only data from correct trials entered entering the model for each feature. SVM
Model 1 score (x-axis) represents the residual difference between predicted (trend
line) and observed value for MCCB WM score

Fig. 4. Scatterplot of Continuous Performance Test-Identical Pairs version (CPT-IP) score
(standardized; t-score) as predicted by SVM Model 1 across the full study sample (N = 52). Multiple regression explained 39 % of the variance in CPT-IP score based on
frontal gamma activity during encoding and central theta 1 activity during retention
for correct trials and occipital gamma activity at retrieval for incorrect trials.
SVM Model 1 score (x-axis) represents the residual difference between predicted (trend
line) and observed value for CPT-IP score

SVM Model 2

To cross-validate the diagnostic classification accuracy of SVM Model 2, derived features
(Tables 5 and 6), were submitted to discriminant function analyses of diagnostic membership (i.e.,
HC vs. SZ) using stepwise entry. The overall Wilk’s lambda, ??=?0.59, ?²
(df?=?4)?=?25.67, p??0.0001, indicated that there was a significant group-wise difference by diagnosis
across four retained EEG features, with group centroids of 1.43 and -.48 for HC and
SZ, respectively. The correlation structure of the discriminant function (Table 8) indicates that HC was classified with higher frontal alpha at retrieval and central
gamma at encoding on incorrect trials, while SZ was associated with higher frontal
theta 1 at baseline and central gamma at retrieval on correct trials. Overall diagnostic
classification accuracy in the full sample was 87 % (sensitivity 90 %, specificity
77 %) with positive predicative power (SZ diagnosis) of 92 % and negative predictive
power of 71 % probability. Leave-one-out cross-validation of this model replicated
classifications with 83 % accuracy.

Table 8. Discriminant Function Structure Matrix