Process evaluation framework, measures, and analyses
The RE-AIM framework was adapted to satisfy the needs of an environmental intervention [26–28]. The definitions and associated process evaluation measures of Reach, Effectiveness, Adoption, Implementation, and Maintenance are described in Table 1. (Effectiveness has been interpreted as potential external influences on the intervention, whereas overall effectiveness of the SLM interventions, measured by student food selection and tray waste, will be reported in a forthcoming paper.) All described process evaluation measures were developed for this study and reviewed by the research team for face validity.
RE-AIM dimensions and application to process evaluation measures
Dimension
Definitiona
Process evaluation measures
Reach
Number of people and percentage of the target population affected by the environmental change (and the extent to which the individuals reached are representative and include those most at risk).
Described using proportional reach, i.e., the number and proportion of enrolled students participating in the National School Lunch Program, including the number of low-incomeb students. Data were retrieved from the New York State Department of Education [29].
Effectiveness
A measure of effects on health behaviors, including positive, negative, and unanticipated consequences.
Described as potential external influences on intervention effectiveness, i.e., contamination or threats to internal validity and aspects of cafeteria environments aligned with protocol prior to implementation, captured by school environmental assessments and lunchroom audits. Note: Student behavioral outcomes will be reported elsewhere.
Adoption
Number, proportion, and representativeness of settings and intervention agents participating, and the extent to which the settings selected are representative of settings that the target population will use or visit.
Number and characteristics of participating schools. Also described as the number of interventionists and providers trained in Smarter Lunchrooms protocol and their reported preparedness to initiate intervention, detailed in provider training progression records, interventionist and provider training evaluations and interviews, and contact logs.
Implementation
Level of adherence to implementation guidelines and the extent to which elements are implemented.
Described as fidelity (including fidelity scores by school service lines) to intervention protocols assessed via contact logs, lunchroom audits and interventionist and provider interviews.
Maintenance
At the setting level, the extent to which change is maintained or new barriers are prevented or reduced.
Described as fidelity scores by school service lines to intervention protocols beyond the intervention end date, assessed via lunchroom audits two weeks post-intervention.
Qualitative and semi-quantitative methods were employed to collect process data for the dimensions of the adapted RE-AIM framework: school environmental assessments, school lunchroom audits, provider training records and evaluations, weekly contact logs, and post-intervention semi-structured interviews.
With assistance from school administrators and staff, interventionists completed school environmental assessment forms created to document school wellness policies and committees, parent and teacher association activities, nutrition and health education in classrooms, and other relevant health promotion activities in participating schools. Researchers reviewed these assessments for evidence of contamination or co-occurring activities that might impact student selection and consumption of foods promoted through the intervention.
School lunchroom audits were conducted in each school before, during, and after the intervention. The mid-intervention audit occurred during week three or four of the intervention to ensure providers had enough time to enact the protocol comfortably, avoiding days in which tray waste was collected to minimize the potential of compounded observer effect. Audits were performed by the research team and included field notes, a comparison of the intervention protocol items with actual lunchroom and service line set-ups, diagrams of each line set-up, and photographs of each line. To generate a semi-quantitative measure of implementation fidelity, each service line received a score for each intervention protocol item: 2 for full compliance, 1 for partial compliance, and 0 for non-compliance. Scores were calculated for all service lines in participating schools. Line scores were averaged within schools, and averages across schools in urban and rural areas were calculated. Averages across all schools were calculated to provide an idea of intervention compliance.
The research team conducted interventionist training. The two-day trainings, designed using Principles of Adult Learning Theory [32] to accommodate multiple learning styles, included an introduction to behavioral economics, project overview and logistics, and a walkthrough of the expected provider training. Interventionists completed evaluation forms following these trainings in which understanding and preparedness were assessed with 12 items using a 5-point Likert scale (strongly disagree – strongly agree). Items included statements such as:
“…I have a better understanding of the study design, including the intervention, process evaluation, and outcome evaluation.”
“…I understand the differences between activities in intervention and control schools.”
“…I feel prepared to deliver training to intervention school staff.”
Item responses were converted to scores and averaged within and across all schools. A total mean across all items was also calculated. Open-ended questions were used to solicit opinions regarding valuable aspects of training and suggestions for improvement. These results were reviewed and summarized for all interventionists.
Interventionists delivered treatment-specific (milk, fruit, or vegetable), one-hour trainings to providers. Control school providers participated in an unrelated kitchen and food safety training created by the National Foodservice Management Institute [33]. Trainings were based on Adult Learning Theory [32] and designed for an audience with a low literacy level in response to provider needs. Specific learning objectives for providers in treatment schools were 1) to develop a working knowledge of SLM rationale, 2) to gain information related to the SLM research project description, purpose, team and timeline, 3) to develop a firm understanding of the specific change to be made in their school’s lunchroom, and 4) to practice using implementation tools. Training records were completed by interventionists and used to document training delivery. Records were reviewed for indicators that the training had been delivered as intended, and any modifications were noted. Providers completed a training evaluation that consisted of ten items assessing understanding and preparedness to implement the intervention using a 5-point Likert scale (strongly disagree – strongly agree), as well as open-ended questions to further solicit training feedback.
Likert scale items included statements such as:
“…I know what changes need to be made in my school lunchroom.”
“…I am comfortable with Smarter Lunchrooms tools (sings, planograms, etc.).”
“…I am prepared to carry out my role in the Smarter Lunchrooms project.”
As with interventionist evaluations, statement responses were averaged and a total mean calculated. Responses to open-ended questions were reviewed and summarized for all providers.
Weekly contact logs were completed by the interventionists to record providers’ experiences with implementation, including any challenges or concerns. Logs were also reviewed for evidence of adoption/implementation fidelity. Additionally, members of the research team conducted post-intervention semi-structured interviews with a sample of interventionists and providers regarding their experiences implementing SLM in their schools in order to understand barriers and facilitators to implementation, determine the strengths and weaknesses of trainings, and clarify which elements of the intervention were maintained. All 16 interventionists and at least one provider from each of the 12 participating treatment schools (typically a manager with knowledge of day-to-day activities) were invited to participate in post-intervention interviews.
Interventionists were asked about recruitment of schools, training they received, training they delivered to providers, and the help and support they received from the research team. Providers identified challenges with implementation, provided feedback on their training, delineated support they received from interventionists, and described any observed intervention outcomes. Interviewees were also asked if there was other information they would like to discuss. Interviews were recorded and transcribed verbatim. Researchers used ATLAS.ti (version 7, 2012, ATLAS.ti Scientific Software Development GmbH, Berlin, Germany) qualitative data management software to code commonalities as well as discrepancies in reported successes and challenges by treatment and across sites.