Exploring the use of research evidence in health-enhancing physical activity policies

Each research team in the partner countries suggested relevant HEPA policies for consideration.
Prior to the selection of HEPA policies for the study, the research team, e.g. principal
researchers in each partner country, looked at the objectives, primary stakeholders
involved, subpopulation specificity of policies, processes leading to release of the
policies, and selection of documents used for the development of the policies. The
policies were then categorised as national, regional, or local. Based on this primary
inventory and in order to focus on the most recent developments, documents were selected
according to the criteria of policies focusing on HEPA, being published by public
authorities and representing HEPA policies in force at the beginning of 2012. The
selected national, regional, and local HEPA policies ensured at least some variation
in the scale and topic of the expected policy change: from major changes, such as
governance structures and legislative perspective, to minor changes such as programmes,
advice services, or building playgrounds. Some of the national, regional, and local
policies belong to a ‘package’ of national, regional, or local policies created after
elections for respective development purposes. They therefore also reflect wider policymaking
processes and possibly the use of a wide base of research evidence. In the case of
‘package’ policies, the study concentrated on the specific policy documents for HEPA
issues only. For the final set of HEPA policies, 21 policies from six European countries
were selected with variation across the types of policies and policymaking processes.
The topics of the sample of HEPA policies included in the analysis contained public
health aspects (prevention, health promotion, and nutrition), HEPA aspects (sport,
movement for health and sport for all, Olympics), infrastructure (transport, walking
and cycling, sports halls and gyms), and places for PA (youth, schools, and neighbourhoods)
29]. The study included countries which already had an evidence-informed approach as
normal practice (England), under development (Finland and the Netherlands), and similar
to normal practice (Denmark, Italy, and Romania). The second criterion was that different
regions within the European Union had to be represented since the project was funded
by the Seventh Framework Programme of the European Commission.

To assess the use of research evidence in HEPA policies, content analyses of policy
documents and semi-structured interviews were used. The content analyses of the policy
documents and the interviews complemented each other to identify the various ways
of using evidence in policymaking. The intention was to facilitate comparisons between
countries in the use of research evidence in policymaking and take into consideration
contextual differences through interviews. In each country, a policy was identified
along with the content, stakeholders, and processes in relation to the relevant research.
A series of questions was put to the stakeholders and policymakers to recall the use
of research evidence. Further questions were allocated depending on the references
to the use of research evidence in speeches, statements, guidelines, and similar background
documents.

The HEPA policy process was split into agenda-setting and policy development phases
in order to identify the use of research evidence at different stages of the policymaking
process on various topics and issues. The agenda-setting phase and the use of research
evidence were mostly traced through interviews and the use of research evidence in
the policymaking process through policies or other supporting documents. The agenda-setting
phase and the use of research evidence were dependent on interviews, as most of the
countries did not have records on the decisions made in the agenda-setting phase,
whereas the policymaking phase was often described in policies or related background
documents.

Content analysis of policy documents

Based on the literature review for the project proposal and its update upon acceptance
of funding for the use of research evidence in policymaking, the study was undertaken
using a qualitative descriptive approach inspired by political sciences 30], 31], public health sciences 15], 32], 33], and the multidisciplinary field of knowledge transfer, knowledge utilisation, and
lesson learning 34]–36]. In many policy processes, colleagues or areas of expertise are also commonly used
for evidence-informed decisions and policymaking 37], 38]. Given the complex processes of policymaking, content analysis of policy documents
and stakeholder interviews was selected to find how tacit (unspoken), implicit, and
indirect knowledge and opinions shaped the policymaking processes and how more explicit
and particular use of research evidence was integrated into policymaking processes.

The content analysis of HEPA policy documents followed the ideas of Ritchie and Spencer
39] and consisted of the process of analysing policy documents by issues and topics with
the help of a set of guiding questions. After mapping the issues and topics, the HEPA
documents were further analysed to establish type and use of evidence for each policy.
Each HEPA policy was reread, sifted, charted, and sorted according to the key issues,
topics and themes, confirming the patterns of research and other evidence used, and
the research evidence was categorised into various types of evidence used. When research
evidence was cited, the content was analysed to establish how the citation supported
the policy statements or position in addressing the issues. The citable research considered
consisted of journal articles, book chapters, and working papers and reports typically
produced by research institutes, universities, and other independent research units.
Other types of evidence considered were readings, media including the Internet and
news, interactions with peers or stakeholders, and the involvement of participants
in hearings, working groups, meetings, and the like.

Where appropriate, a published protocol and tool developed by Lavis et al. 15] and Hanney et al. 11] were used to identify, review, and locate the explicit use of research evidence in
policy documents through content analysis. The REPOPA research team, lead by the work
package leader, prepared a common guideline for all partners for the content analysis
of policy documents and to indicate the information to be retrieved from the selected
policy documents. Thereafter, the findings were reported in English and summarized
into one report. The implementation phase of the policies was not included in the
analysis, since this lies beyond the scope of the present study.

The common guideline that was developed covered the criteria for the selection of
policies, theoretical models of the policymaking phases based on Kingdon 26], the focus of analysis in relation to topics, goals, and processes in the form of
thematic questions, identification of stakeholders, the process description of the
policy analysis and instructions for the HEPA policy analysis of the role of evidence
in policymaking based on Ritchie and Spencer 39], and a schematic example of the analysed text of a policy.

Semi-structured interviews, interview guide, and topics covered

During the HEPA policy document analysis, key informants for each policy were identified
for face-to-face semi-structured interviews in order to verify the findings of the
content analysis stage and gather information on any gaps in the uncited or implicit
use of research evidence. The purposeful sampling of 87 informants in six countries
was based on the selection criteria of the interviewees being directly involved in
the policymaking process and able to report on the use of research evidence or other
kinds of evidence in the policymaking process. In each policy, one of the interviewees
was employed at the organisation responsible for the policy. The interviewees had
been involved in policymaking processes of the policy on which they were interviewed
and were policymakers, researchers, public sector officers, or other influential stakeholders.
All the interviewees were contacted by email or phone by the research team in each
country with basic information on the project and consent forms in the local language.
The interviews were conducted by research team members with backgrounds in health
and social sciences in the local language, recorded when accepted and transcribed
for the analysis. An interview guide was developed by the REPOPA team led by the work
package leader. In the guide, the questions for the policymaking process were split
into agenda-setting and policy development phases. The questions followed the protocols
11], 15] with adaptation to the context and gaps in information after document analysis. A
consent form, description of the research project, and preliminary list of questions
were also provided. The interview questions were based on the analysis of the policy
documents, especially the gaps found in terms of the evidence used. They followed
the same structure, topics, and issues. The interviews were conducted after the document
analysis of the policies.

To adapt the interviews for each context, each country team conducted between one
and three pilot interviews to modify the questions, interview process, and language.

The semi-structured interviews were selected as a method to gain complementary information
and to facilitate the adaptation of questions to each policy and case. The stakeholder
interviews for each policy verified facts identified during the content analysis of
policy documents, gaps in the information gathered in the use of research evidence,
and the needs of policymakers to use evidence in the policymaking process.

In the semi-structured interviews, the interviewees were asked to recall the policymaking
process period, review their files before the interview, and explain the policymaking
process and the parties involved. The main issues in the interviews were how and why
research evidence and other types of evidence did or did not enter into the policymaking,
as well as the origin, influence, and characters of policymaking. The topics covered
included policy changes, their significance, the use of research evidence and other
types of evidence in bringing the content and change into the policy, the factors
describing how and why the issues appeared on the policy agenda, and the factors influencing
the way the policy developed.

Analysis of research evidence in policy documents and interviews

The selected HEPA policy documents and interviews were mapped, coded, and further
analysed using the interview questions as guidance. Each HEPA policy and transcribed
interview was reread, coded, sifted, charted, and sorted according to the key issues
and themes, confirming the patterns of use of research and other evidence, and categorised
into various topics and themes. The interviews were analysed by the country-based
research teams using an interpretative approach derived from a content analysis of
the policy 38]. The coding was performed by the interviewers, and the accuracy of the content analysis
was supported by independent assessments by the team members in the country research
teams. The data from the interviews were enriched by the policy document analysis,
and sometimes with additional available documentation, such as speeches, statements,
guidelines, and similar background documents. However, in most cases, additional documents
were not available and thus the overall analysis of the use of research evidence relied
on data obtained from the analysed policy documents and interviews.

The policies and interviews were further analysed to identify the use of evidence
to support the policies. For the distinction between research evidence and other types
of evidence, a list of evidence with modifications was developed based on Lavis, Ross,
and Hurley 15]. Evidence was considered as:

Demographic and statistical data (facts and reports used as a background or for prioritising
policy areas)

Non-systematically derived peer-reviewed scientific literature (an ad hoc search
for research articles or other similar materials that are not systematically assessed
and applied)

Systematically derived peer-reviewed scientific literature (literature derived from
a systematic literature search in databases, reviewed, summarised, and contextualised
to a specific policy context)

International, transnational, national, regional, or local standards and guidelines

Knowledge derived from community consultations, stakeholder workshops, and in-house
consultations

National reports (national reports on urban development and HEPA or other reports)

Knowledge derived from expert consultations or policy briefs developed by research
institutions

In the content analysis stage, the policy documents were reviewed to identify the
explicit use of research evidence. When research evidence was cited, the content was
analysed to establish the way the citation supported the policy statements or position
in addressing the issues. The research evidence was published as journal articles,
book chapters, working papers, or reports typically produced by research institutes,
universities, or other independent research units. Other types of evidence referred
to readings, information from media, and interaction with peers or stakeholders. Involvement
of people in hearings, working groups, meetings, and the like was also looked for
in the policy documents and interviews.

Finding patterns when research was used required a framework for determining the contexts
in which policymaking occurred 11], 27], 40]. In accordance with Bowen et al. 40], we identified factors influencing policymaking, such as ideas, ideological or political
values, and interests and institutions from the policy documents and interviews. The
content analysis of HEPA policies and interviews detected and defined these items
as evidence used in the HEPA policy development process, e.g. the content analysis
of the document and interview themes categorised and classified them and looked for
evidence of use: what constituted research evidence or other kinds of evidence in
HEPA policy?