Evaluating the implementation of a national clinical programme for diabetes to standardise and improve services: a realist evaluation protocol

Realist evaluation, which allows for the study of context and its influence on outcomes, is appropriate for examining the implementation of the NDP, given the history of regional variation in diabetes services in Ireland. This paper outlines the protocol for a mixed methods evaluation to explore which aspects of the programme are working, for whom and in which circumstances.

Geographic case studies are often difficult to define [52], and this has been a particular challenge in this study given the ill-defined boundaries of health services in Ireland. Catchment areas for health services and hospitals are often fluid, and the organisational structures within the health service have gone through several recent reconfigurations. Furthermore, the results of stage 1 of this study suggest variation in diabetes services within regions and counties depending on the local resources, infrastructure and engagement from stakeholders such as GPs and local management. We have selected cases for stage 2 on the basis of these preliminary findings. The aim of this study is to understand how the NDP is working, for whom and in what circumstances. Therefore, the cases are considered instrumental as opposed to intrinsic [58], that is they are being used to gain a deeper understanding of programme implementation as a whole, as opposed to focusing on the uniqueness of the individual case itself.

We have used a number of strategies to enhance the rigour of this study. Data collection tools including topic guides and surveys have been extensively piloted. Different triangulation techniques will be used to strengthen the validity of findings, including the use of mixed methods, multiple data sources (interviews and documents) and researchers from different disciplinary perspectives (health services research, epidemiology, public health, clinical medicine) [59]. While the limitations of member checking as a strategy to verify overall results have been highlighted, it is considered appropriate to enhance validity in case study research, as case reports maintain the contextual information that allows participants to relate their experiences to synthesised results [52, 60].

Throughout this study, data will be collected and analysed concurrently within each stage to allow emergent lines of enquiry to be explored [60]. The research team has endeavoured to be responsive to the implementation of the programme and changeable context in which it is being rolled out. For example, in October 2015, the Department of Health in Ireland agreed a new contract with GPs which provided financial reimbursement for two structured diabetes review visits in general practice per year. This scheme is known as the Diabetes Cycle of Care. Patients with type 2 diabetes who have a medical card or a GP visit card, which entitles them to free GP care in Ireland, are eligible to be registered by their GP for the scheme. This is a significant influential factor in the context of the NDP. Although not part of the initial programme theories, given its recent introduction, we have adapted our topic guide to explore how the introduction of this financial incentive may influence implementation.

Complex social interventions such as the NDP achieve their outcomes by active input from various stakeholders. Qualitative research is an important part of exploring the reasoning and responses of stakeholders to a programme [37]. Similar to other realist evaluations [61], the results of our interviews with national programme stakeholders, who were also local implementers with context-specific experience, further refined ‘official’ programme theories about which aspects are working, in which circumstances and why. This evaluation builds on previous work by the research team which analysed the many diabetes care policies in Ireland, thereby providing information on some of the contextual factors that preceded the national programme [24]. Pawson suggests that by defining clearly the boundaries of case studies, evaluators are then able to harness the potential of administrative data, for example, relevant to quantifying the outcomes of programmes in realist evaluation [37]. Collaboration with the NDP has enabled us to analyse such administrative information where available. However, we are limited by the lack of a diabetes register in Ireland or national databases on the quality of diabetes care, and patients’ health service interactions and outcomes.

There is increasing interest in the evaluation of health policy and health service implementation. In particular, there is increasing emphasis on theory-based evaluations which aim to establish the context and mechanisms that facilitate successful implementation rather than simply focusing on the achievement of specific endpoints [35]. Realist evaluation has been used for this purpose to study a diverse range of service changes including the introduction of an integrated care pathway for palliative care [41], a multifaceted maternity care programme [49], ‘communities of practice’ [38], oncology teams [54] and quality improvement in primary care [51]. There are very few evaluations of the implementation of programmes or service interventions in Ireland, and to our knowledge, this is the first realist evaluation of a programme in Ireland.

The NDP is constantly moving between planning for future work streams and ongoing implementation of the current work streams. Therefore, the programme offers a potentially unique opportunity to evaluate and inform the implementation of changes in the Irish health system as they emerge and evolve. For example, there has been phased recruitment of integrated care diabetes nurse specialists (known as integrated care nurses (ICNs)) to support the implementation of the national model of integrated care as resources have been secured at national level. A protocol has been developed to clarify the role of the ICN, partly in response to barriers to implementation highlighted in the evaluation. There is close collaboration between the national working group and research team; the principal investigator (PK) is a member of the working group which provides an opportunity to provide formative feedback on implementation to those responsible. Our results should also provide insights relevant to the implementation of other clinical care programmes in Ireland operating in similar contexts. Furthermore, we hope that the findings will be relevant to programmes in other countries, some of which are also evaluating implementation of new care programmes [62].