Protocol for the evaluation of a free health insurance card scheme for poor pregnant women in Mbeya region in Tanzania: a controlled-before and after study

Impact evaluation

Study design

The impact evaluation will employ a controlled before and after study design. Surveys
will be undertaken in one district (Mbarali) in Mbeya region before and after the
introduction of the KfW scheme and also in one comparison district with no scheme
(Kilolo). The comparison district was selected from a neighbouring region and was
similar to the intervention district in terms of baseline CHF coverage, poverty and
literacy rates, population density and population per health facility.

The impact evaluation relies on four tools that will be administered before the scheme
is implemented and 17 months after implementation started: a health facility survey,
a survey of patients exiting facilities, a client-provider interaction observation
checklist, and a household survey of women who delivered in the previous 12 months
(Fig. 2). The facility survey, exit interviews and client-provider observations will be conducted
at 48 sampled facilities across intervention and comparison sites. The household survey
will be administered to 3000 households within the catchment areas of these facilities
to complement the data compiled during the facility survey 11].

Fig. 2. Overview of impact evaluation data collection tools and sample sizes

Data collection tools

The health facility survey aims to measure the effect of the KfW scheme on service
availability and quality at the sampled facilities. The survey captures information
on basic service provision within the facility (e.g. staffing levels, opening hours,
facility management, as well as facility infrastructure), equipment and drug availability,
and service utilization from patient registers and facility expenditures and revenues.
The health facility survey will be administered to the facility in-charge or, in his
or her absence, to a knowledgeable health worker or administrator. It also includes
data extraction from patient registers and facility records.

The exit interview primarily intends to measure the effect of the KfW scheme on patient
experience of care and the cost of services. Respondents eligible for interview include
women of reproductive age (aged between 16 to 49 years) attending antenatal or postnatal
care or childhood immunisation services within three months after delivery. A medical
doctor or nurse will be trained to observe consultations with these patients and to
complete a checklist to assess clinical care in relation to the national clinical
guidelines.

A survey of women who had delivered within the previous 12 months will also be carried
out. The women’s survey addresses the effects of the KfW scheme on care seeking and
associated costs incurred during pregnancy and the postpartum as well as service satisfaction.
The household head is also interviewed to ascertain health care utilisation rates
and out of pocket payments for care seeking in the past month (outpatient care) and
the past year (inpatient care) along with household socioeconomic status. The core
indicators for each of the surveys are shown in Table 1.

Table 1. Overview of core indicators for impact evaluation

Sampling

The health facility is the primary sampling unit. Facilities were sampled from all
facilities accredited by the NHIF within the selected districts. The government hospital
and the health centre in each district were automatically selected (Fig. 2). A random sample of 22 dispensaries out of those which offered reproductive and
child health (RCH) services were selected from each district. The total number of
facilities sampled was 24 per district, representing over 60 % of all facilities in
each of the two districts. The aim of the sampling procedure for the health facility
survey was to seek district representation, therefore, no sample size calculation
was carried out.

A total of 12 exit interviews and client provider observations will be carried out
per facility at each round of data collection. The aim will be to achieve a balance
between antenatal care (ANC) and postnatal care (PNC) or immunization service users
within three months after birth (aiming for 6 ANC clients and 6 PNC or immunization
clients per facility). Patients will be approached upon entry to the health facility
regarding their participation in the exit interview. A series of screening questions
will be used to identify eligible respondents who will be asked for their consent
to participate in the study. Consenting respondents will be monitored from their time
of arrival at the facility until their time of departure, and the waiting and consultation
time will be measured using a stop watch. Patients and providers will also be asked
for their consent for a medically trained interviewer to observe the consultation
and complete an observation check list for ANC and PNC clients. Upon leaving the consultation
room, the patient will then be asked for their consent to participate in the exit
interview in a quiet location within the facility, at distance from providers and
from other patients. At baseline, the criteria for selection will be that patients
are uninsured. At endline the criteria for selection is that patients do not have
any supplementary private health insurance, but patients with a CHF card or an NHIF
card obtained through the KfW scheme will be eligible for interview.

For the household survey, the sample size calculation was based on the formula by
Hayes and Bennett, 1999, adjusted for the cluster design of the study at the facility
level 23]. We estimated that the required sample size to detect an 11 percentage point difference-in-differences
increase in institutional deliveries (from 50 to 61 %), with an assumed coefficient
of variation (standard deviation/mean) of the true rates between clusters within each
group k value of 0.25, 90 % power, significance at 0.05 (two tailed test), and a 90 %
response rate, was 60 households per cluster, equivalent to 1500 recently delivered
women per study arm per round of data collection. Hence, the target sample was a total
of 3000 recently delivered women per round of data collection. In order to identify
eligible households, villages are sampled from the facility catchment area. Three
villages will be sampled from the ward where the facility is located. All hamlets
(comprising approximately 100 households) within the sampled villages will be identified
and a random sample of four hamlets will be sampled. Five households will be sampled
from each of the hamlets, amounting to a total of 60 households within each facility’s
catchment area; households will be selected at random from the selected hamlets using
a modified Expanded Programme of Immunisation (EPI) type sampling scheme that ensures
an equal chance of any household being selected. In the sampled hamlet, the supervisor
will aim to identify on average 3 households that scored “poor” and 2 that scored
“nonpoor” (e.g. “average” or “rich”).

In order to be eligible for interview, households must include a woman who has delivered
within the previous 12 months. At baseline the selection of households was also limited
to those who were uninsured. At endline, eligible households included those who were
uninsured, were insured with the CHF or insured by the NHIF through the KfW scheme.
If there is an eligible woman, the supervisor will then ask permission from the respondent
to complete a form to assess the socio economic status of the household. The supervisor
will score the household from 1 to 3 on questions related to household characteristics
(e.g. type of roof, water source, toilet facilities, average number of meals eaten
per day, daily income, number of children in the house etc.). In the sampled hamlet,
the supervisor will aim to identify on average 3 households that scored “poor” and
2 that scored “average” or “rich”. The objective will be to interview 40 households
who are of poor or average wealth and 20 households who are not (least poor) per facility.
The score sheet is the same tool originally proposed by the NHIF to identify beneficiaries.

Process evaluation

The process evaluation will undertake ongoing descriptive and mixed methods assessment
of the process of implementation, documenting the role and perspectives of key stakeholders
at each stage of the process, and at each level of the health system, to ascertain
how the scheme is implemented in practice. The evaluation will also track the degree
to which implementation has occurred according to the design documentation (fidelity
of implementation). Care will be taken to identify and monitor structural and contextual
factors that may influence the observed implementation and outcomes. Ultimately, through
implementation research we aim to determine what is the “core” of the intervention
– the essential elements always necessary for it to be effective – and what is the
“adaptive periphery” – i.e., those aspects of the intervention that can (and must)
be adapted to fit the context.

The process evaluation will undertake three rounds of data collection at baseline,
14 and 18 months after implementation began in the selected intervention district
(Mbarali). Focus group discussions and in-depth interviews will be carried out among
a purposive sample of consenting individuals at different levels of the health system
(community, facility, district, regional and national) to assess perspectives and
attitudes towards the intervention as well as to routinely identify process bottlenecks.
Three intervention facilities in Mbarali have been selected for an in-depth case study:
a government hospital, health centre and dispensary. Staff at each of these health
facilities will be interviewed, along with village leaders and focus group discussions
with scheme beneficiaries and others in the community. These data will be triangulated
with indicators for monitoring the fidelity of the intervention and its implementation
process. The quantitative measures on content, coverage, frequency and duration will
be derived from the household and exit surveys and project and health facility statistics.
The analysis will also include a review of relevant project and policy documents and
materials.

Economic evaluation

The economic evaluation will assess the incremental cost-effectiveness of the KfW
scheme relative to current practice. The study will be carried out from a societal
perspective, which includes all agencies or bodies that are involved in implementation
or who incur costs or may be affected by the intervention, for example: the implementers
and the beneficiaries.

We will estimate the start-up and ongoing financial costs of the KfW scheme (i.e.
all financial transactions made by the funder), as well as the economic costs, which
values all resources required to set up and implement the scheme. Under economic costing,
donated or subsidised items will be valued at market prices.

Project costs will be measured with reference to project accounts and through interviews
with key implementation stakeholders at national, regional and district levels. Potential
health system costs resulting from increased service use by scheme beneficiaries will
be assessed by measuring any observed changes in staffing levels and bed numbers.
Household costs and care seeking will be captured during the baseline and endline
household surveys. Effectiveness is defined in relation to service coverage measured
in the household survey.

A series of one way sensitivity analyses will be conducted to explore the impact of
uncertainty on incremental cost-effectiveness.

Data management

Household and exit interview data will be collected using hand held devices (Huawei
IDEOS phones and Samsung Galaxy Tablets 7.0) loaded with Pendragon data collection
software with skip and quality check functions to minimize data entry error. Facility
survey data and client provider observations will be captured on paper and double
entered into a pre-designed database. Data will be transferred into a Microsoft Access
Database, and converted to Stata for analysis. Hard copies of questionnaires will
be stored in a locked room. Electronic output will be de-identified.

Interviews and focus groups conducted as part of the process and economic evaluations
will be conducted in Kiswahili by trained research assistants and recorded using audio
digital recorders. Audio files will be transcribed by research assistants who conducted
the interviews and will be translated into English by the bilingual researcher who
also conducts the interviews. All transcripts will be imported into QSR Nvivo 8 for
data management, for the process evaluation, and entered into Microsoft Excel for
the economic evaluation.

Analysis – quantitative data

Consistency checks will be conducted on the data from the baseline and endline surveys
along with data cleaning. A comparison of all variables between intervention and control
arms will be made at baseline through tests of differences in means using the Adjusted
Wald F-test.

Socioeconomic status (SES) indices will be derived from data collected on household
size and characteristics, access to utilities, durable asset ownership, food security,
household expenditures, head of household marital status, highest level of education
attained, and main occupation, using principal component analysis (PCA). We will rank
individuals according to their index score and generate wealth terciles, three equally
sized groups. Patient satisfaction data derived from a 3-point. Likert scale (e.g.,
dissatisfied?=?1, neither satisfied nor dissatisfied?=?2, satisfied?=?3) will be analyzed
by calculating mean scores for each variable.

At endline, we will compare the main outcome indicators for each of the survey tools
between intervention and control arms, using data for 17 months of intervention implementation.

A difference-in-differences regression analysis will be conducted to assess the independent
effect of the KfW scheme on outcomes controlling for all other individual, household
and facility level factors which may influence the given outcome. The ordinary least
squares linear regression model will be used and we will control for facility fixed
effects. For household, exit and patient observations data, we will calculate robust
standard errors, clustered at the facility level, to correct for correlation of the
error terms across patients within facilities, and across households in facility catchment
areas.

Analysis– qualitative data

Qualitative data will be coded based on themes identified in the conceptual framework
and adapted through an iterative process based on the data. Axial (line-by-line) coding
will be conducted using Nvivo 8 software (QSR International Pty Ltd, Australia). A
sample of the transcripts will be coded by a second researcher to insure reliability
of the coding scheme. To validate findings, we will triangulate data across respondents
and across methods.

Ethical issues

The evaluation study was approved by the Institutional Review Board of the Ifakara
Health Institute, the Tanzanian National Institute for Medical Research and the Population
Council (P484). Letters were sent to District Executive Directors (DEDs) copied to
District Medical Officers (DMOs) informing them of the study and its objectives prior
to commencing the study. Prior to each round of data collection, calls were made to
the DMOs to agree on dates for data collection. An information sheet was left at the
DMO’s office. Information sheets and consent forms were provided to all those participating
in the study including patients, providers and households. Written consent was obtained
prior to undertaking all in-depth interviews and focus group discussions conducted
as part of the process evaluation.