Does spending on refugees make a difference? A cross-sectional study of the association between refugee program spending and health outcomes in 70 sites in 17 countries

There are currently numerous large scale and complex simultaneous humanitarian emergencies such as in Syria, Iraq, South Sudan, and Central African Republic. These have strained the host governments’ and international community’s ability to respond adequately, both in terms of personnel, infrastructure and services, and funding. Never before has it been more important to use precious and limited funds in a cost-effective manner to respond to humanitarian crises. Increasingly, large bilateral and multilateral donors are examining ‘value for money’ [19]. This study shows that for a refugee population represented by 20 PPGs including 1.6 million refugees living in 70 refugee sites in 17 different host countries in Africa, South and Southeast Asia, and the Middle East and North Africa, increased system-wide funding for refugee services is positively associated with improved health outcomes.

The results of the regression analyses suggest that total UNHCR spending on protection and assistance programs is positively correlated with a reduction in CMR and U5MR, meaning that higher levels of budgeted spending tended to occur together with better relative health outcomes more frequently than would be expected by chance. Furthermore, increased spending within specific budget categories was found to positively correlate with reduced mortality rates; budgeting for fair protection processes and documentation, and for healthcare services were correlated with both CMR and U5MR; budgeting for external relations, and for logistics and operations support was correlated with CMR; and budgeting for favorable protection environment and for basic needs and essential services was correlated with U5MR. Although these results should not be taken as a definitive recommendation to increase spending in these specific areas, the implications and limitations of these findings raise important considerations for funding programs in humanitarian response.

While spending on basic needs and essential services such as water and sanitation or healthcare would be expected to correlate with health outcomes, the connection between other budget categories and mortality is indirect. Within UNHCR’s Results Framework, fair protection processes and documentation includes activities targeted at improving identification and registration of persons of concern [20]. Since the right to access services and assistance is often tied to documentation of status, effective status determination may improve the ability for crucial assistance such as food, shelter, medicines, and education to reach its intended recipients. Also, effective and complete registration will in turn lead to accurate population planning figures, thus avoiding the problem of inadequate aid allocated because of an undercount of the number of refugees. Spending in the logistics and operations support budget category, which we also found correlated with CMR, should improve the efficiency of service delivery, leading to more effective programs in shelter, water and sanitation, healthcare, and other health-related domains thereby reducing mortality.

Unlike assessments of the impact of individual vertical programs, the analysis presented in this paper evaluates budgeted spending for UNHCR activities at a system-wide, aggregate level. Thus, the findings take into account the potential for loss of effectiveness resulting from “real-world” inefficiencies that occur at each level of the causal pathway between spending and health outcomes. These inefficiencies might arise from poor policy decisions or funding choices, ineffective creation of intermediate outputs or service delivery, the crowding-out of other service providers, or lack of efficacy of the interventions themselves [21]. Such aggregate level analysis more closely resembles the numerous studies of the cross-national impact of aid or public health spending, for which these inefficiencies have been used to explain the empirical lack of correlation between foreign aid and economic growth, or between public spending on health and health outcomes [8, 21, 22].

The results suggest that even when assessed at an aggregate level and including the effect of these real-world inefficiencies, higher levels of UNHCR budgeted spending has an impact on improved health outcomes. This differs from the results of cross-national analyses of the impact of spending on health involving all countries [8, 22], but mirrors analyses limited to more vulnerable, disadvantaged countries or populations [9, 10]. Our finding that refugee program spending has an impact on health outcomes thus likely reflects the unique vulnerability of refugee populations. Dependence on assistance and services provided by humanitarian organizations to refugees, as well as barriers preventing refugees from working or accessing services outside of a refugee site [11, 14], cause refugee populations to be more sensitive to changes in the level of assistance provided by host governments and humanitarian organizations. In addition, the relationship between UNHCR and partner NGOs is vastly different from a typical competitive market; rather than crowding-out other service providers, money spent by UNHCR for services contracted from another organization may in fact “crowd-in” additional services and spending as the partner organization brings its own donor funds to supplement its programming. The geography and organization of a refugee site may also improve the population access and efficacy of certain services such as vaccination, nutrition and food assistance, and access to healthcare, as was observed in a prior study comparing outpatient healthcare service utilization between refugees and the host population [13].

These results should be interpreted within the context of the protracted or chronic phase of a humanitarian situation. This is because the study sample was biased towards more stable refugee sites, reflecting the time needed for setup and implementation of HIS; the exclusion of sites with missing or inconsistent data likely added to this bias. Though this limits the ability to apply these results to all refugee situations, we theorized that unstable refugee situations would have uncharacteristic spending and mortality levels that are more reflective of immediate threats and shocks to the population rather than any stable relationship between spending and health outcomes.

The overall CMRs and U5MRs calculated from the HIS data were low, which was especially apparent when compared to the corresponding mortality rates of the country of asylum. While it has been reported previously that health outcomes among refugees are often better than the health outcomes of their hosts [23, 24], especially in the protracted or chronic phase of a refugee situation, this factor alone does not likely account for the size of the difference between the calculated PPG mortality rates and the country of asylum mortality rates. Another potential reason for the low reported mortality rates is the tendency for surveillance systems such as HIS to underestimate the true mortality rate, as some deaths will not be identified by the formal reporting system [25]. Furthermore, refugee site population censuses tend to be overestimated for a variety of reasons including double counting of refugees and the inclusion of host populations [25, 26]. Thus, the denominator for crude mortality rate calculations is often inflated, thereby further decreasing the calculated rate; this is not the case for under-5 mortality, as live births are used for calculating the denominator in this study, and live births tend to be reliably registered since they translate to increases in food rations.

With assumptions of PPG population size, refugee mortality rates, and country of asylum mortality rates, the estimated regression function can be used to express the health impact of UNHCR budgeted spending in more typical cost-effectiveness terms, such as cost per death averted. For example, if median figures for PPG population and mortality are used, the regression results suggest that increasing spending on healthcare services by an average of US$44,274 (95 %CI: US$31,456 – US$57,091) per year would have resulted in one fewer death. This figure is high when compared to the estimated cost per death averted for the most cost-effective interventions in areas with high disease burden, ranging from a few hundred to a few thousand US dollars [7]. As explained previously, however, the analysis presented in this paper is performed at an aggregate level, and includes the costs of inefficiencies in translating spending into outputs and impact. When compared to the cost per death averted for public spending on health at the cross-national level, estimated at US$50,000 to US$100,000 per year in developing countries [8], this figure falls just below the expected range. The cost per death averted estimated by this analysis, however, may be inflated by the lower than expected mortality rates derived from the HIS data, which may be underreported for the reasons discussed in the previous paragraph. Though this analysis was not designed to estimate cost-effectiveness of spending on refugee healthcare services, the finding that the extrapolated cost per death averted is below the estimated range is nevertheless consistent with the notion of refugee populations being uniquely vulnerable and sensitive to aid spending, and warrants further study.

This analysis was subject to several limitations. First, budget data, rather than actual expense data, were used to reflect the costs to UNHCR. While the phased budget process used by UNHCR ensures that the budget at the “detailed plan” stage closely approximates actual spending (and in fact differs in aggregate by less than 5 %), expenditure data would have been preferable. Furthermore, only UNHCR budgeted spending is included in this analysis; spending from other sources such as NGOs, the host government, private remittances, or local trade were not included. It is unclear if UNHCR spending has a crowding-out or crowding-in effect on spending from partner organizations. At the cross-national level, foreign development assistance for health has been previously reported to have varying effects on increasing or decreasing domestic government health spending depending on whether the assistance was provided to the government or the NGO sector [27]. Thus, the results can only be interpreted in terms of UNHCR spending, and not the overall health impact of humanitarian sector spending in these 70 refugee sites.

Second, the cross-sectional design of the regression analysis limits the findings to correlation; a causal relationship between spending and health outcomes cannot be determined. Reverse causation is another possibility in which health outcomes determine budget levels, since health data from HIS is one of many potential factors considered by UNHCR in the budgeting process. By using budget data for 2011, reflecting a needs assessment and priority-setting process from 2009 to 2010, we attempted to address reverse causation by assuming based on temporal ordering that 2011–2012 mortality did not affect spending levels budgeted before 2011. This assumption potentially breaks down, however, if past mortality rates are correlated to future mortality rates. To test if past and future mortality rates are correlated, we calculated Pearson’s correlation coefficient for CMR and U5MR in 2011 vs. 2012 across all HIS sites included in the analysis. For CMR, the correlation coefficient is 0.368 (p?=?0.001), which is significantly different from the null hypothesis of zero correlation at the ??=?0.05 level, but would be considered a “weak” correlation by Evans’ classification [28]. For U5MR, the correlation coefficient is 0.146 (p?=?0.218), which is not statistically significant, and would be classified a “very weak” correlation.

Third, a weakness in the mortality figures calculated from the HIS data is the lack of age and gender standardization that would especially affect CMR. Variation in the age and gender distributions between site or PPG populations can alter death rates in a manner that does not reflect actual population health status. By using ratios of refugee mortality to country of asylum mortality, we partially correct for being unable to standardize the mortality rates. This assumes that the age and gender distributions of refugee populations are similar to the country of asylum population, or at least dissimilar to an equal degree, though this might not actually be the case.

Fourth, because the regression model used log transformations of the budget variable, several PPGs were dropped from certain objective-specific analyses when the budgeted spending for that objective was $0. This occurred in regressions involving budgeted spending for external relations, water and sanitation, and non-food items, and thus the results for these specific budget categories do not reflect the entire dataset.

Fifth, the granularity of the budget data in Focus was limited to the PPG level. As a result, health outcome data had to be derived by aggregating HIS data from the site level to the PPG level, so the analysis does not take into account differences between sites within the same PPG. Additionally, analysis at the PPG level limited the number of observations in the regression, despite the HIS data representing a relatively large number of refugee sites. Ideally, multiple variable regression analyses could adjust for known correlates of mortality such as education level, access to water, HIV prevalence, and geography, thus allowing for a more robust model. Such multiple variable models could not be used with the small number of available observations (20 PPGs), however, as the resulting few degrees of freedom would increase risk of an over-fitted model. Furthermore, data on these potential covariates at the PPG level for all of the included refugee sites was not available to the authors. Also, the budget data from Focus used in this analysis was limited to a single year because UNHCR PPG definitions shift from year to year, making budget allocations incomparable between years. If several comparable years of budget data were available, this would allow for a greater sample size using the cross-sectional approach described in this paper for analyzing between-PPG differences in mortality and spending; alternatively, multiple years of budget data could also allow for panel data analyses looking at within-PPG differences over time. Though the results-based organization of the budget in Focus makes a health impact analysis of UNHCR spending possible, future studies may benefit from budget or expenditure data disaggregated by sites, and time series data using stable PPG definitions from one budget year to the next.