• WEDNESDAY, OCTOBER 16, 2019
  • ISSN 2399-1623
Lindsay Mallick, Trinadh Dontamsetti, Thomas Pullum, and Julia Fleuret

Abstract

Background

Launched in 2012 in Uganda and Zambia, Saving Mothers, Giving Life (SMGL) was a five-year initiative that aimed to improve access to and use of maternal and newborn health care services, as well as strengthen the quality of these services.

Aim

This study compared the change in coverage of eight related maternal and child health (MCH) indicators between districts included in the SMGL initiative with the change in selected control areas. These indicators included four or more (4+) antenatal care (ANC) visits; two components of ANC including urine sample taken and blood pressure (BP) measured; postnatal care (PNC) by a skilled provider for the mother; and PNC by a skilled provider for the baby. For births in the preceding three years, we examined facility delivery, delivery with a skilled birth attendant (SBA), and Cesarean section (C-section delivery).

Methods

The analysis used data from two Demographic and Health Surveys (DHS) conducted in Uganda, one immediately preceding the initiative (2011) and another in the final year of the initiative (2016). We employed a difference-in-differences analysis to estimate the significance of the difference in the change between the treatment and three different control areas for each indicator. Control areas one and two comprised accumulative districts immediately surrounding the treatment area. The third control area included all districts in the country outside of the treatment area.

Results

The results revealed that while respondents and their children in both the SMGL and control areas demonstrated significant improvements in MCH indicators between 2011 and 2016, the gains were largely not significantly different between the treatment and control areas, with the exception of two indicators: blood pressure measurement among women with ANC and births delivered by cesarean section.

Conclusions

Although the districts hosting the SMGL initiative did not exhibit improvements greater than those of the control areas, the analysis may be limited by the small sample sizes. National progress in Uganda is laudable; however, results of the difference-in-difference analysis should be interpreted with caution.


Maternal and child health in Uganda

The health of mothers and babies is a priority for the Government of Uganda. Maternal and newborn health care services have been central pillars in the Uganda National Minimum Health Care Package (UNMHCP) since its inception in 1999 (1). The specific objectives are to reduce morbidity and mortality related to sexual and reproductive health and rights and to improve newborn health and survival (2). Although women are often expected to bring their own clean delivery kits or sterile equipment when attending facilities to deliver their children, all services included in the UNMHCP are provided without user fees. However, challenges in access to services, uptake of their use, and quality of care have persisted, particularly for emergency obstetric care seeking and provision (3). Thus in 2007 the Government of Uganda launched a strategy specifically focused on reducing maternal and neonatal morbidity and mortality (4).

Some indicators of maternal and child health outcomes and health service delivery improved between surveys in 2000-01 and 2011. The five-year infant mortality rate decreased from 88 to 54 deaths per 1,000 live births, and the proportion of women receiving skilled assistance during delivery during the five preceding years increased from 37% to 58% (5, 6). During the same period the neonatal mortality rate (deaths during the first month of age per 1,000 live births) decreased from 33 to 27. Other indicators still have far to go. According to the 2011 Uganda Demographic and Health survey (UDHS), only about one in every five pregnant women (22%) had a urine sample taken, despite national clinical guidelines requiring urine testing for all women at the first antenatal care visit (7). In 2011, only a third of women (33%) and 11% of newborns received a postnatal checkup in the first two days after birth; nearly two-thirds (65%) of all Ugandan women reported at least one problem accessing health care (5).

One critical indicator, pregnancy-related mortality, showed no significant change between 2000-01 and 2011, as Figure 1 shows. Pregnancy-related mortality includes deaths due to incidental causes and is therefore slightly larger than maternal mortality; both are comparatively rare events and difficult to measure with precision, particularly in the absence of reliable national registration of deaths and determination of cause of death. In addition, there are differences between direct causes of maternal death (puerperal sepsis, hemorrhage, hypertensive disorders, and abortion complications) and indirect causes (anemia, diabetes, HIV, etc.), in their implications for the diagnostic and treatment capacities of health systems (8).

Figure 1.  Trends in pregnancy-related mortality ratio, Uganda Demographic and Health Surveys 2000-01 to 2011. Pregnancy-related mortality ratio for the seven years preceding the survey is expressed per 100,000 births. Pregnancy-related deaths are deaths during pregnancy, delivery, and in the two months following delivery or end of a pregnancy, including deaths due to accidents or violence.
joghr-03-e2019026-f1


Nevertheless, important work has been done to identify the causes of maternal mortality in Uganda. A review of maternal deaths at a regional referral hospital in Western Uganda found that most deaths were due to direct causes. Among women who died of any maternal cause, women who were HIV-positive, who lived in rural areas, did not attend antenatal care, delayed seeking care, or were referred from other facilities were more likely to die (9). When facilities have limited staff/resources, women in labor should be triaged so that women presenting with complications are attended to early enough to avert bad outcomes. Setting up triage systems in Uganda, however, even at the national referral hospital in Kampala, is no simple matter (10). Absenteeism, particularly among doctors, means that women in labor may find no assistance when they arrive at a facility. This is particularly problematic for women with complications who need intensive medical support (11). Inadequate provision of safe obstetric anesthetic is a risk factor for maternal death, but few hospitals in Uganda meet all requirements for safe anesthesia (12). Women who quickly return home after birth or who deliver at home may not recognize when postpartum bleeding qualifies as hemorrhage and therefore may not seek care in time (13).

Saving Mothers, Giving Life

Launched in 2012 in Uganda and Zambia by the United States Secretary of State Hillary Clinton, the Saving Mothers, Giving Life (SMGL) initiative is a five-year public-private partnership led by the United States Agency for International Development (USAID). There were a total 17 contributing agencies and organizations, including the United States Centers for Disease Control and Prevention (CDC), the United States President’s Emergency Plan for AIDS Relief (PEPFAR), Merck for Mothers, and Project CURE, along with implementing agencies in each country. SMGL aims to decrease maternal mortality by 50% and neonatal mortality by 30% during the most dangerous time for women and babies–labor, delivery, and the first 48 hours postpartum–by addressing the “three delays” associated with maternal and newborn deaths. The “three delays” are: delays in seeking appropriate care; delays in reaching care in a timely manner; and delays in receiving high-quality care at a health facility (14, 15). SMGL seeks to ensure that all women have access to safe delivery services and, in the event of an obstetric complication, obtain comprehensive emergency services within two hours of onset.

To address the three delays, the SMGL initiative planned and deployed dozens of activities that were designed to be evidence-based, low-cost interventions. These activities are divided into four dimensions: demand; access; quality; and health system strengthening (16). To help generate demand for giving birth in a health facility, SMGL ran mass media campaigns and trained village health teams to promote facility delivery and birth preparedness; health workers were also expected to distribute hygienic birth supplies, or “Mama Kits”. To improve access to care, SMGL built new and refurbished existing facilities, improved maternity wards, renovated “mothers’ shelters” intended to reduce travel time to care for expecting mothers, and purchased vehicles that could serve as ambulances. Through hiring and training skilled providers, providing supervision, and stocking facilities with essential medicines, SMGL worked to improve quality of care in health services. Strengthening health systems included upgrading facilities to basic or comprehensive emergency obstetric and neonatal care status, improving maternity wards, labs, and pharmacy capacities, and facilitating the use of electronic health information systems.

In Uganda, Phase 1 of the initiative was launched in four districts in 2012, with Phase 2 commencing in another four districts in 2014. From its inception, SMGL was designed to be a district-level program primarily executed by the Ugandan and Zambian governments, building on their existing public and private healthcare systems with expert guidance from USAID and other aid partners (14).

Research focus

The Demographic and Health Surveys (DHS) Program has collected nationally representative data in Uganda every five years since 1988. The DHS surveys cover many topics relevant to women’s health status and health care, including coverage of key indicators of reproductive and maternal and child health (MCH). The two most recent Uganda DHS surveys were conducted in 2011, before the launch of the SMGL initiative, and in 2016, the final year of the SMGL initiative. Although Uganda’s DHS surveys are not designed to be representative at the district level, women from almost every district were sampled in each survey. While this study cannot trace precise inputs of SMGL activities, we compare the overall change in coverage of related MCH indicators between districts included in the SMGL initiative with the corresponding change in selected control districts, using data from respondents in the two most recent UDHS surveys.

DATA

Surveys

This analysis used data from two nationally representative household-based Uganda Demographic and Health Surveys (UDHS) in 2011 and 2016. These surveys were implemented by the Uganda Bureau of Statistics (UBOS) in collaboration with the Uganda Ministry of Health (MOH) (5, 17). The sample of households for each survey was drawn from a two-stage sample design in order to ensure that the sample is representative at the national and regional levels (10 regions in 2011 and 15 in 2016), as well as by urban and rural residence. The first stage was selection of clusters from the strata. Strata are the geographic areas for which representative results are desired. Clusters are based on enumeration areas (EAs), the comprehensive list of small geographic areas into which countries are divided during a census. In Uganda, a typical urban cluster is the size of a city block, and a typical rural cluster is the size of a village. A sampling frame of the list of EAs created during a census offers complete coverage of a country. The two surveys used different approaches to selecting clusters and households.

The 2011 UDHS was designed to enable linked analysis with the 2009-10 Uganda National Household Survey (UNHS). In the first stage for the 2011 UDHS, clusters were selected from the clusters in the UNHS, which used the 2002 Population Census as its sampling frame where clusters were selected with a probability proportional to size in each stratum. In the second stage, 30 households were selected in each cluster; 10 or 20 households (depending on region) that participated in the UNHS, and the remaining 20 or 10 were randomly selected from a complete listing of the households in that cluster.

The 2016 UDHS used the 2014 Uganda Population and Housing Census (UPHC) as its sampling frame and for the first stage, clusters were selected with a probability proportional to size within each stratum. In the second stage, households were randomly selected from a listing of all households in each cluster. Interviewers collected data from respondents in the selected households between May and December 2011 for the 2011 UDHS and between June and December 2016 for the 2016 UDHS.

Respondents included women age 15-49 who provided information about their health status and health-related behaviors and attitudes. All women age 15-49 present in the household on the day of the survey – either permanent residents or visitors who stayed in the household the night before the survey – were eligible to participate in the survey. Women who had a live birth in the five years preceding the survey provided additional data about the care they received during and after pregnancy and childbirth, as well as about the health and care of their children. We restricted our analysis to women who reported having a live birth in the three years preceding each survey. We chose to limit our sample to this period in order to evaluate only women who had been exposed to the SMGL initiative for at least a year and a half at the time of the index birth. This time period coincides with the end of the pilot period of SMGL, which was rolled out between the launch in 2012 through May 2013 (16). The data for this study comprised 4,042 women and their 4,905 live births in 2011, and 7,903 women and 9,228 live births in 2016.

Treatment and control districts

Phase 1 of SMGL was initiated in four contiguous districts in western Uganda: Kabarole, Kamwenge, Kibaale, and Kyenjojo. Figure 2 depicts the location of the intervention areas and control areas. We selected DHS clusters that were located within these districts for inclusion in the treatment area. We constructed three different control areas against which to compare the treatment area.

Figure 2.  Maps of Saving Mothers, Giving Life Phase 1 districts (treatment area) and control areas 1, 2, and 3.
joghr-03-e2019026-f2


Table 1 also specifies the districts included in each control area. While the Government of Uganda continues to redraw its district and regional boundaries, this did not affect the intervention districts between the two most recent DHS surveys, although the district names included in the control areas changed. We structured the control areas such that geographical boundaries of all the control areas remained consistent between the surveys. This study did not employ a matched case-control design nor propensity score matching; rather, this study compares aggregate district-level data between intervention and control areas.

Table 1.  Districts included in each control area
Treatment/Control area Districts Reason for selection Districts affected by redistricting between 2011 and 2016
Treatment (SMGL Phase 1) Kabarole, Kamwenge, Kibaale, Kyenjojo Treatment area Redistricting did not occur between the 2011 and 2016 surveys
Control 1 Hoima, Kyankwanzi, Kiboga, Mubende, Kyegegwa, Kiruhura, Ibanda, Rubirizi All districts contiguous to the treatment area, except three districts with a similar intervention Kiboga divided into Kiboga and Kyankwanzi
Control 2 Control 1 plus Bundibugyo, Kasese and Ntoroko All districts contiguous to the treatment area Bundibugyo divided into Ntoroko and Bundibugyo
Control 3 All districts outside of SMGL Phase 1 districts* For comparison of treatment area against the rest of the country Nationally, 10 regions were reconfigured into 15 regions; however, the total number of districts remained the same (112)

SMGL: Saving Mothers, Giving Life. DHS: Demographic and Health Surveys

*Excludes clusters with missing GPS data.



The first control area comprised districts contiguous to the intervention area, except three districts that received a similar large-scale maternal health program conducted by the Korean International Cooperation Agency (KOICA). The second control area contained districts in the first control area, plus the three districts with the KOICA intervention. We selected the first two control areas on the basis that because these areas are in proximity to each other they also possess similar characteristics – e.g., ethnicity, religion, climate – and were exposed to similar macro-level changes over time, such as economic climate, weather patterns, refugee influx, or political atmosphere. The final control area was used to compare the intervention districts with the rest of the country. In 2011, four clusters without global positioning system (GPS) data were included only in the final control area. Table 2 presents descriptive characteristics and the numbers of women and live births in each area that are included in the analysis.

Table 2.  Background characteristics of women and births in the intervention and control groups
Percent of women Number of:
Rural Lowest wealth quintile No education Under 20 years old at most recent birth Not currently married Women age 15-49 with a birth in the preceding 3 years Children born in the preceding 3 years
2011:
SMGL Intervention 82.5 10.8 11.6 10.1 17.3 278 339
Control 1 95.6 14.5 23.7 12.9 15.4 266 333
Control 2 89.7 11.0 21.6 13.7 15.3 372 457
Control 3 85.5 22.9 13.6 15.3 14.2 3,764 4,566
2016:
SMGL Intervention 87.0 11.8 15.1 19.4 22.4 463 540
Control 1 85.7 9.5 11.8 16.1 18.7 575 680
Control 2 83.1 7.1 10.8 17.2 17.5 832 973
Control 3 77.5 22.8 9.5 17.1 16.7 7,441 8,688

SMGL – Saving Mothers, Giving Life, DHS – Demographic and Health Surveys



METHODS

Indicators

The SMGL initiative focused on improving access to maternal and newborn health care and the quality of maternal and newborn health care services. To assess the results of this initiative, this analysis examined eight potentially related outcomes for women with a birth in the three years preceding the survey, and also for births occurring in the three years preceding the survey. For women, these outcomes included: four or more (4+) antenatal care (ANC) visits; two components of ANC including urine sample taken and blood pressure (BP) measured; postnatal care (PNC) by a skilled provider for the mother; and PNC by a skilled provider for the baby. For births in the preceding three years, we examined facility delivery, delivery with a skilled birth attendant (SBA), and Cesarean section (C-section delivery).

Analysis

This analysis relied on cross-sectional data and employed a difference-in-differences analysis, a type of quasi-experimental design that can serve as a substitute for experimental design in the absence of randomly assigned treatment and control areas (18). This type of analysis, though intended for use with longitudinal data, can be applied with repeated cross-sectional data (19) and provides an estimate of the causal relationship between the treatment (SMGL interventions) and selected outcomes. Our analysis compared the change in MCH care indicators (above) over time in the intervention and control areas in order to assess whether the districts included in the SMGL initiative improved more than districts that were not included.

First, we estimated the coverage of each indicator, or the proportion of women or births obtaining each MCH service or outcome, using data from both Uganda surveys (2011 and 2016), for the treatment and each control area. We then calculated the percentage point difference between the two surveys for each area by subtracting the proportions in 2011 from those in 2016. After appending the two datasets, we conducted logit regression to test the significance of the change between the two surveys. Next, to obtain the difference-in-differences estimate, we subtracted the percentage point difference of coverage of each indicator in each control area from the percentage point difference in the treatment area. We tested the significance of the difference-in-differences estimate by, again, conducting logit regression, one model for each pair of treatment-control areas, using the appended dataset and respective survey weights. The unadjusted models contain dichotomous variables for the year of the survey (coded as 1 for mothers in 2016 and 0 for mothers in 2011); for residence (coded as 1 in the treatment districts and 0 in the control districts); and for the interaction between the two. The interaction term is the product of year of interview and residence in either treatment or control district:1 if the mother resided in the intervention area in 2016 and 0 if the mother resided in the control area in 2016 or was a respondent in the 2011 survey. Adjusted models controlled for mother’s characteristics, including type of place of residence (urban or rural), wealth quintile, education (none, primary, secondary or higher), maternal age at birth (under 20, 20-34, 35 and above), and marital status (either currently married or not). We determined statistical significance (α=0.05) of the difference-in-differences from the p-value for the interaction term in the unadjusted and adjusted models. We conducted all analysis using Stata 15 (StataCorp, College Station, TX, USA) with adjustments made for the complex sample design, sampling probability, and non-response using the svy command, which incorporates individual weights, the primary sampling unit, and strata for each survey.

RESULTS

As Table 3 shows, overall, in both the treatment districts and in the control areas there were substantial improvements in the indicators of maternal and child health, care-seeking behavior, and key life-saving interventions for mothers and children during pregnancy and birth. While gains were made for PNC for the mother and baby, in both treatment and control areas, PNC still lagged far behind the coverage of other indicators. For example, in the SMGL treatment area, although 70% of births were delivered in a health facility, only 34% of mothers reported receiving a check-up in the two days after delivery, and only 40% of babies were examined by a skilled provider. Appendix S1 in the Online Supplementary Document includes corresponding 95% confidence intervals of the estimates.

Table 3.  Estimates and percentage difference in the estimates in the SMGL intervention areas and control areas, Uganda DHS 2011 and 2016
SMGL intervention Control 1 Control 2 Control 3
% 2011 % 2016 Difference† % 2011 % 2016 Difference† % 2011 % 2016 Difference† % 2011 % 2016 Difference†
Among women with a birth in the last 3 years:
Four or more ANC visits 43.0 59.3 16.3*** 32.5 60.0 27.5*** 41.6 61.9 20.3** 47.5 60.3 12.8***
Urine sample taken during ANC 15.8 32.7 16.9** 13.5 37.1 23.6*** 23.8 38.2 14.4** 22.4 38.7 16.3***
BP measured during ANC 41.6 67.2 25.6*** 49.5 60.4 10.9ns 58.6 69.1 10.5ns 59.3 71.2 11.9***
PNC in 2 days by a skilled provider (mother) 26.7 33.9 7.2ns 19.9 38.3 18.4*** 25.5 42.9 17.4*** 34.2 55.7 21.5***
PNC in 2 days by a skilled provider (baby) 6.3 39.5 33.2*** 3.3 38.1 34.8*** 9.9 43.9 34.0*** 11.0 57.0 46.0***
Among births in the last 3 years:
Facility delivery 51.8 69.9 18.1* 46.8 70.6 23.8* 51.3 71.3 20.0** 59.6 75.5 15.9***
Skilled birth attendance 48.8 71.0 22.2** 48.8 71.1 22.3* 54.0 73.0 19.0** 60.3 76.1 15.8***
C-section 2.7 5.9 3.2ns 7.2 5.5 -1.7ns 9.0 8.5 -0.5ns 5.6 6.7 1.1*

SMGL – Saving Mothers, Giving Life, DHS – Demographic and Health Surveys, BP – blood pressure, PNC – postnatal care, ANC – antenatal care

†Percentage point difference between 2011 and 2016 with significant tests for the difference in proportions, P: *<0.05; **<0.01; ***<0.001, and ns=non-significant.



Logit regressions assessing the significance of the change between the two surveys in the treatment area revealed a significant change for most of these indicators, including 4+ ANC visits, urine sample taken during ANC, blood pressure measurement during ANC, PNC for the baby within two days of birth by a skilled provider, facility delivery, and skilled birth attendance. Each of these indicators improved by over 15 percentage points between the two surveys. C-section rates increased from 3% to 6%, though a significant change was not detected.

The difference-in-differences analysis revealed mixed findings, as Table 4 shows. Some indicators demonstrated larger gains in the treatment area than in the control areas, including blood pressure measurement during ANC for a woman’s last pregnancy in the previous three years and C-sections among all births in the preceding three years. The analysis also revealed many negative difference-in-differences, such that the treatment area showed smaller gains than the control areas. This pattern appeared for the proportion of women who received PNC and whose babies received PNC. For other indicators, such as 4+ ANC, urine sample, facility delivery, and SBA, the direction of the difference-in-difference estimate varied when the treatment area was compared with the various control areas.

Table 4.  The difference-in-differences between SMGL intervention areas and control areas, Uganda DHS 2011 and 2016 with unadjusted (UA) and adjusted (A) tests of significance†
SMGL – Control 1 UA P-value A P-value SMGL - Control 2 UA P-value A P-value SMGL - Control 3 UA P-value A P-value
Among women with a birth in the last 3 years:
Four or more ANC visits -11.2 ns ns -4.0 ns ns 3.5 ns ns
Urine sample taken during ANC -6.7 ns ns 2.5 ns ns 0.6 ns ns
BP measured during ANC 14.7 ns * 15.1 ns * 13.7 * *
PNC in 2 days by a skilled provider (mother) -11.2 ns ns -10.2 ns ns -14.3 ns ns
PNC in 2 days by a skilled provider (baby) -1.6 ns ns -0.8 ns ns -12.8 ns ns
Among births in the last 3 years:
Facility delivery -5.7 ns ns -1.9 ns ns 2.2 ns Ns
Skilled birth attendance -0.1 ns ns 3.2 ns ns 6.4 ns Ns
C-section 4.9 * * 3.7 ns ns 2.1 ns Ns

SMGL - Saving Mothers, Giving Life, DHS: Demographic and Health Surveys

†The significance of the difference-in-differences was obtained from logit regressions, which test the null hypothesis that the change for each outcome from 2011 to 2016 is the same in the intervention area as in the control areas. Unadjusted (UA) and adjusted (A) P: *<0.05; **<0.01; ***<0.001, and ns=non-significant. Adjusted models control for place of residence, wealth, education, maternal age at birth and marital status.



With a few exceptions, the difference-in-differences analysis largely failed to detect a significant advantage in the intervention area. Measurement of blood pressure during ANC increased from 42% in 2011 to 67% in 2016 (25 percentage points) in the treatment area (P < 0.001), while in the rest of the country (control 3), blood pressure measurement increased only 12 percentage points, from 59% to 71%. Although the change in the treatment area was double that in control area 3, the significance was only at the .05 level in the unadjusted (UA) and adjusted (A) logit models. We found a significant difference-in-difference between the treatment area and control area 1 for C-sections, because in control area 1 the proportion of births delivered by C-section actually decreased by 2 percentage points, rendering a significant difference-in-difference estimate of 5 percentage points (P <.05) in the unadjusted and adjusted models. Appendix S2 – S9 in the Online Supplementary Document contain the results of each regression model, with one table for each outcome examined in this analysis, and include unadjusted and adjusted odds ratios as well as corresponding P-values.

Power

We conducted a post-hoc power analysis to determine the probability of correctly rejecting the null hypotheses given that an alternative hypothesis is in fact true. Incorrectly retaining the null hypothesis can occur simply because the sample size is insufficient. We found that this analysis lacked sufficient power (with a standard of 80%) to detect a significant change (α <0.05) between the two surveys among women (or births) in the treatment area for several indicators, including PNC for the mother and C-section. With relatively rare events and little change over time for indicators such as, for example, C-section, which increased from 2.7% to 5.9% of births, 1,382 births would be needed to detect a significant difference with power >0.80 and α <0.05, given equal sample sizes between two surveys.

Robustness

We conducted several robustness checks for our analysis. First, we examined the change in the same outcomes in the treatment and control areas using the 2006 and 2011 surveys – the two conducted in Uganda before the launch of the SMGL initiative. An important assumption for this type of analysis–the common trends assumption – is that the treatment and control areas shared similar trajectories before any intervention (20). In Figure 3 we present the trends before and during the SMGL initiative – that is, between the 2006 and 2011 surveys and between the 2011 and 2016 survey –f or four indicators: 4+ ANC visits, blood pressure measurement during ANC, SBA, and C-section. In each graph, a solid line between two surveys represents a significant change from one survey to the next, and a dotted line indicates no significant change. As the graphs show, the common trends assumption holds most strongly when comparing the treatment area with control area 3 (the rest of Uganda) for the indicators 4+ ANC, blood pressure measurement, and SBA, but not C-section.

Figure 3.  Trends in selected maternal and child health indicators, 2006-2016. A solid line indicates a significant change between two surveys, while a dotted line indicates no significant change.
joghr-03-e2019026-f3


In the treatment area the proportion of births delivered by C-section was smaller among births in the three years preceding the 2011 survey than births in the three years preceding the 2006 survey. Between 2011 and 2016, however, the proportion increased two-fold (as discussed above). Conversely, in almost all control areas there was a larger increase between 2006 and 2011, followed by minimal change or even declines between 2011 and 2016. However, as discussed above, the treatment area and the first two control areas lacked sufficient power to detect a significant difference between the two most recent surveys (2011 and 2016).

The graphs in Figure 3 further depict the findings of our difference-in-differences analysis. Compared with the control areas, the treatment area did not demonstrate larger improvement in 4+ ANC visits or SBA. However, more rapid improvements appear for blood pressure measurement during ANC and C-section within the treatment area, even if the gains compared with the control areas are mostly non-significant in the analysis.

A second robustness check explored additional indicators not directly related to the program, including use of modern contraception, intermittent preventative treatment (IPTp) for malaria (3 doses), immediate breastfeeding practices (among facility births), and receipt of the 3-series diphtheria-tetanus-pertussis vaccine (results not shown). We found no significant difference-in-differences for these indicators, with exception to IPTp consumption. IPTp use significantly decreased during the study period while at the same time increased in the control areas with the results indicating a significant negative difference-in-difference.

DISCUSSION

Overview of findings

This study applied a difference-in-differences analysis to assess the potential impact of the SMGL initiative in the districts subject to the interventions on indicators that reflect the use and quality of MCH services compared with selected control areas. Using data from a sequence of UDHS surveys, we found that, among mothers sampled in the treatment districts, there were improvements in most of the indicators of care received during pregnancy and birth. We found that PNC still lags behind other indicators of service utilization and that C-section rates are still far below the 10-15% of births where the procedure may be medically indicated (21).

The improvements in the SMGL area were mostly not significantly different from the differences in the three control areas. Two exceptions were in the differences-in-differences of blood pressure measurement during ANC between the treatment area compared with the rest of Uganda (control 3) and C-section in the comparison between the treatment area and a limited number of contiguous districts (control 1). These results may suggest that SMGL was perhaps more effective at strengthening the quality and health care system capacity than generating demand and improving the use of services, with exception to PNC. If a woman is discharged soon after birth and has not yet had a postnatal check, she may face additional barriers to PNC, including travel time to health facilities, as well as lack of knowledge about the importance of PNC, or concerns about the health system (22-24). The activities of the initiative centered around increasing service utilization and improving quality of care for facility delivery; perhaps promoting postnatal care was either neglected or ineffective against the co-existing barriers to care. However, all results should be interpreted with caution due to the small sample sizes.

Addressing the “three delays”

Globally, similar programs aimed at addressing the “three delays” of maternal health care have demonstrated success. An evaluation of a rural participatory women’s group community intervention and a quality improvement intervention at health centers focusing on maternal, perinatal, and neonatal mortality was conducted in three districts of Malawi (25). Similar to SMGL, these interventions emphasized community mobilization (demand) and facility improvement (quality). Using a randomized control trial, the authors found a significantly larger decrease in neonatal mortality in districts with both interventions, and a significantly lower perinatal mortality rate in community-intervention locations (25) compared with the control areas. Another program, the Maternal and Neonatal Implementation for Equitable Systems (MANIFEST) study in eastern Uganda, a multisectoral approach to address access to maternal and newborn care, was evaluated using a quasi-experimental pre- and post-comparison design (26). This program also focused on community mobilization (demand) and health provider capacity strengthening (quality) as vehicles to improve use of antenatal, skilled delivery, and postnatal care, as well as care for newborns. The authors conducted a survey of the target and control areas and, similar to our approach, used a difference-in-differences analysis. Their analysis revealed increases in early ANC and facility delivery and showed that these differences were greater than those in the control area.

An external evaluation of SMGL in Kalomo, Zambia, was conducted using a difference-in-differences method, focusing on facility-based birth and delivery with a skilled birth provider between the Kalomo district and three comparison districts (27). While we used DHS data to evaluate the impact of SMGL, White and colleagues used data from the Zambia Chlorhexidine Application Trial (ZamCAT) to conduct their analysis. They found an increase in facility-based births from 55% before the intervention to 65% during the intervention and a non-significant increase in the proportion of women delivering with a skilled birth attendant at health facilities in Kalomo (27). This suggests that while SMGL was successful in creating demand for facility delivery, the training of health providers did not match those successes.

Strengths

The present study has several strengths. First, the DHS data were collected independently from SMGL. Using an external data source may eliminate the potential for biased reporting. Second, the two most recent DHS surveys were well timed to capture groups of women before and after the launch of the SMGL initiative. As the DHS survey is nationally representative, with samples of women drawn from most districts in the country, we were able to examine various control areas to gain a more robust picture of the effect of the SMGL program.

Limitations

An important limitation of the study is the sample size, which likely affected both the statistical significance of the tests and the precision of the estimates. Post-hoc power analysis and robustness checks show that the analysis may have been underpowered to detect significant differences and that the MCH care-seeking behaviors in the treatment area were not consistent with those in all of the control areas prior to the study. The estimates for the four treatment districts were based on only 500 women interviewed in the 2016 UDHS who had given birth in the previous three years. Given that the population size for these four districts exceeded 2 million people, according to the 2014 census (28), our estimates should be interpreted with caution. Furthermore, given the small sample sizes at the sub-regional level, DHS data are not intended to be representative by districts; thus we cannot conclude that this sample reflects all women who might have been exposed to the program. Based on this study alone, there is not sufficient evidence to conclude that the SMGL program was ineffective. As described above, studies have shown that community mobilization and facility improvement activities are successfully improving facility delivery, delivery with skilled birth attendants, and neonatal and perinatal survival. That this analysis does not show these results may reflect the limitation of our study rather than the program in Uganda, as the analysis was performed on a small sample size of mothers with a birth or births occurring in the three years preceding the survey.

Another limitation is that we were unable to state clearly that the control areas – or our treatment area for that matter – were not subject to the effects of other programs. Per USAID’s Foreign Aid Explorer, approximately 300 basic health care and reproductive health care projects that include USAID assistance were active during the period between fiscal years 2012 and 2017 (29), overlapping the active period of the SMGL initiative. This tally does not account for projects that were implemented without USAID’s involvement, such as those organized by other foreign donors. An external evaluation conducted at the Columbia University Mailman School of Public Health identified two nearby districts that had no similar intervention as controls in their study (16). However, the sample of women in these two districts in the DHS survey was even smaller than the treatment area (92 mothers in 2011 and 159 mothers in 2016). Assuming that other nearby districts were without a similar intervention, it may be plausible that women in the control areas would bypass care at their nearby facilities to seek better care in the intervention area, choosing to use services in the program districts instead, and thus potentially diminishing the differences seen in this analysis.

Finally, the sample in our study was limited to mothers with a birth or births occurring in the three years preceding the survey. The oldest birth or pregnancy in the treatment area could have occurred as early as 2013. Therefore, women with births at that time may not have been exposed to the program for long enough to affect their care-seeking behavior. Behavior change can evolve over many years, according to the Transtheoretical Model of Behavior Change (30). We attempted to examine only pregnancies and births in the year preceding the survey; however, the sample size was even smaller. Future studies may benefit from using the next round of DHS data collected in Uganda. In addition, we recommend using propensity score matching to specify the control area(s) for further analysis.

CONCLUSION

Improving access to and use of high-quality maternal health care services is critical to preventing preventable maternal and newborn deaths. The SMGL initiative in Uganda supported activities intended to achieve these objectives, while also bolstering the health system in the districts where SMGL was implemented. Overall, many improvements in maternal and child health appear in the districts under the purview of the SMGL initiative, surrounding districts, and Uganda as a whole. Our analysis was unable to detect many improvements in MCH in the geographic area of the SMGL program that significantly exceeded those in the rest of Uganda, possibly due to the study’s small sample size. Although we believe our study benefits from using external data that may be free from biases, more investigation is needed.

Acknowledgements

The authors would like to thank Robert Cohen for conceptualization of the research question; Marc Cunningham for providing insight for the study design and feedback on the manuscript; Mona Steffen for imparting critical knowledge about the SMGL initiative; Tom Fish for map design; Bryant Robey for editing the manuscript; and the Uganda Bureau of Statistics staff and fieldworkers who collected the data that made this analysis possible.

Notes

[1] Financial disclosure Funding: This research is carried out with support provided by the United States Agency for International Development (USAID) through The DHS Program (#AID-OAA-C-13-00095). The views expressed are those of the authors and do not necessarily reflect the views of USAID or the United States Government.

[2] Conflicts of interest Competing interests: The authors completed the Unified Competing Interest form at www.icmje.org/coi_disclosure.pdf (available upon request from the corresponding author), and declare no conflicts of interest.

REFERENCES

1Wright J. Essential Package of Health Services Country Snapshot: Uganda. Bethesda, MD: Health Finance & Governance Project, Abt Associates Inc., 2015.

2Ministry of Health RoU. Health Sector Strategic and Investment Plan: Promoting People’s Health to Enhance Socio-economic Development 2010/11 – 2014/15. Kampala, Uganda: Ministry of Health, 2010.

3AK Mbonye, JB Asimwe, J Kabarangira, G Nanda, and V Orinda. . Emergency obstetric care as the priority intervention to reduce maternal mortality in Uganda. Int J Gynaecol Obstet. 2007;96:p.220–5. DOI: 10.1016/j.ijgo.2006.12.017. [17292370]

4Ministry of Health RoU. Roadmap for Accelerating the Reduction of Maternal and Neonatal Mortality and Morbidity in Uganda 2007-2015. Kampala, Uganda: Ministry of Health, 2007.

5Uganda Bureau of Statistics(UBOS). ICF International. Uganda Demographic and Health Survey 2011. Kampala, Uganda: UBOS and ICF International, 2012.

6UBOS. O. R. C. Macro. Uganda Demographic and Health Survey 2000-2001. Calverton, Maryland, USA: UBOS and ORC Macro, 2001.

7Ministry of Health RoU. Uganda Clinical Guidelines 2010 National Guidelines for Management of Common Conditions. Kampala, Uganda: Ministry of Health, 2010.

8World Health Organization. The WHO application of ICD-10 to deaths during pregnancy, childbirth and puerperium: ICD MM. Geneva, Switzerland: WHO, 2012.

9J Ngonzi, YF Tornes, PK Mukasa, W Salongo, J Kabakyenga, and M Sezalio. Puerperal sepsis, the leading cause of maternal deaths at a Tertiary University Teaching Hospital in Uganda. BMC Pregnancy Childbirth. 2016;16:p.207 . DOI: 10.1186/s12884-016-0986-9. [27495904]

10J Forshaw, S Raybould, E Lewis, M Muyingo, A Weeks, and K Reed. Exploring the third delay: an audit evaluating obstetric triage at Mulago National Referral Hospital. BMC Pregnancy Childbirth. 2016;16:p.300 . DOI: 10.1186/s12884-016-1098-2. [27724846]

11L Ackers, E Ioannou, and J Ackers-Johnson. . The impact of delays on maternal and neonatal outcomes in Ugandan public health facilities: the role of absenteeism. Health Policy Plan. 2016;31:p.1152–61. DOI: 10.1093/heapol/czw046. [27142803]

12I Epiu, A Wabule, A Kambugu, H Mayanja-Kizza, JVB Tindimwebwa, and G Dubowitz. . Key bottlenecks to the provision of safe obstetric anaesthesia in low- income countries; a cross-sectional survey of 64 hospitals in Uganda. BMC Pregnancy Childbirth. 2017;17:p.387 . DOI: 10.1186/s12884-017-1566-3. [29149877]

13S Ononge, ES Okello, and F Mirembe. . Excessive bleeding is a normal cleansing process: a qualitative study of postpartum haemorrhage among rural Uganda women. BMC Pregnancy Childbirth. 2016;16:p.211 . DOI: 10.1186/s12884-016-1014-9. [27503214]

14Saving Mothers GL. Saving Mothers, Giving Life Primer. In: Saving Mothers GL, editor. 2014.

15Saving Mothers GL. Who We Are. 2017. Available: http://savingmothersgivinglife.org/who-we-are.aspx. Accessed.

16Kruk ME, Galea S, Grepin K, Rabkin M. Final report: External evaluation of Saving Mothers Giving Life. New York: Columbia University, 2013.

17UBOS. ICF. Uganda Demographic and Health Survey 2016. Kampala, Uganda: UBOS and ICF, 2018.

18Gertler P. Chapter 6: Difference-in-Differences. Washington, DC: The World Bank; 2011.

19D Basu. . Randomization Analysis of Experimental Data: The Fisher Randomization Test. J Am Stat Assoc. 1980;75:p.575–82. DOI: 10.1080/01621459.1980.10477512

20R Blundell and MC Dias. . Evaluation Methods for Non-Experimental Data. Fisc Stud. 2000;21:p.427–68. DOI: 10.1111/j.1475-5890.2000.tb00031.x

21AP Betran, M Torloni, J Zhang, and A Gülmezoglu. . WHO Statement on Caesarean Section Rates. BJOG. 2016;123:p.667–70. DOI: 10.1111/1471-0528.13526. [26681211]

22SN Kiwanuka, E Ekirapa Kiracho, S Peterson, O Okui, M Hafizur Rahman, and D Peters. Access to and Utilisation of Health Services for the Poor in Uganda: A Systematic Review of Available Evidence. Trans R Soc Trop Med Hyg. 2008;102:p.1067–74. DOI: 10.1016/j.trstmh.2008.04.023. [18565559]

23M Mrisho, B Obrist, JA Schellenberg, RA Haws, AK Mushi, and H Mshinda. The use of antenatal and postnatal care: perspectives and experiences of women and health care providers in rural southern Tanzania. BMC Pregnancy Childbirth. 2009;9:p.10 . DOI: 10.1186/1471-2393-9-10. [19261181]

24SK Nabukera, K Witte, C Muchunguzi, F Bajunirwe, VK Batwala, and EM Mulogo. Use of postpartum health services in rural Uganda: knowledge, attitudes, and barriers. J Community Health. 2006;31:p.84–93. DOI: 10.1007/s10900-005-9003-3. [16737170]

25T Colbourn, B Nambiar, A Bondo, C Makwenda, E Tsetekani, and A Makonda-Ridley. Effects of quality improvement in health facilities and community mobilization through women’s groups on maternal, neonatal and perinatal mortality in three districts of Malawi: MaiKhanda, a cluster randomized controlled effectiveness trial. Int Health. 2013;5:p.180–95. DOI: 10.1093/inthealth/iht011. [24030269]

26E Ekirapa-Kiracho, R Muhumuza Kananura, M Tetui, G Namazzi, A Mutebi, and A George. Effect of a participatory multisectoral maternal and newborn intervention on maternal health service utilization and newborn care practices: a quasi-experimental study in three rural Ugandan districts. Glob Health Action. 2017;10:p.1363506 . DOI: 10.1080/16549716.2017.1363506. [28871853]

27EG Henry, DM Thea, DH Hamer, W DeJong, K Musokotwane, and K Chibwe. The impact of a multi-level maternal health programme on facility delivery and capacity for emergency obstetric care in Zambia. Glob Public Health. 2018;13:p.1481–94. DOI: 10.1080/17441692.2017.1385824. [28994352]

28UBOS. ICF. National Population and Housing Census 2014 Main Report. Kampala, Uganda: 2014.

29USAID. Foreign Aid Explorer: The official record of U.S. foreign aid. Available: https://explorer.usaid.gov/. Accessed: April 16, 2018.

30JO Prochaska and WF Velicer. . The transtheoretical model of health behavior change. Am J Health Promot. 1997;12:p.38–48. DOI: 10.4278/0890-1171-12.1.38. [10170434]