Recent evidence suggests that women in many low- and middle-income countries (LMIC) have negative experiences such as physical, verbal or sexual abuse or neglect during childbirth.^1–3 In countries with a high burden of maternal mortality and morbidity, the expectation of poor quality of care, including mistreatment, may deter women from seeking care in health facilities^2,4,5 and, in turn, contribute to otherwise preventable maternal deaths.^2,5 Studies in Nigeria have reported that two-thirds of women experience at least one form of negative maternity care experience² and that respectful and dignified care is an important determinant of women’s decision for facility childbirth.⁶

The World Health Organisation (WHO) defined quality of care framework consisting of eight domains for improvement, three of which pertain to maternity care experience: effective communication, respect and preservation of dignity and emotional support.^4,7 Measures of these experiences should be used to assess and improve the quality of services.^4,8 But collecting accurate and reliable information about negative and positive childbirth care experiences poses multiple challenges: there are many possible tools and data collection modes to apply, operational definitions of measures may vary by context, and the optimal recall period for self-report is a subject of debate.⁹ Validation research is sparse, reporting mixed results, limiting evidence-informed decision-making on how best to derive measures of maternity care experiences.^10,11

The present study from northeast Nigeria contributes to the need to improve the measurement of experience of childbirth care. We measured the experience of positive and negative maternity care practices through childbirth observations (the reference standard) and exit interviews with women upon discharge following facility childbirth and tested the validity of the self-reported measures at the exit.

METHODS

Study setting

The study was conducted in Gombe State, north-east Nigeria, where maternal and newborn health services are provided primarily through public health facilities, mostly by Community Health Extension Workers (CHEWS), Junior CHEWS and Health Officers. In 2018, uptake of maternal and newborn health services in the State was estimated to be relatively low, with just 44% of women having at least four antenatal care visits and just 28% of women having facility childbirth, compared to 57% and 39% nationally.¹²

The sampling methods for the present study have been described elsewhere.^13,14 Briefly, in November 2015, we selected ten high-volume facilities from 57 primary health care (PHC) facilities for an in-depth assessment of maternal and newborn health care quality. The selected facilities had the highest number of births in the preceding six months based on recorded births in the maternity registers: mean number of births per month of 15.7 (SD 12.0), compared with the Gombe State average of 4.3 (SD 6.3) births per facility per month. This sampling aimed to achieve a sufficiently large number of observations while minimising the duration of data collection.

Data collection

Observations of childbirth

Data for the study were collected in two rounds of childbirth observations in August 2018 and August 2019. Each round of data collection lasted three weeks, during which time the data collectors aimed to conduct a total of 350 childbirth observations. In each study facility, two trained female observers (all local midwives but not staff at the assigned PHC) and one clinical supervisor worked 8 or 12 hours shifts to ensure continuous data collection. The observers stayed in the same room with women from the first point of contact through the first hour after birth. Data collectors used a structured clinical observation checklist programmed on a Lenovo A3300 tablet using CSPro V.7.0 (US Census Bureau and ICF Macro, Suitland, Maryland, USA), to continuously document labour and childbirth care processes and birth attendant–client interactions. The data observation checklist was piloted and modified to the Gombe context.¹⁴

Women were eligible and invited to participate in the study at admission. Recruitment involved providing all potential participants with a study information sheet and a consent form in English and Hausa, reading the information sheet, explaining the purpose of the study, the risks and benefits of participating and answering questions before obtaining written informed consent from eligible women to observe the childbirth care received and verbal consent from the healthcare worker attending. Care was taken to ensure the inclusion of any support persons accompanying potential participants in the consent taking. In cases where women could not sign or write their names, they were asked to provide a thumbprint on the consent form. Women were informed of their right to participate or the freedom to withdraw at any time.

Observers were trained for four days before each round of data collection on how to conduct unobtrusive observations of childbirth while maintaining the safety and confidentiality of the participants, following a specified protocol. The protocol described procedures on how to prioritise the safety of the women and newborns over data collection and on how observers should seek help in the event of any complication or life-threatening event during childbirth since observers had no legal right to intervene in clinical care during childbirth observation in their assigned health facilities. Data quality assurance involved clinical supervisors conducting spot checks of observers and data throughout the observation period.

Exit interviews

All women observed during labour and childbirth, who were discharged with a live baby, and provided informed consent, were invited to participate in an exit interview. The exit interview instrument was also written to tablets using CSPro, and covered questions consistent with the information recorded during the observations of childbirths, including the content of care provided to the mother and the newborn and women’s experiences of care. Questions about the experience of care required ‘experienced an event’ (yes) or ‘not experienced an event’ (no) or ‘don’t know responses’.¹⁵ The exit interviews were conducted in the Hausa language in an area reserved for the interviews or in a separate room within the health facilities to ensure privacy and confidentiality. In each health facility, the exit interviews were conducted by the same team that conducted the observations of childbirth.

Indicator selection

Twenty-four maternity care experience indicators, including eight positive maternity care experiences and sixteen negative maternity care experiences were selected. The negative maternity care experience indicators captured physical abuse, verbal abuse, sexual abuse, stigma & discrimination, failure to meet professional standards of care, poor rapport and communication between women and providers, and health systems conditions and constraints drawn from the typology of mistreatment.¹ The positive maternity care experience indicators consisted of patient-centred practices that recognise women’s preferences and needs identified from the literature on improving the quality of maternal and newborn care in health facilities.^16–20 The final list of indicators is provided in Table S1.

Statistical analysis

Data from August 2018 and August 2019 were combined for the analysis. Observations of childbirth and exit interviews were matched by unique participant id. All analyses were conducted using STATA version 16 (www.stata.com). Observations were used as the reference standard and compared to exit interview responses. We computed the characteristics of women observed during labour and childbirth and interviewed at discharge. We tabulated the prevalence of positive and negative maternity care experiences documented during childbirth observations and self-reported by mothers.

To evaluate the individual level validity metric, we constructed two-by-two tables of women’s responses in exit interviews (Yes, No) to observations (Yes, No), for all 24 indicators separately. Missing and don’t know responses were excluded from the two-by-two tables. We then computed the sensitivity and specificity of women’s recall, and calculated per cent agreement between the exit and observations. We determined the area under the receiver operating characteristic curve (AUC) for the measures with at least 5 counts per cell, with AUC<0.60, AUC 0.60<AUC<0.70, and AUC≥0.70 reflecting low, moderate, and high individual level accuracy, respectively.^21–23

For the evaluation of the population level accuracy, we computed the inflation factor (IF). The IF represents the degree to which an indicator would be over- or underestimated in a population-level survey: the prevalence of the measure under consideration that would be obtained for respondents in a survey (Pr) divided by the true prevalence (P) from the reference standard. IF estimates of 0.75<IF<1.25, 0.50<IF<0.75 or 1.25<IF<1.5, and IF<0.50 or IF>1.5, reflecting low, moderate and large degrees of bias. Positive and negative maternity care experience measures with moderate or high individual level accuracy and low population level bias are considered to have acceptable validity.^21,22 We presented findings below in line with the STROBE statement.²⁴

Ethics approval

The study was conducted with approvals from the Nigeria Health Research Ethics Committee (reference NHREC/01/01/2007), the State Ministry of Health Gombe State, Nigeria and the London School of Hygiene & Tropical Medicine (reference 12181). The free signed and verbal informed consent of all interviewees was obtained.

RESULTS

Characteristics of study participants

The characteristics of the study participants are presented in Table 1. A total of 754 women were observed during childbirth care, of whom 724 women participated in the exit interviews, 30 of whom did not meet the inclusion criteria for the exit interview. Women’s ages at the time of childbirth ranged from 15 to 47 years. Close to half of the women had four or more prior childbirths. Further, nearly half of the women had no formal education, and lower cadre health workers conducted most childbirths. Of the 724 women, 716 were married, seven were single, and one was widowed.

Positive maternity care experience measures

From the observations, the prevalence of maternity care practices that recognise women’s preferences and needs (positive maternity care experience) ranged between 25% for the indicator ‘were you asked which position you would like to deliver in’ and 96% for the indicator ‘were you respectfully greeted by health workers when they first saw you’. Agreement between the observation and the exit interviews ranged between 55% and 97% (Table 2). Very few don’t know responses, and all indicators had at least 5 values per cell in the two-by-two tables. Exit interviews with women showed high sensitivity (89% to 99%) and moderate to high specificity (44% to 84%) for reports of positive maternity care experience events that were in accordance with observations. Six of the eight indicators had high overall validity, meaning AUC≥0.70 for high individual-level accuracy and 0.75<IF<1.25 for low population-level bias (Table 2).

Negative maternity care experience measures

The prevalence of negative maternity care experience events documented during the observations ranged between 0.1% (observed only once) for the indicator ‘woman being physically restrained, tied, or gagged during labour and delivery’ and 18% for the indicator’ Staffing shortages during labour and childbirth’. The reported prevalence for all negative indicators is shown in table S2 where we also observe agreement between observed and exit interview reports to be over 85% for all indicators. However, the low prevalence for many negative measures resulted in just six of the 16 indicators having at least 5 counts per cell in the two-by-two tables: validation results for these six measures are shown in table 3. For these six measures, the exit interviews showed low to moderate sensitivity (32% to 74%) and high specificity (97% to 100%) for the detection of negative maternity care experience events. The six negative maternity care experience measures met at least one validity criterion for high individual level reporting accuracy or low population level bias (Table 3). Two of the indicators related to the domain’ failure to meet professional standards of care’, including ‘lack of informed consent process (e.g., examine without permission)’ and ‘skilled attendants absent at time of childbirth’, met the individual-level validity metric but not the population-level validity metric. ‘Staffing shortages (perceived) during labour and childbirth’ and ‘lack of privacy’, the two indicators related to the domain’ health system conditions and constraints’ had high individual-level validity but did not meet the population-level validity metric. ‘Denial or lack of birth companions during labour and childbirth’, one of the two indicators related to the domain’ poor rapport between women and providers’ that met the individual-validity metric also had low population-level bias (Table 3).

DISCUSSION

Our study adds to the growing but still limited literature on the validity of exit interviews to derive self-reported measures of maternity care experience. We found that maternity care experiences self-reported in exit interviews with women conducted at the time of facility discharge were consistent with the observation of childbirth for all eight positive maternity care experience indicators and six of the sixteen negative maternity care experience indicators investigated.

For the positive maternity care experience measures assessed, exit interviews with mothers had high sensitivity but lower specificity. These findings indicate that, for this study population, exit interviews can be used to detect maternity care practices that recognise women’s preferences and needs. Exit interviews with women had high individual level reporting accuracy, with all the eight positive maternity care experience measures meeting the individual level validity metric and most of the measures (6 out of 8) having low population reporting bias. Some but not all of our findings are consistent with previous studies. For example, positive maternity care experience events, such as support persons present during labour and childbirth and women encouraged to move and change position during labour, consistently have high overall validity.^13,22,25 But the measures on whether women were allowed to drink or eat during labour and childbirth or encouraged to move and change position during labour were found to have low population level reporting bias in two studies in Kenya and Nigeria.^13,22,25 While in studies in Mexico and Kenya, maternity care experience measures, including whether women were allowed to drink or eat during labour and childbirth, encouraged to move and change position during labour and encouraged to have some light food during labour and delivery met none of the validity metrics.^22,26

The low prevalence of the negative maternity care experience measures resulted in only six of the sixteen negative maternity care indicators meeting the requirement for the validation analysis. For these six measures, the exit interviews showed low to moderate sensitivity and high specificity for the detection of negative maternity care experience events. The low to moderate sensitivity of the exit interviews to detect negative maternity care experience events showed the likelihood of the method erroneously labelling maternity care experience event as positive, potentially masking prevalent mistreatment. The low frequency of the negative maternity care experience events or the possible bias related to self-report of negative maternity care experiences could explain the sensitivity and specificity observed.^27,28 In a study in Tanzania, observers reported a higher prevalence of disrespect and abuse than mothers in exit interviews at baseline and endline.¹¹ In India, a comparison of mistreatment of mothers by providers showed mothers’ self-report of mistreatment to be slightly lower than reported in childbirth observations.¹⁰ Conversely, women were more likely to report the quality of maternity care in a more positive light than in childbirth observations.²² Nonetheless, the six negative maternity care experience measures that met the condition for validation analysis had high individual level reporting accuracy. But only one of the measures had low population level reporting bias ‘Denial or lack of birth companions during labour and childbirth’. Studies on the validity of exit interview to derive maternity care experience measures are few, and none of the previous studies that we have reviewed reported on the any of the negative maternity care experience measures evaluated in this study.

Implication for research, practice, or policy

Out of the 14 measures evaluated for validity, including eight positive and six negative maternity care experience indicators, seven met both individuals, and population level validity criteria and seven met only the individual level validity criterion. Not meeting both validity criteria does not invalidate a measure for all measurement purposes.^22,26 Those exit interview measures that only met the individual criterion may still be suitable to assess service quality at the health facility.^22,26

Study limitations

While the study provides new insights regarding the validity of positive and negative maternity care experience measures, there were some limitations. The study was conducted in ten primary health care facilities, and our findings may be more reflective of the population accessing this level of care in this setting. The exit interviews were conducted with women following facility childbirth, upon discharge, and before they exited the health facility. Hence, the short recall period may have resulted in a more accurate self-report of experiences with maternity care. It may also be possible for a mother to be exhausted from childbirth or not to have had enough time to process her experiences with facility childbirth care.

Conclusions

In this setting, exit interviews with women following facility-based childbirth care provided responses consistent with the observation of childbirth for eight positive and six negative maternity care experience indicators.

Acknowledgments

The authors would like to thank the Gombe State Primary Health Care Development Agency, Gombe State Ministry of Health, our partners, the Society for Family Health and Pact Nigeria. Mr Dahiru Ribadu (DRMC), fellow researchers and collaborators who supported the study. Finally, we are most grateful to mothers, their families, the health workers and data collectors.

Funding

This work was supported by IDEAS - Informed Decisions for Actions to improve maternal and newborn health (http://ideas.lshtm.ac.uk), which is funded through a grant from the Bill & Melinda Gates Foundation to the London School of Hygiene & Tropical Medicine. (Gates Global Health Grant Number: INV-007644).

Authorship contributions

NU, TM, conceptualised and designed the study; NU, NUS & IBJ coordinated data collection; NU, AB, & TM conducted the analysis; NU drafted the manuscript; All authors revised the manuscript and provided critically important intellectual content; all authors read and approved the final manuscript.

Competing interests

The authors completed the Unified Competing Interest form at www.icmje.org/coi_disclosure.pdf (available upon request from the corresponding author), and declare no conflicts of interest.

Correspondence to:

Nasir Umar, Assistant Professor, Department of Disease Control, Faculty of Infectious and Tropical Diseases, London School of Hygiene & Tropical Medicine, London, United Kingdom. Keppel St, London WC1E 7HT, United Kingdom, Email: [email protected]; [email protected].