Measurement of respectful maternity care in exit interviews following facility childbirth: a criterion validity assessment in Nigeria

Valid methods for assessing women’s experiences of maternity care are essential to the global efforts toward providing positive childbirth environments for all women in all health facilities.   This criterion validity study used observation of childbirth as the reference standard compared to exit interviews with women upon discharge with a live baby, usually within 24 hours of childbirth. We investigated eight positive and sixteen negative maternity care experience indicators. Data were collected from ten primary healthcare facilities in Gombe State, northeast Nigeria, in August 2018 and August 2019. Data analysis involved tabulation of demographic characteristics of women and childbirth context, computation of individual level validity metrics including the area under the receiver operating curve (AUC) and estimating population level validity using the inflation factor (IF).   A total of 724 women were observed and interviewed at the time of discharge (exit) following facility childbirth, 15% of whom were adolescents, 99% were married, 43% had no formal education, and a skilled birth attendant had attended only 12%. The prevalence of positive maternity care experience indicators ranged between 25% and 96% in childbirth observations. For these positive indicators, the agreement between childbirth observations and exit interviews ranged from 55% to 97%. Six of the eight positive maternity care experience indicators had high overall validity, meaning AUC≥0.70 and 0.75<IF<1.25, with high sensitivity (89% to 99%) and moderate to high specificity (44% to 84%). The prevalence of the 16 negative maternity care experience indicators ranged between 0.1% and 18% in childbirth observations. For these negative indicators, agreement ranged from 87% to 99%. Just six of the 16 negative maternity care experience indicators met the criteria for validation analysis; and these showed low to moderate sensitivity (32% to 74%), high specificity (97% to 100%) and moderate (0.60<AUC<0.70) to high (AUC≥0.70) individual-level validity.   In this high mortality setting with relatively low coverage of skilled attendance at birth, exit interviews with women following facility-based childbirth care provided responses consistent with the observation of childbirth for eight positive and six negative maternity care experience indicators.

ria have reported that two-thirds of women experience at least one form of negative maternity care experience 2 and that respectful and dignified care is an important determinant of women's decision for facility childbirth. 6he World Health Organisation (WHO) defined quality of care framework consisting of eight domains for improvement, three of which pertain to maternity care experience: effective communication, respect and preservation of dignity and emotional support. 4,7Measures of these experiences should be used to assess and improve the quality of services. 4,8But collecting accurate and reliable information about negative and positive childbirth care experiences poses multiple challenges: there are many possible tools and data collection modes to apply, operational definitions of measures may vary by context, and the optimal recall period for self-report is a subject of debate. 9Validation research is sparse, reporting mixed results, limiting evidenceinformed decision-making on how best to derive measures of maternity care experiences. 10,11he present study from northeast Nigeria contributes to the need to improve the measurement of experience of childbirth care.We measured the experience of positive and negative maternity care practices through childbirth observations (the reference standard) and exit interviews with women upon discharge following facility childbirth and tested the validity of the self-reported measures at the exit.

METHODS STUDY SETTING
The study was conducted in Gombe State, north-east Nigeria, where maternal and newborn health services are provided primarily through public health facilities, mostly by Community Health Extension Workers (CHEWS), Junior CHEWS and Health Officers.In 2018, uptake of maternal and newborn health services in the State was estimated to be relatively low, with just 44% of women having at least four antenatal care visits and just 28% of women having facility childbirth, compared to 57% and 39% nationally. 12he sampling methods for the present study have been described elsewhere. 13,14Briefly, in November 2015, we selected ten high-volume facilities from 57 primary health care (PHC) facilities for an in-depth assessment of maternal and newborn health care quality.The selected facilities had the highest number of births in the preceding six months based on recorded births in the maternity registers: mean number of births per month of 15.7 (SD 12.0), compared with the Gombe State average of 4.3 (SD 6.3) births per facility per month.This sampling aimed to achieve a sufficiently large number of observations while minimising the duration of data collection.

OBSERVATIONS OF CHILDBIRTH
Data for the study were collected in two rounds of childbirth observations in August 2018 and August 2019.Each round of data collection lasted three weeks, during which time the data collectors aimed to conduct a total of 350 childbirth observations.In each study facility, two trained female observers (all local midwives but not staff at the assigned PHC) and one clinical supervisor worked 8 or 12 hours shifts to ensure continuous data collection.The observers stayed in the same room with women from the first point of contact through the first hour after birth.Data collectors used a structured clinical observation checklist programmed on a Lenovo A3300 tablet using CSPro V.7.0 (US Census Bureau and ICF Macro, Suitland, Maryland, USA), to continuously document labour and childbirth care processes and birth attendant-client interactions.The data observation checklist was piloted and modified to the Gombe context. 14omen were eligible and invited to participate in the study at admission.Recruitment involved providing all potential participants with a study information sheet and a consent form in English and Hausa, reading the information sheet, explaining the purpose of the study, the risks and benefits of participating and answering questions before obtaining written informed consent from eligible women to observe the childbirth care received and verbal consent from the healthcare worker attending.Care was taken to ensure the inclusion of any support persons accompanying potential participants in the consent taking.In cases where women could not sign or write their names, they were asked to provide a thumbprint on the consent form.Women were informed of their right to participate or the freedom to withdraw at any time.
Observers were trained for four days before each round of data collection on how to conduct unobtrusive observations of childbirth while maintaining the safety and confidentiality of the participants, following a specified protocol.The protocol described procedures on how to prioritise the safety of the women and newborns over data collection and on how observers should seek help in the event of any complication or life-threatening event during childbirth since observers had no legal right to intervene in clinical care during childbirth observation in their assigned health facilities.Data quality assurance involved clinical supervisors conducting spot checks of observers and data throughout the observation period.

EXIT INTERVIEWS
All women observed during labour and childbirth, who were discharged with a live baby, and provided informed consent, were invited to participate in an exit interview.The exit interview instrument was also written to tablets using CSPro, and covered questions consistent with the information recorded during the observations of childbirths, including the content of care provided to the mother and the newborn and women's experiences of care.Questions about the experience of care required 'experienced an event' (yes) or 'not experienced an event' (no) or 'don't know responses'. 15he exit interviews were conducted in the Hausa language in an area reserved for the interviews or in a separate room within the health facilities to ensure privacy and confidentiality.In each health facility, the exit interviews were con-Measurement of respectful maternity care in exit interviews following facility childbirth: a criterion validity assessment...

Journal of Global Health Reports
ducted by the same team that conducted the observations of childbirth.

INDICATOR SELECTION
Twenty-four maternity care experience indicators, including eight positive maternity care experiences and sixteen negative maternity care experiences were selected.][18][19][20] The final list of indicators is provided in Table S1.

STATISTICAL ANALYSIS
Data from August 2018 and August 2019 were combined for the analysis.Observations of childbirth and exit interviews were matched by unique participant id.All analyses were conducted using STATA version 16 (www.stata.com).Observations were used as the reference standard and compared to exit interview responses.We computed the characteristics of women observed during labour and childbirth and interviewed at discharge.We tabulated the prevalence of positive and negative maternity care experiences documented during childbirth observations and self-reported by mothers.
To evaluate the individual level validity metric, we constructed two-by-two tables of women's responses in exit interviews (Yes, No) to observations (Yes, No), for all 24 indicators separately.Missing and don't know responses were excluded from the two-by-two tables.We then computed the sensitivity and specificity of women's recall, and calculated per cent agreement between the exit and observations.We determined the area under the receiver operating characteristic curve (AUC) for the measures with at least 5 counts per cell, with AUC<0.60,AUC 0.60<AUC<0.70,][23] For the evaluation of the population level accuracy, we computed the inflation factor (IF).The IF represents the degree to which an indicator would be over-or underestimated in a population-level survey: the prevalence of the measure under consideration that would be obtained for respondents in a survey (Pr) divided by the true prevalence (P) from the reference standard.IF estimates of 0.75<IF<1.25,0.50<IF<0.75or 1.25<IF<1.5,and IF<0.50 or IF>1.5, reflecting low, moderate and large degrees of bias.Positive and negative maternity care experience measures with moderate or high individual level accuracy and low population level bias are considered to have acceptable validity. 21,22e presented findings below in line with the STROBE statement. 24

ETHICS APPROVAL
The study was conducted with approvals from the Nigeria Health Research Ethics Committee (reference NHREC/01/ 01/2007), the State Ministry of Health Gombe State, Nigeria and the London School of Hygiene & Tropical Medicine (reference 12181).The free signed and verbal informed consent of all interviewees was obtained.

CHARACTERISTICS OF STUDY PARTICIPANTS
The characteristics of the study participants are presented in Table 1.A total of 754 women were observed during childbirth care, of whom 724 women participated in the exit interviews, 30 of whom did not meet the inclusion criteria for the exit interview.Women's ages at the time of childbirth ranged from 15 to 47 years.Close to half of the women had four or more prior childbirths.Further, nearly half of the women had no formal education, and lower cadre health workers conducted most childbirths.Of the 724 women, 716 were married, seven were single, and one was widowed.

POSITIVE MATERNITY CARE EXPERIENCE MEASURES
From the observations, the prevalence of maternity care practices that recognise women's preferences and needs (positive maternity care experience) ranged between 25% for the indicator 'were you asked which position you would like to deliver in' and 96% for the indicator 'were you respectfully greeted by health workers when they first saw you'.Agreement between the observation and the exit interviews ranged between 55% and 97% (Table 2).Very few don't know responses, and all indicators had at least 5 values per cell in the two-by-two tables.Exit interviews with women showed high sensitivity (89% to 99%) and moderate to high specificity (44% to 84%) for reports of positive maternity care experience events that were in accordance with observations.Six of the eight indicators had high overall validity, meaning AUC≥0.70 for high individual-level accuracy and 0.75<IF<1.25 for low population-level bias (Table 2).

NEGATIVE MATERNITY CARE EXPERIENCE MEASURES
The prevalence of negative maternity care experience events documented during the observations ranged between 0.1% (observed only once) for the indicator 'woman being physically restrained, tied, or gagged during labour and delivery' and 18% for the indicator' Staffing shortages during labour and childbirth'.The reported prevalence for all negative indicators is shown in table S2 where we also observe agreement between observed and exit interview reports to be over 85% for all indicators.However, the low prevalence for many negative measures resulted in just six of the 16 indicators having at least 5 counts per cell in the two-bytwo tables: validation results for these six measures are shown in table 3.For these six measures, the exit interviews showed low to moderate sensitivity (32% to 74%) and high Measurement of respectful maternity care in exit interviews following facility childbirth: a criterion validity assessment... specificity (97% to 100%) for the detection of negative maternity care experience events.The six negative maternity care experience measures met at least one validity criterion for high individual level reporting accuracy or low population level bias (Table 3).Two of the indicators related to the domain' failure to meet professional standards of care', including 'lack of informed consent process (e.g., examine without permission)' and 'skilled attendants absent at time of childbirth', met the individual-level validity metric but not the population-level validity metric.'Staffing shortages (perceived) during labour and childbirth' and 'lack of privacy', the two indicators related to the domain' health system conditions and constraints' had high individual-level valid-ity but did not meet the population-level validity metric.'Denial or lack of birth companions during labour and childbirth', one of the two indicators related to the domain' poor rapport between women and providers' that met the individual-validity metric also had low population-level bias (Table 3).

DISCUSSION
Our study adds to the growing but still limited literature on the validity of exit interviews to derive self-reported measures of maternity care experience.We found that maternity care experiences self-reported in exit interviews with women conducted at the time of facility discharge were consistent with the observation of childbirth for all eight positive maternity care experience indicators and six of the sixteen negative maternity care experience indicators investigated.
For the positive maternity care experience measures assessed, exit interviews with mothers had high sensitivity but lower specificity.These findings indicate that, for this study population, exit interviews can be used to detect maternity care practices that recognise women's preferences and needs.Exit interviews with women had high individual level reporting accuracy, with all the eight positive maternity care experience measures meeting the individual level validity metric and most of the measures (6 out of 8) having low population reporting bias.Some but not all of our findings are consistent with previous studies.For example, positive maternity care experience events, such as support persons present during labour and childbirth and women encouraged to move and change position during labour, consistently have high overall validity. 13,22,25But the measures on whether women were allowed to drink or eat during labour and childbirth or encouraged to move and change position during labour were found to have low population level reporting bias in two studies in Kenya and Nigeria. 13,22,25While in studies in Mexico and Kenya, maternity care experience measures, including whether women were allowed to drink or eat during labour and childbirth, encouraged to move and change position during labour and encouraged to have some light food during labour and delivery met none of the validity metrics. 22,26he low prevalence of the negative maternity care experience measures resulted in only six of the sixteen negative maternity care indicators meeting the requirement for the validation analysis.For these six measures, the exit interviews showed low to moderate sensitivity and high specificity for the detection of negative maternity care experience events.The low to moderate sensitivity of the exit interviews to detect negative maternity care experience events showed the likelihood of the method erroneously labelling maternity care experience event as positive, potentially masking prevalent mistreatment.The low frequency of the negative maternity care experience events or the possible bias related to self-report of negative maternity care experiences could explain the sensitivity and specificity observed. 27,28In a study in Tanzania, observers reported a higher prevalence of disrespect and abuse than Measurement of respectful maternity care in exit interviews following facility childbirth: a criterion validity assessment...

Journal of Global Health Reports
mothers in exit interviews at baseline and endline. 11In India, a comparison of mistreatment of mothers by providers showed mothers' self-report of mistreatment to be slightly lower than reported in childbirth observations. 10Conversely, women were more likely to report the quality of maternity care in a more positive light than in childbirth observations. 22Nonetheless, the six negative maternity care experience measures that met the condition for validation analysis had high individual level reporting accuracy.But only one of the measures had low population level reporting bias 'Denial or lack of birth companions during labour and childbirth'.Studies on the validity of exit interview to derive maternity care experience measures are few, and none of the previous studies that we have reviewed reported on the any of the negative maternity care experience measures evaluated in this study.

IMPLICATION FOR RESEARCH, PRACTICE, OR POLICY
Out of the 14 measures evaluated for validity, including eight positive and six negative maternity care experience indicators, seven met both individuals, and population level validity criteria and seven met only the individual level validity criterion.Not meeting both validity criteria does not invalidate a measure for all measurement purposes. 22,26hose exit interview measures that only met the individual criterion may still be suitable to assess service quality at the health facility. 22,26UDY LIMITATIONS While the study provides new insights regarding the validity of positive and negative maternity care experience measures, there were some limitations.The study was conducted in ten primary health care facilities, and our findings may be more reflective of the population accessing this level of care in this setting.The exit interviews were conducted with women following facility childbirth, upon discharge, and before they exited the health facility.Hence, the short recall period may have resulted in a more accurate self-report of experiences with maternity care.It may also be possible for a mother to be exhausted from childbirth or not to have had enough time to process her experiences with facility childbirth care.

CONCLUSIONS
In this setting, exit interviews with women following facility-based childbirth care provided responses consistent with the observation of childbirth for eight positive and six negative maternity care experience indicators.