In recent years, there has been a growing global commitment to enhance maternal and newborn health outcomes, leading to the implementation of various initiatives and interventions. These efforts aim to reduce mortality rates and enhance the well-being of mothers and their children. Ensuring quality health services is particularly crucial in low and middle-income countries, such as Ethiopia, where maternal (412 maternal deaths per 100,000 live births)1 and neonatal (33 neonatal death per 1000 live births)2 mortality rates remain high despite recent improvement in access to health services.3–6

Monitoring progress in healthcare requires more than just measuring the coverage of health services provided. Equally important is the assessment of the quality and impact of these services on the health of populations.7,8 In this context, measuring effective coverage has gained recognition as a valuable method for assessing the impact and extent of high-quality healthcare services.3–6,9 Taking a cascade approach, effective coverage indicators quantify the proportion of the population in need of healthcare services that receive those services to a level of quality that should result in maximum possible health gain. Besides, effective coverage identifies gaps in service quality that may inform improvement strategies.10 For maternal, newborn and child health, effective coverage indicators focus on the potential of various healthcare contacts, such as antenatal care, skilled birth attendance, and care for sick newborns or children, to contribute to positive health outcomes.

Following the first introduction of the concept by Dr T. Tanashi in 1978,11 many have defined and used effective coverage to measure the quality of health service.12 More recently, in 2021, a Think Tank group came together to propose a standardised cascade for effective coverage indicators that incorporate data points on health care (1) contacts, (2) inputs, (3) interventions, (4) processes, (5) adherence, and (6) outcomes, drawing on previous studies.13 Nevertheless, there continue to be large variations in the definition and measurement approach of effective coverage indicators.12,14 Moreover, relatively few examples of effective coverage measures are being implemented on a large scale within country plans. Challenges to operationalising effective coverage measurement have been hypothesised to include resource constraints, data quality issues, and the complex nature of measuring healthcare outcomes comprehensively.15

Considering these challenges, the aim of our study was to explore the demand for effective coverage measurement among key Ethiopian stakeholders and identify the actions needed to enhance effective coverage measurement implementation in Ethiopia’s healthcare system.

METHODS

Study setting and design

Ethiopia has made important strides in recent years towards improving the quality of healthcare services and the collection of essential healthcare data. With a diverse population exceeding 120 million people, Ethiopia faces the challenge of providing adequate and quality healthcare services to its citizens. To address this, the Ethiopian government has demonstrated a strong commitment to enhancing the nation’s healthcare infrastructure, providing quality health service and harnessing the power of data-driven decision-making.16,17

This study included national level participants from selected directorates of the Ministry of Health (MoH), the Ethiopia Public Health Institute (EPHI), as well as partners who were working in the area of maternal, newborn, and child health (MNCH) service quality improvement; and sub-national actors from three Ethiopian regions (Amhara, Oromia, and Gambella) and one administrative city (Addis Ababa).

We employed an exploratory qualitative study design, and collected data through semi-structured interviews, allowing for in-depth exploration of participants’ perspectives, experiences, and insights. This qualitative piece is part of a mixed method study that included desk review and secondary analysis of data from the district health information system-2 (DHIS2).

Sampling process

Nineteen key informants were purposefully selected using a snowball sampling technique; individuals with expertise and familiarity in tracking quality of health service were intentionally chosen. Six respondents were from the regions, two were from Addis Ababa, and the rest were from the national government and partners. A validation workshop was then held with 13 experts and policymakers in the field of health service quality and effective coverage measurement, along with five members of the research team. The size of the sample was decided based on the saturation level of the data during the interview.

Data collection

The semi-structured interview guide was developed to explore participant perceptions of the need for effective coverage measurement in Ethiopia, the challenges faced in measuring effective coverage indicators, and recommendations to overcome those challenges. The interview guide was initially prepared in English and translated into the local languages Amharic and Afan Oromo. The tool was back-translated to English to ensure consistency. The interview guide and data collection protocol was pilot tested and updated accordingly.

The qualitative interviewers had at least masters level training and prior experience in qualitative research at the Ethiopian Public Health Institute. Before the data collection, five days refresher training was given on principles of qualitative interview, research ethics and on the aim and methods of this research work. The interview guide was also discussed and reviewed before pilot testing.

During data collection one interviewer and one note taker conducted interviews together. Participants were interviewed in a quiet and private place where they could give their opinions comfortably. Interviews were audio recorded for later transcription, alongside the notes taken during the interview. The transcriptions were initially done in the local languages and then translated into English for data analysis purposes. During the validation workshop, the research team presented a preliminary report on the qualitative interview data. Participants were encouraged to provide their feedback on the preliminary result. Participants provided their feedback on the trustworthiness of the qualitative result and their concerns regarding the way forward in measuring effective coverage for maternal, neonatal and child health indicators in Ethiopia. Further analysis was conducted, incorporating the feedback from the workshop participants.

Data analysis

Interview transcripts were thematically analysed using a deductive approach, guided by the research objectives. Four researchers participated in coding the qualitative data. A list of codes was prepared by the research team after reading sample transcripts, and new codes were added to the list as needed during the coding process. We used NVivo 12 software to identify codes and categorise emerging themes from the data. Interpretations were made to draw meaningful insights from participants’ narratives.

RESULTS

Three main themes emerged. First, we present findings pertaining to the existing practice of tracking health service quality in Ethiopia, providing important context about the backdrop against which effective coverage measurement is being considered. Second, we characterise the demand for effective coverage measurement as expressed in two different ways: the need for measures to monitor health system performance and the need for measures to identify gaps in individual patient care. Third, we summarise the four key actions identified by respondents to advance the measurement agenda: the need for high-quality data, the need for clarity of effective coverage metrics, the need for leadership, and the need for sustained motivation.

1. The current practice of tracking health service quality

There is a strong commitment from MoH and its partners to track and provide quality health services, as evidenced by the national health sector transformation plan as well as the different health system guidelines. Alongside this, various approaches have been used to track progress and to identify gaps in health service provision. As shared by one of the respondents, the application of the “Donabedian model” was prominent in quality measurement discussions, focusing on inputs, qualified personnel, patient-centeredness and satisfaction, and health outcomes.

“We primarily employ the “Donabedian model” as our framework for assessing quality. This involves checking essential elements like inputs, qualified staff, equipment, infrastructure, and data systems. Another crucial aspect is evaluating how care is delivered, focusing on the practice and its appearance. Our indicators often gauge the process of service provision, particularly its patient-centeredness for mothers and children. By combining inputs and process, we then analyse the outcomes, using triangulation to verify expected or interim results.” (MoH National)

Respondents also indicated that multiple additional approaches have been implemented to measure quality at the health facility level to inform patient care practice. Standard tools, including the clinical audit tool and bottleneck analysis, are available for this purpose.

“This [clinical audit] is not formal research, but it is a tool used by frontline health workers to assess, for instance, the service given to mothers and children in the last three months and examine whether it was of quality or not. This is self-assessment, and based on the result, the health worker/health facility is expected to design QI activities to fill identified gaps from the assessment.” (MoH Reginal)

Another respondent from the national level also said:

“We have a new initiative called system bottleneck focused reform. We are working on eight hospitals to identify system bottlenecks and carry out improvement works. There are also quality measures in each service area. So, we are tracking it almost every month.” (MoH National)

Quality health service is also monitored to track the progress of quality improvement (QI) projects designed by health facilities based on identified gaps in quality health service provision. Core outcome and process indicators are evaluated to define gaps in health service quality using root cause analysis at the facility level. Sometimes, these findings are also aggregated to inform MoH.

“After a root cause analysis, for instance, if the problem is identified as not filling the partograph and not taking appropriate measures based on the information recorded on the partograph, we design a QI project, and measure this process and the outcome (the number of mothers who died after delivering their baby).” (Partner National)

In addition, both practitioners and policymakers at national and subnational levels emphasised the importance of implementing effective coverage measurement for MNCH. This collective perspective underscores the alignment with real-world needs.

“…the crude comparisons often do not show the health impact, … We can’t reduce mortality simply by having a mother arrive at a healthcare facility. I think it is better to measure the outcome by showing how many interventions she has received.” (MoH National)

The Ministry of Health has recently adopted the concept of effective coverage within the framework of these multiple approaches to measuring healthcare quality.

“… there is a clear demand for such [quality adjusted] measurement…. We recently completed a five-year global initiative on MNCH service quality. In 2020, we started a new strategy called National Quality Initiative II. In this, one of our objectives is to establish effective coverage measurement for all service areas that include maternal neonatal health (MNH).” (MoH National)

2. Characterising demand for effective coverage measurement

Demand for a comprehensive method to effectively monitor and track health service quality was expressed in two different ways: to monitor health system performance and to identify gaps in the provision of individual patient care.

Monitoring health system performance

There was a strong emphasis on the importance of achieving Sustainable Development Goal targets related to health and well-being, and a growing sense of urgency to measure and ensure the effectiveness of healthcare services.

“Instead of solely counting ANC visits, it is preferable to measure the specific packages of care received. …shifting to quality measurement aligns with the health transformation agenda outlined in the health sector transformation plan (HSTP)” (MoH National)

The priorities identified for the application of effective coverage measurement included antenatal care, childbirth and postnatal care, family planning services, immunisation, and chronic disease services. A participant from the national office expressed the following perspective:

“MCH has been a priority for thirty years…. There is better data there because everyone has been collecting it. The partner’s interest is there, and the government’s interest is there… so, it will be easier to process.” (Partner National)

Further, a more detailed picture of healthcare provision for high-risk complications was needed. Challenges during childbirth and the postpartum period, including maternal haemorrhage, eclampsia, and caesarean sections, were discussed in terms of needing effective coverage measurement to identify gaps, enhance care quality, and improve health outcomes in these areas. Respondents were clear in their collective concern that some health outcomes had shown limited improvement at the population level despite increased access to the services in the last decade. Respondents acknowledged the need for a closer look at the service provision processes and the availability of comprehensive amenities. As described by one respondent,

“Access to services has reached up to the community through the health extension program. However, the reduction in maternal death is not as we expected it to be….. it is not the routine contact with the health service that matters or brings change. When a pregnant woman comes to a health service, there is a service package that she should get, and mothers are not getting the complete package or content of care during their health facility visits… It is just that mothers come and go; mothers visit the health facility four times during pregnancy, yet they die of haemorrhage, hypertension, or infection.” (MoH, Regional)

Respondents appreciated the potential benefit of effective coverage measurement in creating access to comprehensive evidence on the quality of health services, thereby facilitating rapid decision-making at all levels of the health system.

“I think it’s beneficial for decision-making as it reveals each [step in the quality] cascade. It facilitates swift policy decisions and enables informed choices. I guess there will be something to see. Because of this, policymakers should use it, regional health offices and national level too.” (MoH National)

“The service quality gap is enormous… if we start measuring quality adjusted coverage, focusing on the content of care, we could reduce this health service quality gap. Because we can assess the availability of input, interventions, or content of care…we can also assess whether mothers and their newborn received the minimum package of services during their visit to a health facility or not.” (MoH, Regional)

Identifying gaps in individual patient care

Effective coverage measures were also identified as a potential facility-level service quality improvement tool. Some respondents strongly argued that effective coverage measurement had to be done at the health facility level, where data on the content of care is readily available from registers and individual patient charts.

“…if you go to the registers, you will find everything needed for patient care, they have so much data.” (National Partner)

“To be frank, a closer examination is essential…. it becomes imperative to delve into the underlying factors associated with healthcare facilities and individual infant care.” (MoH National)

A participant at the validation workshop also recommended effective coverage metrics to evaluate the success of QI projects in health facilities.

“Health facilities are conducting QI projects and collecting data to track their progress, so it is good to consider such data in effective coverage measurement approaches to inform the QI projects.” (Participant validation workshop)

3. Actions needed to advance effective coverage measurement

Participants described four actions needed to meet the demand for effective coverage measurement in Ethiopia.

Need for high-quality data

Almost all of the respondents raised the need for high-quality data but also commented that existing data sources in the health system should be the starting point, citing existing national household and facility surveys and the routine health information system (including patient charts and DHIS2). Survey data were considered to be of better quality compared to the routine health information system data, although the periodicity of surveys made them less useful for routine quality improvement efforts.

“First, we can get the data (routine) every month even if it is not in real-time. But we get DHS retrospectively every 5 years. But this one [DHIS 2] is very useful for service improvement actions. This is because you get it every month and having recent data in real time is better for taking appropriate action.” (MoH National)

Respondents acknowledged the investments that have been made by the government of Ethiopia to revolutionise its routine health information system by introducing DHIS2 in the last decade and encouraged the use of this data source for decision-making as a mechanism to improve data quality further.

“We need to properly map the available data; I believe DHIS2 has a lot of data also the client card/chart. …” (Partner National)

However, some respondents were concerned about the high number of data elements and indicators already included in DHIS2 and that the demands of effective coverage measurement could overburden the system.

“…the DHIS2 system is already generating too much data. If we add more inputs, I think if more data are in, it will compromise the system…” (MoH National)

Need for clarity on effective coverage

Though respondents understood and appreciated the importance of tracking health service quality, many were not yet clear about the methods underpinning effective coverage measurement. A need for a better and more widespread understanding of the effective coverage cascade, indicator definitions and measurement methods were mentioned as crucial. For example, one of the participants questioned why health facility infrastructure should be included in an indicator definition:

“Input selection should focus on the inputs that have a significant impact or are closely related to the outcome of interest. For instance, to test for hepatitis, you only need a test kit, not electricity.” (Participants, validation workshop)

Another area of uncertainty was the method for data aggregation. According to the respondents, understanding regional variations in health priorities and healthcare delivery was crucial and guidance was needed to examine national, regional, zonal or district disaggregation of some effective coverage indicators, for example, malaria indicators. Challenges could also arise when attempting to aggregate measures on a national scale due to the knowledge that there were differences in data quality by region.

“To achieve this goal, it’s essential to define the scope of the specific [health] area accurately. As you broaden the perspective, it might become less refined due to the aggregation of data…certain procedures may not be as prevalent in our current practices.” (MoH Regional)

Need for leadership

According to the participants, to advance implementation of the measures requires collaborative efforts, tailored services, proficient data handling, and a skilled workforce. Leadership was needed to foster that collaboration:

“Besides the data, engaging stakeholders and having the necessary human resources are essential inputs. I believe there’s a significant potential in this regard. This effort is not solely the responsibility of a single institution or ministry; it necessitates collaboration. For instance, involvement from the government, the health sector, research institutions, and the ministry is vital, along with universities and partners.” (Partner National)

Need to stay motivated

Respondents worried that applying the effective coverage cascade could result in discouraging findings; it was inevitable that adjustment for dimensions of quality would result in effective coverage estimates that were lower than crude coverage estimates since many health facilities lack basic amenities. Managing expectations in this regard was thought to be an important part of the process of normalising effective coverage measures.

“Let’s speak the truth, how many health facilities have basic amenities, if there is a handwashing structure, the water may not be there. You find hospitals with their big buildings with no access to water supply. So, considering these inputs in the effective coverage measurement will give us zero value. And it is frustrating…” (Partner National)

DISCUSSION

This paper revealed that many actors in health care quality in the Ethiopian health sector are talking about effective coverage measurement and want to see more quality-adjusted measures used for decision-making. The demand for this measurement is characterised by the need to use effective coverage measures for individual patient care or to track health system performance; currently, there is limited clarity about the distinction between these two needs. To advance the effective coverage measurement agenda in Ethiopia, respondents recommend building confidence in high-quality data sources, more guidance on how to generate measures, and government leadership to coordinate these efforts. The importance of effective coverage measurement in Ethiopia is aligned with the government’s commitment to providing high-quality health care and also reflects the strong advocacy for improved health system quality that has gained momentum over the last decade.3,6,13 Nevertheless, despite this growing momentum, there continue to be considerable gaps in knowledge about how to define and construct effective coverage indicators,12,13 making the measures difficult to interpret or replicate at the country level. In part, this lack of clarity has emerged because of the need to reflect on quality health care within a given context, with respondents in this study even highlighting sub-national differences as the burden of disease changes between regions in a diverse country like Ethiopia. Moving forward, it will be important to properly engage with country health systems and their priorities when defining effective coverage indicators.

Respondents strongly suggested that the availability of high-quality data was crucial in deciding what was practical in effective coverage measurement of a given health intervention.18 From experience, respondents acknowledged that the periodically conducted population and health facility-based surveys were the main sources of data used to date. This experience was consistent also with the studies reported in published literature,5,12,19 with only one study published so far using routine health information to measure effective coverage.20

However, in this study, the recommendation to use the routine health information system data was partly linked to the need for effective coverage measures to improve individual patient care as well as to track progress at the health facility or higher level. Despite data quality challenges, it was thought that in Ethiopia, the routine health information data could have an important temporal advantage over surveys and the continued commitment to quality routine health information data by MoH could improve individuals’ trust and desire to use the routine data.

This study captures the voices of national and sub-national actors in a country where there is a commitment to take the effective coverage measurement agenda forward. The trustworthiness of our findings was checked through a validation workshop, where summary themes were reported back to respondents and experts on the subject for further discussion. Nonetheless, the study is constrained by the relatively early stage of implementation of effective coverage measures in the country, and we expect that views may change as more experience is gained at the country level.

CONCLUSIONS

In Ethiopia there is a clear need for effective coverage measurement to monitor health system performance, supported by a positive policy environment that encourages further testing of effective coverage measurement.

Acknowledgements

We would like to thank all the key informants for their time and invaluable participation. We also appreciate the support the study team received from Ministry of Health Ethiopia, and Ethiopian Public Health Institute.

Ethics statement

Ethical approval was obtained from both the Ethiopian Public Health Institute [EPHI-IRB-478-2022] and the London School of Hygiene & Tropical Medicine (ref 17541) before data collection. A formal letter was sent to the key informants’ institutes and received permission to interview. Informed written consent was obtained from all participants, and they were assured of the confidentiality and anonymity of their responses. They were also informed of their right to withdraw from the study at any time without facing any negative consequences.

Data availability

All data is derived from qualitative interviews, with stakeholders where only one individual holds a position, either within federal or state government, facilities, or NGOs. Every care has been taken to ensure anonymity of the data in the submitted manuscript, but the authors feel strongly that making data freely available would jeopardise the conditions of informed consent. We therefore request to be exempt from the requirement to make data available.

Funding

This work was funded by the Bill & Melinda Gates foundation via grant INV-007644 to TM at LSHTM, United Kingdom. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.

Authorship contributions

SL was involved in the conception and design of the work; the acquisition, analysis, and interpretation of data for the work; and drafting the work, reviewing the manuscript critically. TM was involved in the conception and design of the work; interpretation of data for the work; and reviewing the manuscript critically. AT, FG, and HA were involved in the design of the work, acquisition, analysis, and interpretation of data; and drafting the work, and reviewing the manuscript critically. BY and TG were involved in the design of the work and reviewing the manuscripts critically. All authors approved this final version to be published and agreed to be accountable for all aspects of the work.

Disclosure of interest

The authors completed the Unified Competing Interest form at http://www.icmje.org/disclosure-of-interest/ and it is available upon request from the corresponding author and declare no conflict of interest.

Correspondence to:

Name Seblewengel Lemma
Institution London School of Hygiene and Tropical Medicine
Address
Country Ethiopia
[email protected]