Data for local decision-making, not a mere reporting requirement: development of an index to measure facility-level use of HMIS data

Health Management Information Systems (HMIS) involve integrated processes whereby health data are recorded, stored, retrieved, and processed.^1–3 By producing reliable and timely health data, a well-functioning HMIS enables decision-makers at all levels of the health system to identify needs, make evidence-based decisions, and optimally allocate resources – all of which are essential elements in achieving large-scale health improvements.⁴ The Ethiopian Federal Ministry of Health (FMoH) is highly invested in strengthening the country’s HMIS.⁵ By the end of 2013, 98% of public hospitals and 87% of health centers in Ethiopia had already adopted HMIS.⁵ The first Health Sector Transformation Plan (HSTP 2015-2020) called for an “information revolution” in the way that health data are collected, analyzed, and disseminated to deliver quality and equitable health services.⁵ This national plan gave rise to several initiatives to enhance HMIS data use and data quality, including capacity-building training on data recording and reporting, data analysis, quality assurance, and guidance on policymaking using routine health data.^6–9 The District Health Information Software (DHIS-2) – a web-based data capturing and analyzing platform – was introduced in 2018, in parallel with a nationwide campaign to improve internet connectivity in health facilities.¹ The most recent data available showed that >95% of health facilities employed either the online or offline version of DHIS-2. The reporting rate using DHIS-2 varied by region and ranged from about 80% for disease reporting to 90% for health service reporting.¹⁰

It has been well established that information management and use serve as the basis for evidence-based decisions in health facilities.¹¹ These decision-making processes include priority setting, annual health planning and budgeting, health resource allocation, and strategies to promote greater health service utilization. Data-driven health decisions contribute to a culture of transparency and accountability, as the allocation of human resources, finances, and commodities are based on evidence of needs.¹² In Ethiopia, studies show that between 33% and 69% of health care professionals use HMIS data as part of their jobs.^13–15 This practice appears to differ by region. Bogale found that 64% of hospitals in Southwest Shoa Zone, Oromia, analyzed their HMIS data on a monthly or quarterly basis and 56% reported making decisions based on these data during performance review meetings.¹⁶ However, Endriyas et al. found that <10% of health facilities in the Southern Nations Nationalities and People’s Region prepared action plans after reviewing performance using their HMIS data.¹⁷

Recognizing its significance, previous research in low-and-middle-income countries (LMICs) has investigated ways to bridge HMIS data collection and use. A study that assessed HMIS implementation and use in five sub-Saharan African countries (Ghana, Mozambique, Rwanda, Tanzania, and Zambia) found that while specific strategies to link data with decision-making varied by country, successful linkages shared common features. These included engaging frontline staff in data management and reviews, enhancing feedback mechanisms from facility and district health managers, and using the data to design interventions that are in line with national strategies.¹¹ A recent systematic review further identified factors such as the actual availability and quality of health facility data, having dedicated human resources and financial autonomy to set budget and allocate funds at the facility-level to be significantly associated with local decision-making using HMIS data in LMICs.¹⁸ Studies in Ethiopia have similar findings, showing that lack of training, shortage of designated staff, low computer literacy, and electricity or internet interruptions were major barriers to HMIS data use for decision-making.^8,19 Conversely, supportive supervision from facility leadership, competency in data interpretation, and favorable staff attitudes towards data use were associated with greater use of HMIS data at the facility-level.^13,20 While providing insights into factors associated with HMIS data use, these studies were not representative of entire regions or the country. Furthermore, to our knowledge, no LMIC study to date has employed a standardized approach for assessing HMIS data use in health facilities.

Since being declared in March 2020, the COVID-19 pandemic has posed unprecedented challenges to Ethiopia’s health system, including service disruptions, commodity shortages, and health care professional burnout and turnover.^21,22 It is during critical times like this pandemic that the value of HMIS data can be most prominent. Indeed, in Ethiopia, utilization of HMIS data to monitor trends in essential health services during the COVID-19 pandemic was a priority task set by the FMoH.²³ The availability of health facility data from Ethiopia collected 6 months before and 8 months after the declaration of the COVID-19 pandemic by the World Health Organization led to our conducting this study to: 1) develop an index to measure the utilization of HMIS data at the facility level, and 2) assess changes in HMIS data use before and after the COVID-19 pandemic.

METHODS

Study setting

The Ethiopian health service delivery is structured into a three-tier system: primary, secondary, and tertiary levels of care. Facilities providing primary-level health care consist of health posts, health centers, and primary hospitals. Staffed by health extension workers, health posts provide a variety of preventative and health promotion services such as family planning, antenatal care, postnatal care, and child immunization. In addition to services offered by health posts, health centers provide curative services, including comprehensive abortion care, skilled delivery care, and management of newborn illnesses. Primary hospitals offer inpatient and ambulatory services in addition to an expansion of health center maternal health service provision, including emergency surgical services (e.g., cesarean sections) and blood transfusions. The second tier is comprised of general hospitals, which receive referrals from primary hospitals, while specialized hospitals comprise the tertiary level of care in the country. The latter provide more advanced care and receive referrals from general hospitals.²⁴

Figure 1 shows how information flows within Ethiopia’s HMIS hierarchy, by facility type. In health posts, health extension workers collect data using, largely, a paper-based system and send the data to their designated health center once a month. Health centers compile these data and submit them to the woreda (district) health office where digitized data from health centers and health posts are compiled and sent to the zonal health department, then the corresponding regional health bureau, and finally to FMoH. Primary and general hospitals may send their data directly to the woreda health office, zonal health department, or regional health bureau digitally. Specialized hospitals submit digital data directly to the FMoH.²⁵

Figure 1.Hierarchy of health management information system reporting in Ethiopia

Study design and procedures

This study uses two rounds of cross-sectional data collected from a sample of public service delivery points (SDPs) in all regions and two city administrations in Ethiopia in 2019 (September-December) and 2020 (October-December), respectively. Eligible SDPs included all public hospitals, health centers, and health posts whose catchment areas cover a sampled enumeration area (EA) selected for a linked population-based household survey. Detailed information on sampling, including sample size calculations, is provided elsewhere.²⁶ This study was implemented as part of Performance Monitoring for Action (PMA) Ethiopia, a project implemented by the Addis Ababa University School of Public Health and the Johns Hopkins Bloomberg School of Public Health. All SDPs that completed the 2019 survey were eligible for follow-up in 2020. Of note, for the 2020 survey, security concerns and conflict made data collection in Tigray infeasible. After consultation with the FMoH, PMA Ethiopia ended all data collection in Tigray, resulting in a total of 95 eligible SDPs not being followed up in 2020. Of the 339 public hospitals and health centers eligible for follow-up in 2020 (after excluding those in Tigray), 316 completed the 2020 SDP survey, yielding a response rate of 93.2%. Our analytic sample is comprised of 305 public hospitals and health centers that completed the SDP survey in both years and reported having a functional HMIS. Health posts and health clinics were excluded since they have no independent HMIS.

Trained interviewers obtained oral consent to conduct interviews with the head of the facility or designated authority, and collected data using a standardized, publicly available questionnaire using smartphones.²⁶ The survey collected information on key facility readiness indicators, including provision of reproductive, maternal, and newborn care services and outcomes, HMIS capacity and use, decision-making by performance monitoring teams, and referral networks.

Measurement

We identified 14 questions from the SDP survey that provided HMIS-relevant information and could be used to construct an HMIS data use index. To assess content validity, items identified were mapped using the Performance of Routine Information System Management (PRISM) framework,²⁷ reflecting important aspects of HMIS information use: feedback to health facilities, quality of feedback, and discussion of HMIS data during performance monitoring and review meetings in facilities. Using this item mapping, our team consulted with five additional Ethiopian experts, including two experts at the FMoH, and selected 10 of the question items for inclusion in the index. We excluded two items that was largely universal (>95%) in health facilities (i.e., producing HMIS report monthly or more often, and having a performance monitoring team); and two items that provided closely related information (i.e., reporting data for the Maternal and Perinatal Death Surveillance Response and reviewing maternal deaths monthly or more often) to a third item selected for inclusion (i.e., review maternal deaths) because it was more likely to vary by facility level. Each item was given a score of either 0 for ‘no’ (i.e., no, facility did not receive feedback) or 1 for ‘yes’ (i.e., yes, facility received feedback), and item scores were added together to generate a summary index score, with a minimum value of 0 and maximum value of 10.

Statistical analysis

We first employed the Kaiser-Meyer-Olkin (KMO) test to determine the suitability of this data for factor analysis. Then, we used the 2019 survey data and performed exploratory factor analysis (EFA) to identify underlying constructs of the index and assess the internal consistency of the items included with Cronbach’s alpha. An index was created based on EFA and expert guidance. We examined differences in mean index scores by key facility characteristics (facility type, region, provision of basic and comprehensive emergency obstetric care) using one-way ANOVA. We employed t-tests to compare the mean index score by the same key facility characteristics within and between 2019 and 2020. To confirm our results comparing 2019 and 2020 data, we conducted a sensitivity analysis and fitted four different types of regression models: fixed-effect linear regression, fixed-effect logistic regression, random-effect logistic regression, and logistic regression with robust variance estimation. Significant findings were considered at the P<0.05 cutoff, although P<0.1 results were also shown given the relatively small sample used. All analyses were conducted in Stata version 17.

RESULTS

Sample characteristics

Our sample included 39% hospitals and 61% health centers (Table 1). The three largest regions of the country (Amhara, Oromia, and SNNP) contributed 72.8% of all facilities in the sample. Approximately four in ten (43.3%) facilities performed all basic emergency obstetric and newborn care (BEmONC) signal functions, while three in ten (30.8%) provided all comprehensive emergency obstetric and newborn care functions (CEmONC) in the past 3 months before the 2019 survey.

Index development

Figure 2 presents the distributions of all 14 HMIS measures and the 10 HMIS data use items included in the index. The KMO measure of sampling adequacy for the 2019 data was 0.798, indicating that the correlation matrix was appropriate for factor analysis and index development. Using the number of eigenvalues greater than one in the scree plot criterion for item reduction, the EFA indicated a one-factor solution for the index (Online Supplementary Document, Figure S1). The 10 items in index achieved a Cronbach’s alpha of 0.74 in 2019 data, indicative of acceptable internal reliability for the measure; the Cronbach’s alpha for the index in 2020 data was 0.76. Indicative of content validity, items reflect important components of HMIS information use included in the PRISM framework: feedback to health facilities, discussion, and decision-making forums and processes. While our index does not capture all technical, behavioral, and organizational determinants of HMIS performance identified in the PRISM framework, the index meets its intended purpose as a short measurement tool focused on HMIS data use and relevant to the local context. Expert review confirmed the acceptability of items as pertains to national priorities and initiatives to promote HMIS data use in Ethiopia.

Figure 2.Health management information system (HMIS) data use index items and difference in HMIS functionality and utilization between 2020 and 2019, among Ethiopian government hospitals and health centers (N=305)

†Items included in the index; For health centers, receiving feedback on HMIS report from administrative health offices refers to feedback from woreda health offices; for hospitals, feedback from administrative health offices include any feedback from woreda health office, zonal health department, regional health bureau, and/or FMoH. The distribution of all types of feedback received is shown here but only the composite variable describing any feedback received is included in the index.
HMIS, health management information system; MPDSR, Maternal and Perinatal Death Surveillance Response
*P<0.1, **P<0.05, ***P<0.01

Changes in HMIS utilization pre- and post-COVID19

As expected from functional HMIS during the COVID19 pandemic, there were significant differences in HMIS utilization between 2019 and 2020 (Figure 2). The proportion of facilities that received feedback from facility leadership and zonal health department increased by 19.3% and 8.2%, respectively, during this period, although no differences were seen in overall feedback from administrative health offices. Improvements were observed on facilities’ benefiting from action-oriented feedback based on HMIS report where 12.1% and 7.5% more facilities received recommendations on resource allocation and examining performance target and actual performance, respectively, in 2020 versus 2019. Also, 15.1% more facilities reported reviewing maternal deaths in 2020 than 2019. However, the proportion of facilities that held participatory performance review meetings monthly or more often decreased significantly by 13.8% during the same time. Results from sensitivity analyses showed consistent results (Online Supplementary Document, Table S1) irrespective of methodology used to examine differences in HMIS functionality and data use.

The overall mean index score increased by 0.7 (95% CI: 0.3-1.0) on the 0-10 range between 2019 and 2020 (Figure 3). Significant improvements at p<0.05 or better were seen in subgroups by facility characteristics, with facilities in Oromia (mean score difference 1.2; 95% CI: 0.7-1.7) and those providing CEmONC functions (mean score difference 1.0; 95% CI: 0.4-1.6) having the largest improvements. These findings are consistent with results from our sensitivity analysis using linear regression modeling to test for changes in mean index scores between 2019 and 2020 (Online Supplementary Document, Table S2).

Figure 3.Comparison of health management information system data use index score by facility characteristics: 2019 versus 2020 (N=305)

DISCUSSION

To our knowledge, this is the first study to develop a standardized instrument to measure facility-level HMIS data use. The 10-item HMIS data use index exhibited good internal consistency and content validity. We also document an expected and desired increase, albeit small, in HMIS data use during the COVID-19 pandemic in Ethiopia in 2020 compared to 2019. With COVID-19 serving as a “natural experiment” demonstrating the utility of HMIS data, our study finds significant improvements in several aspects of HMIS data use at the facility level, with only the timeliness of participatory review meetings appeared to be negatively affected by the COVID-19 pandemic, which we discuss below.

The value of greater use of HMIS data for policy decisions, especially the use of HMIS data elements such as the absolute number of clients served as opposed to coverage indicators, has been cited in prior research.²⁸ During the COVID-19 pandemic, HMIS data were greatly needed to not only track COVID-19 cases but to monitor facility readiness and availability of essential health services and to inform emergent health system needs.²⁹ Our proposed HMIS data use index covers such key areas in data use, with good utilization indicated by receiving feedback on facilities’ HMIS report; receiving actionable recommendations on specific quality improvement areas such as resource allocation, performance examination and review of personnel responsibilities; using HMIS data during performance review meetings; and review of maternal deaths, which are rare events clinically and statistically and an accepted marker of a country’s population health, health inequalities, and health system functioning.³⁰

The overall positive trends in HMIS data utilization we found in Ethiopia can be attributed to the various national and regional initiatives to promote use of these data. Most recently, in 2020, an integrated data use, data quality, and DHIS-2 training was provided to health care providers at all levels (personal communication with O. Esmale). It was intended for the training to improve data ownership by program staff and the use of these data for decisions. Previously, the connected woreda strategy (CWS), an assessment tool that evaluates woredas on HMIS capacity, data quality, and information use was introduced by the FMoH in the five-year HSTP.⁵ Its implementation in over 400 woredas in Amhara, Oromia, SNNP, and Tigray showed significant increases in data use and the proportion of “model woredas” – i.e., woredas that meet the highest standard of HMIS performance.^31,32 Other initiatives such as the Capacity-Building and Mentorship Partnership Program presented new opportunities to improve data use through enhancing health workers’ data-use competencies, supporting the implementation of DHIS-2, and providing guidance on data triangulation and evidence generation.^33,34 We also expected the COVID-19 pandemic to improve facilities’ utilization of HMIS data. Since the early stages of the pandemic, the government of Ethiopia made strong leadership commitments to sustaining essential health services while swiftly enacting several pandemic response strategies.^23,35,36 The National Comprehensive COVID-19 Management Handbook advised that all health facilities collaborate with political leadership for resource allocation guidance for COVID-19 response.³⁷ Furthermore, at the beginning of the pandemic, a command post was organized to monitor the status of essential service provision and uptake at regional and central levels, with a specially designed tool to monitor these services on a weekly basis.²⁵ These efforts likely promoted HMIS data use and encouraged facility leadership to make actionable recommendations, as reflected in our findings.

However, we found a decline in the timeliness of participatory performance review meetings. Participatory performance reviews are meetings where facilities meet monthly with administrative health offices and other stakeholders to monitor performance and ensure implementation of proper strategies that are in line with the woreda-based and national Health Sector Development Plan.³⁸ There are several ways in which COVID-19 could have contributed to this decline. First, during early phases of the pandemic, health facilities adopted various work from home and rotation arrangements. Health workers were also, at times, affected by movement restrictions, hindering the possibility of in-person performance review gatherings. The option of remote performance review was challenging for facilities with internet connectivity issues. Second, there was likely an increase and divergence of health care worker responsibilities caused by COVID-19. According to the National Comprehensive COVID-19 Management Handbook, all health facilities were required to establish a multidisciplinary planning committee to specifically address COVID-19 preparedness planning.³⁷ Over 90% of health facilities in our study reported having such teams by the time of the 2020 interview. The Handbook also advised shifting of staff responsibilities (e.g., suspending certain administrative activities, having supervisory staff work clinically) to accommodate surging clinical demands. Third, COVID-19 caused fears among health care workers just like it did among the public.^39,40 Fears of going to a health facility, seeing patients, and handling equipment such as recording devices have all likely contributed to the disruptions in performance review sessions.

Although not statistically significant, our finding a decline in receiving woreda-level feedback is potentially concerning. Woreda health offices play a critical role in creating alignment in planning, budgeting, monitoring, and reporting within the Ethiopian tier-based health system.⁴¹ Using the “top-down, bottom-up” approach, woreda offices disseminate federal and regional-level indicator plans to health facilities while reflecting grass-root level issues to the national level. Woreda-based planning – a planning process where various stakeholders convene to make evidence-based decisions using HMIS data – is now the formal planning process in most regions.⁴² As with performance reviews, the importance of sustaining and strengthening HMIS reporting and feedback system was brought to light during COVID-19, as these functions are critical to ensuring the health facilities have adequate support, resources, and guidance to carry out regular services in addition to emergency pandemic response activities.

Our study has several strengths, but also limitations. First, we used a national sample of health facilities in Ethiopia to develop a facility-level HMIS data use measure. Second, using COVID-19 as a “natural experiment”, our study allowed us to not only validate its construct validity but to also assess the potential effects of the COVID-19 pandemic on Ethiopian health facilities’ HMIS performance and to provide direction for future HMIS strengthening policies. However, since health facilities were selected based on population distribution and catchment area, larger-level facilities such as hospitals were oversampled. Thus, our sample does not reflect Ethiopia’s health facility composition at the national level, but findings are approximately representative within each facility type (e.g., hospital, health center). The survey focus was on reproductive, maternal and child health, and while it covered key aspects of HMIS data use, issues like data display, data use for performance evaluation and planning, or the actual implementation of received recommendations were not assessed. Neither did the survey measure the organizational (e.g., information culture, structure) and behavioral (e.g., knowledge, skills, motivation) determinants that influence HMIS data use at the facility level.⁴³ In sum, while recognizing that our data were not specifically collected to develop this index, our approach provides a meaningful starting point for further refinement and validation of an HMIS data use measure at the facility-level, which will be highly valuable in strengthening HMIS capacity in Ethiopia and other LMICs.

Our findings call for several recommendations to promote greater use of HMIS data. First, we recommend strengthening HMIS data use training, mentorship and supervision, with special attention paid to engaging facility leadership in such activities. Despite significant improvement, our data shows that about four in ten health facilities do not receive feedback from facility leadership. As previously suggested in the literature, lack of feedback disincentivizes data use and hinders data quality.^44–46 Thus, addressing this barrier is essential to reinforcing the positive feedback mechanism between data quality, data use, and evidence-based decision-making.^6,44,47 Moreover, in addition to providing technical assistance, future HMIS training should also focus on creating accountability systems (e.g., including data use as a performance measure, recognizing data champions, etc.), and incorporating progress monitoring following the training.⁴⁸ Second, there is an ongoing need to strengthen the HMIS workforce. Our results suggest that human resource shortages may have played a major role in the disruption in performance reviews during COVID-19. Performance monitoring and review – a process that strengthens district health systems through establishing clinical and administrative standards and systematic managing of performance – is a key element highlighted in the HSTP to realize sector-wide reforms.^5,49 It is critical to ensure sufficient clinical, administrative, and designated HMIS staff to cover all responsibilities along the HMIS reporting and feedback pathway, and that HMIS activities do not compete with clinical duties. Third, we recommend implementing user-centered approaches to cultivate the culture of data use within health facilities and administrative health offices. Data use should be driven by the value of using HMIS data for local decision-making rather than a mandatory requirement to report to higher levels. The key to using a user-driven approach is involving HMIS users beyond data generation (i.e., engaging HMIS staff in data analysis and decision-making processes), making health workers and program staff embrace their facility data and strive towards improving its quality and translating data into action.^15,50

CONCLUSIONS

The facility-level use of HMIS data for ongoing monitoring of health programs, resource allocation, and quality improvement is essential to improving health service effectiveness in Ethiopia and like settings. We propose an instrument to measure HMIS data use in health facilities and encourage future research to refine and validate this tool in order to promote greater data use via strengthened training, dedicated HMIS workforce, and a culture of data use.

Acknowledgements

PMA Ethiopia project relies on the work of many individuals. We thank the special contributions of Ayanaw Amogne, Selam Desta, Solomon Emyu Ferede, Bedilu Ejigu, Celia Karp, Ann Rogers, Sally Safi, Aisha Siewe, Shannon Wood, Mahari Yihdego, and Addisalem Zebene. Finally, we express gratitude to the Ethiopia country team, resident enumerators, and survey participants who are ultimately responsible for the success of PMA Ethiopia.

Ethics statement

This study was approved by the Addis Ababa University (075/13/SPH) and Johns Hopkins Bloomberg School of Public Health (FWA00000287) Institutional Review Boards.

Data availability

The data can be requested at https://www.pmadata.org/data/request-access-datasets

Funding

This study was supported by the Bill & Melinda Gates Foundation (INV 009466). Under the grant conditions of the Foundation, a Creative Commons Attribution 4.0 Generic License has already been assigned to the author-accepted manuscript version that might arise from this submission.

Authorship contributions

JQ and AAC developed the research aims. JQ performed data analysis and drafted the initial manuscript. AAC supervised this work and provided guidance on study construct and interpretation. SS, AS, and LZ were the Principal Investigators of PMA Ethiopia and oversaw the project implementation. OEE, WD, SS, AS, and ES provided critical input on background and HMIS index development. All authors participated in writing and approving the final manuscript.

Disclosure of interest

The authors completed the Unified Competing Interest form available upon request from the corresponding author, and declare no conflicts of interest."

Additional material

This article contains additional information as an Online Supplementary Document.

Correspondence to:

Jiage Qian
Johns Hopkins University
3400 N. Charles Street
Baltimore, MD 21218, USA
Email: [email protected]