Data for local decision-making, not a mere reporting requirement: development of an index to measure facility-level use of HMIS data

Well-functioning health management information systems (HMIS) enable decision-making at all health system levels. This study develops an index to measure the use of HMIS data at the facility level


Background
Well-functioning health management information systems (HMIS) enable decision-making at all health system levels.This study develops an index to measure the use of HMIS data at the facility level.

Methods
We used two rounds of cross-sectional data collected from 305 health facilities in Ethiopia in 2019 (pre-COVID-19) and 2020 (post-COVID-19).We constructed a summative, 10-item index using exploratory factor analysis and 2019 index development data; and used Cronbach's alpha to assess reliability.To examine content validity, we mapped items against a previously published conceptual framework and consulted Ethiopian experts.We then employed one-way ANOVA and t-tests comparing the mean index scores overall and by key facility characteristics between 2019 and 2020.

Results
The 10-item index loaded on one factor (Cronbach's alpha=0.74),and the index scores did not differ significantly by facility characteristics in 2019.The mean index score increased from 7.2 in 2019 to 7.9 in 2020 (P<0.01).During this period, more facilities received feedback on HMIS reports from facility leadership (19.3% difference); received actionable recommendations on performance targets and resource allocation (7.5% and 12.3% difference, respectively); and reviewed maternal deaths (15.1% difference); conversely, the proportion of facilities that held participatory performance review meetings monthly or more often decreased by 13.8% (all P <0.05).

Conclusions
We propose a facility-level HMIS data use index and document an upward trend in HMIS data use in Ethiopia immediately after the COVID-19 pandemic was declared.Future research should further evaluate and refine the proposed index to support the measurement of HMIS data quality and utilization in Ethiopia and like settings.
2][3] By producing reliable and timely health data, a well-functioning HMIS enables decision-makers at all levels of the health system to identify needs, make evidence-based decisions, and optimally allocate resources -all of which are essential elements in achieving large-scale health improvements. 4The Ethiopian Federal Ministry of Health (FMoH) is highly invested in strengthening the country's HMIS. 5 By the end of 2013, 98% of public hospitals and 87% of health centers in Ethiopia had already adopted HMIS. 5 The first Health Sector Transformation Plan (HSTP 2015-2020) called for an "information revolution" in the way that health data are collected, analyzed, and disseminated to deliver quality and equitable health services. 57][8][9] The District Health Information Software (DHIS-2) -a webbased data capturing and analyzing platform -was introduced in 2018, in parallel with a nationwide campaign to improve internet connectivity in health facilities. 1 The most recent data available showed that >95% of health facilities employed either the online or offline version of DHIS-2.The reporting rate using DHIS-2 varied by region and ranged from about 80% for disease reporting to 90% for health service reporting. 10t has been well established that information management and use serve as the basis for evidence-based decisions in health facilities. 11These decision-making processes include priority setting, annual health planning and budgeting, health resource allocation, and strategies to promote greater health service utilization.Data-driven health decisions contribute to a culture of transparency and accountability, as the allocation of human resources, finances, and commodities are based on evidence of needs. 124][15] This practice appears to differ by region.Bogale found that 64% of hospitals in Southwest Shoa Zone, Oromia, analyzed their HMIS data on a monthly or quarterly basis and 56% reported making decisions based on these data during performance review meetings. 16However, Endriyas et al. found that <10% of health facilities in the Southern Nations Nationalities and People's Region prepared action plans after reviewing performance using their HMIS data. 17ecognizing its significance, previous research in lowand-middle-income countries (LMICs) has investigated ways to bridge HMIS data collection and use.A study that assessed HMIS implementation and use in five sub-Saharan African countries (Ghana, Mozambique, Rwanda, Tanzania, and Zambia) found that while specific strategies to link data with decision-making varied by country, successful linkages shared common features.These included engaging frontline staff in data management and reviews, enhancing feedback mechanisms from facility and district health managers, and using the data to design interventions that are in line with national strategies. 11A recent systematic review further identified factors such as the actual availability and quality of health facility data, having dedicated human resources and financial autonomy to set budget and allocate funds at the facility-level to be significantly associated with local decision-making using HMIS data in LMICs. 18tudies in Ethiopia have similar findings, showing that lack of training, shortage of designated staff, low computer literacy, and electricity or internet interruptions were major barriers to HMIS data use for decision-making. 8,19Conversely, supportive supervision from facility leadership, competency in data interpretation, and favorable staff attitudes towards data use were associated with greater use of HMIS data at the facility-level. 13,20While providing insights into factors associated with HMIS data use, these studies were not representative of entire regions or the country.Furthermore, to our knowledge, no LMIC study to date has employed a standardized approach for assessing HMIS data use in health facilities.
Since being declared in March 2020, the COVID-19 pandemic has posed unprecedented challenges to Ethiopia's health system, including service disruptions, commodity shortages, and health care professional burnout and turnover. 21,22It is during critical times like this pandemic that the value of HMIS data can be most prominent.Indeed, in Ethiopia, utilization of HMIS data to monitor trends in essential health services during the COVID-19 pandemic was a priority task set by the FMoH. 23The availability of health facility data from Ethiopia collected 6 months before and 8 months after the declaration of the COVID-19 pandemic by the World Health Organization led to our conducting this study to: 1) develop an index to measure the utilization of HMIS data at the facility level, and 2) assess changes in HMIS data use before and after the COVID-19 pandemic.

STUDY SETTING
The Ethiopian health service delivery is structured into a three-tier system: primary, secondary, and tertiary levels of care.Facilities providing primary-level health care consist of health posts, health centers, and primary hospitals.Staffed by health extension workers, health posts provide a variety of preventative and health promotion services such as family planning, antenatal care, postnatal care, and child immunization.In addition to services offered by health posts, health centers provide curative services, including comprehensive abortion care, skilled delivery care, and management of newborn illnesses.Primary hospitals offer inpatient and ambulatory services in addition to an expansion of health center maternal health service provision, including emergency surgical services (e.g., cesarean sections) and blood transfusions.The second tier is comprised of general hospitals, which receive referrals from primary hospitals, while specialized hospitals comprise the tertiary level of care in the country.The latter provide more advanced care and receive referrals from general hospitals. 24igure 1 shows how information flows within Ethiopia's HMIS hierarchy, by facility type.In health posts, health extension workers collect data using, largely, a paper-based system and send the data to their designated health center once a month.Health centers compile these data and submit them to the woreda (district) health office where digitized data from health centers and health posts are compiled and sent to the zonal health department, then the corresponding regional health bureau, and finally to FMoH.Primary and general hospitals may send their data directly to the woreda health office, zonal health department, or regional health bureau digitally.Specialized hospitals submit digital data directly to the FMoH. 25

STUDY DESIGN AND PROCEDURES
This study uses two rounds of cross-sectional data collected from a sample of public service delivery points (SDPs) in all regions and two city administrations in Ethiopia in 2019 (September-December) and 2020 (October-December), respectively.Eligible SDPs included all public hospitals, health centers, and health posts whose catchment areas cover a sampled enumeration area (EA) selected for a linked population-based household survey.Detailed information on sampling, including sample size calculations, is pro-Data for local decision-making, not a mere reporting requirement: development of an index to measure facility-level use of...

Figure 1. Hierarchy of health management information system reporting in Ethiopia
vided elsewhere. 26This study was implemented as part of Performance Monitoring for Action (PMA) Ethiopia, a project implemented by the Addis Ababa University School of Public Health and the Johns Hopkins Bloomberg School of Public Health.All SDPs that completed the 2019 survey were eligible for follow-up in 2020.Of note, for the 2020 survey, security concerns and conflict made data collection in Tigray infeasible.After consultation with the FMoH, PMA Ethiopia ended all data collection in Tigray, resulting in a total of 95 eligible SDPs not being followed up in 2020.Of the 339 public hospitals and health centers eligible for follow-up in 2020 (after excluding those in Tigray), 316 completed the 2020 SDP survey, yielding a response rate of 93.2%.Our analytic sample is comprised of 305 public hospitals and health centers that completed the SDP survey in both years and reported having a functional HMIS.Health posts and health clinics were excluded since they have no independent HMIS.
Trained interviewers obtained oral consent to conduct interviews with the head of the facility or designated authority, and collected data using a standardized, publicly available questionnaire using smartphones. 26The survey collected information on key facility readiness indicators, including provision of reproductive, maternal, and newborn care services and outcomes, HMIS capacity and use, decision-making by performance monitoring teams, and referral networks.

MEASUREMENT
We identified 14 questions from the SDP survey that provided HMIS-relevant information and could be used to construct an HMIS data use index.To assess content validity, items identified were mapped using the Performance of Routine Information System Management (PRISM) framework, 27 reflecting important aspects of HMIS information use: feedback to health facilities, quality of feedback, and discussion of HMIS data during performance monitoring and review meetings in facilities.Using this item mapping, our team consulted with five additional Ethiopian experts, including two experts at the FMoH, and selected 10 of the question items for inclusion in the index.We excluded two items that was largely universal (>95%) in health facilities (i.e., producing HMIS report monthly or more often, and having a performance monitoring team); and two items that provided closely related information (i.e., reporting data for the Maternal and Perinatal Death Surveillance Response and reviewing maternal deaths monthly or more often) to a third item selected for inclusion (i.e., review maternal deaths) because it was more likely to vary by facility level.Each item was given a score of either 0 for 'no' (i.e., no, facility did not receive feedback) or 1 for 'yes' (i.e., yes, facility received feedback), and item scores were added together to generate a summary index score, with a minimum value of 0 and maximum value of 10.

STATISTICAL ANALYSIS
We first employed the Kaiser-Meyer-Olkin (KMO) test to determine the suitability of this data for factor analysis.Then, we used the 2019 survey data and performed exploratory factor analysis (EFA) to identify underlying constructs of the index and assess the internal consistency of the items included with Cronbach's alpha.An index was created based on EFA and expert guidance.We examined differences in mean index scores by key facility characteristics (facility type, region, provision of basic and comprehensive emergency obstetric care) using one-way ANOVA.We employed t-tests to compare the mean index score by the same key facility characteristics within and between 2019 and 2020.To confirm our results comparing 2019 and 2020 data, we conducted a sensitivity analysis and fitted four different types of regression models: fixed-effect linear regression, fixed-effect logistic regression, random-effect logistic regression, and logistic regression with robust variance estimation.Significant findings were considered at the P<0.05 cutoff, although P<0.1 results were also shown given the relatively small sample used.All analyses were conducted in Stata version 17.

SAMPLE CHARACTERISTICS
Our sample included 39% hospitals and 61% health centers (Table 1).The three largest regions of the country (Amhara, Oromia, and SNNP) contributed 72.8% of all facilities in the sample.Approximately four in ten (43.3%) facilities performed all basic emergency obstetric and newborn care (BEmONC) signal functions, while three in ten (30.8%) provided all comprehensive emergency obstetric and newborn Data for local decision-making, not a mere reporting requirement: development of an index to measure facility-level use of... care functions (CEmONC) in the past 3 months before the 2019 survey.

INDEX DEVELOPMENT
Figure 2 presents the distributions of all 14 HMIS measures and the 10 HMIS data use items included in the index.The KMO measure of sampling adequacy for the 2019 data was 0.798, indicating that the correlation matrix was appropriate for factor analysis and index development.Using the number of eigenvalues greater than one in the scree plot criterion for item reduction, the EFA indicated a one-factor solution for the index (Online Supplementary Document, Figure S1).The 10 items in index achieved a Cronbach's alpha of 0.74 in 2019 data, indicative of acceptable internal reliability for the measure; the Cronbach's alpha for the index in 2020 data was 0.76.Indicative of content validity, items reflect important components of HMIS information use included in the PRISM framework: feedback to health facilities, discussion, and decision-making forums and processes.While our index does not capture all tech-nical, behavioral, and organizational determinants of HMIS performance identified in the PRISM framework, the index meets its intended purpose as a short measurement tool focused on HMIS data use and relevant to the local context.Expert review confirmed the acceptability of items as pertains to national priorities and initiatives to promote HMIS data use in Ethiopia.

CHANGES IN HMIS UTILIZATION PRE-AND POST-COVID19
As expected from functional HMIS during the COVID19 pandemic, there were significant differences in HMIS utilization between 2019 and 2020 (Figure 2).The proportion of facilities that received feedback from facility leadership and zonal health department increased by 19.3% and 8.2%, respectively, during this period, although no differences were seen in overall feedback from administrative health offices.Improvements were observed on facilities' benefiting from action-oriented feedback based on HMIS report where 12.1% and 7.5% more facilities received recommendations on resource allocation and examining performance target and actual performance, respectively, in 2020 versus 2019.Also, 15.1% more facilities reported reviewing maternal deaths in 2020 than 2019.However, the proportion of facilities that held participatory performance review meetings monthly or more often decreased significantly by 13.8% during the same time.Results from sensitivity analyses showed consistent results (Online Supplementary Document, Table S1) irrespective of methodology used to examine differences in HMIS functionality and data use.
The overall mean index score increased by 0.7 (95% CI: 0.3-1.0) on the 0-10 range between 2019 and 2020 (Figure 3).Significant improvements at p<0.05 or better were seen in subgroups by facility characteristics, with facilities in Oromia (mean score difference 1.2; 95% CI: 0.7-1.7)and those providing CEmONC functions (mean score difference 1.0; 95% CI: 0.4-1.6)having the largest improvements.These findings are consistent with results from our sensitivity analysis using linear regression modeling to test for changes in mean index scores between 2019 and 2020 (Online Supplementary Document, Table S2).

DISCUSSION
To our knowledge, this is the first study to develop a standardized instrument to measure facility-level HMIS data use.The 10-item HMIS data use index exhibited good internal consistency and content validity.We also document an expected and desired increase, albeit small, in HMIS data use during the COVID-19 pandemic in Ethiopia in 2020 compared to 2019.With COVID-19 serving as a "natural experiment" demonstrating the utility of HMIS data, our study finds significant improvements in several aspects of HMIS data use at the facility level, with only the timeliness of participatory review meetings appeared to be negatively affected by the COVID-19 pandemic, which we discuss below.
Data for local decision-making, not a mere reporting requirement: development of an index to measure facility-level use of...The value of greater use of HMIS data for policy decisions, especially the use of HMIS data elements such as the absolute number of clients served as opposed to coverage indicators, has been cited in prior research. 28During the COVID-19 pandemic, HMIS data were greatly needed to not only track COVID-19 cases but to monitor facility readiness and availability of essential health services and to inform emergent health system needs. 29Our proposed HMIS data use index covers such key areas in data use, with good utilization indicated by receiving feedback on facilities' HMIS report; receiving actionable recommendations on specific quality improvement areas such as resource allocation, performance examination and review of personnel responsibilities; using HMIS data during performance review meetings; and review of maternal deaths, which are rare events clinically and statistically and an accepted marker of a country's population health, health inequalities, and health system functioning. 30he overall positive trends in HMIS data utilization we found in Ethiopia can be attributed to the various national and regional initiatives to promote use of these data.Most recently, in 2020, an integrated data use, data quality, and DHIS-2 training was provided to health care providers at all levels (personal communication with O. Esmale).It was intended for the training to improve data ownership by program staff and the use of these data for decisions.Previously, the connected woreda strategy (CWS), an assessment tool that evaluates woredas on HMIS capacity, data quality, and information use was introduced by the FMoH in the five-year HSTP. 5 Its implementation in over 400 woredas in Amhara, Oromia, SNNP, and Tigray showed significant increases in data use and the proportion of "model woredas" -i.e., woredas that meet the highest standard of HMIS performance. 31,32Other initiatives such as the Capacity-Building and Mentorship Partnership Program presented new opportunities to improve data use through enhancing health workers' data-use competencies, supporting the implementation of DHIS-2, and providing guidance on data triangulation and evidence generation. 33,34We also expected the COVID-19 pandemic to improve facilities' utilization of HMIS data.Since the early stages of the pandemic, the government of Ethiopia made strong leadership commitments to sustaining essential health services while swiftly enacting several pandemic response strategies. 23,35,36 Te National Comprehensive COVID-19 Management Handbook advised that all health facilities collaborate with political leadership for resource allocation guidance for COVID-19 response. 37Furthermore, at the beginning of the pandemic, a command post was organized to monitor the status of essential service provision and uptake at regional and central levels, with a specially designed tool to monitor these services on a weekly basis. 25These efforts likely promoted HMIS data use and encouraged facility leadership to make actionable recommendations, as reflected in our findings.

Journal of Global Health Reports
However, we found a decline in the timeliness of participatory performance review meetings.Participatory performance reviews are meetings where facilities meet monthly with administrative health offices and other stakeholders to monitor performance and ensure implementation of proper strategies that are in line with the woreda-based and national Health Sector Development Plan. 38 There are several ways in which COVID-19 could have contributed to this decline.First, during early phases of the pandemic, health facilities adopted various work from home and rotation arrangements.Health workers were also, at times, affected by movement restrictions, hindering the possibility of inperson performance review gatherings.The option of remote performance review was challenging for facilities with internet connectivity issues.Second, there was likely an increase and divergence of health care worker responsibilities caused by COVID-19.According to the National Comprehensive COVID-19 Management Handbook, all health facilities were required to establish a multidisciplinary planning committee to specifically address COVID-19 preparedness planning. 37Over 90% of health facilities in our study reported having such teams by the time of the 2020 interview.The Handbook also advised shifting of staff responsibilities (e.g., suspending certain administrative activities, having supervisory staff work clinically) to accommodate surging clinical demands.Third, COVID-19 caused fears among health care workers just like it did among the public. 39,40ears of going to a health facility, seeing patients, and handling equipment such as recording devices have all likely contributed to the disruptions in performance review sessions.
Although not statistically significant, our finding a decline in receiving woreda-level feedback is potentially concerning.Woreda health offices play a critical role in creating alignment in planning, budgeting, monitoring, and reporting within the Ethiopian tier-based health system. 41sing the "top-down, bottom-up" approach, woreda offices disseminate federal and regional-level indicator plans to health facilities while reflecting grass-root level issues to the national level.Woreda-based planning -a planning process where various stakeholders convene to make evidence-based decisions using HMIS data -is now the formal planning process in most regions. 42As with performance reviews, the importance of sustaining and strengthening HMIS reporting and feedback system was brought to light during COVID-19, as these functions are critical to ensuring the health facilities have adequate support, resources, and guidance to carry out regular services in addition to emergency pandemic response activities.
Our study has several strengths, but also limitations.First, we used a national sample of health facilities in Ethiopia to develop a facility-level HMIS data use measure.Second, using COVID-19 as a "natural experiment", our study allowed us to not only validate its construct validity but to also assess the potential effects of the COVID-19 pandemic on Ethiopian health facilities' HMIS performance and to provide direction for future HMIS strengthening policies.However, since health facilities were selected based on population distribution and catchment area, larger-level facilities such as hospitals were oversampled.Thus, our sample does not reflect Ethiopia's health facility composition at the national level, but findings are approximately representative within each facility type (e.g., hos-Data for local decision-making, not a mere reporting requirement: development of an index to measure facility-level use of... pital, health center).The survey focus was on reproductive, maternal and child health, and while it covered key aspects of HMIS data use, issues like data display, data use for performance evaluation and planning, or the actual implementation of received recommendations were not assessed.Neither did the survey measure the organizational (e.g., information culture, structure) and behavioral (e.g., knowledge, skills, motivation) determinants that influence HMIS data use at the facility level. 43In sum, while recognizing that our data were not specifically collected to develop this index, our approach provides a meaningful starting point for further refinement and validation of an HMIS data use measure at the facility-level, which will be highly valuable in strengthening HMIS capacity in Ethiopia and other LMICs.
Our findings call for several recommendations to promote greater use of HMIS data.First, we recommend strengthening HMIS data use training, mentorship and supervision, with special attention paid to engaging facility leadership in such activities.Despite significant improvement, our data shows that about four in ten health facilities do not receive feedback from facility leadership.5][46] Thus, addressing this barrier is essential to reinforcing the positive feedback mechanism between data quality, data use, and evidence-based decision-making. 6,44,47Moreover, in addition to providing technical assistance, future HMIS training should also focus on creating accountability systems (e.g., including data use as a performance measure, recognizing data champions, etc.), and incorporating progress monitoring following the training. 48Second, there is an ongoing need to strengthen the HMIS workforce.Our results suggest that human resource shortages may have played a major role in the disruption in performance reviews during COVID-19.Performance monitoring and review -a process that strengthens district health systems through establishing clinical and administrative standards and systematic managing of performance -is a key element highlighted in the HSTP to realize sector-wide reforms. 5,49It is critical to ensure sufficient clinical, administrative, and designated HMIS staff to cover all responsibilities along the HMIS reporting and feedback pathway, and that HMIS activities do not compete with clinical duties.Third, we recommend implementing user-centered approaches to cultivate the culture of data use within health facilities and administrative health offices.Data use should be driven by the value of using HMIS data for local decision-making rather than a mandatory requirement to report to higher levels.The key to using a user-driven approach is involving HMIS users beyond data generation (i.e., engaging HMIS staff in data analysis and decision-making processes), making health workers and program staff embrace their facility data and strive towards improving its quality and translating data into action. 15,50

CONCLUSIONS
The facility-level use of HMIS data for ongoing monitoring of health programs, resource allocation, and quality improvement is essential to improving health service effectiveness in Ethiopia and like settings.We propose an instrument to measure HMIS data use in health facilities and encourage future research to refine and validate this tool in order to promote greater data use via strengthened training, dedicated HMIS workforce, and a culture of data use.

Figure 2 .Figure 3 .
Figure 2. Health management information system (HMIS) data use index items and difference in HMIS functionality and utilization between 2020 and 2019, among Ethiopian government hospitals and health centers (N=305) †Items included in the index; For health centers, receiving feedback on HMIS report from administrative health offices refers to feedback from woreda health offices; for hospitals, feedback from administrative health offices include any feedback from woreda health office, zonal health department, regional health bureau, and/or FMoH.The distribution of all types of feedback received is shown here but only the composite variable describing any feedback received is included in the index.HMIS, health management information system; MPDSR, Maternal and Perinatal Death Surveillance Response *P<0.1, **P<0.05,***P<0.01

Table 1 . Basic facility characteristics
Note: SNNP, Southern Nations Nationalities and People's Region; BEmONC: basic emergency obstetric and newborn care, including administration of parenteral antibiotics, parenteral anticonvulsants, parenteral uterotonic drugs, manual removal of the placenta, assisted vaginal deliveries, and neonatal recitation; CEmONC: comprehensive emergency obstetric and newborn care which includes all BEmONC functions with the addition of cesarean section delivery and blood transfusion.Assessment of both BE-mONC and CEmONC signal functions presented used 2019 data.