Monitoring, Learning and Evaluation of the Uttar Pradesh Technical Support Unit

Number of Units Surveyed: Baseline: 15,000 HHs, Midline: 28,000 HHs, Endline: 35,000 HHs
To strengthen results and to evaluate performance and outcomes of the TSU, an external Monitoring, Learning and Evaluation (MLE) led by Sambodhi will evaluate the extent to which the TSU achieves intended outcomes and support learning and programming adjustments. The TSU is working in 25 High Priority Districts in UP, with an objective of improving RMNCH+A outcomes. At the community level, key interventions of TSU involved providing support to ASHAs in provision of care to mothers during pregnancy as well as post child birth. Home-based newborn care also formed a key component of ASHAs activities supported by TSU. The objective of this external MLE effort is to support ongoing program steering and learning from evidence to strengthen results. The key objectives of MLE are:
• To monitor and evaluate the contribution of the TSU-RMNCH+A initiative using rigorous study designs and multiple data collection approaches;
• To build state capacity to undertake rigorous measurement and evaluation of RMNCH+A activities with a focus on vulnerable populations; and
• To facilitate knowledge sharing, including documentation and dissemination of best practices
• To strengthen results and monitor and evaluate the performance and outcomes of the  TSU, an embedded Augmented Monitoring, Learning and Evaluation (MLE+A) framework has been planned.

The following activities are being conducted by Sambodhi:
• Evaluation of the TSU: To assess the impact of TSU, Sambodhi will examine the attribution, contribution, and diffusion through a quasi-experimental design employing a mix method design. Baseline-midline-endline data is being collected from the primary target group (women having children in the age group of 0 – 23 months) on key RMNCH+A health outcomes. In addition, a survey of frontline workers (ANM, AWW,
ASHA) and health facilities is being conducted to capture service level statistics. Sambodhi will also conduct a cost-effective analysis (CEA) to provide insights into the cost-efficacy of TSU strategies.
• Process Monitoring and Process Evaluation: Process monitoring of TSU interventions provides feedback for implementation support and link outcome level changes with the process and output changes to understand the TSU story of change. Process maps have been developed for all the TSU processes at state, district and sub district level. Further, TSU monitoring data has been used to capture the intensity of the processes implemented on ground. A comprehensive index to measure the implementation strength of the TSU’s intervention has been developed. In addition, Sambodhi develops videos (stories for change) to document some of the changes at ground.
• Strengthening Data Systems: The TSU aims to strengthen the quality and use of data for informed decision-making to be effective for policy development and program management. Sambodhi undertakes the exercise of ensuring data quality at community and facility level. The data quality assurance includes validity checks, range checks and consistency checks on the primary data collected by the TSU. Additionally, we are also conducting analysis of the Health Management Information Systems (HMIS) data to assess the data quality. Since, the HMIS data is important for program planning and monitoring, its quality of data should be accurate. We track changes in HMIS data quality on various parameters like missing values, outliers, false reporting, inconsistencies etc.
• Need-based Studies: Need-based studies/implementation research will cross-examine findings from the monitoring and evaluation and address implementation questions that emerge. The MLE works with the GoUP, TSU, BMGF and other stakeholders to identify topics for need based studies and implementation research to address gaps in knowledge and strengthen evidence-based decision-making.
• Nutrition Cohort Study: The three categories of individuals are being interviewed for two rounds to understand the care received by the identified vulnerable individuals as well as their improvements or deterioration over time and the causes for the same. The findings would assist the program (TSU) in identifying critical aspects along the pathways of care for undernourished children and mothers that need focus, to improve overall maternal and child nutrition outcomes.
• Capacity Building on Monitoring and Evaluation: The consortium undertakes capacity-building activities on M&E for the GoUP staff. The target audience for the trainings is block, district and divisional-level NRHM personnel anchoring M&E functions. This includes the Block Programme Manager (BPM), District Programme Managers (DPM) and Divisional Programme Managers. A comprehensive need assessment was undertaken to identify the training needs of the target group.
• Financial Report Cards: The MLE develops report cards as analytical and diagnostic tool to measure, track and suggest improvement in financial expenditure. The report card is generated as a measure of governance and health system management by tracking inputs specifically fund flows (including timing of funds), utilization, supplies, human resource positions and output indicators.
• Assessment of Quality of Care
TSU supported the GoUP to strengthen the service provision for key RMNCH services at different levels of health facilities through its Nurse Mentoring (NM) program. The overall objective of the NM strategy was
• To unlock the potential of health services providers through onsite mentoring
• To improve quality of RMNCH service delivery across facilities through facility assessments and mentoring
At the facility level, the TSU supports GoUP in improving the availability and utilization of institutional delivery services along with basic quality of care of services during labor and delivery and early neonatal period (Annexure for detailed ToC). The intervention was provided at the community health centers (CHCs) and block primary health centers (BPHCs) through the Nurse Mentoring Program combined with overarching technical support from specialists posted at District and Zonal levels. The NM program was initiated by TSU in 100 TSU blocks and 50 non-TSU blocks within 25 High Priority District in the state. All the 150 Nurse Mentors (NMs) were trained by the TSU; however, the 50 NMs in non-TSU blocks were on the payroll of National Health Mission (NHM). The NM program involved improving the knowledge, skills, and practices of staff-nurses and ANMs on intrapartum and immediate post-partum care along with complications management and referral. The NM intervention utilized job-aids like casesheets, mini-skill labs for mentoring on various topics, and supported use of self-assessment checklists and assessments of skills of delivery care nurses using Objective Structured Clinical Examinations (OSCE) as quality improvement processes. The NM intervention also focused on improving the facility readiness (availability of drugs, equipment and organization of labor room and other infrastructure) to provide quality RMNCH services. The health systems interventions (details in subsequent section) aimed at strengthening stocks and supplies, improving availability of human resources and improving infrastructure within the facilities would also work in conjunction with the NM intervention to improve the overall quality of care at these facilities.
To measure the changes in quality of care at facilities, three primary studies were undertaken by Sambodhi:
• Quality of care (QoC) study- this included observation of deliveries (DoD), interviews with SN to assess their skills and knowledge, along with facility readiness (availability of drugs, equipment and organization of labor room and other infrastructure) using checklists
• Facility outlier study- This study was undertaken to analyses at the facility outliers (from TSU and Non-TSU geographies) on key parameters identified during the QoC study based on their facility scores respectively
• Follow-up study- women who delivered in public facilities (where QoC study was undertaken) but were not part of the DoD were interviewed at their homes two weeks postpartum to assess utilization of post-partum services and newborn care practices.
Facility surveys focusing on quality of care, infrastructure, standard treatment protocols, direct observation, checklists, patient satisfaction surveys and medical audits are an integral part of the assignment.