Challenges of Using Technology in MEL
In the last decade, the use of technology in monitoring, evaluation, and learning has become commonplace to support systematic inquiry into interventions, their impact, and evidence-based policymaking. Technology is leveraged in various stages of MEL – from diagnosis, design, data collection, and monitoring to analysis, evaluation, reporting, forecasting, and decision facilitation (Raftree and Bamberger, 2014; Tilton et al 2020). In South Asia, the use of tablets and phones for data collection, software to clean and aggregate data, and statistical packages for analysis and visualization are commonplace in the development sector, even if it has not received adequate academic scrutiny yet.
Technology-enabled MEL has the advantage of providing adaptive management, real-time, faster, and high-quality data collection, remote data collection, constant production of self-reported data, and advanced analytics of both numeric and non-numeric data. It has made data collection faster and increased the breadth of participants covered. Recent research on MERL-Tech identifies three waves of use of technology in MEL:
- technology for traditional MERL wherein digital technology is used to conduct traditional MEL activities in more efficient ways, which has become integral for many practitioners in international development,
- use of big data and data science, which emerged in India in the 2010s, but is still sparsely used for MEL activities, and
- emerging approaches and technologies to collect, organize, store and analyze data such as application data, sensor data, blockchains, and text analytics (Raftree 2020).
However, the advancement of technology in MEL is accompanied by complex ethical and epistemic questions. Some of the challenges include:
- selectivity bias,
- overreliance on digital data,
- low institutional capacities,
- issues of privacy and anonymity,
- missing lexicons for the development sector,
- reduced evaluator control,
- additional costs involved in acquiring digital infrastructures,
- overcollection of data,
- potential risks in use of the cloud and so on (Raftree and Bamberger 2014; Tilton et al. 2020).
In deeply divided and unequal societies in the context of countries like India, it is also important to reflect if newer technologies sharpen existing vulnerabilities and inequities instead of becoming a tool to unpack and address them. Perhaps, top-down technological innovations need to be accompanied by ground-up, community-driven, community-owned, and community-first approaches to contextualizing data produced for M&E activities. GIS data or health systems data, for example, can otherwise miss out on caste- or gender-based dynamics and complexities that mark real-world problems.
After all, data, which is being produced at dizzying speeds and in unimaginable quantum today due to the advancement of technology in MEL, is never inert or asocial. It is ‘produced’ and mediated by its social and material contexts.
However, there is a clear lack of systematic research, documentation, and critical reflection on MEL-Tech and digital data production.2 Within South Asia’s development sector, most academic publications around this theme are emerging from Sub-Saharan Africa.
A recent panel discussion titled “Challenges and Opportunities of using technology and digital data in MEL in South Asia,” organized during the 2022 Evaluation Conclave by Community of Evaluators – South Asia (in partnership with Sambodhi), opened up this debate on MERL and technology in India. The event featured M&E practitioners sharing examples from the ground of challenges, biases, and pitfalls of using technology for MEL.
A recurring point of discussion was the issue of inherent biases in digital data collection, data modeling, and in teams that collect and analyze data for MEL3. In my experience, such biases, and corresponding exclusions, instead of going away with the digitization of MEL data, can become even more entrenched due to the use of technologies and complex techniques which are far removed from the lived experiences of the community. When collecting data on the field using CAPI, I often faced confused looks and questions from research participants regarding how the data will be used to benefit them – questions that the MEL community must be held accountable for. Another significant theme in the panel discussion was the ethical challenges of digital data collection, the ethics of development itself, the nuances associated with ‘informed consent’ in data collection, and the need for uniform policies for data governance and management (privacy, security, transparency)4.
As the government of India and non-profits move towards adopting more rigorous results frameworks to measure and evaluate policies and programs, it is important that MEL practitioners reflect on how MEL data is produced and tempered in South Asian context, and the inclusions and exclusions that production of digital data entails. Enabled by technology, data collection in MEL has become increasingly fast-paced today, which is important for timely course-correction of development interventions.
However, the MEL sector must look inwards and ask how much data is too much data and if we are guilty of over-producing and over-extracting data now. MEL practitioners like Swapnil Shekhar have called for the use of public-intend data for MEL activities – perhaps that idea is worth exploring today. While moving away from the extractive nature of data collection, the MEL community should also strive to become more accountable to the communities from which data is being collected and who are the rightful owners of this data to center them in the development dialogue. This, in turn, also needs research and reflection on data ownership, data governance and accountability, and data democratization from the lens of equity to understand what works and what doesn’t in the MEL-tech-data space.
End Notes: 1 Acknowledgment: Rajib Nandi for his careful review and feedback on earlier drafts. 2 Data production encompasses processes of data collection, cleaning, and aggregation. 3 This point was raised by Krishanu Chakraborty (IDinsight) and Shruti Viswanathan (Athena Infonomics). 4 As highlighted by Sohini Mookerjee (CLEAR/JPAL) and Akashi Kaul (Sambodhi) during the panel discussion.
References:
Raftree L, Bamberger M (2014) Emerging Opportunities: Monitoring and Evaluation in a Tech-Enable World. New York: The Rockefeller Foundation.
Raftree, Linda. (2020). MERL Tech State of the Field: The Evolution of MERL Tech.
Tilton, Z., Harnar, M., Raftree, L., Perrin, P., Bruening, G., Banerji, S., Gordley, J., McGuigan, M., Foster, H. and Behr, M. (2020). What we know about traditional MERL Tech: Insights from a scoping review. MERL Tech
Bruce, K., Vandelanotte, J., Gandhi, V. (2020). Emerging Technologies and Approaches in Monitoring, Evaluation, Research, and Learning for International Development Programs.
Mahima Taneja – Guest Author
