Data on a Dime: Designing an Effective Monitoring & Evaluation Strategy on a Budget

By Jacquie Cutts and Stephanie Hackett

For many CHMI programs, Monitoring & Evaluation (M&E) is a high priority for achieving operational success. M&E provides managers with better means for learning from past experiences, improving health service delivery, planning and allocating resources, and demonstrating results. Within the development community, there is often a very strong focus on results—yet there is also concern that effective M&E processes are expensive and resource-heavy. Although a good practice to follow, many small organizations find it difficult to raise and allocate funds for M&E when there are often demands for direct programmatic support and resources in the communities they serve. But does it have to be this way? Not according to Safe Mothers, Safe Babies. 

SAFE, a maternal, newborn, and child health (MNCH) program operating in rural Uganda, thinks otherwise. Through their participatory and community-based program, SAFE is able to provide a range of services to ensure that no woman or child dies during childbirth by addressing the Three Delays. Read more about how SAFE tackles the Three Delays to prevent maternal and newborn death. Creating an M&E system to both quantify impact and guide strategy has been key for SAFE’s growth.

As a part of the peer-learning Primary Care Adaptation Partnership (PCAP), SAFE shared their low-cost M&E strategies with their Adaptation Partner, Healthy Entrepreneurs. SAFE offered technical and programmatic support to the Healthy Entrepreneurs’ team, in order to put these affordable strategies into action.

In this blog, SAFE’s CEO, Jacquie Cutts and Vice-President, Stephanie Hackett discuss how SAFE has refined a host of M&E techniques in order to better quantify the impact they are having on MNCH without breaking the bank. 

CHMI: How did you design SAFE's Monitoring & Evaluation strategy and what are its core components?

SAFE: At the crux of SAFE's approach to M&E is the high value we place on evidence-based interventions. As public health practitioners, it is essential that we monitor our programs to ensure that they are working, and then develop strategic ways to fix errors [in these programs] to expand on what works. Fundamentally, we value understanding where we are succeeding and where we are failing because that is how we can have the greatest impact on the mothers, children, and families we care very deeply about. With that in mind—but also recognizing that we are a small organization with very limited resources—our M&E strategy consists of conducting robust mixed- methods evaluations in inexpensive ways. This requires a lot of creativity, as well as dedicated professionals who are adept in a range of evaluation techniques, including quantitative and qualitative strategies.

CHMI: Why do you think M&E is considered to be so expensive?

SAFE: In the current public health landscape, M&E is often expensive. The people who developed the methods and often employ them have large (at least large from our perspective) grants from big funders, and they can afford the best. They have also often used paper-based tools that require tremendous amounts of printing (which is very expensive); rely on external parties that can be exploitative in billing for certain tasks; hire very experienced enumerators that often come with the price tag commensurate with that experience, etc. This is not to speak ill of their work at all— we are truly grateful to have benefited so much from the data they acquire. But in prectice, some of those approaches don’t work for us.

SAFE's strategy towards M&E was developed out of necessity. We believe that M&E is not only a moral imperative, but that it needs to be practical and achievable for ALL ranges of projects, including small organizations likes ours. Simply put, when you don't have as many resources, you learn how to get creative in order to keep as much robustness as possible while also cutting costs wherever you can.

CHMI: How did you design your M&E budget (any examples you looked to)?

SAFE: Truthfully, when we first designed an M&E budget, we didn't really know much about what other people paid, and honestly that was probably a good thing since most M&E projects had a lot more capital to work with than we did. We just looked at the overall categories of other budget templates (translation, staffing, training, survey materials/technology, logistics and census mapping) and adapted them to our context, budget, and available resources.

Also, if you need help implementing cost effective M&E solutions, don’t be afraid to find a really dedicated MPH (or other professional) student. We work with highly qualified and well-trained students in a lot of our work; it saves us a lot of money, gives them really good experience, and gets us much better data in the end.

CHMI: What did you find to be the highest cost components/items when you were planning your M&E strategy? How did you approach making trade-offs to keep it affordable?

SAFE: We had to make a few compromises [to balance costs]. For example:

• Labor was a compromise. We had to hire moderately-skilled workers that could be trained without having to meet the high salaries most NGOs pay.

• We found it best to use tablets instead of paper (you won't have to print, buy ink and paper or do data entry afterwards, thus eliminating a lot of costs). We set up a server to host data through Google AppSpot. We also currently use a free, open-source software, called Open Data Kit, to develop the electronic surveys in Excel. For all of our data collection and storing results, we paid $1.00 total (not including the tablets), and the $1 only accounts for times when our submissions exceeded the allotted free pings/second, which was rare.

Mapping or conducting a census of the communities where you work can also be expensive, but they can also make sure that your results are meaningful.If you want to do it, there are ways to cut costs. Shape files are accessible online that other people have generated for most countries. These can be imported into "MyMaps" on Google Maps, which can be used to look at the satellite imagery to trace on your roads [or local landmarks]. We would be happy to connect further on this.

CHMI: What are three tips you would give programs that would like to conduct an evaluation? 

SAFE: I would say the following three things:

• Do it! There are ways to make [M&E processes] work effectively for your budget, and being able to demonstrate that you are successful [through M&E] is really important, both ethically and for future funding. Conversely, if you learn that what you're doing isn't working, you should be open to addressing those challenges.
• Don't be afraid of potentially "undesirable" results. Everyone fails at some point. The important thing is to learn from past mistakes.

• Communicate your results. If you can't afford to publish your results in an expensive journal, then publish them in an inexpensive one or publish them on your blog and tag related partners. Go to conferences, present your results and be willing to talk not just about the good stuff, but also the not so good stuff. The only way that we can make a real difference for real people is if we know what works, and what doesn't. 

CHMI: What tips do you wish you had received?

SAFE: Overall, to: (1) Over-budget what you think you need; (2) Use mixed methods for capturing data (qualitative and quantitative); and (3) Ask for help from people in a similar position to you, who understand your type of constraints (like SAFE). Also—network, network, network! And collaborate.

For the practical and day-to-day:

• If you’re planning to use tablets, buy more tablets than you think you need—you WILL lose one, have it stolen, drop it in mud, etc. Expect it, plan for it, have a contingency plan.
• Hire 1 or 2 more interviewers than you need; people will fall sick, quit, etc.
• Don't underestimate translation. It takes forever, it’s expensive, and it’s a very subtle, important science.
• Training should never be shorter than a week, and you should always have at least 2 days of testing the instrument AFTER that week of classroom training.

For more information about SAFE’s Monitoring & Evaluation strategy, please feel free to email them directly at info@safemotherssafebabies.org or visit their blog. 

Top photo: SAFE trained enumerators share in their excitement as they explore the tablets they will be using for data collection for the first time. 

All photos  c/o SAFE