CHMI has recently announced the launch of its Reported Results initiative – an effort to capture performance and impact information from programs profiled in our database. To support this program and help highlight the benefits and challenges of tracking operational data, CHMI is featuring a series of blogs that take a closer look at the monitoring and evaluation approaches of programs reporting results.
Here, Nicholas Gordon of Operation ASHA, a New Delhi based program that is helping to curb the national tuberculosis epidemic by bringing DOTS treatment to the doorsteps of the poor, shared with us the program’s patient data collection strategy and some of its exciting achievements.
Programs collect a wide range of information to support monitoring and evaluation. What indicators and other data is important for your program to track?
NG: Tracking the millions of patients we serve is a challenge, as well as monitoring the resources that we use in treating TB patients. To eliminate human error, OpAsha employs an Electronic Medical Record (EMR) that maintains all data and generates periodic reports. Crucial factors tracked include the number of patients enrolled in treatment, demographics, average distance of a treatment center from the patient’s house, TB detection rate per 100,000 persons, cure rate, default rate, death rate, cost of treatment, financial leverage, and social return on investment (for funding purposes).
Tell us a little bit about how your program tracks performance. What methods do you use (e.g., internal monitoring, external evaluations)? How often is performance data collected? Do you have targets/objectives that you aim to reach?
NG: OpAsha uses an Electronic Record System that maintains all data on patients and inventories, generating statistical reports at specified intervals. Each month, the Public Health Department (Government of India) verifies the data’s accuracy in each of our reports by cross-checking our data with government statistics. Each month, the government’s District TB officer also receives a report indicating the new and total patients enrolled at OpAsha, an inventory of TB meds and supplies used, and report of potential MDR or XDR patients. Internally, the EMR generates weekly, monthly, quarterly, and annual reports that help control default, facilitate counseling and operations, and assist in TB detection.
How does performance tracking influence operational decisions? Can you give us an example of how a decision was made based on information obtained from M&E?
NG: All of OpAsha’s operations are dependent on the reports generated by the EMRR. Performance is tracked, as stated above, on a weekly, monthly, quarterly and annual basis. The different steps in tracking allow us to evaluate our performance on multiple levels, and identify discrepancies between reports. For example, the weekly report allows counselors working on the ground to see exactly how many patients missed a dose that week and who they are. Then, that counselor goes directly to the patient’s home to ensure that they resume their medication schedule and prevent a case of regular tuberculosis from becoming MDR-TB (multiple-drug resistant TB), a deadlier and more costly form of the disease caused by failure to complete a full course of treatment for regular TB. A monthly report shows a more comprehensive view of performance including the number of patients enrolled, treated, defaulted, and died, information that affects the day-to-day performance of our operations.
Talk about some of the results you've obtained thus far. What are some of your successes? What would you like to improve in the future?
NG: Key achievements include a default rate of less than 3%, a rate 30 times less than the average observed in other urban slums worldwide. Of TB patients enrolled with OpAsha, 92% are successfully treated. The World Health Organization set a minimum of 85% success rate for NGOs. We are also the largest nonprofit treating TB in India. Our success has earned us an election to the Coordinating Board of the Stop TB partnership by WHO, where we represent NGOs from the developing world.
We plan to expand operations in India to serve a population of 80 million by 2014. OpAsha’s mission is to eradicate tuberculosis worldwide. To that end, we have already replicated our model in Cambodia, completed groundwork in Morocco, and are in discussions with organizations in Ghana.
Do you think it's important for programs to disseminate performance information more broadly? Do you disseminate your results? If so, how?
NG: We believe that maintaining transparency and disseminating our results is of utmost importance, especially as we continue to expand. Our main outlet for publishing our results is our website: www.opasha.org. The default rate, the social return on investment, number of patients we serve, etc. can all be found on the website. We also provide access to our databases for the Massachusetts Institute of Technology who performs controlled trials to determine the optimal workload and salaries for our counselors. They also analyze OpAsha’s model and cost effectiveness. MIT’s involvement has lent us credibility with both donors and governments and we plan to distribute their results as soon as they become available. Finally, in September 2010, OpAsha was named one of the 11 top public private models in the world alongside the Grameen Bank and BRAC in a report presented by the UN Secretary General to the Millennium Development Summit of the UN. The recognition is wholly a result of disseminating our performance information and maintaining a high level of internal and external performance evaluation.
What are the biggest challenges you see to performance monitoring? How can the global health community help you overcome some of these challenges?
NG: With massive expansion on the horizon, investing in the resources to continue effective performance monitoring will become a challenge. Private donors and institutions fund our one-time expenses such as the biometric devices, donations that will have to increase exponentially to expand our model on the scale we hope to achieve. Generating the reports from the raw data collected from the biometric devices and our counselors on the ground also requires hardware such as desk and laptops, Internet access, and software programs, as well as the personnel to operate them. Internally, the challenge is ensuring expansion is created on a model that applies across cultures, language and geographies for the long-term.