One of the more challenging tasks in the monitoring and evaluation world is the effective development of indicators. When dealing with prevention activities the challenge is often compounded since proving the desired outcome can often be an ‘unknown’ factor. So what makes a good indicator and how can we show evidence of achieving outcomes?
In a recent conversation I had with a program specialist we were talking about this very topic and she gave me a great example. She had been working with fire prevention officers who conducted community outreach activities. During a performance appraisal session the following question was asked: “So, how many fires have you actually prevented this year?” I hope everyone can see the dilemma posed by this, rather unfair, question. The problem is that the question assumes prevention is the same as impact when, in fact, the responsibility of the fire prevention officers was to carry out outreach activities (e.g. house calls). How many fires they prevented is of course ‘unknown’. To gain insight into impact we can look at whether fires are declining over time, but we need a different source of data for this indicator. Potential indicators could be the following:
Disaster risk reduction, wildlife protection, and prevention of pandemics all face similar issues. As with fire prevention outreach, these programs engage in various activities to prevent negative outcomes. The challenge, however, is that higher level indicators are often more difficult to conceptualize. Several factors contribute to this:
- Most cities around the world would have records of the number of fires that take place, providing an often-reliable source of information. Wildlife trafficking, however, is an illegal activity and accurate information at the outcome and impact levels will seldom be available.
- Many fires can be prevented. However, we don’t have control over natural disasters. We can only apply disaster risk reduction to lessen the impact.
- Fires are, to some extent, predictable. In contrast, new viruses emerge all the time and some have the ability to mutate. Therefore, predicting where and when the next pandemic threat will happen is very difficult.
The first step towards developing effective indicators for prevention is to recognize that prevention takes place at the output level; it is not an outcome per se. Prevention is the first step, followed by detection and response.
Indicators at the prevention level are typically developed from key program activities and should be standardized so they can be aggregated across locations. Focus on a few ‘key’ indicators as too many can make monitoring difficult. But we can’t prevent everything so when disaster strikes we need a good smoke detector. For disaster risk reduction and pandemic prevention, detection could take the shape of an early warning system, or customs officials in the case of wildlife protection. The challenge here is that programs may not detect every potential threat, and hence, they become subject to the ‘detection trap’. That is, the number of detected risks increases as the program becomes better at detecting potential threats leading to the interpretation that the program is failing. A better indicator is to focus on the quality of risk detection such as larger coverage, better prediction levels or time taken to detect a potential threat. Finally we have response, and again, the focus should be on the quality of the response and the extent to which negative impacts can be reduced.
Know anyone who is working on prevention? Share this blog with them.
Connect With Rapid Asia: Facebook l LinkedIn
About the Author: Daniel Lindgren is the Founder of Rapid Asia Co., Ltd. a management consultancy firm based in Bangkok that specializes in evaluations for programs, projects, social marketing campaigns and other social development initiatives. Learn more about our work on: www.rapid-asia.com.