For the past twenty years nonprofits have been told, first by funders, and now by researchers and academics, that they need to measure the outcomes of their work. On the surface it is hard to argue with this point. We all work too hard not to wonder about the impact of our efforts. Are the children we counsel succeeding in school? How long and how much money does it take to preserve an acre of land? Are atrocities averted by our advocacy efforts? We all want to know and funders and policymakers have a legitimate need to ask.
What does seem arguable is the belief that anything can be measured, and what is beyond arguable, is in fact laughable, is the proposition that a nonprofit should be able to make the resources available to do the measuring.
Here’s an example. A six year old enters a counseling program after his teacher finds him impossible to manage in the classroom, disruptive, inattentive, and aggressive toward other students. After two years of counseling, his behavior improves and he leaves the program. The counselor thinks of this case as a success.
But based on what metric or proof? How do we measure his improvement? There are standardized tools which could have been applied pre- and post-intervention, but they only measure current behavior.The key question is not whether the child is doing better at the moment counseling ends, but whether he continues to do well for years to come. This would require tracking and periodic retesting, which is expensive and difficult, as well as intrusive.
But let’s assume the resources are available to re-test this child in a year. And let’s further assume that the child’s behavior has deteriorated. Is the program then deemed a failure? To get at that question we would have to know what this child’s life has been like for the past year.
Answers to some of these questions could be found if enough resources were devoted to the research. But the cost of this effort could easily be far greater than the cost of the counseling intervention it is studying. And since every individual’s situation is unique, following just a sample of kids would not necessarily allow you to generalize to the entire program’s effectiveness.
Apply this example to your own work. Yes, we want to know if our programs are successful. But are we really willing to invest the resources it would take to provide a real answer?