March 15, 2018

By Peter James, Senior Manager of Impact Evaluation

I’ve got to admit, “impact evaluation” probably isn’t the most tangible job in the world. What is someone like me doing all day, beyond squinting at spreadsheets (although, yes, there’s a bit of that)? The answer is surprisingly simple: my job is to figure out what impact our programs are trying to achieve, and to then gather evidence to evaluate whether or not we’re meeting those goals.

Let me give an example.

OneJustice runs pro bono legal clinics to help people with criminal record clearance. So let’s say we run 10 clinics and serve 180 clients. Ok, that’s great! But is it enough to know that those clients have simply met with a lawyer – or could we learn more by delving deeper? We might start asking: do clients typically leave the clinic with a completed petition, and how many successfully file the petition in court? What happens to clients that we refer to other organizations for more in-depth assistance? What is this whole experience like for our clients, and does it meet their needs and goals?

As you can imagine, these discussions about a program’s goals quickly become complex – and that’s before you start devising methods to assess whether these goals are being achieved.

So why go to all this effort? Again, my answer is fairly simple: because our clients’ legal problems matter deeply and often have high stakes. We owe it to them to critically assess what we are doing and make adjustments where necessary. In the example above, studying our criminal record clearance clinics may help us to identify ways that we can improve our service – for example, by changing the kind of information provided to clients before they attend the clinic, or by adjusting the training offered to pro bono attorneys who volunteer their time.

It’s an exciting time to be doing this work. Scholars in universities and law schools are pushing forward a reinvigorated research agenda on civil justice that seeks to answer difficult questions. How often do people experience civil justice problems? Do factors such as race and class influence how people respond in these situations? What is the nature of our civil justice infrastructure? How do we measure the effectiveness of legal interventions and services? Part of the job of us data folks is ensuring that the learning from these academic studies actually contributes to thinking within legal services organizations.

In addition to this type of academic research, some recent major investments in legal aid programs have included funding to evaluate impacts, such as the New York Immigrant Family Unity Project. Other organizations in the civil justice area, notably the Self-Represented Litigation Network, are using GIS mapping to provide a spatial lens to planning and analysis.

Impact evaluation is just one of many ways in which programs can assess (and reassess) their strategies. While managers and program staff are always observing what’s going on and making improvements to their work, the advantage of impact evaluation is being able to step back and take a more systematic perspective. This can bring into focus patterns that are difficult to spot on a day-to-day basis and incorporate feedback from a wider range of voices, including clients and partner organizations.

Peter James

OneJustice is at the very beginning of our work in this area. We will be learning from the academics and other legal services organizations who have been pushing this research forward. As we move ahead with our own research and impact evaluation initiatives, we plan on sharing our learning widely with the legal services community. I’m excited to have recently launched the OneJustice Research Newsletter, for example, and we’re looking forward to creating more spaces for others to share their experiences and ideas. So look out for news from us, and we can’t wait to hear from you!

Questions? Want to sign up for the Research Newsletter? You can reach out to Peter with questions and ideas at research@one-justice.org.