maxwell logo
Evaluation of International Programs and Projects
 

1. The Craft of Evaluation

January 12, 13 and 14, 2009

Empirical evaluation of public programs is well over 75 years old.  Starting in the 1920's with some of the earliest studies of industrial sociology, through the great empirical analyses during World War II, to the present, the concept that programs should be subject to proof that they are achieving results is central of public administration thinking.

Evaluation of international programs has been a concern since the beginning of the United Nations, for many of the same reasons that it has become part of national administrative practices.  Some would argue that it is even more important for international programs since, in the main, they are funded by national governments who need to be able to demonstrate the worth of deploying scarce public resources to international organizations rather than using them nationally.

While evaluation of international programs shares many of the concepts that would be used at national and sub-national levels, it also faces more difficulties because, generally, international programs act indirectly.  The lecture explores this issue.

Questions covered
  • What is evaluation and what is monitoring?
  • What are the main elements of evaluation?
  • How is evaluation in the public sector different from evaluation in the private sector?
  • How is evaluation different at the international level than at the national?
  • What is the role of baselines?

Lecture

Required readings
  • United Nations Department of Economic and Social Affairs, Systematic Monitoring and Evaluation of Integrated Development Programmes, (United Nations publication, sales number E.78.IV.11, Chapter 1.  This publication was produced by a team consisting of Inge Kaul, who was Director of UNDP's Office of Development Studies; Clinton Rapley, who retired from the United Nations in March 2003 and John Mathiason.  It shows basically what U.N. evaluation orthodoxy sounded like 25 years ago.
    World Bank Group, Operations Evaluation Department, International Program for Development Evaluation Training (IPDET): Course Modules, Module 1 (Presentation and Paper) and Module 2 (Presentation and Paper).  These are rather simple material produced for a refresher course, but summarize the Bank's current thinking.  They are pdf files and you can either read them on-line or download them. 
    International Atomic Energy Agency, How Well Are We Doing? A Guide to Programme and Project Evaluation for Agency Managers, IAEA Evaluation Series, Vol. 1, December 2002,  Excerpts from the Forward and Introduction (used with permission).

Recommended readings
  • United Nations Development Programme, Handbook on Monitoring and Evaluating for Results, (2002), Part I, The Monitoring and Evaluation Framework 
    World Health Organization, Roll-Back Malaria, Framework for Monitoring Progress and Evaluating Outcomes and Impact, 2000. Shows how a major United Nations specialized agency has decided to integrate monitoring and evaluation into one of its main areas of work.
  • RealWorld Evaluation, Chapters 1, 16
  • If you want to go more into the general orthodoxy of evaluation, these are two books that are used in courses on program evaluation at both Maxwell and the Wagner School of NYU.  Interestingly, their first editions date to the 1970's, which suggests that plus ça le change, plus c'est le même chose.
    Peter Rossi, Howard Freeman, and Mark Lipsey (1999) Evaluation: A Systematic Approach, 6th ed. Sage Publications.
    Weiss, Carol. Evaluation (1998), 2nd edition. Upper Saddle River, NJ: Prentice Hall.
Simulation

The course will include a simulation in which we design and, to the extent possible, conduct an evaluation of an international program or project.  For the first session, you should read about the program your section will evaluate.  You will be the Evaluation Section of the Monitoring, Evaluation and Consulting Division of the Office of Internal Oversight Services.  You will be reporting to the Assistant Secretary-General for Monitoring and Evaluation (a new position created as a result of the 2006 United Nations reform that emphasized accountability and oversight) who has taken a personal interest in your evaluation.

The Monday and Tuesday sections will be evaluating the United Nations' Internet Governance Forum. We will be particularly looking at the results promised in a series of program documents. To see how the program, which is managed by the a special secretariat funded from extra-budgetary sources, is presented, you should consult IGF website.

The Wednesday section will be evaluating a set of institutions concerned with climate change. These are the Global Environment Facility, the Clean Development Mechanism managed by the United Nations Framework Convention on Climate Change Secretariat, the United Nations Environment Programme and the United Nations Convention to Combat Desertification.

Each week you will have an assignment that is part of the evaluation process.  It will be sent by e-mail and posted on the syllabus page and the respective session page.

© 2003, 2004, 2005 John R. Mathiason. All Rights Reserved.
Revised: January 13, 2009 .