Do development projects reach their stated objectives, such as reducing poverty, improving skills, creating jobs, etc.? This turns out to be a complicated question about project impact that a simple before-and-after measurement would not help answer. Why? The answer is also complicated, of course, but here are two points to consider:

• First, when a country goes through a rapid modernization process, as is arguably the case in Georgia, most development indicators improve over time regardless of specific donor interventions.
• Second, even if we observe a greater-than-average improvement in, say, jobs or incomes of the “treated” individuals or communities (project beneficiaries), such improvements cannot be automatically attributed to the project in question. A project may appear to perform spectacularly well if it selectively “treats” (i.e. provides training, machinery, or technical assistance) the most talented individuals or the best-organized communities. These may have done just as well (or even better!) without the project.

A case in point is the almost $400 million investment by the Millennium Challenge Corporation (MCC) in Georgia’s agriculture, road, and water infrastructure. On October 4th, ISET hosted researchers from NORC at the University of Chicago who presented the results of an impact evaluation they conducted for MCC’s Samtskhe-Javakheti road rehabilitation and agribusiness development projects.

According to the presenters, these two projects were designed to allow for a rigorous assessment of their impacts. Thus, MCC’s Agribusiness Development Activity (ADA) included an element of randomization mimicking an experimental setup that is more common in the natural sciences.

This worked as follows: primary agricultural producers (mostly households who have registered as businesses) applying for ADA grants were scored on a set of objective criteria. ADA awarded grants to all top scorers (85% and above), and some of the second-tier applicants (scoring in the 70-85% range) who were chosen through a lottery (i.e. randomly). Those who were not chosen through the lottery were supposed to be excluded from assistance for the first two years of the project. Because grants were assigned in a random fashion, those who did not win the lottery would have constituted an ideal “control group”: they were similar to other businesses in the same (70-85%) score range except that they were not supposed to receive the ADA treatment (grant assistance) until the third year of the project. This approach would have allowed for counterfactual analysis, or an understanding of what would have happened to those in the treatment group had they not been treated.

Unfortunately for NORC, due to a lack of qualified applicants, businesses assigned to the control group were released into the treatment group (i.e. awarded grants) already in the second year of the project, thereby spoiling the experiment. As a result, NORC was not able to detect any significant differences in the performance of businesses that have been initially assigned to different groups. This might be due to the earlier-than-expected release of the control group or other reasons, but what is sure is that the reliability of NORC’s results has been seriously undermined.

One indicator on which the project did have some impact is the tendency of households to register as businesses. This is not surprising as it was necessary to register in order to be eligible. It also affected the chances of businesses to survive: in 2008-2009 only 8% of “treated” business entities ceased to exist as compared to 28% for the non-treated population. Two years later, in 2010-12, ADA-financed entities were still less likely to disappear, but the gap shrank to 11 percentage points (19% vs. 30%). Unfortunately, this does not tell us anything about the impact of the program on the profitability of businesses or about the probability of their long-term survival.

The obvious – and quite tangible – benefits brought about by the Samtskhe-Javakheti (SJ) road rehabilitation program, did not make the task of rigorously estimating its overall economic impacts any less problematic. The project dramatically shortened the travel time from Tbilisi to Akhalkalaki (from 8 to less than 3 hours), essentially integrating the entire SJ plateau into the Georgian market. A very simple, back-of-the-envelope analysis by Eric Livny suggests that the project’s economic impact was staggering:

In 2009-2011, household incomes grew in all Georgian regions, yet, SJ experienced the highest income increases in both years under examination: 38% and 20% in 2010 and 2011, respectively. As a result, in just two years, this – until recently isolated – region registered a cumulative income growth of 66% (well more than twice the average rate of income growth (29%) for all other regions!) and went from 5th to 1st in the income per household ranking of Georgia’s regions. While not all of this improvement should be attributed to the MCC road project, there is no doubt on our mind that this investment has had a major positive impact.

NORC’s approach to evaluating the impact of the SJ road rehabilitation project was quite a bit more sophisticated. Since there was nothing random about the selection of SJ road, the team used the so-called “propensity score matching” technique to match SJ households and communities with comparable households and communities in proximity to similar Georgian roads. The results of the evaluation thus crucially depended on the choice of roads to serve as controls.

Unfortunately for the results of NORC’s evaluation, instead of picking similarly isolated regions, the development of which was held back by poor road conditions and travel time to Tbilisi, their control group included stretches of the main East-Way highway. This choice of controls affected the outcomes of the evaluation: MCC investment appears to have had no impact on household income or profits in the treated region. However, this seems unlikely to be the case. It is indeed possible that SJ households did not see their condition improve more than Imeretian households sitting on the main East-Way highway. But as GeoStat data suggest, they did much better than those Georgian households and communities that remained economically and socially isolated from the Georgian mainland, in Imereti and all other Georgian regions.

The results illustrate how challenging it can be to conduct rigorous impact evaluations, especially when projects do not allow for randomized assignment or when the randomization strategy falls short because of unforeseen changes.