Commentary on the review by Dr. Gordon Cleveland, University of Toronto, member of Peripheral Review Group

April 15, 2006

As a member of the Peripheral Review Group, I have serious reservations about the interpretation of results, and therefore the conclusions drawn, in the recently published Early Years Review entitled “What is known about the long-term economic impact of centre-based early childhood interventions?” (Penn et al., 2006). Let me explain.

The Review concludes that we do not know very much (see the Summary and Chapter 5 – Conclusions and Implications). We do know from the studies reviewed that the children involved in the specific interventions were better off, but it is concluded that we cannot make any inferences to other populations based on this evidence. We are told that “there may well be long-term outcomes from early childhood interventions….” (p. 4), but that “the widespread, international, use of the most favourable headline findings, and in particular of the Perry High/Scope study, is unjustified.” (p. 3). We also receive considerable warnings about the difficulty of using longitudinal studies, and the weaknesses of cost-benefit analysis. However, the authors are unable to answer the question “Do early childhood interventions save money over the long term?” (p. 3) with a positive response.

This reticence on the part of the authors is surprising. There is now a very substantial literature on the positive effects on cognitive and language development of early childhood education (Vandell, 2004; Cleveland, 2005). The evidence about effects on social competence and behaviour is more mixed, but generally very positive effects on behaviour have been found for children from low-income families. In particular, the evidence from experimental studies of early childhood interventions has been strongly positive (Karoly et al, 1998). Further, there have been very good studies of the impacts on children of the large Head Start program in the United States, which serves close to a million children per year in 20,000 centres (Currie and Thomas, 1995, 2000; Garces, Currie and Thomas, 2002). Head Start has been found to have long-term and substantial positive impacts on children, through to adulthood.

In this context, I think it astonishing that this EPPI Centre review concludes that we do not know anything substantial about the long-term economic impact of centre-based early childhood interventions.

How was this conclusion reached? The problem, I believe, starts with the inclusion and exclusion criteria for the Review. The Review (p. 9) decided that it would only look at studies which evaluate centre-based experiments using longitudinal data and which follow the children up to at least age 15. Further, these evaluations have to include “the assignment of costs to inputs and outcomes” (i.e., must include a complete cost-benefit analysis). The net was cast so narrowly that only three studies were included (although there were a large number of reports and secondary literature on these three experiments).

These inclusion and exclusion criteria meant that the review did not look at any of the careful statistical analyses (NICHD, 2005; NICHD and Duncan, 2003) about effects on children from the NICHD (National Institute of Child Health and Human Development) longitudinal study in the U.S., nor did they look at similar positive results from the Cost, Quality and Child Outcomes Study (Peisner-Feinberg et al, 2001), or other U.S. studies. They did not take into account recent studies (Lefebvre and Merrigan, 2005) of the very strong maternal labour supply effects of the Quebec (Canada) child care reforms of 1997, with very positive effects on society’s tax revenues (Baker, Gruber and Milligan, 2005). They did not take into account the positive academic and behavioural findings by Andersson (1989, 1992) from Sweden. They did not take into account positive results from the EPPE study in the U.K (Sylva et al, 2003). They did not take into account the above-mentioned longitudinal studies of the Head Start program in the U.S. They did not consider cost-benefit studies (Cleveland and Krashinsky, 1998) that were not based on a specific longitudinal experiment, but instead gathered cost and benefit data from numerous sources.

Forced into an intellectual straightjacket by the inclusion and exclusion criteria, the authors of the Review felt constrained to answer the very broad question “What is known about the long-term economic impact of centre-based early childhood interventions?” from the very narrow evidence base of three very specialized studies on very disadvantaged U.S. children. The studied experiments were the Perry Preschool/High Scope experiment, the Abecedarian Project and the Chicago Child-Parent Centres Project. I disagree with the authors’ interpretation of the evidence precisely because I think the narrow evidence from the three studies should have been interpreted within the context of what we know from many other good quality statistical studies.

Based on the evidence from the three studies, the EPPI Centre review’s authors use a number of arguments to justify the opinion that no valid conclusions about the magnitude or existence of positive economic effects are possible:

1. The children studied are very disadvantaged low-income children
2. The children are African-Americans in a racist society
3. The interventions happened a long time ago
4. The databases used to assess the value of benefits are highly specific U.S. databases
5. The studies vary in whether they assess health effects on children and effects on mothers’ employment
6. Cost-benefit studies are inherently problematic because they focus on measurable effects and do not include equity concerns.

Let’s look through these arguments in turn:

1. The children are very disadvantaged low-income children

This is certainly true, and we would want, therefore, to be quite cautious about claiming that these results would apply to all children, or even to all low-income children. However, our inferences will be affected by what else we know. And we do know things. There has been a considerable amount of other research on the effects of good quality centre-based early childhood education on children and families, but it was excluded by definition from this review. In particular, there have been well-done studies on the long-term effects of Head Start programs on what is now nearly a million American children per year. And these show that there are substantial positive long-term effects on both white and black (i.e., Caucasian and African-American) low-income children in the United States. Not just highly disadvantaged children, and not just African-American children.

2. The children are African-Americans in a racist society

The position of African-Americans in the U.S. is special, and the U.S. is particularly racist in the ghettoization of African-Americans into poorer schools and poorer neighbourhoods and poorer lives. However, it’s not clear how this should make us amend the inferences from these studies that have found strong positive effects of early childhood interventions on African-American children.

Racism may have “distorted the results”, but how? I would think that strong racism against African-Americans should have meant that even after attending a centre-based early childhood intervention (with positive emotional, social and cognitive effects), African-American children might have found it impossible or difficult to get continuing good schooling, to get into college or university, to get a good job and keep it (Currie and Thomas, 2000). In other words, systematic racism might be expected to make the “treatment” group of children and the “control” group of children share the same fate after the early childhood intervention was over. How much more surprising, then, to find a strong surplus of benefits over costs in long-term economic assessments for African-American children in the U.S.

3. The interventions happened a long time ago

This is certainly true, and it’s true that the preconceptions lying behind the studies may be outdated and even abhorrent. These are good points to note. However, if we really thought that information from studies started 20 or 30 years ago was irrelevant, then why did the authors seek to review only evidence from longitudinal studies? To have solid evidence about the effects on children when they are grown into adults, we would have to have data from this long ago.

4. The databases used to assess the benefits are highly specific U.S. databases

I don’t know of a single cost-benefit study that does not use highly specific databases from the country in which the experiment was conducted in order to calculate the benefits from employment, or health effects, or reduction in criminality, or whatever. What did the authors expect they would use? The point really is that the authors think the estimates of the value of reductions in crime are too high in the Perry Preschool Study and the Chicago Child-Parent Centre Study and that if the same reduction in crime occurred in the U.K. or Canada, the benefits would be less. This is not a criticism of the original studies; it is an issue of interpretation and inference. The literature review makes no attempt to offer an alternative valuation of benefits that the authors believe would better reflect the U.K. realities (i.e., benefits from reduction in criminal behaviour, and from lowered public costs due to improved abilities in school).

The EPPI Centre review approach is in contrast with that in Karoly et al. (1998), the Rand study which reviewed studies on the costs and benefits of early childhood interventions. Karoly and her colleagues decide that the “intangible” benefits from crime reduction in some of the studies reviewed (that is, the pain and suffering of the victims of crime) are too speculative to rely on. They decide to include only the tangible benefits of crime reduction (less than 25% of the total, including property loss, medical expenses and income lost while injured) in completing their review of the costs and benefits from the Perry Preschool program. This recalculation reduces the ratio of the benefits to the costs to 4.11 to 1 instead of over 7 to 1. Benefits of crime reduction are now about 1/5th of total calculated benefits (21.4%). After these adjustments are made, the clear inference is that similar programs would still have benefits considerably greater than costs, but the EPPI Centre review ignores this conclusion.

5. The studies vary in whether they assess health effects on children and effects on mothers’ employment

The Abecedarian cost-benefit analysis includes both health effects (probability of smoking) and effects on the employment rates of mothers of the children receiving the “treatment”. The other studies do not. I think this is a problem with, and a valid critical comment to make about, the cost-benefit analyses of the Perry Preschool Program and the Chicago Child-Parent Centres Program. It was a problem with the original data collected for these programs; they were not broad enough in their consideration of possible impacts of an early childhood intervention (at least partly due to antediluvian views about the appropriate roles of mothers in society). However, it is obvious that these weaknesses in the cost-benefit analyses should lead the EPPI Centre review to infer that the benefits of a similar program in some other jurisdiction, when properly measured, might well be larger than found in the measurements for these two experiments.

6. Cost-benefit studies are inherently problematic because they focus on measurable effects and do not include equity concerns

It is true that cost-benefit studies, in practice, do focus on measurable effects and do not include equity concerns. How could they do otherwise? Because they are an attempt at measurement, they rely on what is measurable. Because a cost-benefit study is an assessment of economic efficiency, it does not make judgements about the contribution of early childhood intervention to equity in society. However, the EPPI Centre review is a study of the long-term economic effects of centre-based early childhood interventions in which it has chosen explicitly to restrict its literature review to cost-benefit studies (and has rejected other studies of economic effects). If the authors thought that cost-benefit analysis was methodologically unsound, I can’t understand why the review made the restrictions that it did. Obviously some comment about what cost-benefit analysis may not be including is valid. However, the conclusions to the EPPI Centre review appear to be saying that cost-benefit analysis techniques are fatally flawed. In assessing economic effects, which is what the EPPI Centre review is supposed to be doing, I believe that cost-benefit analysis does a better job than any alternative. Supplementing a cost-benefit analysis with evidence about effects on children’s immediate well-being and on equity in society will make this analysis even more persuasive.

From the material analyzed in the EPPI Centre review and from points (1) to (6) above, I would say that the review should conclude that there is good evidence that centre-based early childhood intervention programs do have effects on children – that those children are likely to do somewhat better in school, are likely to have better eventual experiences in the job market, and that their mothers (or perhaps, fathers) will be permitted to develop and maintain a stronger labour force attachment than they would otherwise have done. For at least some populations of children, there will also be visible reductions in criminal activity. And there may well be other effects, such as health improvements, and increased happiness of the child in the here and now. Further, we can be confident that, for the three programs examined, the long-term benefits for children and families have been greater than the costs.

None of this necessarily argues that targeting of early childhood services to extremely disadvantaged children makes sense. I don’t think it does. We have enough additional evidence of the effects of good quality early childhood programs delivered to much broader groups of children (or delivered universally) to conclude that these effects for severely disadvantaged children are but the tip of an iceberg.

The review ends up concluding that that there is no usable information about long-term economic effects of early childhood interventions that comes out of these three longitudinal cost-benefit studies. On the contrary, I believe (and tried to convince the EPPI Centre authors before publication of the review) that there is information in these studies, and the methodology that produces it is reasonably sound (with criticisms and qualifications), and it is reasonable to infer that there would likely be positive effects of similar (or, perhaps, much broader) interventions if they were repeated in other countries. However, I also believe it is probably not sensible to replicate these highly targeted intervention studies, because delivery of early childhood services only to low-income disadvantaged children does not make sense on other grounds.

References

Andersson B-E (1989) Effects of Public Day-Care: A Longitudinal Study. Child development 60: 857-66.

Andersson B-E (1992) Effects of Day-Care on Cognitive and Socioemotional Competence of Thirteen-Year-Old Swedish Schoolchildren. Child Development 63: 20-36.

Baker M, Gruber J, Milligan K (2005) Universal childcare, maternal labor supply, and family well-being. National Bureau of Economic Research Working Paper 11832. Cambridge, Massachusetts. Retrieved from http://www.nber.org/papers/w11832

Cleveland G (2005) Knowledge, Policy and Research about Early Learning and Child Care. Paper for the Expert Roundtable on Child Development, Dec.8-9, 2005, Social Development Canada, Government of Canada.

Cleveland G, Krashinsky M (1998) The benefits and costs of good child care: the economic rationale for public investment in young children. Toronto: University of Toronto Centre for Urban and Community Studies, Childcare Resource and Research Unit.

Currie J, Thomas D (1995) Does Head Start Make a Difference? American Economic Review 85: 341-64.

Currie J, Thomas D (2000) School quality and the longer-term effects of Head Start. Journal of Human Resources 35: 755-74.

Garces E, Thomas D, Currie J (2002) Longer term effects of Head Start. American Economic Review 92: 999-1012.

Karoly L, Greenwood P, Everingham S, Hoube J, Kilburn R, Rydell P, Sanders M, Chiesa J (1998). Investing in Children: What we know and what we don’t know about the costs and benefits of early childhood interventions. Santa Monica, CA: RAND Corporation.

Lefebvre P, Merrigan P (2005) Low-fee ($5 per day) regulated child care policy and the labor supply of mothers with young children: a natural experiment from Canada. Working paper 05-08. Centre interuniversitaire sur le risqu, les politiques economiques, et l’emploi, Montreal.

NICHD, Early Child Care Research Network (ed.) (2005) Child Care and Child Development: Results from the NICHD Study of Early Child Care and Youth Development. New York: The Guildford Press.

NICHD, Early Child Care Research Network, Duncan GJ (2003) Modeling the impacts of child care quality on children’s preschool cognitive development. Child Development 74: 1454-75.

Peisner-Feinberg ES, Burchinal MR, Clifford RM, Culkin ML, Howes C, Kagan SL, Yazejian N (2001) The relation of preschool child-care quality to children's cognitive and social developmental trajectories through second grade. Child Development 72: 1534-53.

Penn H, Burton V, Lloyd E, Potter S, Sayeed Z, Mugford M (2006) What is known about the long-term economic impact of centre-based early childhood interventions? Technical Report. In: Research Evidence in Education Library. London: EPPI Centre, Social Science Research Unit, Institute of Education, University of London.

Sylva K, Melhuish E, Sammons P, Siraj-Blatchford I, Taggart B, Elliot K (2003) The Effective Provision of Pre-School Education (EPPE) Project: Findings from the Pre-School Period. London: Institute of Education, University of London and SureStart.

Vandell, DL (2004) Early Child Care: The Known and the Unknown. Merrill Palmer Quarterly Journal of Developmental Psychology 50: 387-414.