A systematic review is needed in order to ask the question: What is the effect of grammar teaching on the accuracy and quality of 5 to 16 year-olds’ written composition?
This perennial question has haunted the teaching of English for over a century. Although there have been extensive reviews of the question, views remain polarised, with a belief among some teachers, newspaper editors and members of the public, that such teaching is effective, and among others that it is ineffective. A systematic review is therefore required to provide an authoritative account of the results of research into the question.
The objectives of the review are as follows:
- To map the field of research on the effects of text- and sentence-level grammar teaching on writing in English-speaking countries for pupils aged between 5 and 16
- To undertake two distinct but complementary in-depth reviews in the field of sentence-level grammar: the effect of teaching syntax on accuracy and quality in written composition (in 2003-4); the effect of teaching sentence-combining on accuracy and quality in written composition (in 2004-5).
The present review concerns the effect of teaching syntax on the accuracy and quality of written composition.
One previous systematic review has been published in the broader field of the effect of grammar teaching on written composition. In 1986, Hillocks published a meta-analysis of experimental studies designed to improve the teaching of written composition. He analysed the experimental research between 1960 and 1982 and concluded that grammar instruction led to a statistically significant decline in student writing ability, the only instructional method of those examined not to produce gains in writing ability.
Methods used in the review
Systematic review methods were used throughout this review, using the Evidence for Policy and Practice Information and Co-ordinating Centre (EPPI Centre) guidelines and tools for conducting a systematic review.
Studies were included in the systematic map if they looked at the effect of grammar teaching in English on 5 to 16 year olds’ accuracy and quality in written composition. The criteria for including and excluding studies for the in-depth review on the effect of teaching ‘syntax’ were refined after the systematic map was drawn.
Reports were identified from the following sources:
- searching of electronic bibliographic databases: Educational Resources Information Center (ERIC); PsycINFO; and Social Science Citation Index (SSCI)
- citations in reference lists of all included systematic and non-systematic reviews
- personal contacts.
We applied the inclusion and exclusion criteria successively to the titles and abstracts and the full reports with quality assurance (QA) screening supplied by the EPPI Centre.
The studies remaining after application of the criteria were keyworded using the EPPI Centre’s Core Keywording Strategy and online database software, EPPI-Reviewer. Additional review-specific keywords which are specific to the context of the review were added to those of the EPPI Centre. Again, QA was provided by the EPPI Centre.
Studies identified as meeting the inclusion criteria for the in-depth review were analysed in depth using the EPPI Centre’s detailed Data-Extraction Guidelines, together with its online software, EPPI-Reviewer®. Three components were identified to help in making explicit the process of apportioning different weights to the findings and conclusions of different studies. Such weights of evidence are based on the following:
(A) The soundness of studies (internal methodological coherence), based upon the study only
(B) The appropriateness of the research design and analysis used for answering the review question
(C) The relevance of the study topic focus (from the sample, measures, scenario, or other indicator of the focus of the study) to the review question.
(D) an overall weight taking into account (A), (B) and (C)
The data were then synthesised to bring together the studies which answered the review question and which met the quality criteria relating to appropriateness and methodology. A narrative synthesis was undertaken. It was not felt to be appropriate to conduct a statistical meta-analysis.
Data extraction and assessment of the weight of evidence brought by the study to address the review question was conducted by pairs of Review Group members, working first independently and then comparing their decisions before coming to a consensus. Members of the EPPI Centre helped in data extraction and quality appraisal of a sample of studies.
Identifying and describing studies: results
A total of 4,566 potentially relevant papers were identified from the initial searches. After screening for relevance to the review using the pre-established inclusion and exclusion criteria, 58 papers were included in the systematic map of research in the field. These comprised 25 papers containing 24 systematic and non-systematic reviews, and 33 papers containing 31 primary studies. All the included primary studies were evaluations: 30 researcher-manipulated evaluations and one naturally-occurring evaluation. Of the 30 researcher-manipulated evaluations, seven were randomised controlled trials (RCTs), 13 were controlled trials (CTs), eight were pre- and post-test studies, and two were evaluations of ‘other’ designs.
Sixteen out of the 24 reviews explored the teaching of ‘syntax’. Of these, 12 provided a conclusion about the effect of syntax teaching on the accuracy and quality of pupils’ writing. None of these 12 reviews of the teaching of syntax concluded that teaching traditional or transformative/generative grammar had a positive effect on the quality and accuracy of 5 to 16 year-olds’ written compositions. The results of these reviews provide the context for our discussion of the results of our review.
Of the 28 studies that reported on sentence-level grammar teaching, 20 focused on sentence combining and 10 focused on other aspects of syntax (the focus of the in-depth review). A much smaller proportion focused on punctuation (n = 3), and only one study focused on sentence diagramming. Three studies investigated the teaching of both sentence combining and syntax. One study focused on sentence combining and punctuation; one on syntax, punctuation and sentence diagramming; and one on punctuation alone.
In-depth review: results
Ten studies were identified for the in-depth review. These studies were identified through the application of the review-specific keyword ‘syntax’ to the primary studies in the map.
The ten studies selected for in-depth review were all researcher-manipulated experimental studies, of which two were randomised controlled trials (Bateman and Zidonis, 1966; Fogel and Ehri, 2000); two were controlled trials (Elley et al., 1975, 1979; Stock, 1980); four used pre- and post-tests (Hilfman, 1970; McNeill, 1994; Roberts and Boggase, 1992; Rousseau and Poulson, 1985); one was a curriculum evaluation (Satterfield and Powers, 1996); and one a single subject ABACA design (Stone and Serwatka, 1982).
The narrative overview must begin with the studies rated high and high/medium or medium/high. These are Elley et al. (1975, 1979) (high to medium); Bateman and Zidonis (1966) (medium to high); and Fogel and Ehri (2000) (high).
It is not possible to synthesise systematically the results of the Elley et al. and Bateman and Zidonis studies. First, the transformational grammatical approach of Elley et al., based as it is on materials from the Oregon Curriculum (Kitzhaber, 1968), uses – we assume – different intervention materials from the unspecified ‘special grammatical materials’ of Bateman and Zidonis. Second, the analytical framework of the two studies is different, with Elley et al. using 12 variables for analysis and Bateman and Zidonis, 46. Third, we cannot rule out from either study, for different reasons, methodological invalidity or unreliability. Fourth, there is insufficient detail given in Bateman and Zidonis of the intervention or of the analytical tools used (hence the lower rating than Elley et al. in terms of weight of evidence). Fifth, there is no clear comparability between the two studies because Elley et al. use what they call a ‘transformational’ approach, and Bateman and Zidonis use a ‘generative’ approach to transformational/generative grammar. The relationship between the two, and to transformational and generative grammars and theories, is not clearly articulated.
In summary, Elley et al. conclude that syntax teaching, whether traditional or transformational, has virtually no influence on the language growth of typical secondary school students. Bateman and Zidonis conclude, tentatively, that a generative grammar approach does make a difference to syntactic quality and to the control of malformed sentences. Because of the relative quality of the two studies methodologically, the results of the Elley et al. study have a higher weight of evidence. However, neither study can be said to be conclusive. Fogel and Ehri present a different kind of study in which mastery of standard English written forms is improved for elementary school African-American pupils by a process of exposure, strategies for labelling and identifying grammatical features and, crucially, practising writing in these forms and receiving teacher feedback. However, short-term feedback is not enough to cause change in pupils of this age. As the authors point out:
further research is needed to determine whether more extensive and repeated use of the procedures would result in increased achievement; [because] instruction was limited to six forms … it is not clear whether findings would generalize to other more complex syntactic forms [nor] whether the performance differences that were observed would be maintained over time. These remain questions for further research (Fogel and Ehri, 2000, p 230).
Findings and implications
The results of the present in-depth review point to one clear conclusion: that there is no high-quality evidence to counter the prevailing belief that the teaching of the principles underlying and informing word order or ‘syntax’ has virtually no influence on the writing quality or accuracy of 5 to 16 year-olds. This conclusion remains the case whether the syntax teaching is based on the ‘traditional’ approach of emphasising word order and parts of speech, or on the ‘transformational’ approach, which is based on generative-transformational grammar.
Nearly all our included studies were experimental (i.e. researcher-manipulated as opposed to naturally-occurring evaluations), a highly appropriate design for testing causality.
In terms of practice, the main implication of our findings is that there is no high-quality evidence that the teaching of grammar, whether traditional or generative/transformational, is worth the time if the aim is the improvement of the quality and/or accuracy of written composition. This is not to say that the teaching of such grammar might not be of value in itself, or that it might lead to enhanced knowledge and awareness of how language works, and of systems of language use. But the clear implication, based on the available high-quality research evidence, is that the evidence base to justify the teaching of grammar in English to 5 to 16 year-olds in order to improve writing is very small.
It was not our brief in the present review to suggest what does work in improving the quality and accuracy of writing in English for 5 to 16 year-olds, but the implication is that, if there is little evidence that formal grammar teaching of syntax works, then practices based on theories such as ‘you learn to write by writing’ need to be given more credence and subject themselves to further systematic review. Whether there is space in the curriculum to teach syntax for its own sake, or for other purposes, remains to be seen.
The implications for further research are various. Despite a hundred years of concern about the issue of the teaching of grammar and thousands of research studies, the high-quality research base for claiming the efficacy of syntax teaching is small. The first implication, then, is that there should be a conclusive, large-scale and well-designed randomised controlled trial to answer the question about whether syntax teaching does improve the writing quality and accuracy of 5 to 16 year-olds. Such a study should have a longitudinal dimension to test whether any significant effects are sustained.
While we do not claim the final word on the question, the present review has been the largest systematic review in the history of research on the topic to date. This does not mean that other reviews of different aspects of the question of the relationship between grammar teaching and writing quality and accuracy cannot be undertaken. The specific focus of this review has been on the teaching of syntax and a complementary review we are undertaking is on sentence combining, both of which come under the umbrella of ‘grammar’ teaching.
There are limitations to this particular review. The teaching of traditional grammar or syntax to improve written composition tends to ignore the levels of language immediately below and above the sentence: morphological structures in language below the level of the sentence; and paragraph and textual levels above the level of the sentence.
Despite the above reservation, we hope to have established a landmark in studies on the effectiveness of syntax teaching in the development of writing quality and accuracy in school-age children. If this is a landmark, it points the way to further research in the field, where the territory of debate will be somewhat different. We now know that there is no high-quality evidence that teaching of traditional grammar or syntax (or the direct teaching of formal or generative/transformational grammars) is effective with regard to writing development. Having established that much, we can now go on to research what is effective, and to ask clearer and more pertinent questions about what works in the development of young people’s literacy.
Bateman DR, Zidonis FJ (1966) The Effect of A Study of Transformational Grammar on the Writing of Ninth and Tenth Graders. Champagne, IL, USA: National Council of Teachers of English.
Elley WB, Barham IH, Lamb H, Wyllie M (1975) The role of grammar in a secondary school curriculum. New Zealand Council for Educational Studies 10: 26-41.
Elley WB, Barham IH, Lamb H, Wyllie M (1979) The Role of Grammar in a Secondary School Curriculum. Educational Research Series No 60. Wellington: New Zealand Council for Educational Research. ERIC document number ED185588.
Fogel H, Ehri LC (2000) Teaching elementary students who speak black English vernacular to write in standard English: effects of dialect transformation practice. Contemporary Educational Psychology 25: 212-235.
Hilfman T (1970) Can second grade children write more complex sentences? Elementary English 47: 209-214.
Hillocks G Jr (1986) Research on Written Composition: New Directions for Teaching. Urbana, IL, USA: National Conference on Research in English. ERIC document number ED265552.
Kitzhaber AR (ed.) (1968) The Oregon Curriculum: A Sequential Program in English. New York: Holt, Rinehart and Winston.
MacNeill TB (1982) The Effect of Sentence-Combining Practice on the Development of Reading Comprehension and the Written Syntactic Skills of Ninth Grade Students. Alberta: University of Alberta. ERIC document number ED217415.
Roberts CM, Boggase BA (1992) Non-intrusive grammar in writing. Paper presented to the Annual Conference on Computers and Writing. Indianapolis, USA: May 1-3. ERIC document number ED348684.
Rousseau MK, Poulson CL (1985) Using Sentence-combining to Teach the use of Adjectives in Writing to Severely Behaviorally Disordered Students. New York: City University of New York. ERIC document number ED342153.
Satterfield J, Powers A (1996) Write on! Journals open to success. Perspectives in Education and Deafness 15: 2-5.
Stock R (1980) The Effect of Teaching Sentence Patterns on the Written Sentence Structures of Grade Two Children. Manitoba, Canada. ERIC document number ED208414.
Stone AK, Serwatka TS (1982) Reducing syntactic errors in written responses of a retarded adolescent through oral patterning. Education and Training in Mental Retardation and Developmental Disabilities 17: 71-74.
This report should be cited as: Andrews R, Torgerson C, Beverton S, Locke T, Low G, Robinson A, Zhu D (2004) The effect of grammar teaching (syntax) in English on 5 to 16 year olds’ accuracy and quality in written composition. In: Research Evidence in Education Library. London: EPPI Centre, Social Science Research Unit, Institute of Education.