Background
The broad background to this review is that there is a growing concern internationally that the investment in Information and Communication Technology (ICT) in schools is not impacting on literacy development. This concern arises from a belief held by many - including governments as well as schools - that ICT is beneficial to learning and specifically literacy learning. The question is a specific one and has to be seen within a wider political, social and technological context in which the symbiosis between new technologies and new literacies (and thus literacy learning) is acknowledged.
In this systematic in-depth review of the impact of ICT on literature-related literacies in English, 5-16, Locke and Andrews set out to determine - as far as they could, given the range of research available and the conceptual complexities of the field - the nature of the impact of what have become known as new technologies within a wider notion of the symbiosis between ICT and literacies.
The background anticipates that research studies of the impact of ICT on literature-related literacies are thin on the ground. It sets out key definitions and explores the conceptual issues in the present review. It also summarises briefly the policy, practice and research background. The review uses systematic research review methodology developed by the EPPI Centre.
Review questions
The overall research question for the two-year project is:
What is the impact of ICT on literacy learning in English, 5 - 16?
The research question for this sub-review is:
What is the impact of ICT on literature-related literacies in English, 5-16?
Structure of review
The structure of this review is unusual in that it includes a two-stage 'mapping' process, followed by an in-depth review.
In the descriptive map of the overarching review, the process of identifying, including and characterising the studies for the systematic review of the impact of the ICT on literacy learning, is described. This map is an updated version of the original map described in the first review. In total, a series of five sub-reviews have been undertaken to address aspects of the original research question. In the present review, the literature-based literacies map describes the process of identifying, including and characterising the studies for one of the five sub-reviews.
Methods
Overarching descriptive map
Defining relevant studies for the descriptive map of the overarching review: inclusion and exclusion criteria
The earlier systematic review mapped the research on the impact of ICT on literacy learning in English, 5-16. The relevant research was searched for, located, sent for and mapped for the years 1990-2001. In addition to updating the searches for the period 2001-2002 and screening for inclusion of any potentially relevant studies for the period 2001-2002, all the included studies in the original map were re-keyworded, using revised generic and review-specific keywording sheets. The English Review Group working document for the inclusion and exclusion of potentially relevant studies was updated to reflect changes made to the keywording sheets, both generic and review-specific.
Defining relevant studies: inclusion and exclusion criteria
The English Review Group working document for the inclusion and exclusion of potentially relevant studies was updated to reflect the changes made to the keywording sheets, both generic and review-specific.
Identification of potential studies: search strategy
The potential studies for this review were identified through an updating of the original electronic searches and handsearches.
Screening studies: applying inclusion and exclusion criteria
The updated database for 2001-2002 that included potentially relevant studies published after October 2001 was screened by a member of the review team using titles and abstracts and the updated working document with inclusion and exclusion criteria.
Characterising included studies: EPPI Centre and review-specific keywording
All the studies included in the original database from the review of 2001 were re-keyworded by members of the Review Team using the new guidelines from the EPPI Centre. The studies retrieved for the updated database were keyworded by a member of the Review Team, with assistance from other members of the Review Team and the EPPI Centre where there was any doubt about keywording. The database was fully annotated with the keywords by another member of the team. For pragmatic reasons, the database for 2002 was closed on 30 November 2002. Any studies received after that time will be included in the next update.
Identifying and describing studies: quality assurance process
For the purposes of quality assurance, two members of the Review Team and one member of the EPPI Centre screened a random sample (10%) of the studies in the updated database. Screening was undertaken independently, using the inclusion/exclusion criteria working document. After double-screening, the inter-rater reliability scores were calculated using the Cohen's Kappa.
Sub-review on literature-based literacies
Screening and keywording
First, the updated (2001-2002) database for all the studies on the impact of ICT on literacy learning was screened. This database included all potentially relevant studies published after 2001 as well as those that appeared between 1990 and 2001. Then studies keyworded as literature-related were re-screened by Terry Locke and Richard Andrews. For purposes of quality assurance, two members of the review team and one member of the EPPI Centre screened a random sample (10%) of the studies in the updated database.
For keywording, a random sample from the updated database of 18 papers was double-keyworded by two members of the EPPI Centre. Two reviewers independently re-screened the studies retrieved from the database and then compared results. All disagreements were resolved without recourse to further EPPI Centre involvement.
In-depth review
Data-extraction was undertaken by two reviewers working independently, using the 2002 version of EPPI Reviewer. Any disagreements were discussed and resolved. Any included studies that were data-extracted in the 2001-2002 review on networked ICT were re-data-extracted.
The four criteria for weighing the evidence of research were used to gauge the quality of evidence: the methodological quality of each study (A), the appropriateness of research design for answering the review question (B), the relevance of the particular focus for addressing this question (C) and the overall weight of evidence in the light of responses to the first three criteria (D). In respect of the second criterion, a detailed list of issues to bear in mind was drawn up and is listed in the full report.
Narrative synthesis
A narrative synthesis was undertaken, including the weight of evidence judgements as set out above. No meta-analysis was possible in the case of this in-depth review because of the largely qualitative nature of the evidence. Publication bias was not addressed as it was not possible to draw a funnel plot on the basis of the evidence gathered.
Results
Overarching descriptive map: results
A total of 2,319 potentially relevant reports were identified for the current review. Of these 2,319 reports, 1,891 (just over 81%) were excluded by screening titles and/or abstracts and 428 were sent for. Of the 428 reports, 34 (fewer than 8%) were not received within the timeframe of the review or were unavailable. A reading of the full report resulted in the exclusion of a further 182 reports, leaving a total of 212 that met the criteria for inclusion in the mapping study.
Quality assurance results
Screening
The inter-rater reliability scores were good or fair.
Keywording - EPPI Centre generic keywording sheet
Inter-rater agreement was very high. Out of a total possible 180 'keywords', disagreement occurred in only 30 keywords (i.e. 16.7%).
Keywording - English Review Group ICT and literacy keywording sheet
Agreement was again very good. Out of a total possible 794 keywords, disagreement occurred in 88 cases (i.e. 11%).
Sub-review on literature-based literacies: results
Synthesis
Twelve studies were identified as relevant to the focus of the in-depth review on the impact of ICT on literature-based literacies. Of these, half were from the US and half from Australia/Canada. One-half focused on the primary years, with 40 percent on the secondary years and the others bridging the two phases. One-third of the studies explored conceptual and other relationships, while two-thirds were researcher-manipulated or naturally-occurring evaluations.
The twelve studies were reduced to seven on further analysis. Four studies were excluded as insufficiently focusing on literature-related aspects of literacy, and one on the basis that it was insufficiently focused on evaluation.
The weight of evidence of the remaining studies that formed the basis of the in-depth review was high to medium overall.
Conclusions: in-depth review
A common theme emerging from the research studied is that teachers matter more than technology. 'Impact' is mediated by teachers, and specifically by the discourses that teachers and students use. The actual outcomes of the two-way relationship between ICT and literature-related literacy learning are determined by a third factor: the ideology, values and practices of the teacher or teachers who set up the interaction.
Another firm finding is that there is a mismatch between commercially available multimedia literature software packages and response-based teaching. However, two studies indicate the possibility of multimedia literature-based software packages that encourage both literary reading and writing. A number of studies reviewed comment on motivational aspects of impact when ICT is introduced into the literature programme. There is, however, a suggestion that duration of exposure to a technology can affect motivation, and that there may be a connection between demotivation and cognitive aspects of what happens when readers engage with digital texts structured in certain ways. Other themes this review have identified are e-credibility and a need for more emphasis to be given in future research on literary writing (in contrast to literary reading) as a focus for classroom practice and research.
A number of studies specifically frame literature-based literacy within a response-based tradition and assume a degree of orthodoxy in this position, especially in the US. Others confirm a more problematic status for literature as a category and assume, for example, that it has expanded beyond print-based mediation. It is also a common thread that a carefully conceptualised account of literature-based literacy lends itself more easily to the identification of desirable learning outcomes, with both social and cognitive dimensions. A number of the studies suggested positive ways in which ICT can foster collaborative meaning making around texts. There was less deliberate focus on the ways in which cognitive aspects of textual response might change as texts become transformed technologically or in response to new, digitally mediated text forms.
On the basis of these findings, this report indicates the problematical nature of literature, the changing nature of all texts (including 'literary' ones) and textual practices under technological pressure, the problematic nature of narrowly conceived learning outcomes, and the mediation of pedagogical discourses. It concludes that these implications are relevant to policy, practice and the future research agenda.
This report should be cited as: Locke T, Andrews R (2004) A systematic review of the impact of ICT on literature-related literacies in English 5-16. In: Research Evidence in Education Library. London: EPPI Centre, Social Science Research Unit, Institute of Education, University of London.