This summary draws out some of the key messages from this systematic review for policy-makers in curriculum and assessment and teacher professional development. It gives the rationale for the review, signposts some of the features of the review which make it an important read in its own right, and identifies - alongside the policy implications the writers have identified from the review findings - some important factors to consider in operationalising them.

Background

The review was commissioned by the EPPI Centre from the Assessment and Learning Research Synthesis Group. The authors are Professor Wynne Harlen and Dr Ruth Deakin-Crick.

Rationale

It was important to conduct the review because:

• There is a growing array of programmes aimed at developing 'higher level thinking skills', ranging from the content of Curriculum 2000 to the aspirations of the OECD (McFarlane and De Rijker 1999, p 9) for lifelong learning. Some of these higher order metacognitive skills have been identified as critically important in formative assessment processes.
• There is a resource-hungry set of initiatives aimed at improving and widening the use of information and communication technologies (ICT) in teaching and learning. Much 2020 visioning for education lists creative and critical thinking high on the agenda along with the role of ICT.
• ICT plays a growing role in teaching and learning, and increasingly now in assessment.
• The development of online assessment and testing are of considerable interest to system designers as they are seen by some as having potential to solve the problems of our complex and failing labyrinth of examinations and marking - and to play a key role in a more reliable, much swifter as well as diagnostic and evaluative, national summative assessment system.

In short:

• higher order thinking is an increasingly valued purpose and outcome of our education system;
• 'what is valued has to be assessed' (this review, technical report: p 10) for both formative (improvement of learning) and summative (indication of progression and value) reasons;
• although 'in theory using ICT in assessment should have the same advantages and disadvantages as its use in teaching and learning' (ibid, p 15) there are specific issues related to assessment and testing which need to be specified in order to promote the best use of ICT in assessment of creative and critical thinking skills;
• we know from research on assessment (ibid, p 16) that if we do not get assessment design, feedback and use of assessment information with the learner right then it can have a negative impact on student learning and motivation (Harlen and Deakin-Crick 2002) and that when we DO get it right it can have a dramatic impact on improving pupil learning and achievement (Black and Wiliam, 1998; Black et al. 2002).

Scope of the review

The review focused on 12 studies from an initial trawl of 103. Two provided high-weight dependable evidence, but significant findings were also drawn from the other ten studies.

The authors reviewed literature on higher order thinking skills and on assessment research findings. This section of the report forms a useful summary of recent work in this area, which it notes has been described as a 'conceptual swamp' (Cuban 1984). The authors use the terms 'creative and critical thinking skills' to define the 'higher order thinking skills' the review focuses upon.

Findings

Overall the study shows that computers can enhance assessment in a number of ways:

• They can be used for assessment of creative and critical thinking skills through the use of concept mapping. They can improve performance through presenting abstract relationship and ideas in more concrete and manipulable forms such as concept maps. (One high-weighted paper by Osmundson et al. (1999) is worth reading in its own right on this).
• They can provide feedback and enhance later performance on the same material.
• They can promote greater confidence for teachers and allow them to focus feedback on the areas the ICT does not cover.
• They can present assessment problems in more visual, interactive and authentic ways which enable students to demonstrate higher attainment (although this affects the performance of girls).
• They can act to extend working memory, allowing the student to demonstrate higher levels of achievement than if s/he had to hold the complex problems in mind at the same time as processing them.
• Using ICT makes it possible to collect data on learners' processing which can be used diagnostically to improve later learning.

Performance is affected by pupils' attitudes towards and experience of computers.

Policy implications

The writers list the following findings for policy-makers to consider.

These include:

• The need for policy-makers to recognise the diversity and complexity of ICT in education and assessment and the problems created when it is treated as a single entity.
• The use of any form of ICT in education must start from consideration of how it can be integrated into learning, teaching and assessment.
• It is important firstly to develop a shared system-wide map of creative and critical thinking skills before then bringing into play knowledge of how ICT can be used in developing and assessing creative and critical thinking.
• Teachers, teacher educators and advisers need professional development in the role that ICT can take in assessing higher order thinking. Teachers are the route to better teaching, learning and assessment though ICT in the classroom and they must therefore have first-hand understanding and a repertoire of practice in practical ways if using ICT for these purposes.
• Teachers should be provided with ways and means of evaluating software in terms of its potential value in developing and assessing creative and critical thinking - especially in new areas.
• Importantly, with only 12 useable studies, the writers call for more research to be commissioned into this area.

System dilemma

The review findings reinforce a major dilemma for the system evident in other recent studies - namely that promotion of creative and critical thinking and a context of repeated high-stakes testing are incompatible.

Although it can be argued that assessment of creative and critical thinking skills promotes better teaching and learning of such skills, there is contrary evidence that teachers are capable of training pupils to pass any kind of tests. In addition, Harlen and Crick (2002) have provided evidence that high stakes testing is demotivating for many students and that currently there are too many high-stakes tests in the system. Black and Wiliam (1998) have further demonstrated that creative and critical thinking skills form a vital part of formative assessment itself. They have been careful to separate these from the formative use of summative assessment. This review provides evidence that ICT can help in storing and processing assessment information and data.

Policy-makers should explore ways of enabling ICT to improve the formative use of summative assessment of creative and critical thinking skills.

This will go some way to closing the gap inevitably created when, from September 2003, schools are provided with comparative analyses of pupil-level data which indicate questions they need to ask about relative progress of pupils in different subject areas.

Continuing Professional Development (CPD)

The call for professional development for teachers in this area is critical. CPD in this area is inevitably occurring in a context of rapid change. Students are still frequently more familiar and confident than their teachers with numerous computer applications (home PCs, computer games) which call on problem solving and other critical or creative thinking skills. Many young people however, do not have access to ICT outside of school - save through neighbourhood or city learning centres and homework clubs.

CPD will only work in the area of creative and critical thinking skills if advice to schools counters the current situation where teachers with responsibility for ICT in learning are often technically knowledgeable about ICT but less so about learning across the curriculum. Conversely, teachers with expertise in developing critical and creative thinking in areas of the curriculum are often not aware of the potential of ICT to support or assess these. Many schools have spent their National Grid for Learning (NGfL) resources on computer suites which allow whole-class teaching but which do not promote so easily curriculum-based independent learning to support learning and assessment of ongoing critical or creative thinking in the classroom.

Prior to CPD in use of ICT in the assessment of creative and critical thinking skills, it is necessary to create better understanding for education professionals of creative and critical thinking skills themselves across the curriculum. To design effective CPD, it will be important to investigate further work being done with concept mapping and ICT-based assessment of creative and critical thinking skills. This area encompasses ICT in an assessment process which is at once potentially both formative and summative.

CPD training can usefully begin in areas of creative and critical thinking where there is already a training and development focus and a developing technical lexis. One such area is in the critical thinking demands of assessment for learning.

CPD developers should aim to create CPD tools which embody and model the learning and ICT technological understanding and skills they are hoping teachers and other education professionals will gain in order better to build ICT-focused assessment into the development of creative and critical thinking skills.

Test and examination development

The review demonstrates that mistakes are already being made in this field. 'Where computers are used [in testing] they tend to mimic paper and pencil tests' (McFarlane and Rijke, 1999).

Although this review indicates that computer-based assessment can in some circumstances improve performance and be used to provide feedback which is helpful, this should not be taken as a green light for such system development. Online assessment and testing must have built into its specification the fact that it should demonstrably possess the outcomes for learning such as feedback and guidance, identified in the review.

Account must also be taken of the fact that there is evidence that high-stakes testing and frequent testing are demotivating learners.

References

Black P, Harrison C, Lee C, Marshall B, Wiliam D (2002) Working Inside the Black Box: Assessment for learning in the classroom. London: King's College.

Black P, Wiliam D (1998) Assessment and classroom learning. Assessment in Education 5: 7-74.

Cuban L (1984) Policy and research dilemmas in teaching of reasoning: unplanned designs. Review of Educational Research 54: 655-681.

Harlen W, Deakin Crick R (2002) A systematic review of summative assessment and tests on students' motivation for learning (EPPI Centre Review). Click here to read the review.

McFarlane A, De Rijke FJM (1999) Educational use of ICT, OECD Quality Assurance working paper for the Educational Software Working Group.

Osmundson E, Chung G, Herl H, Klein D (1999) Knowledge Mapping in the Classroom: A tool for examining the development of students' conceptual understandings. University of California, USA: Center for the Study of Evaluation.