PublicationsSystematic reviewsAssessment and ICTAssessment and ICT - summary
A systematic review of the impact on students and teachers of the use of ICT for assessment of creative and critical thinking skills. Summary

Background

The review reported here was prompted by the rapid changes associated with the 'information age'. New technologies have created both the need for education to provide students with what are described as 'higher level thinking skills' and the opportunity to teach and assess these skills. There is also evidence from two previous reviews of assessment (Harlen and Deakin Crick, 2002; Black and Wiliam, 1998) that, on the one hand, what is assessed for summative purposes is what is valued in the curriculum, and, on the other hand, that formative assessment of what is taught leads to improved learning. It follows that, if valued goals of education are to be taught effectively, they need to be assessed effectively for both formative and summative purposes. The reported neglect of creative and critical thinking in assessment is therefore a cause for concern, given the prominence it is accorded in current discussion of the education that students need in preparation for life in a rapidly changing society and for life long learning.

This review has focused on critical and creative thinking, since despite a lack of clarity as to the meaning of higher level thinking, there is a degree of agreement that critical thinking and creative thinking are key elements of it. Both are involved in problem-solving to varying degrees depending on the nature of the problem. The definitions adopted here are as follows:

  • critical thinking: involves the evaluation of arguments or propositions in relation to evidence, reasoning, drawing conclusions
  • creative thinking: involves relating together principles, ideas, information and entities in new and original ways to generate new entities or ideas.

The review was funded by the EPPI-Centre at the Institute of Education, University of London, and conducted on behalf of the Assessment and Learning Research Synthesis Group (ALRSG) by the authors, with the guidance of the Review Group and the participation of members at various stages.

Aims

The aims of the review were as follows:

  • to gather evidence of any impact on students and teachers of the practice of using information and communication technologies (ICT) in assessing creative and critical thinking skills for both formative and summative purposes
  • to determine the positive and the negative aspects of using ICT for assessing creative and critical thinking skills, in particular, any differential impact relating to student characteristics and experience
  • to make recommendations for policy and practice based on these findings
  • to identify questions needing to be addressed by research so that decisions on policy and practice relating to using ICT in assessment can be evidence-based.

Review questions

The review was designed to answer the following main question:

  • What evidence is there of the impact of the use of ICT for assessment of creative and critical thinking skills on students and teachers?

In order to achieve all the aims of the review, it was necessary to address the further subsidiary questions:

  • How does any impact vary with the experience of the students and the conditions of the assessment?
  • How does any impact vary according to whether the purpose of the assessment is formative or summative?
  • What are the implications for assessment policy and practice of these findings?

Methods

The review was conducted according to the procedures developed by the EPPI-Centre for the review of educational research. These procedures set out well-defined stages and require careful documentation of all decisions and actions. This means that the evidence base for the findings is made explicit and the review can be updated in the future, building on and extending what has already been done.

The stages were as follows: specifying the question and conceptual framework; developing the protocol detailing methods for the review; searching for studies; applying criteria to abstracts; obtaining full texts, keywording and summarising the characteristics of studies; applying refined criteria for final selection; extracting data and evaluating weight of evidence; synthesising; and consultation. Through these stages, the studies found were filtered using various screening processes in order to identify those which provided the most sound and relevant evidence for answering the review questions.

The search for studies was guided by selection criteria. Studies were sought that were written in English, reported on the formative or summative assessment of creative and critical thinking using ICT, and concerned students aged 4 to 18 in schools. The full texts of studies potentially meeting these criteria were obtained, read, again screened against the inclusion criteria; reasons for exclusion were recorded. Those meeting the criteria were categorised using a core set of keywords (EPPI-Centre Educational Keywording Sheet) and some review-specific keywords. This resulted in further studies being excluded, the reasons being recorded. The remaining studies were used in mapping the identified relevant research in terms of the keywords. These mapped studies were then subjected to in-depth study and analysis in which data were extracted using the Guidelines for Extracting Data and Quality Assessing Primary Studies in Educational Research (EPPI-Centre, 2002) and review-specific questions.

Judgements were made as to the weight to be attached to the evidence from each study in relation to its methodological soundness (as reported) and to the appropriateness of its design and relevance of its focus for this review.

Quality assurance procedures involved 20 of the studies being keyworded by two people, all data extraction being carried out independently by at least two people, and differences reconciled by discussion. The authors and four members of the Review Group took part in keywording and data extraction. Two members from the EPPI-Centre also carried out a quality assurance role in the keywording and data-extraction processes for a sample of studies.

The synthesis of information extracted from the studies involved identifying, in the selected studies, relationships between the affordances (that is the opportunities made available by the use of the information technologies), the processes and the outcomes of using ICT for the assessment of critical and creative thinking skills. The studies provided evidence of three types of impact: on (i) teachers' knowledge of students' processing and learning, (ii) their summative knowledge of students' attainment, and (iii) students' achievement or performance. The main findings were reported in relation to each of these impacts. Other outcomes, inferred or used as explanatory factors by the authors of the studies, but not based on systematic evidence, are reported as implications.

Results

Identification and categorisation of studies

A total of 103 studies were found as a result of the search for studies potentially meeting the inclusion criteria. Full texts were obtained for 94 of these, from which 62 were excluded because they did not meet all of the following criteria: concerned with assessment; ICT-based; concerned with students within the age range aged 4 to 18 in school; concerned with critical or creative thinking skills; reporting research. The remaining 32 studies were keyworded, a process that resulted in a further 20 being excluded, as detailed study showed that they did not meet the inclusion criteria. Of the 12 included studies, eleven were evaluations of situations set up by the researchers, of which four involved random allocation of students to groups.

Evidence of impact

Two of the 12 studies meeting the inclusion criteria provided evidence of high weight in relation to the review question, itself a significant finding of the lack of research focusing on this question and providing dependable evidence.

Evidence from the two high-weight studies was as follows:

  • Computer-based concept mapping with automated scoring can be used to provide summative assessment of critical and creative thinking about complex relationships.
  • The use of ICT can help teachers by:
    - storing and recording information about how students are developing understanding of new material;
    - taking over some of the role of assessing and providing feedback to students so that the teacher can focus on ways of supporting learning that are beyond the program's mediation.
  • Feedback from the computer during the use of test material improves student performance in later use of the same test material.

The following findings were from nine studies providing medium-weight evidence in relation to the review questions and should thus be treated more cautiously. Unless otherwise indicated, the evidence is derived from individual studies:

  • Use of computers to assess teamwork did not provide evidence that aspects of collaboration result in increased problem solving (as measured by computer-based knowledge-mapping).
  • Automated collection and scoring of the processes used in problem solving provided additional information relevant to problem-solving performance.
  • Using a computer program both to test and give feedback to students can increase the level of performance as compared with students taking the same tests on paper.
  • Using a computer-based program for solving problems of balancing chemical equations enabled students to achieve a higher level of performance than solving equations on paper.
  • A conventional multiple-choice test gave a false impression of understanding compared with the analysis of multimedia presentations on the same topic.
  • Subject matter used in computer assessment of problem solving affected the outcome for girls more than boys.
  • The performance of middle-school students was not improved by training in the use of computer graphics aimed at helping them make meaning of new material.
  • Use of a computer program involving diagrammatic representation provided useful information about students' causal reasoning in thinking through analysis of their diagrams but not from the log files of their computer moves.
  • There was conflicting evidence from two studies relating to the impact on performance of using the web to search for information.

Evidence of variation of impact relating to the conditions and purposes of testing and the experience of the students

  • Several studies showed that interacting with a computer provides feedback that supports better performance even if this only reflects back to the students the moves and links they made in a visual representation of relationships. These studies provide high and medium-weight evidence.
  • Computers were shown to provide information about processes in reaching a solution that gives additional feedback to students and teachers.
  • There was evidence that the choice of content influences the performance of girls to a greater degree than boys.
  • Working with a team at a distance was found not to improve performance in knowledge mapping.
  • Computer-based assessment enabled students to achieve at a higher level than in equivalent paper-based tasks.
  • Using technology probed students' understanding to a greater degree than conventional tests.
  • Students' experience with computers and attitude towards them can influence computer-based test performance.

Other impacts reported as explanatory factors of the effects of using ICT

The authors of several studies included comments on other impacts and outcomes of computer-based assessment, such as motivation, meaning making, awareness of learning processes and feelings of competence. Since none of the studies included quantitative evidence of these impacts or offered systematic qualitative evidence, the points made have therefore to be treated as tentative. However, they are included in the report because they point to some possible explanations of the impact of the computer on students' achievement and relate to the affordances of ICT.

  • Analysis of computer concept mapping suggests that students develop understanding incrementally, but there is a lack of evidence of students being aware of their learning processes.
  • Studies showed that real-life problems can be presented and represented in ways that call upon creative and critical thinking.
  • The visual representation afforded by the use of computer programs in several studies was suggested as supporting students meaning-making.
  • The concrete form of abstract relationships provided by computer programs, particularly in concept and knowledge mapping, was considered to be a factor in improved performance.
  • There were several studies which indicated that a range of products and processes can be used to assess creative and critical thinking, often through problem solving.
  • The use of ICT made practicable the collection and use of information about the moves and processes that learners use in tackling assessment tasks, thus enabling this information to be used in supporting learning.
  • The use of a computer program was considered to relieve demand on working memory by providing an 'external memory' for the student.
  • In several studies, authors reported information evidence of the use of computer programs resulting in improved motivation for the assessment task, either through the use of the computer per se or through the feedback it provided.
  • The impact of human mediation may be increased by computer mediation because the computer enhances the feeling of competence given by the teacher mediation.

Implications for policy, practice and research

In order to extend the thinking in relation to implications, a short consultation conference was held with researchers, practitioners and students of ICT in education in the UK. The outcomes of this discussion are included in the following list of implications.
 

Policy

  • The term ICT covers a range of diverse programs, applications, hardware and software, with varied affordances for assessment and for learning. It is important to recognise this and not to treat ICT as if it were a single entity.
  • The use of any form of ICT in education should start from consideration of how it can be integrated into learning, teaching and assessment.
  • Priority has to be given to conceptualising learning as having higher order thinking as key features as a basis for developing and assessing creative and critical thinking. Knowing how to use ICT should go hand in hand with understanding how to bring about higher level thinking.
  • There is need for professional development of teachers, teacher educators and advisers in the role that ICT in its various forms can take in learning, teaching and assessing higher order thinking.
  • The route to effective use of ICT for learning, teaching and assessment in the classroom is through the teachers. Thus it is important for teachers to recognise the affordances of ICT at first hand. Familiarity with using forms of ICT in their work and professional development to create and develop their own knowledge and understanding is a precursor to effective use of ICT in the classroom.
  • Teachers should be provided with ways of evaluating software for its potential in developing and assessing critical and creative thinking, and opening up aspects of the subject not available elsewhere.

Practice

  • Learners need to be made aware of creative and critical thinking as explicit learning objectives so that they can use the feedback provided by ICT for formative self-assessment.
  • Teachers should recognise their own role in exploiting the potential of ICT to provide intrinsic motivation for learning, through facilitating self-regulation, allowing learners more control of their learning and focusing their own role on aspects of mediation that are beyond the computer.
  • The role of ICT in facilitating reflection on learning processes - the development of metacognition - should be made explicit to learners.
  • Teachers need to be able to review software critically in relation to its potential for providing information for formative and summative assessment of higher level thinking.
  • Teachers need to be clear about the relationship/distinction between the roles that ICT can take in learning and assessment.

Research

  • The small number of studies identified, with a quarter of these emanating from one institution, is an indication of the need for a greater range of research studies on impact at the school level.
  • There is need also for studies with different foci, in particular on the impact of specific affordances, such as non-linear representations, or feedback to teachers and students.
  • More general studies of the affordances of ICT are needed, providing qualitative data as well as quantitative data that facilitate understanding of whether and, if so, how, critical and creative thinking can be developed and assessed.
  • It is important that all research is reported and not suppressed on account of lack of impact or commercial interests.

References

Black P, Wiliam D (1998) Assessment and classroom learning. Assessment in Education 5: 7-74.

EPPI-Centre (2001) Guidelines for extracting data and quality assessing primary studies in educational research (version 0.94). London: EPPI-Centre, Social Science Research Unit.

Harlen W, Deakin Crick R (2002) A systematic review of summative assessment and tests on students' motivation for learning (EPPI-Centre Review). Click here to see this review.

This report should be cited as: Harlen W, Deakin Crick R (2003) A systematic review of the impact on students and teachers of the use of ICT for assessment of creative and critical thinking skills. In: Research Evidence in Education Library. London: EPPI-Centre, Social Science Research Unit, Institute of Education, University of London.

  
Copyright 2019 Social Science Research Unit, UCL Institute of Education :: Privacy Statement :: Terms Of Use :: Site Map :: Login
About::Projects::Publications::Research Use::Courses & seminars::Resources::Databases::Blog