PublicationsSystematic reviewsAssessment and ICTAssessment and ICT - Initial Teacher Trainer
A systematic review of the impact on students and teachers of the use of ICT for assessment of creative and critical thinking skills. User perspective: an ITT provider

The review methodology

This report has been set quite a difficult challenge in addressing a very complex question: What evidence is there of the impact of the use of information and communication technologies (ICT) for assessment of creative and critical thinking skills on students and teachers? There seems to be a very large number of factors involved in the review: 'students and teachers', ' ICT', 'assessment', 'creative and critical thinking skills' and, not least, 'impact'. The search for relevant research studies that bring together all of these factors necessarily made the range of studies examined rather limited once the rigorous exclusion process was completed. This does mean that the studies finally included will be of a high degree of relevance to the fairly narrow question, but also means that some interesting and important studies may have been excluded for not meeting all the criteria, even though they would have had some useful insights into this topic. This is a general feature of the EPPI methodology, but it seems that this study may have been affected more than some others because of the nature of the question examined.

The methodology employed involves making a wide-ranging search for studies meeting certain criteria. If a study seems to match the criteria judging from the title and abstract, the full text is obtained and categorised using agreed keywords, some general and some specific to the topic of the review. This process results in some more studies being excluded. The remaining ones (12 in this case) are analysed in depth and judgements are made as to the weight of evidence relevant to the review provided by each study.

This set of procedures is worthy of further examination. As an initial teacher training (ITT) provider (i.e. one who works with intending teachers on a PGCE course) I am concerned to induct future teachers into aspects of research, so that they will see their future roles as being active ones in the area of research - both as users and as possible practitioners. It is therefore important for my students (I will use the word 'student' to refer to an intending teacher and 'pupil' to refer to school learners) to be able to evaluate the research they come across - no easy matter when they are faced with conflicting results from studies with a range of different methodologies. This kind of review is therefore useful in providing an example of how research can be rigorously examined through the application of clear criteria. I might suggest that my students start with a slightly less complex question, but the general principle is an interesting one for them to work with. I would also add that they could learn from such a study that however rigorous one's methodology, one should treat outcomes with caution. In the case of this review, many of the findings are the outcomes of individual pieces of research and it is important to remember how these outcomes will have been affected by the context of the research and therefore need to be critically examined, even after the thorough and rigorous scrutiny of the EPPI review procedures.

Another interesting factor involved in this review is the focus on creative and critical thinking. These may not be skills that we immediately associate with many ICT applications and it is valuable to broaden our thinking to recognise what ICT has to offer in this area. I would be encouraging new teachers to be examining their use of ICT with pupils to ensure that it does offer scope for creative work - the further step to assessment of these skills using ICT is an extra level of complexity that I would address as students' skills in this area develop. It is valuable to think of ICT as having a role in this area, as students' perspectives on assessment can be quite narrow and summative - this is an opportunity to encourage more pupil-focused approaches to assessment to support teaching and learning.

The findings

So to some of the key findings which might have an impact in work with new teachers. Some findings indicated increased levels of pupil achievement through use of specific pieces of software, for example that feedback from the computer during assessment was associated with improved performance in subsequent assessment of the same kind. Such reported outcomes were in particular contexts, e.g. balancing chemical equations, and so we have to be cautious about generalising to other contexts. However, students can move from this finding to consider other contexts and examine for themselves whether similar outcomes might occur.

Other interesting findings include ones which should lead students to question the type of assessment methods they use and the results they get. For example the finding that a conventional multiple choice test gave a false impression of understanding compared with the analysis of multimedia presentations on the same topic, and another that the subject matter used in assessing problem solving by computer affected the performance of girls more than boys, have implications for considering the 'fairness' and accuracy of multiple choice assessment and possible impacts of pupil gender on outcomes. Feedback is another factor which is important to work on with students. It has previously been held that, for some pupils, feedback from a computer can be more effective as it is seen as less judgemental, personal and critical than that of a teacher. One study in this review offers support to this view, although without ascribing reasons for this.

Whilst there are clear findings to show that it is feasible to use computer programs, with automated scoring, to assess critical and creative thinking, the picture is not, however, clear and unambiguous as there are examples of studies where the findings are not all positive in terms of pupils' learning - so we have to remember the effect of context and the subtlety of some of the results obtained in these studies. It is important for users of ICT to remain critical and questioning of the choices they make in the material they offer to pupils and the way pupils interact with it. The motivation factor of working with computer-based materials is one which is identified in some studies (and positively motivating pupils to be engaged with assessment is pretty useful!) but again all the factors already mentioned have to be considered in making decisions about what is appropriate for particular pupils.

I have been commenting throughout on what I think the impact of this review might be for me as a teacher educator. This can be summed up by saying that students should be encouraged to use a review like this to consider critically their own developing practice in this area. One important and valuable aspect of the study is its focus on creative and critical thinking - these are skills we would want developed in our teachers as well as in pupils and such a study highlights some of the possibilities. The wider role of metacognition (i.e. awareness of and reflection on one's own learning) is a key to developing effective, independent learners and this study highlights the role ICT can have in both developing and assessing this attribute.

Implications

The set of implications identified by the review, separated out into those for policy, practice and research, seem to offer an important and practical agenda for work by those in various roles in education. For me, the set on practice summarise neatly what I have been commenting on above:

  • 'Learners need to be made aware of creative and critical thinking as explicit learning objectives so that they can use the feedback provided by ICT for formative self-assessment.
  • Teachers should recognise their own role in exploiting the potential of ICT to provide intrinsic motivation for learning, through facilitating self-regulation, allowing learners more control of their learning and focusing their own role on aspects of mediation that are beyond the computer.
  • The role of ICT in facilitating reflection on learning processes - the development of metacognition - should be made explicit to learners.
  • Teachers need to be able to review software critically in relation to its potential for providing information for formative and summative assessment of higher level thinking.
  • Teachers need to be clear about the relationship/distinction between the roles that ICT can take in learning and assessment.'

The role of the teacher is crucial in making principled and practical decisions about how ICT can support their pupils' learning in these areas. ICT is not a 'quick fix' but a tool which teachers need to learn to use like any other. It does however have potential in areas which we might not at first consider and this review helps us to open our thinking to that wider potential. As a researcher too, and as an educator wanting to encourage new teachers to think of themselves as researchers, I think there is also a powerful message in this review about what is needed in the further development of knowledge about effectiveness of ICT use. Several research priorities are identified and I would see them as markers against which to consider my own interaction with relevant ICT applications so that I can begin to consider how our understanding in these areas can be improved.

  
Copyright 2019 Social Science Research Unit, UCL Institute of Education :: Privacy Statement :: Terms Of Use :: Site Map :: Login
About::Projects::Publications::Research Use::Courses & seminars::Resources::Databases::Blog