Adult participation

A systematic review of effective strategies to widen adult participation in learning. Summary

Background

The consultation document on the widening adult participation strategy of the Learning and Skills Council (LSC), published in March 2003, emphasises the importance of having the best evidence, based on high quality research into what helps widening participation (Learning and Skills Council, 2003a). It notes that much research evidence on 'what works' exists, but that it has not yet been synthesised, so that we do not always 'know what we know'. The LSC proposed commissioning a systematic review on 'what works' in order to provide a solid foundation for future research on participation and strategies for widening it.

The Learning and Skills Development Agency (LSDA) organised an international research seminar on widening participation in 2001. In preparation for that seminar, it commissioned an independent review of literature on international evidence and practice in attracting new learners. The review was conducted by the Institute for Employment Studies (Hillage and Aston, 2001). The current systematic review takes account of what LSDA learned from this earlier, international work and is designed to contribute towards improving knowledge about the most effective means of widening participation, based on trustworthy evidence.

Aims

The review's aims are as follows:

to identify effective strategies for widening adult participation
to contribute to the development of an objective, comprehensive and authoritative basis for action by local LSCs to extend good practice
to support the LSC's national strategy for widening adult participation by providing a sound basis for improvements in practice at local LSC level
to contribute to the LSC's programmes of action research and quality improvement on widening adult participation

Review questions

The research questions are as follows:

Which strategies have been reliably proven to raise, or not to raise, participation in learning by adults with traditionally low participation?

Which strategies do, or do not, offer compelling evidence that they raise participation in learning by adults with traditionally low participation?

With reference to strategies that provide evidence of raising participation in learning by adults with traditionally low participation, how and why do these strategies work?

Methods

The review began by establishing a clear process for involving users, including policy-makers and planners who, via the Advisory Group, played a key role at the outset in deciding the scope and conceptual framework for this review.

The Review Group then agreed a set of criteria for including and excluding studies. To be included in the systematic map a study had to:

report on a strategy to widen participation in learning by adults, including strategies to reach and engage learners, help and support success of adult learners, and enable progression of learners
include adults who are educationally or economically disadvantaged and in either formal or informal learning
be an evaluation and published after 1992.

Effort was made to identify as many studies as possible undertaken from 1992 onwards that might answer one or both of the review questions.

Studies were identified by systematic searches of electronic databases and websites. In addition, relevant journals were handsearched and those with specialist knowledge in the field were contacted and asked to recommend relevant studies. As well as these methods, bibliographies in other reviews and relevant papers were examined to uncover studies not already identified.

A search log was created to record details of each search, and details of studies found were stored on two EndNote databases. The first contained details of all studies found and the second contained details of all studies that were judged to meet the review's inclusion and exclusion criteria on the basis of title and abstract only.

Great care was taken to define the criteria used to screen papers, so that those screening against inclusion and exclusion criteria would interpret them in the same way. To ensure that all reviewers were making the same judgements, they each screened the same first 30 independently. Where there were discrepancies, these were discussed. Checks of the decisions made on 20 random studies (1.9% of the total studies screened at full document stage) were made by a member of the Evidence for Policy and Practice Information and Co-ordinating Centre (EPPI Centre).

Studies meeting the review's inclusion and exclusion criteria were assigned descriptive keywords in order to produce a descriptive map of research activity. One generic set of core keywords was provided by the EPPI Centre and these were applied, using definitions specified in the EPPI Centre's Keywording Strategy. Additional keywords, specific to the context of this review, were devised and also applied.

A descriptive map was produced, which described research activity in the field, and this was also used to identify the most suitable subset of studies for answering the review questions. From 1,058 studies, 82 met the criteria for inclusion and were keyworded, using the EPPI Centre's generic keywording tool as well as specific keywords developed for use in this systematic review.

The decision was then taken to narrow the field to a coherent and manageable set of studies for the in-depth review. These formed a subset of studies that were relevant to the review topic, covering a wide range of topics and containing 'robust' evidence. Exclusion criteria were based on the review-specific keywords, to ensure the inclusion of a group of studies which were relevant to the review question.

To be included in the in-depth review, studies had to meet the following criteria:

They had to be carried out in a developed country.
They had to be coded with the topic focus of 'widening adult participation'.
They had to be coded as an external evaluation.
They had to to have among their objectives, at least three of the following from the objective of the intervention, including one from each of engagement and achievement and progression:
- Engagement intervention objectives (A.7.1 Outreach, A.7.2 Targeting, A.7.3 Stimulating demand, A.7.4 Information, advice and guidance)
- Achievement and progression intervention objectives (A.7.8 Retention, A.7.9 Achievement, A.7.10 Qualification or assessment reform, A.7.11 Progression)

Application of these criteria left 17 studies to be included in the in-depth review. These studies identified as meeting the criteria, were analysed in-depth using the EPPI Centre's software (EPPI-Reviewer) and data-extraction tool. EPPI Centre tools were used to make explicit the weight of evidence apportioned to each study.

The evidence was synthesised to group and amalgamate material from the studies which answered the review questions and which met the quality criteria for appropriateness and methodology. This was done by grouping studies according to three main dimensions of widening adult participation and by indicating, for each study in the in-depth review, the overall review-specific weight of evidence.

Data extraction and assessment of the weight of evidence brought by the study to address the review question were conducted by pairs of reviewers, working independently, and then comparing their decisions and coming to a consensus. A member of the EPPI Centre carried out a reliability check by extracting data from a subset of the sample of studies.

The first review question seeks reliable proof of success or failure in widening adult participation; and the second, 'compelling evidence' - a less demanding criterion, applied to studies that are judged by the reviewers on the basis of the relevance of the study and thus the study's ability to offer useful insights.

It was decided that the findings in this report would be based on studies that were judged to be of either a high or medium overall weight of evidence (WoE) (in data extraction terms, this is WoE D, and relates to the category 'compelling' in the review question). However, studies of low weight are also included since they provide supporting evidence and contribute to the overall conclusions. However, to be judged as offering 'reliable proof', the Review Group expected studies to be high-rated overall (WoE D).

Studies were considered to be 'compelling' using the weight of evidence (WoE) judgements applied to all studies included in the in-depth review. Two decisions were applied to studies based firstly on WoE C (relevance of the study to answering the review question): one decision for studies rated as high and one decision for studies rated as medium.

The following criteria applied:

Studies rated high on WoE C were judged as 'compelling'
Studies rated medium on WoE C and at least medium on D (overall weight of evidence) were judged as 'compelling'.

The idea of 'compelling' is situated around the relevance of the topic and the extent to which studies can offer some guidance on effective widening participation strategies. Studies with high WoE C would closely match our target population and would therefore automatically qualify as providing some compelling guidance for policy and practice in the UK. Studies judged as medium WoE C would be less closely matched to our target population of under-represented groups, who have traditionally low participation and would need stronger methodological/internal coherence in order to provide 'compelling' evidence (WoE D).

Results

The 17 studies meeting the criteria for the in-depth review contained a higher proportion of British studies compared with those in the systematic map. The criteria for inclusion favoured interventions which addressed engagement, achievement and progression, and a high percentage of studies in the in-depth review were about interventions related to targeting, stimulating demand, achievement and particularly progression.

In general, the in-depth review was representative of populations in the map. However, there were sizeably higher proportions of studies focusing on those who were reluctant learners, unemployed, not in education, employment or training (NEET), or those who were homeless.

The weights of evidence for each of the 17 studies included in the in-depth review were as follows:

Two studies received a high rating.
One study received a medium-high rating.
Seven studies received a medium rating.
One study received a medium-low rating.
Six studies received a low rating.

It should be remembered that the findings of low weight of evidence have to be treated with caution due to methodological weaknesses. However, in a large majority of cases, studies of low weight of evidence did not differ greatly in their conclusions from the medium- and high-weighted studies.

The findings of each study were related to one or more of the three key aspects of widening participation:

outreach, targeting and engagement
participation and retention
achievement and progression

The evidence from six studies that provided evidence on the issue of outreach, targeting and engagement, and which were rated medium and low (on WoE D), suggests that the strategies which appear to widen participation in this area share some of the following elements:

Presence of initiatives within the community through outreach work, and more specifically, person-to-person recruitment (word of mouth), is more likely to attract potential learners from minority communities. This finding emerges from Tyers et al. (2003), rated medium, and is also present in Field et al. (2001) and HA Associates (2002) work, both rated low.
There is evidence to suggest in the study by Tyers et al. (2003), rated medium, that tailored flexible support and provision created through networking and partnerships between key organisations which is responsive to individual learner needs is more likely to engage 'hard to reach' learners. This is also supported by the work of two low-rated studies by HA Associates (2002), and Hawaii University and College of Manoa (1992).
A sound understanding of the needs of target group(s) and clarity about the provider can go some way to pre-empting disengagement. Additionally, funding projects to target the needs of certain 'hard to reach' groups can be successful in engaging learners from these groups. This finding is from Tyers et al. (2003), rated medium, and was also evident in Field et al. (2001), HA Associates (2002) and Squirrell's (2001) studies, all of which were rated low.
Catalysts, such as intermediary bodies (e.g. Basic Skills Agency (BSA) and the National Institute for Adult and Community Education (NIACE)) or trade unions can play a key role in engaging new learners, through developing effective partnerships with community-based organisations and employers. This finding was drawn from Shaw and Armistead (2002), rated medium; it was also a finding of the study by Field et al. (2001), rated low.

The evidence from six studies that provided evidence on the issue of participation and retention, and which were rated medium-high to low (on WoE D), suggests that the strategies which appear to widen participation in this area have some of the following themes in common:

One of the findings from Paris (1992), a medium-high rated study, found that a shared understanding between learners, learning providers and employers about what motivates learners and what are their key barriers to accessing learning opportunities can be used to create responsive learning opportunities and shape learning provision to encourage participation and retention. This also emerges out of the Grief and Taylor (2002) study, rated medium.
The delivery method, the nature of support offered, and the suitability of a learning programme's design to the needs of the learners together with the characteristics of the staff delivering the programme (experience, knowledge of target community) seem to be some of the important ingredients which can enhance learner participation and retention. This finding emerges from Paris' (1992) study, rated medium-high; it also emerges from the studies of Grief and Taylor (2002) and Young et al. (1995), rated medium, and from the studies by Field et al. (2001), and Hawaii University and College of Manoa's (1992), both rated low.
Some evidence shows that provision of funding which can be used flexibly to support additional costs incurred by learning providers can help to set up and sustain innovative learning provision for hard to reach groups. This finding comes from the studies by Grief and Taylor (2002), and Young et al. (1995), both of which were rated medium.
Evidence from Young et al.'s (1995) study, rated medium, indicates that sustained attendance in a learning programme appears to depend on the appropriate level of support available to learners, in accordance with their needs, during the early stages after enrolment; and the efforts made by the providers to link the learning programme to outcomes desired by the learner. This finding is also backed up by findings in the studies by Robinson and Hughes (1999), rated medium-low, and Field et al. (2001), rated low.

The evidence from ten studies that provided evidence on the issue of achievement and progression, and which were rated high to low on WoE D, suggests that the strategies which appear to widen participation in this area have some of the following features in common:

Tailored learning programmes which address learners' desired outcomes; skilled trainers; clarity about the expectations of the learners and the extent to which they can be met by the learning programme, together with appropriate and sensitive assessment of progress, can help learners to achieve accreditation and/or progress to suitable employment, or other learning programmes. This finding emerges from the BMRB study (2001), rated high, and the study by Taylor (2002), rated medium. It also emerges from the studies undertaken by Coats (1999), Field et al. (2001) and McRoberts and Leitch (1998), all of which were rated low).
The findings from St Pierre et al. (1998) found that a high level of learner support, especially to those from 'hard to reach groups'; skilled and experienced staff in the provider organisation; and effective networking and collaboration between learning provider and local agencies to improve learning pathways and support progression, can all contribute to positive outcomes for learners. Similar findings are also reported by Taylor (2002), rated medium.
Embedding or tailoring basic skills training to the needs of employers and employees in workforce development programmes can lead not just to improvements in learners' self-confidence and self-image, but also to improving the quality of their work, and their economic position. This finding emerges from three medium-rated studies by Paris (1992), Scheer (1993), and Zandniapour and Conway (2001).
Information, Advice and Guidance (IAG) services can have a useful role to play in the planning of learning provision and progression, both in the community and the workplace settings; and also in identifying smaller steps which learners can take to gain confidence and achieve substantial and measurable progress. This finding comes from two medium-rated studies by Taylor (2002) and Tyers et al. (2003).

Themes that appeared to be important in the success of interventions to widen adult participation in learning were then identified. Each of the seven themes are highlighted in at least two studies as being important success factors. At least one of the two studies, on which the seven themes are based, are rated as either high, medium-high, or medium on WoE D:

Sufficient, suitable resources including quality support services: Young et al. (1995), Tyers et al. (2003) - both medium-rated studies.
Effective use of resources and good management of interventions: Grief and Taylor (2002), Taylor (2002), and Tyers et al. (2003) - all medium-rated.
Suitable ways to measure learning gains: Taylor (2002) - medium-rated.
Listening to learners, responding to feedback, encouraging realistic expectations about what learning programmes offer: BMRB (2001) - high; Grief and Taylor (2002) and Tyers et al. (2003) - medium.
Steps that break down barriers to learning: Paris (1992) - medium-high, Grief and Taylor (2002), Tyers et al. (2003), and Young et al. (1995) - medium-rated.
Flexible and tailored delivery and support: Paris (1992) - medium-high; Grief and Taylor (2002), Scheer (1993), and Tyers et al. (2003) - all medium-rated.
Networking and partnership, including the use of intermediary organisations: Grief and Taylor (2002), Shaw and Armistead (2002), Taylor (2002), and Tyers et al. (2003) - all medium-rated.

Conclusions

No studies provide evidence of strategies that can be confidently described as 'reliably proven' to raise, or not to raise, participation in learning by adults. Thus, the review confirms the earlier finding by Hillage and Aston (2001) that there is a shortage of effective evaluation evidence in this field.

The following conclusions are drawn from the 17 documents subjected to in-depth review:

Twelve studies provide evidence that is compelling. (The criteria used to make this judgement are provided in the 'Methods' section of this chapter). Of these 12 studies, two studies (rated high on WoE D) show that the programmes they evaluated failed or provided outcomes that are inconclusive, or that the interventions were not of great help to participants (BMRB, 2001; St Pierre et al., 1998).
Ten studies (eight medium and two low-rated studies on WoE D) provide compelling evidence on strategies to widen participation, but each is marred to some extent by methodological and reporting weaknesses: Grief and Taylor, (2002), Paris (1992), Scheer (1993), Shaw and Armistead (2002), Taylor (2002), Tyers et al. (2003), Young et al. (1995), and Zandniapour and Conway (2001) - all rated medium; Field et al. (2001) and Squirrell (2001)- both rated low on WoE D.
Five studies cannot be confidently described as offering compelling evidence: Coats (1999), Hawaii University and College of Manoa (1992), McRoberts and Leitch (1998), Robinson and Hughes (1999) and HA Associates (2002) - all rated low on WoE D.

Based on the analysis of studies included in the in-depth review, the most promising strategies (emerging from high, medium-high or medium-rated studies on WoE D) to raise participation in learning by adults with traditionally low participation appear to involve:

a substantial degree of flexibility in learning provision and support services, tailored to learners' needs (e.g. Tyers et al., 2003; Grief and Taylor, 2002).
programmes tailored to the needs of employees and the workplace, including occupationally specific learning (e.g. Paris, 1992; St Pierre et al., 1998).

These findings echo the results of previous studies that call for 'appropriate, targeted provision' (Macleod, 2003 p 16). It is also in tune with the learner-centred approach advocated by the LSC's widening adult participation strategy, Successful participation for all (Learning and Skills Council, 2003a), which proposes action to support learner interests, promote demand for learning opportunities, develop the supply of diverse learning opportunities, and create 'a learning environment for adults'.

It is therefore not surprising to note that the outcomes of interventions to widen participation are rarely clear-cut and that success or failure can be the product of interaction between many different factors.

Implications

National and local LSCs may wish to consider the following interventions:

Work with providers to increase the flexibility and tailoring of provision and support
Steps to ensure adequate funding and resourcing of interventions to widen adult participation
Work with providers to ensure the effective management of such interventions
Incorporation of widening adult participation strategies into workforce development programmes
Piloting of interventions to widen adult participation, coupled with evaluation strategies that enable impact to be measured with confidence.

Learning providers may wish to consider:

Steps to increase flexibility and tailoring of provision and support, including provision tailored to workplace requirements
Steps to ensure the effective management of widening participation interventions.

The research community may wish to consider:

What changes might be needed in research design to ensure that the methods adopted are strong enough to yield evaluation evidence suitable for informing policy and practice
Steps to ensure adequate recording and reporting of research methodology, evidence and results, to ensure that useful findings and conclusions can be used with confidence by planners and policy-makers
Further use of the findings of this review. For example, the tables in Chapter 3 of the technical report highlight potential gaps in research evidence and could help inform decisions on future research priorities in this field.

Strengths and limitations

Strengths

These can be summarised as follows:

a rigorous search strategy and systematic methodology for assessing the value of studies, using procedures developed by the EPPI Centre
the high volume of references identified and keyworded
a review that is informed by the LSC widening adult participation strategy, providing a clear context
confirmation that there are few sound evaluations available in the English language in this field.

Limitations

It is worth noting that the following limitations arise from the literature available for this review:

lack of studies that evaluate the impact of interventions by comparison with a baseline or control group
lack of testing of recommended practice
the extent to which lessons are transferable to the UK context
weaknesses in the reporting of method and evidence
only including references and studies in the English language, meaning that effective strategies reported in other languages have been overlooked.

It may be argued that the limitations raised mean that there are concerns about the usefulness of the findings drawn from this review. However, their value lies in the fact that the features or common themes presented in the syntheses under each of the three key aspects of widening participation, and, indeed, in the section entitled Key issues emerging from the studies in the technical report, recur in more than one study, at least one of which is rated high, medium-high or medium. The key issues drawn from the evidence selected for the in-depth review, that comprise the various syntheses and highlight some of the critical ingredients required to formulate strategies for successfully raising participation in learning by adults, have emerged from studies that are rated either high, medium-high or medium. These have often been corroborated by other high-rated or medium-rated studies.

References

BMRB International Ltd (2001) Further Investigation of a Disadvantaged Group on Pre-Vocational Training. London: Department for Education and Skills. Available from: http://www.dfes.gov.uk/research/data/uploadfiles/RR253.doc

Coats M (1999) Lifelong learning policy and practice: the impact of accreditation on education and training provision for adult women in the UK. In: Alheit P, Beck J, Kammler E, Taylor R, Olesen HS (eds) Lifelong Learning inside and outside schools. Contributions to the Second European Conference on Lifelong Learning, Bremen, 25-27 February 1999, Collected Papers. Roskilde: Roskilde University; Bremen: Universität Bremen; Leeds: Leeds University, pages 586–609.

Field J, Spencer D, Tight M, Blaxter L, Byrd P, Merrill, B (2001) Evaluation of the Adult and Community Learning Fund. London: Department for Education and Skills. Available from: http://www.dfes.gov.uk/research/data/uploadfiles/RR284.doc

Grief S and Taylor C (2002) Final Report: Evaluation of the basic skills and ESOL in local communities projects. Coventry: Learning and Skills Council. Available from: http://www.lsc.gov.uk/National/Documents/SubjectListing/Research/Externalresearch/Final-Report_Evaluation-Basic-Skills.htm

HA Associates (2002) Evaluation of Pioneer and Pathfinder UK Online Centres: Follow-up study. London: Department for Education and Skills. Available from: http://www.dfes.gov.uk/research/data/uploadfiles/RR362.doc

Hawaii University and College of Manoa (1992) Skills Enhancement Literacy Project of Hawaii (II). Final Project Performance Report. Washington, DC: Office of Vocational and Adult Education.

Hillage J, Aston J (2001) Attracting New Learners: A Literature Review. London: Learning and Skills Development Agency.

Learning and Skills Council (2003a) Successful Participation for All: Widening Adult Participation Strategy for Consultation. Coventry: Learning and Skills Council. McRoberts R, Leitch R (1998) Targeting social need in a divided society: an evaluation of a community-based adult education initiative. Research in Post-Compulsory Education 3: 71–91.

Paris KA (1992) Evaluation of the Third Year of Implementation of the Wisconsin Workplace Partnership Training Program. March 1, 1991 through August 31, 1992. Washington, DC: National Workplace Literacy Program, Office of Vocational and Adult Education.

Robinson C, Hughes P (1999) Creating a Sense of Place: Indigenous Peoples in Vocational Education and Training. Adelaide: National Centre for Vocational Educational Research. Available from: http://www.ncver.edu.au/publications/382.html

Scheer RL (1993) READS Program. National Workplace Literacy Program. Final Report. Washington, DC: Office of Vocational and Adult Education, National Workplace Literacy Program .

Shaw N, Armistead C (2002) Evaluation of the Union Learning Fund Year 4, London: Department for Education and Skills. Available from: http://www.dfes.gov.uk/research/data/uploadfiles/RR378.pdf

Squirrell G (2001) Evaluation of DfEE Funded Pilots for Rough Sleepers: Off the Streets and into Work. London: Department for Education and Employment. Available from: http://www.dfes.gov.uk/research/data/uploadfiles/RR261.PDF

St. Pierre R, Gamse B, Alamprese J, Rimdzius T, Tao F, (1998) Even Start: Evidence from the Past and a Look to the Future. National Evaluation of the Even Start Family Literacy Program. Washington, DC: Planning and Evaluation Service, US Department of Education.

Taylor S (2002) Learning Pathways for Adults in Oxfordshire, Milton Keynes and Buckinghamshire. Summary Report. London: Learning and Skills Development Agency. Available from: http://www.lsda.org.uk/files/PDF/LPathways2002.pdf

Tyers C, Aston J, Barkworth R, Willison R, Taylor, J (2003). Evaluation of Adult Guidance Pilots. London: Department for Education and Skills. Available from: http://www.dfes.gov.uk/research/data/uploadfiles/RR491.pdf

Young MB, Fleischman H, Fitzgerald N, Morgan MA (1995). National Evaluation of Adult Education Programs. Executive Summary. Washington, DC: US Department of Education.

Zandniapour L, Conway M (2001) Gaining Ground: The Labor Market Progress of Participants of Sectoral Employment Development Programs. SEDLP Research Report No. 3, Washington, DC: The Aspen Institute. Available from: http://www.aspeninstitute.org/bookdetails.asp?i=&d=90

This report should be cited as: Taylor S, MacLeod D, Houghton N, Zwart R, Sachdev D (2005) A systematic review of effective strategies to widen adult participation in learning. In: Research Evidence in Education Library. London: EPPI Centre, Social Science Research Unit, Institute of Education, University of London.