Catch Up® 1 Numeracy is a one to one intervention for learners who are struggling with catchup.uk/portals/3/cu%20research/bjep919.pdf [27
54 KB – 32 Pages
PAGE – 2 ============
Education Endowment Foundation 2 The Education Endowment Foundation (EEF) The Education Endowment Foundation (EEF) is an independent grant – making charity dedicated to breaking the link between family income and educational achievement, ensuring that children from all backgrounds can fulfil their potential and make the most of their talents. We aim to raise the attainment of children facing disadvantage by: Identifying promising educational innovations that address the needs of disadvantaged children in primary and secondary schools in England; Evaluating these innovations to extend and secure the evidence on what works and can be made to work at scale; Encou raging schools, government, charities, and others to apply evidence and adopt innovations found to be effective. The EEF was founded in 2011 by lead charity The Sutton Trust , in partnership with Impetus Trust (now part of Impetus The Private Equity Foundation) , with a £125m grant from the Department for Education . With investment and fundraising income, the EEF intends to award as much as £200m by 2026. T ogether, the EEF and Sutton Trust are the G overnment – designated What Works C entre for I mproving E ducation O utcomes for S chool – A ged C hildren. For more information please contact: Robbie Coleman Research and Communications Manager Education Endowment Foundation 9th Floor, Millbank Tower 21 – 24 Millbank SW1P 4QP p: 020 7802 1679 e: robbie.coleman @eefoundation.org.uk w : www.educationendowmentfoundation.org.uk
PAGE – 3 ============
Education Endowment Foundation 3 About the evaluator The project was independently evaluated by a team from the National Foundation for Educational Research , led by Simon Rutt . Contact details: National Foundation for Educational Research The Mere Upton Park Slough SL1 2DQ p: 0 1753 637396 e: s.rutt@nfer.ac.uk
PAGE – 4 ============
Education Endowment Foundation 4 Contents Executive Summary .. .. .. . 5 In troduction .. .. .. 7 Methodology .. .. .. . 9 Impact evaluation .. .. .. 15 Process evaluation .. .. .. . 23 Conclusion .. .. .. .. 28 References .. .. .. .. 30 Technical Appendix .. .. .. 31
PAGE – 5 ============
Executive Summary Education Endowment Foundation 5 Executive Summary The project Catch Up ® 1 Numeracy is a one to one intervention for learners who are struggling with numeracy. It consists of two 15 – minute sessions per week, delivered by teaching assistants (TAs) . The approach is based on research indicating that numeracy is not a single skill, but a compo site of several component skills that are relatively discrete. The intervention breaks numeracy do wn into ten elements , including counting verbally, counting objects, word problems and estimation . Pupils are assessed on each component and instruction is targeted on those areas requiring development. In this evaluation, the intervention was run for 30 weeks and delivered to Year 2 – 6 pupils who were struggling with numeracy, as identified by TAs . The Catch Up Numeracy intervention was compared to who received the same amount of one to one teaching by TAs , but did not use Catch Up Numeracy. Those TAs delivering Catch Up Numeracy were supplied with detailed session plans and receive d three half – day training sessions , led by Catch Up and Dr Ann Dowker of the University of Oxford . The project ran from September 2012 to July 2013 . What impact did it have? was +0.21, meaning the programme led to a noticeable improv ement in numeracy outcomes. T his effect size suggests that, on average, pupils receiving the interventions would make approximately three additional months of progress over the course of the year compared to pupils that did not. However equivalent effect is likely to be a result of regular and sustained one to one teaching, rather than an intrinsic benefit of Catch Up Numeracy . The study demonstrate s that one to one tea ching with TAs is an effective strategy to increase numeracy skills in Year 2 – 6 pupils. Sub – group analysis did not identify any differential effects for pupil gender or eligibility for free school meals. The process evaluation indicated that most TAs value d Catch Up Numeracy and believed it had a problems. Group N umber of pupils Effect size Estimated m progress 95% confidence interval (CI) Evidence strength * Catch Up Numeracy vs . control 108 +0.21 + 3 0.01 – 0.42 Equivalent time one to one support vs . control 102 +0.27 + 4 0.06 – 0.49 * Evidence ratings are a new measure under development based on a number of factors including study type , size and drop – out. Ratings are provisional and are not given for sub – group analyses, which will always be less secure than overall findings. For more information about ratings visit: www.educationendowmentfoundation.org.uk/evaluation . 1 Catch Up® is a not – for – profit UK registered charity (1072425). Catch Up ® is a registered trademark
PAGE – 6 ============
Executive Summary Education Endowment Foundation 6 How secure is this finding? The evaluation was set up as an effectiveness trial to test the impact of Catch Up Numeracy in comparison to a ntervention group, with the developer leading the training and overseeing the provision of the intervention. Effectiveness trials aim to test whether an intervention can work at scale, in a large number of schools. The findings are based on a three – arm ra ndomised controlled trial using an intent to treat analysis (i.e. pupils were compared in the groups to which they were originally randomly assigned). Six pupils from each of 54 primary schools (two with two sites within the same school) were randomly assi gned to one of three groups: a control group that received normal teaching, a Catch Up Numeracy intervention two 15 minute sessions a week without Catch U p Numeracy, to replicate the one to one nature of the intervention. The primary outcome measure was numeracy ability, as measured by the Basic Number Screening test. Blind marking of test papers was undertaken. There was relatively low drop out, relatively evenly spread across the control and intervention groups. The main threat to the internal validity of this trial is the po ssibility that TAs delivering Catch Up Numeracy passed on knowledge of the intervention to those TAs in the group within the same school called cross – contamination. There is some evidence that the time equivalent group of TAs had some knowledge of Catch Up Numeracy and amended their approach in light of this knowledge, although it is unclear as to whether this had an impact on the results of the trial. Overall, the evaluators consider that the effect of one – to – one teaching is robust . H owever, the differences between the time equivalent group and the Catch Up group are harder to identify. The study findings are consi stent with the wider evidence base on one to one tuition, and a smaller number of studies evaluating the use of TAs for one to one support. The process evaluation revealed that there was some variance in the way in which the intervention was delivered, inc luding a failure to deliver two 15 – minute sessions each week for the full 30 weeks, as the trial intended. It is suggested successful implementation would benefit from TAs having sufficient time to plan and prepare for the sessions, with time scheduled spe cifically within the existing timetable. How much does it cost? The cost of the approach is estimated at £130 per pupil. This estimate includes resources (estimated at £2.00 per pupil), direct salary costs of TA (£95 per pupil), initial training (£17.50 pe r pupil) and on – going monitoring and support (£8.75 per pupil). Estimates are based on a school delivering the interven tion to 40 pupils and training two TAs and one teacher as the Catch Up Coordinator who supports the TAs but does not work directly with p upils. Key C onclusions 1. Within this trial, o ne – to – one support by TAs led to a significant gain in numeracy skills . 2. Catch Up makes similar significant gains , but t here is little evidence tha t Catch Up Numeracy provided any additional gains in numeracy outcomes over and above those from one to one teaching itself. 3. Schools can find it challenging to run two 15 minutes sessions per week , due to timetabling and other issues. 4. Structured interventions, such as Catch Up Numeracy, should be planned into the t imetable from the start of the new school year to ensure they are given priority and status.
PAGE – 8 ============
Introduction Education Endowment Foundation 8 Overall, the research behind Catch Up Numeracy is quite strong. The intervention is rooted in an evidence – based understanding of how children learn arithmeti c, and the only trial so far has shown promising results. The logical next step is a full randomised trial that includes enough pupils and TAs to detect educationally relevant effects with reasonable confidence . Evaluation ob jectives The primary research question was to identify the impact of the Catch Up N umeracy intervention on individual pupils over a 30 – week intervention period . Pupil numeracy skills were measured by performance on the Basic Number Screening Test. The process evaluation explore d the implementation and scalability of the intervention . Project team The internal evaluation team was led by Dr Ann Dowker of Oxford University and Dr Graham Sigley from Catch Up . These evaluators were responsible for the recruitment of schools and TAs , t he training of TAs to use the Catch Up intervention , the administration of the numeracy tests and undertaking follow – up sessions with TAs at the end of the intervention. The external evaluation team at the NFER was led by Simon Rutt, Head of Statistics . W hil e the overall proj ect and the impact evaluation were led by Simon Rutt the process evaluation was led by Claire Easton , a Senior Research Manager within the Research Department . The NFER team were responsible within the impact evaluation for the randomisation of TAs a nd pupils, the analysis of test data and the writing of a final evalu ation report . The NFER was responsible for the whole of the process evaluation and its contributi on to the final report. Ethical review The NFER has a well – developed Code of Practice that contains detailed ethical protocols . These protocols govern all research undertaken by NFER and the trial lies within them. Parent s gave active written consent for a ll eligible pupils put forward for the intervention and testing , and the Catch Up team confirm ed that consent had been received before continuation of the trial .
PAGE – 9 ============
Methodology Education Endowment Foundation 9 Method ology Trial d esign This was a multi – centre, three – arm , parallel – group pupil – randomis ed trial within schools in England . The trial involved 5 4 schools (two with two test sites within the same school) with two TAs and six eligible pupils within each school . Teaching assistants were randomised to one of two intervention groups : on e delivering a time equivalent intervention and the other delivering the Catch Up intervention . The s ix pupil s within each school were ran domly allocated to one of three groups : Catch Up , time equivalent and control . This design was developed from earlier evaluations that noted a Catch Up effect but could not differentiate this effect from the quality of the TA , or the potential bias in pupil allocation . The randomisation of TAs and pupils eliminate d th e s e potential source s of bias, whil e the time equivalent group of pupils will allow analysis to identify any Catch Up effects over and above the effects of one – to – one teaching. Eligibility Schools were selected and approached by Catch Up prior to the involvement of the external evaluator . These were originally planned to be around three main areas : Oxford, Southend and Thurrock . NFER therefore had no input into the early phases of the tri a l set up . Each school paid £ 275 to take part in the trial ; this is a little less than the normal cost of Catch Up Numeracy materials and training of £350 per person . The costs associated with the trial may result in a specific type of school that is predisp osed to the use of interventions , and one that is prepared to invest in such activities, agreeing to take part. However, this recruitment strategy has the advantage of being similar to reality since schools were charged a cost . The normal intervention process is for e ligible pupils to be chosen by the nominated individual after they had received training from Catch Up . For the purposes of the trial pupils were selected prior to any training. It was agreed that TAs would be able to identify those pupils who were struggling with numeracy and would benefit the most from the Catch Up intervention . This occurred before any randomisation so is free from bias that could impact on trial findings. Intervention Catch Up Numeracy is a one – to – one intervention for pu pils w ho are struggling with numeracy. It consists of two 15 – minute sessions per week which are usually d elivered by a trained TA outside of the usual teaching class. T o prepare them for delivering the intervention, TAs are supplied with detailed session p lans and receive three half – day training sessions. The intervention breaks numeracy that the tutor always addresses the exact area of weakness. The interv ention is made up of four key stages : assessment s for learning, identifying an appropriate focus, individual sessions and ongoing monitoring. These are outline d below. Stage one: a ssessment s for learning a bank of easy – to – administer assessments to determine what the learner can do and where their needs lie set ting the learner’s level and identify ing the appropriate starting point for the intervention.
PAGE – 10 ============
Methodology Education Endowment Foundation 10 Stage two: i dentifying an appropriate focus using the results of the assessments for le arning t o complete the Catch Up Numeracy learner profile using the learner profile to set the target level and identify an appropriate focus for numeracy intervention . Stage three: i ndividual sessions The trial instructed TAs to deliver two 15 minute sessions each week for a 30 week period . These 15 minute sessions break down into three sub parts; Stage four: ongoing monitoring The learner profile is revisited and the Catch Up Numeracy target level is reviewed. The r evi ew and introduction : r emind s the learner of what was achieved in the previous session and outline s the focus of the current session confirm s to the learner the number range being used introduce s and review s key vocabulary link s the focus of the current se appropriate . The n umeracy activity : gives an opportunity for the learner to work on the focus of the session in a range of learning styles; e.g. Visual; Aural; Verbal; Physical familiarise s the learner with vocabulary appropriate to the focus of the session . The l ink recording : reinforce s the skills learned during part two of the individual session, using writing/recording as an additional approach to learning provide s focused teaching based on observed miscues (i.e. mistakes). Below is a description of one of the Catch Up Numeracy activities . Cool fans! Tell the learner that the number fan is very good at making numbers but needs help to keep the numbers in order. Use the fan to make a 1, 2, or 3 digit number as appropriate for the number range . Write the number on the whiteboard . Repeat for two further numbers Ask the learner to point to the numbers in order of size, smallest first. Repeat the activity for other s ets of 3 numbers Repeat the activity, asking the learners to point to the numbers in order of size, starting with the largest number.
PAGE – 11 ============
Methodology Education Endowment Foundation 11 Pupils were assigned to one of three groups as part of the trial : a pure control grou p that received normal teaching; an equivalent time intervention group that received two 15 – minute sessions a week to replicate the one – to – one nature of the intervention ; and a Catch Up intervention group that received the intervention as described above . The equival ent time group could receive any form of numeracy support as long as it was not Catch Up and the TAs were asked to keep a log that recorded the actual focus of their sessions . It became evident during the process evaluation interviews and from manipulatio n check questionnaires that the some of the equivalent TAs deviated from the original delivery protocol . The interpretation section in the conclusion discusses this issue in more detail. Outcomes The primary outcome measure focused on numeracy ability , as measured by the Basic Number Screening Test 2 . Secondary outcome data was collected by Catch Up and the University of Oxford on reading ability, using the Salford Sentence Reading Test 3 , and on general ability, using the Non – Reading Intelligenc e Tests 1 – 3 4 . Dr Dowker and the Catch Up office recruited research assistants from am ong students at the University of Oxford to administer the tests . Dr Dowker trained the research assistants in test administration ; they were also prov i ded with the nec essary test manuals. The Catch Up office contacted each school and agreed a suitable date and time for a research assistant to visit the sc hool and administer the tests. The visits were organised on a geographical basis in order to minimise costs and the test schedule took place in as short a time period as possible . Pre – test s were administered between September and November 2012 with post – test s administered between June and July 2013 The research assistants then visited the schools and administered the tests . For the pre – intervention tests, the research assistant did not know which of the three groups the pupils had been assigned to and did not know who they were testing until they ar rived at the school . Where a pupil was absent, arrangements were made for a research assistant to re – visit the school, where possible . In a small number of cases, a Catch Up a pproved t rainer who had no connection with th is tri a l administered tests to absen tees. These trainers also administered the tests with no knowledge of group membership . A similar process was followed for the post – intervention re – testing. All test papers were sent to Dr Dowker who marked all the papers and generated the raw scores , su bsequent maths ages and standardised scores . Dr Dowker did not know the membership of each group until she had finished marking the post – test data . All data w as sent to NFER via a secure data transfer portal . The process of test administration and test ma rking was not evaluated by the external EEF evaluator but there is no evidence to suggest that this has not been conducted by following appropriate protocols . 2 http://www.hoddertests.co.uk/tfsearch/numeracy – maths/bnst.htm 3 http://www.hoddertests.co.uk/tfsearch/reading/nssrt.htm 4 http://www.hoddertests.co.uk/tfsearch/reasoning/nrit1 – 3.htm
54 KB – 32 Pages