Jump Trial
Kerrie Gingrich
Copyright: 2019 Solomon et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Funding: This work was supported in part by funding and in-kind support from Jump Math, a registered Canadian charity, and by funding from the Institute of Education Sciences, U.S. Department of Education, through Grant R305A120184 to the Hospital for Sick Children. The opinions expressed are those of the authors and do not represent views of the Institute or the U.S. Department of Education. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Efforts to improve mathematics achievement have focused increasingly on the early school years as early difficulties in mathematics have been linked to later academic and professional success. Individual differences in mathematical skill are already apparent upon Kindergarten entry and predict later academic achievement more strongly than early reading, attention, or socioemotional skills [9]. Persistent difficulty in mathematics is associated with lower rates of high school graduation and college entry and high school mathematics achievement predicts college graduation, career earnings and earnings growth [10, 11].
We investigated the effectiveness of JUMP Math (JUMP), a distinctive approach to math instruction, developed by a Canadian mathematician based on a wealth of experience working with children with diverse math skills and challenges [18, 19]. The program is fully developed for K-8 and currently reaches more than 210, 000 students worldwide, with rapidly growing interest in Canada and the United States. It has been adapted to meet regional curriculum expectations (e.g. there is an American common core version), translated into multiple languages including French, Spanish and Bulgarian (translation into Inuktitut, a Canadian indigenous language, is also underway), and aims to be affordable (JUMP Math is a registered charity). Informal study in diverse settings suggested the potential for a positive impact on math learning [19]. JUMP instruction includes problem solving but the program is distinct from problem-based learning, the prevailing approach to math instruction in a number of countries, and its key ideas are empirically supported.
To elucidate the nature of JUMP instruction, it is instructive first to consider the central features of problem-based learning. Table 1 provides a summary of the key contrasts between the two curricula.
In a typical problem-based lesson, the teacher presents a real-world word problem that is open-ended (has multiple possible solutions). Students then work collaboratively, in small groups, to solve the problem using a variety of materials and recording their work using graphics, symbols and text. Teachers circulate to check on progress, facilitate thinking and provide encouragement. The class then reconvenes and the teacher facilitates discussion as the groups share their problem-solving strategies and solutions.
The pilot study involved grade 5 teachers and students in a rural school board (district; SB1) in Ontario, Canada. Participating schools were randomly assigned to use either JUMP Math (JUMP) or their business-as-usual approach to math instruction (SB1) for about 5 months, including mandatory school holiday breaks of about 3 weeks. We tracked student progress in math achievement, collected demographic data from the teachers and solicited feedback regarding their experience working with their assigned curricula (see S1 Text).
The Research Ethics Boards (REB) at the Hospital for Sick Children, Toronto, Ontario, Canada and the participating school board approved the study procedures. Participation was voluntary. We obtained written, informed consent from parents and teachers prior to random assignment to curricula and assent from the children prior to testing. Other than the JUMP Math materials distributed to teachers in the JUMP group (at no cost), teachers did not receive any compensation for participating.
Twenty-one principals consented to participate. Schools were randomly assigned to either the JUMP group (11 schools) or to the SB1 group (10 schools). However, 2 schools in the SB1 group declined after random assignment, which together with variation in the size of the participating schools, resulted in an imbalanced number of participating teachers and students in the two groups. The elementary student population in the participating school board was predominantly white, with middle to lower middle class backgrounds and low average, to average academic achievement (see Table 2 and S1 Text).
Teachers were eligible to participate if they were accredited to teach in Ontario, in good standing, and did not plan to take a leave of absence during the study period. Eighteen JUMP and 11 SB1 teachers in the participating schools agreed to take part. As a group, the SB1 teachers had somewhat more teaching experience and somewhat stronger math backgrounds. (see S1 Text).
Teachers in both groups received 2 full days (1 in the fall, 1 mid-year) of professional development (PD), according to their assigned curriculum (JUMP or SB1). The content and delivery of the PD was determined by the JUMP organization and by the participating school board, respectively. Members of our research team attended all of the PD sessions to ensure alignment with assigned curricula. We asked teachers to use only their assigned curriculum for the study duration.
The math achievement outcome measures included the math fluency, calculation and quantitative concepts scales from the Woodcock-Johnson Achievement Battery (WJ-III [45]). We measured reading achievement with the letter word identification test also from the WJ-III, IQ with the Kaufman Brief Intelligence Test (KBIT; [46]), and verbal working memory with the backwards version of the non-word letter span test adapted from the Wechsler Intelligence Scale for Children (WISC-III [47]). We assessed math and reading achievement at baseline and post-intervention, and IQ and working memory at baseline only (see Table C in S1 Text).
Students were tested individually in a quiet area in their school by a local team of retired teachers who were familiar with the school board but blind to the study hypothesis and to random assignment to curricula. The study measures were administered in two sessions that took place on separate days. Breaks were given as needed.
Results are based on standard scores and therefore indicate progress relative to same aged peers, which is represented by the 0 line. Vertical lines indicate 95% confidence limits around mean change scores. P-values and effect sizes (ES) are for the difference between the group means. Vertical lines that do not intersect the zero line indicate mean change that is significantly different from expected change, based on available norms.
Note that effect sizes (ES) were derived from estimates from the repeated measures hierarchical regression models. We calculated standard deviation for the pre-post change by multiplying the standard error of the estimate of change for each curriculum by the square root of its degrees of freedom. Effect size is the change in the JUMP group minus the change in the SB1 group, divided by the pooled standard deviation. Hence, a positive value indicates better performance in the JUMP group, a negative value better performance in the SB2 group. An effect size of .25 is considered educationally meaningful [48]. A recent review of educational intervention research found that effect sizes are rarely as large as .30. Hence, values of .50 would be considered very large indeed [49].
Teachers in the JUMP group rated their experience teaching math during the study period significantly more positively than teachers in the SB1 group (mean ratings 7.2 SB1, 8.2 JUMP, p = .03). All of the teachers who used JUMP indicated that they would use it again and that they would recommend it to other teachers.
The Research Ethics Board (REB) at the Hospital for Sick Children, Toronto, Ontario, Canada and the participating school board approved the study procedures. Participation was voluntary. We obtained written informed consent from teachers; in year 1, prior to randomization to curricula and in year 2, after the curricula were already in place. We obtained separate written, informed consent from teachers for the videotaping. We obtained written, informed consent from parents for both years of the study and assent from the students prior to participation.
JUMP Math resources were distributed to teachers in the JUMP group at no cost. To help equate the distribution of new math resources across groups, all of the teachers in the SB2 group received the equivalent of CDN$250 in gift cards to purchase math-related classroom materials of their choosing. Funds were offered after teachers had already consented to participate in the study.
Elementary students attending the participating school board were predominantly white; 3% percent were born outside Canada, 5% had a non-English first language and 2% were English language learners (corresponding regional values 11%, 5%, 11%). Twenty-four percent were identified as requiring special education (regional value 19%, see note [50], [51]).
Teacher eligibility criteria were the same as for the pilot study; accredited to teach in Ontario, in good standing, and not planning to take a leave of absence during the study period. Members of our research team contacted eligible teachers either by email or in person and provided an overview of the study along with a detailed teacher consent form. Questions were encouraged and addressed immediately. In year 1, 115 teachers agreed to take part but 9 (2 JUMP, 7 SB2) declined following random assignment citing overwhelming schedules. Nine teachers who participated (5 JUMP, 4 SB2) left the study in year 1 due either to the end of a contract or for maternity leave and were replaced by new teachers. Thus, there were 106 teachers in year 1; 58 JUMP (31 grade 2, 27 grade 5) and 48 SB2 (26 grade 2, 22 grade 5). In year 2, 92 new teachers agreed to take part but 5 later declined (3 JUMP, 2 SB2) citing overwhelming schedules. However, 20 teachers who participated in year 1 also agreed to participate in year 2 (they taught grade 2 or 5 in year 1 and then inherited study students in grade 3 or 6 in year 2, in the same school). Ten teachers who participated (8 JUMP, 2 SB2) left the study in year 2 for the same reasons cited by teachers who left in year 1, and were replaced by new teachers. Thus, there were 107 teachers in year 2; 59 JUMP (34 grade 3, 25 grade 6) and 48 SB2 (27 grade 3, 21 grade 6). Replacement teachers were recruited to the study immediately and received a combination of the relevant PD and support implementing their assigned curriculum. As teacher changes tended to occur either relatively early or relatively late in the school year, student data were yoked to the predominant teacher and only the data from the predominant teacher were included in our analyses.
c80f0f1006