The continued development and improvement of Study Edge resources hinges on the efficacy of those resources and the results obtained. To ensure that the resources Study Edge provides improve student outcomes, measuring the effectiveness of these resources is critical. The research studies below show a wide-ranging set of positive outcomes utilizing Study Edge resources.
1. Increased Passing Rates for Re-Takers with Frequent Use over Multiple Years
A study was conducted to determine the impact of Algebra Nation on students who had previously failed the Florida Algebra 1 End-of-Course Exam in Spring 2015. A sample of 8,746 students who re-took the Algebra 1 exam in the 2015-2016 school year was analyzed and categorized based on usage frequency. Path analyses and logistic regressions were used to evaluate relationships between Algebra Nation usage indicators and Algebra scores, controlling for number of absences, free/reduced lunch eligibility, Hispanic/Latino origin, race, and gender. The results indicate that higher levels of logins, video views, and practice questions answered were related to higher scores when the students re-took the exam. Students who utilized Algebra Nation at least 30 times over the school year demonstrated an improvement in scores, including students who are typically most at risk for failure. There was some indication that higher use of the Test Yourself module in which students practiced solving problems similar to those they would encounter on the exam had more impact than viewing instructional videos alone (Test Yourself responses were related to increased odds of passing the exams, whereas video views were not). Regarding students who historically struggle in End-of-Course Exams, the results indicate that struggling students with higher usage of Algebra Nation scored better on the exam than did comparable students with less usage.
Leite, W. L. , Cetin-Berber, D. D., Huggins-Manley, A. C., Collier, Z. K., & Beal, C. R. (2019). The relationship between Algebra Nation usage and high-stakes test performance for struggling students. Journal of Computer Assisted Learning, 35(5). doi: 10.1111/jcal.12360
2. A Comparison Between Low-Usage and High-Usage Schools in Florida
An analysis of school-level Algebra Nation usage and student achievement in Algebra data was conducted to determine what differences exist in student performance between low-using and high-using schools in Florida during the 2014-2015 school year. Usage data from the Algebra Nation platform compared with Florida Algebra 1 End-of-Course Exam passing rates from Spring 2015 test administration showed that high-using schools had an average Algebra 1 End of Course exam pass rate of 83%, a 20-point higher average than low-using schools’ rate of 63%.
Collier, Z. & Leite, W. (2017). A comparison of three-step approaches for auxiliary variables in latent class and latent profile analysis, structural equation modeling. A Multidisciplinary Journal, 24(6), 819-830. doi: 10.1080/10705511.2017.1365304
3. An Evaluation of Study Edge’s Resources in South Carolina
A yearlong evaluation of Study Edge’s Algebra Nation use and its impact in South Carolina was conducted by an independent team of educators under contract with the South Carolina Education Oversight Committee. Data gathered from a survey of teachers using Algebra Nation as well as classroom observations of teachers’ use with students indicated that the materials positively impacted the rigor and quality of instruction. Teachers also cited how Algebra Nation materials allowed them to provide meaningful instruction for periods of time when school was canceled or when they were absent for health or professional reasons. Additionally, 80% of the surveyed teacher respondents felt that the use of the Algebra Nation materials had a positive impact on their students’ learning of algebra, and 90% of teachers surveyed felt that providing South Carolina teachers with access to Algebra Nation should be a high or moderate priority.
Dickey, E. (2018). An evaluation of Algebra Nation in South Carolina: 2017-2018. Unpublished Manuscript. South Carolina Education Oversight Committee, Columbia, SC.
4. Overall Effect of Study Edge’s Algebra Nation as an Online Resource
Adoption of online resources to support instruction and student performance has grown with technological advances and increased standards for mathematics education. The objective of this analysis was to evaluate the effects of one online resource, Algebra Nation, on the performance of schools with respect to student scores and percentage of students passing the Algebra I End-of-Course Exam. Results indicate that the effects of Algebra Nation usage are both statistically significant and substantively important according to the What Works Clearinghouse criteria.
Mitten, C., Collier, Z., & Leite, W. (2015). Online resources: Do they affect student learning? Lastinger Center for Teaching and Learning, University of Florida, Gainesville, FL.
5. Effect on Math Pass Rates in Title 1 Schools
An analysis was performed for Title 1 schools in Florida using 2017- 2018 Algebra Nation usage and Algebra 1 End-of-Course Exam performance data to investigate whether Algebra Nation usage presents a similar effect on the passing rate for the Algebra 1 End-Of-Course Exam. The average passing rates for low usage users was 60.93 and 77.01 for high usage users. Thus, Title 1 schools frequently utilizing the Algebra Nation platform had proficiency rates 17 percentage points higher than low-usage Title 1 schools.
Qiu, Y. (2019). The effect of Algebra Nation usage on Title 1 school passing rates for the Florida Standards Assessment Algebra 1 EOC. Lastinger Center for Teaching and Learning, University of Florida, Gainesville, FL.
6. Impact on Passing the Algebra 1 End-of-Course Exam
An analysis of school-level Algebra Nation usage and student achievement in Algebra was conducted to determine what differences exist in student performance between low-usage and high-usage schools in Florida. Usage data from the Algebra Nation platform was analyzed for the 2017-2018 school year and compared to Florida Algebra 1 End-of-Course Exam passing rates from the Spring 2018 test administration. Schools identified as high-usage had a significantly higher mean passing rate comparing to low-usage schools. The average passing rates for low-usage schools was 59.06 and 85.70 for high-usage schools. Schools that used Algebra Nation frequently had a 26.6-point higher average End-of-Course Exam pass rate compared with schools identified as infrequent users.
Qiu, Y. (2018). Summary of analysis of Algebra Nation usage and Florida Algebra 1 EOC. Lastinger Center for Teaching and Learning, University of Florida, Gainesville, FL.
7. Analysis of Study Edge’s Algebra Nation Usage and Student Achievement
An analysis of 82 schools was conducted to determine whether a relationship existed between the use of Algebra Nation and student achievement on the State Algebra 1 Mississippi Academic Assessment Program (MAAP) test. High usage schools had an 8.6 percent higher pass rate than the state average. Schools with higher usage exceeded the State performance levels of “proficient” through “advanced”. Although schools with low usage exceeded the State passing rate, they performed below the State average in all other categories.
Seymour, D. (2017). Algebra Nation program evaluation: Fall 2016 status report. Mississippi State University Research & Curriculum Unit Research Report.
8. Analysis of Study Edge’s Algebra Nation Usage and Student Achievement
The successful Algebra Nation intervention found that a high usage of Algebra Nation correlated with a 5.3% increase in PSAT math scores for Detroit Public Schools students.
Sawilowsky, S. (2019). Statistical analysis of the 2018 Algebra Nation (AN) project in Detroit, MI. Unpublished manuscript. Department of Education, Wayne State University, Detroit, MI.
9. Effect of Teacher Experience with Study Edge’s Algebra Nation
A study analyzed how the Algebra Nation platform contributes to increasing student exam scores by examining user logs over a three-year period. The Algebra Nation platform was used by more than 36,000 students in the first year, increasing to nearly 67,000 by the third year, and therefore researchers were able to see how usage patterns evolved and influenced student performance at scale. The results show that as teachers became more familiar with the platform, student usage increased. Additionally, the findings showed that higher number of test-yourself questions answers were significantly correlated with higher scores.
Niaki, S.A., George, C.P., Michailidis, G., & Beal, C.R. (2019). Investigating the usage patterns of Algebra Nation tutoring platform. In The 9th International Learning Analytics & Knowledge Conference (LAK19), March 4-8, 2019, Tempe, AZ. doi: 10.1145/3303772.3303788
10. Instructor Presence Positively Impacts Learning
This study explored the impact of instructor presence on learning, visual attention, and perceived learning in mathematics instructional videos of varying content difficulty. Findings suggest that instructor presence attracted considerable visual attention. Additionally, instructor presence positively influenced participants’ perceived learning and satisfaction of topics and led to a lower level of self-reported mental effort for difficult topics.
Wang, J., & Antonenko, P. (2017). Instructor presence in instructional video: Effects on visual attention, recall, and perceived learning. Computers in Human Behavior, 71, 79-89. doi: 10.1016/j.chb.2017.01.049.
11. Eye Tracking Leads to Usability
Students generally found Algebra Nation, an online learning environment, user-friendly and were confident in using it.
Wang, J., , Antonenko, P., Celepkolu, M., Jimenez, Y., Fieldman, E., & Fieldman, A. (2019) Exploring relationships between eye tracking and traditional usability testing data. International Journal of Human–Computer Interaction, 35(6), 483-494. doi: 10.1080/10447318.2018.1464776
12. High Usage of Algebra Nation Makes Learning Algebra Easier
These research findings are mirrored in the positive student learning outcomes experienced by schools utilizing Algebra Nation the most. Schools with high Algebra Nation usage on average scored higher in the 2017-2018 Algebra 1 MAAP proficient and advanced levels compared to the state average. It is also clear from the results of the student survey that students who use Algebra Nation feel more confident using Algebra. Nearly 80% of respondents agreed that Algebra Nation made learning Algebra easier because of the way Algebra Nation demonstrates how to work out problems and explains key concepts.
Sibley, D., & Hauser, J. (2019). Algebra Nation Final Report. Unpublished manuscript, Research and Curriculum Unit, Mississippi State University, Starkville, MS.