Identification of Comparable Elementary Schools:
Fact or Fiction?
Rita O'Sullivan
ritao@unc.edu
University of North Carolina - Chapel Hill
Brent Cooper, University of North Carolina - Chapel Hill, EvAP School of
Education, CB 3500, Chapel Hill, NC 27599
"cooperjb@email.unc.edu" <cooperjb@email.unc.edu>
Emily Cook, EvAP School of Education, CB 3500, Chapel Hill, NC 2759
emily cook <emilyemilyc@yahoo.com>
Mary Julia Moore, EvAP School of Education, CB 3500, Chapel Hill, NC 2759
"mjmoore@erskine.edu" <mjmoore@erskine.edu>
Beth Bader, EvAP School of Education, CB 3500, Chapel Hill, NC 2759
"bbader@email.unc.edu" <bbader@email.unc.edu>
Currently much emphasis is placed on the use of experimental or
quasi-experimental designs is education and social science research, which
require the use of equivalent control or comparison groups. When the school is
the unit of analysis, typically researchers “match” schools based on a variety
of criteria and then randomly assign one of the matched pairs into the treatment
and the other becomes part of the control groups. The effectiveness of this
procedure has been questioned in the past (O’Sullivan et al, 2003; O’Sullivan
et al 2005) and shown to be somewhat suspect with one large data set from an
urban school district in the United States. The purpose of this study is to
further expand the discussion around the usefulness of different matching
procedures to form equivalent groups. Using data from the North
Carolina Schools Report Card reports from 2001-06, this study will test the
following research questions for the four largest school districts in North
Carolina:
o Which predicators are the best for
determining equivalent school achievement?
o How well do the
best predictors of achievement hold up across multiple years to support school
equivalency?
Correlational analyses and group mean differences will be used to answer the
research questions, holding school districts constant and controlling for
school size (large, medium and small). Preliminary results suggest that
identifying equivalent schools, even in the same school district, can be
problematic.
Objectives
Currently much emphasis is placed on the use of experimental or
quasi-experimental designs is education and social science research, which
require the use of equivalent control or comparison groups. When the school is
the unit of analysis, typically researchers “match” schools based on a variety
of criteria and then randomly assign one of the matched pairs into the
treatment and the other becomes part of the control groups. The effectiveness
of this procedure has been questioned in the past (O’Sullivan et al, 2003;
O’Sullivan et al 2005) and shown to be somewhat suspect with one large data set
from an urban school district in the United States.
The purpose of this study is to further expand the discussion around the
usefulness of different matching procedures to form equivalent groups.
Using data from the North Carolina Schools Report Card reports from
2001-06, this study will test the following research questions for the four
largest school districts in North Carolina:
o Which predicators are the best for
determining equivalent school achievement?
o How well do the
best predictors of achievement hold up across multiple years to support school
equivalency?
Perspective or Theoretical Framework
According to past research,
equivalence is often assumed on the basis of only one or two school
characteristics. For example, Josendal, Aaro, Torsheim, and Rashbash
(2004 divided schools into groups to test the effectiveness of a school-based
smoking program. The only factor they took into consideration when
determining equivalence was school size. They assumed that as long as
schools were within plus or minus 10% of the same size, they were equivalent.
Another common practice used
when establishing equivalent schools is matching schools based on demographic
characteristics. Lynch, Taymans, Watson, Ochsendorf, Pyke, and Szesze
(2007) matched five pairs of diverse middle schools on several demographic
characteristics including: gender, ethnicity, free and reduced lunch status,
and English as a Second Language (ESOL) status. Riordan and Noyce (2001)
used a similar set of criteria to put schools into comparison groups based on
the following demographics: (1) Eligibility for Free/Reduced Lunch, (2) Limited
English Proficiency, (3) Special Education, or (4) Ethnicity (Asian, Black,
Hispanic, White) as well as prior school performance. Using a combination
of demographic variables and prior school performance, as indicated by
achievement scores appears to be one of the most common methods of establishing
equivalency in school-based research. Both McDougall, Saunders, and
Goldenberg (2007) and Zhang, Fashola, Shkolnik, and Boyle (2006) also used a
combination of prior years achievement scores and demographic variables to
establish equivalent schools.
O'Sullivan, et al (2003) used data from a large urban school district to
examine equivalence of 122 elementary schools. Using school size, as a
matching variable they examined student's standardized achievement test scores
in mathematics and reading. Their findings showed less than 10% of school were
equivalent on both measures. These
percentages declined as more matching criteria were added. A second study,
using the same data set revealed that pre-achievement was the best predictor of
current achievement but that consistency from Year 1 to Year 2 was very
unreliable.
Methods or Techniques
The sample for this research study began with all the elementary schools within
the four largest school districts in the state of North Carolina for the
2001-06 school years. Elementary schools were chosen because end-of-grade test
score data exist for all elementary schools for grades three through five. The
four largest school districts were chosen to control for school district
effects as well as because there were large numbers of schools to use for
comparisons.
Within the school districts several factors were considered consideration as
potentially important predictor variables: average student expenditure, student
needs, percentage of students eligible for free and reduced lunch, percentage
of students identified as Limited English Proficient (LEP), and grade range of
students within a school.
Pairs of elementary schools in each of the four school districts were chosen
that had similar school characteristics according to the previous factors
identified as important in the literature. Specific school
characteristics this research group selected as important when choosing pairs
of schools in each district included: (1) K-5 grade configuration, (2) school
size; small, medium, or large, and (3) previous achievement of students at
these schools as determined by reading end-of-grade test scores. Data for
the 2002-03, 2003-04, and 2004-05 school years were used to test for school
equivalence amongst pairs of schools chosen for each school district.
Median splits were used to distinguish small, medium, and large schools.
Pairs of schools were selected as potentially equivalent based on the
percentage of students scoring three or above on the reading end-of-grade test
in 2001 and 2002. Effect sizes were calculated for each school once
standard deviations and means were calculated. Pairs of schools were
defined as equivalent if difference in effect size was less than or equal to
.5.
Data Source & Preliminary Results
Data from the 2001-02, 2002-03, 2003-04, and 2004-05 school years from the
North Carolina ReportCard data were used to test for school equivalence amongst
pairs of schools chosen for each school district. Correlational analyses and
group mean differences will be used to answer the research questions, holding
school districts constant and controlling for school size (large, medium and
small). Preliminary results suggests that identifying equivalent schools, even
in the same school district, can be problematic.
References
Josendal, O., Aaro, L., Torsheim, T., & Rashbash, J. (2004). Evaluation of
the school-based smoking prevention program “BE smokeFREE”. Scandinavian
Journal of Psychology, 46, 189-199.
Lynch, S., Taymans, J., Watson, W. A.,& Ochsendorf, R. J., Pyke, C.,
Szesze, M. J. (2007). Effectiveness of a highly rated science curriculum unit
for students with disabilities in general education classrooms. Council for
Exceptional Children, 73, 202-223.
McDougall, D., Saunders, W. M., & Goldenberg, C. (2007). Inside the black
box of school reform: Explaining the how and why of school change at getting
results schools. International Journal of Disability, Development, and
Education, 54, 51-89.
O'Sullivan, R. G., Broadway, E.,, Bundy M., Gould, T., Hedgpeth, M. W.,
Petterson, M. & Poole, L. (2005). Exploration of Comparison of School
Equivalence. Paper presented at the annual meeting of the North Carolina
Association for Research in Education.
O'Sullivan, R. G., Fedora, P., Levine, S., MacKInnon-tucker, D. McCullough, A.
K. & Shaw, T. (2003). Comparison School Equivalence: Reality or Myth.
Paper presented at the annual meeting of the North Carolina Association
for Research in Education, Holly Springs, NC.
Riordan, J. E., & Noyce, P. E. (2001). The impact of two standards-based
mathematics curricula on student achievement in Massachusetts. Journal for
Research in Mathematics Education, 32, 368-98.
Zhang, Y., Fashola, O., Shkolnik, J., & Boyle, A. (2006). Implementation of
comprehensive school reform and its impact on increases in student achievement.
Journal of Education for Students Placed at Risk, 11, 309-329.