Educators feel increased pressure to seek technology-based instructional products whose outcomes have been proven through scientifically based research. However, educators are often confused about vendor claims. The No Child Left Behind (NCLB) Act of 2001 (2002) requires that “educational practice, professional development, technical assistance and educational interventions be based in scientifically validated knowledge.” Additionally, improved student achievement on state tests is driving districts’ and schools’ purchases of all curriculum materials. Given these realities, how can educators know the difference between proven efficacy and oversold promises?

To help answer this critical question, this chapter of the Technology Coordinator’s Handbook details important points in understanding scientifically based research. It also introduces an online educational software chart with standardized ratings for software products. The chart shows which software companies provide evidence that they are meeting the scientifically based research requirements mandated by NCLB.

In order to demonstrate that instructional software products provide evidence of compliance with NCLB’s scientifically based research requirement, vendors must provide proof that they are attempting to comply with the following four components:

Appropriate Research Design

While both quantitative and qualitative research modalities provide valuable information, quantitative research is based on empirical measurement, large sample groups, and establishing correlations or cause-and-effect relationships. Quantitative research uses such means as standardized tests and surveys/questionnaires. Therefore, quantitative research is the kind required by NCLB.

Within the boundaries of quantitative research, the research design also needs to be carefully considered. Research designs include nonexperimental, quasi-experimental, and true experimental. A true experimental research design incorporates randomized field trials in which participants are randomly assigned to one of two conditions: treatment (intervention) or control (no intervention). To illustrate, if a school district wants to conduct a reading intervention program study of its entire fourth-grade student population, it could put the names of every fourth-grade student into a large hat, then draw out 100 names to participate in the study—hence, a random sample. That random sample of 100 fourth-grade students would then be put into another hat, with 50 names drawn out for the treatment/intervention group and 50 names drawn out for the control/nonintervention group—hence, random assignment. The experimental research design with random assignment is the gold standard for all scientifically based research, including that required by NCLB.

Although the quasi-experimental design is more of a “silver standard” regarding its reliability and validity, it is commonly used in educational settings when conducting a true experiment may not be practical, cost-effective, or ethical due to the complex human factors present in any school. An example of a quasi-experimental design in education would be to test the impact of a math software program on eighth-grade students by selecting one eighth-grade classroom to receive the math software along with regular instruction (the treatment/intervention group) and one eighth-grade classroom that would not receive anything beyond regular instruction (the comparison group). Note that the term “control group” is not used with the quasi-experimental design because unless participants have been randomly assigned to either group, true “control” of the study situation is not possible.

Empirical Measurement

Scientifically based research must also be based on empirical measurement or observation. This means that the study defines (a) dependent variables (i.e., the outcome of interest); (b) independent or predictor variables (i.e., factors—such as race/ethnicity, age, gender, or socioeconomic status— that might influence the outcome); and (c) control/confounding variables (i.e., variables, such as socioeconomic status, that might get in the way of clearly seeing the outcome of interest).

Replicable and Applicable Research

Additionally, scientifically based research must be both replicable and applicable. First, the research is able to be reproduced in a straightforward, cost-effective way. Additionally, other groups should be identified that might be affected by the research beyond the actual study participants; in an educational context, such groups could include principals, teachers, students, parents, central office staff, school board members, and community organizations.

Peer Review Process

As the final step in conducting scientifically based research, the peer review process is used to establish persuasiveness. The most rigorous form of peer review is publication in a professional journal. However, a school district may allow qualified colleagues to carefully examine and critique the completed research project if publication is not a feasible option.

MAR*TEC Educational Software Chart

As a way to incorporate the principles of scientifically based research into the practical realities of choosing educational software, MAR*TEC has designed a software review chart. In addition to revealing what different software products offer (e.g., free demo software, teacher and parent resources), the chart indicates which software companies provide evidence that they are meeting the scientifically based research requirements mandated by NCLB.

Keep in mind that since each company listed typically offers more than one software package, the categories are designed as a general checklist for the company’s software as a whole, rather than for any specific software title.

If you would like to submit the names of additional educational software companies for inclusion on this site, or if you have any questions or comments about the chart, please contact Jill Stoltzfus, Project Director, at jillstol@temple.edu.

The chart is available at http://www.temple.edu/martec/onlinetools/software.pdf

While understanding the foundation of scientifically based research will help you evaluate vendor claims, such knowledge will not detail which technology-based instructional products are right for your students and your school. As Beghetto (2003) cautions, “Ultimately, the difficult decisions of selecting programs and practices will rely on the professional discretion and expertise of school leaders” (p. 1).

We’re interested in hearing your own successes and lessons learned regarding the issue of scientifically based research and vendor claims. What evaluation process has your district put in place to consider the promises made by vendors? Using hindsight, what do you know now that you wished you knew when you first began considering this process?

E-mail your stories to Patricia Hendricks at phendric@temple.edu

References:

Beghetto, R. (2003, April) Scientifically based research . Retrieved February 10, 2005, from http://eric.uoregon.edu

No Child Left Behind Act of 2001, Pub.L. No. 107-110, 115 Stat 1425 (2002). Retrieved February 2005, from http://www.ed.gov/legislation/ESEA02/107-110.pdf