 |
 |
Temple University Center for Research
in Human Development and Education |
|
|
Organization and Clarity of Questions. Good
instruments use questions that are specific and
understandable and omit the ambiguity of "double-barreled
questions." An example of one such question is: "should
teachers be required to supervise and verify students'
acceptable use of the Internet?" An illustrative example is
ACT Now! Teacher Survey (Johnston & Barker, 2002). (See
"appendix 1" for survey). Terms are defined and items are
clearly written for the target audience. For example,
respondents are asked to rate the extent to which they agree
or disagree with attitudinal items and report the frequency
of specific classroom practices using technology.
Internal and External Validity. The term
internal validity refers to a question's strength and
appropriateness. Do questions measure what they intend to
measure? External validity refers to whether results can be
generalized to a larger population rather than only
participants in the study. Validity is improved through
pilot tests and revision. Typically, high quality
instruments have been pilot-tested with an audience whose
characteristics closely match the individuals completing the
finalized assessment. For instance, if the survey is
designed for inservice elementary teachers, it should be
pilot tested with inservice elementary teachers. LoTi
(Learning Quest, Inc., 2002) uses a 50-item questionnaire
that was tested for reliability, internal consistency, and
validity with several different samples, all of which
provided confidence that the LoTi Technology Use Profile
accurately measures the teachers' level of technology
implementation in the classroom. |
|
| |
Timeframe for recall questions. Recall
questions should reflect a relatively short window of time
(e.g., last week or last month) to reduce "memory decay."
Longer recall periods generate less reliable responses. The
following question is an example of a poor question that
puts too great a burden on respondents' memory: "From Fall
2001 to the present, how many hours per month did you use
websites to obtain information about educational
technology?" A better question would limit the recall period
to one week or month. An illustrative example is Technology
Proficiency for California Teachers-Technology Assessment
Profile (2002). Select "guest login" and "My Technology Use
Survey." This survey asks teachers to recall how often they
use technology tools for classroom management. (Options
included daily, 2-4 days/week, and between once a month and
weekly.)
Multiple Contexts. Survey designers recognize
that a survey is only one tool. For a more comprehensive
snapshot, it is best to employ multiple data points. For
instance, cross-referencing self-reported data with observed
data such as a technology inventory provides a more complete
picture. An illustrative example is the Maryland State
Department of Education (2001) Technology Inventory Summary
"Where Do We Stand in 2002 Digital Divide Results?" By
cross-referencing the Technology Inventory results with data
from the Free and Reduced Meal Program (FARMS), Maryland
State Department of Education and the Maryland Business
Roundtable have produced a dynamic report that details the
pervasive effects of the digital divide in Maryland schools.
User Feedback. Online surveys provide the
option of offering immediate feedback to participants. For
instance, an online survey can quickly create a visual
representation of the user's responses. An online survey can
also enable users to compare responses with others in a
specific group or community. Developers can give something
back to participants by presenting an immediate visual
depiction of their skill and proficiency. An example is
Profiler (High Plains Regional Technology in Education
Consortium, 2002). In the use of Profiler, a visual profile
is created that links each user's strengths and needs with
those in the community who have also participated. It allows
each community member to track his or her progress over
time, compare the profile to group results, find experts
within the community, or view a tutorial for a specific
skill. The user profile is available only to the members of
the community who receive a login for that particular
survey.
Educators are now faced with the challenge of making
decisions based on data collected through assessment
instruments. Surveys can provide useful and specific data.
However, educators must consider the qualities of a good
survey before selecting and implementing one themselves.
Designing or selecting a survey takes time and skill, and
educators seeking implementation should consult with
experienced practitioners on critical issues such as
sampling, coverage error, and nonresponse. For additional
information on the fundamentals of a survey, see the "What
is a Survey?" series published by the American Statistical
Association (2002) and the US Department of Education's
Planning and Evaluation Services "Frequently Asked
Questions"(2000, October 16).
References
American Statistical Association. (2002). What is a survey?
Retrieved January 2003, from
http://www.amstat.org/sections/srms/brochures/survwhat.html
California Technology Assistance Project. (2002).Technology
assessment profile. Retrieved January 2003, from
http://ctap2.iassessment.org/
High Plains Regional Technology in Education Consortium.
(2002). Profiler online collaboration tools.
Retrieved January 2003, from
http://profiler.hprtec.org/
Johnston, J., & Barker, T. (2002). Assessing the impact
of teaching and learning: A sourcebook for evaluators.
Ann Arbor, MI: University of Michigan. Institute for Social
Research. Retrieved January 2003, from
http://www.dlrn.org/star/TechSbk.pdf
Learning Quest, Inc. (2002). Welcome to LoTi.
Retrieved January 2003, from
http://www.learning-quest.com/LoTi/index.html
Maryland State Department of Education. (2001).
Technology inventory summary. Retrieved January 2003,
from
http://msde.aws.com/digitaldivide.asp
North Central Regional Educational Laboratory. (2002).
enGauge online assessment intro. Retrieved January 2003,
from
http://www.ncrel.org/engauge/assess/assess.htm
Pennsylvania Department of Education eTechPlanner. (2000).
PDE inventory 2001-District survey questions.
Retrieved January 2003, from
http://www.etechplanner.org/inventory/01districtsurvey.pdf
Slowinski, J. (2000). School staff education technology
needs assessment. Retrieved January 2003, from
http://chadwick-k12.com/ssetna/
Sun-Associates Educational Technology Integration. (2002,
August 14). Sample survey instrument. Retrieved
January 2003, from
http://www.sun-associates.com/eval/samples/samplesurv.html
U.S. Department of Education. Planning and Evaluation
Service. (2000, October 16). Frequently asked questions.
Retrieved January 2003, from
http://www.ed.gov/offices/OUS/PES/eval_faq.html#main_eval
Utah Technology Awareness Project. (2002). Rubrics.
Retrieved January 2003, from
http://www.uen.org/cgi-bin/websql/utahlink/UTAPdomains.hts
Previous
Page
1 2
|
|
|
Copyright 2001 © MARTEC
|
|
