The use of the internet to complete surveys and experiments involves a
number of issues. There are many issues to keep in mind when conduction
any research project. Some issues are common to any survey method
(e.g., response rate, participant selection), though there are some special
considerations when conducting Web-based projects.
The information below would be beneficial as a synopsis of these issues.
This is not an attempt at a complete survey of the literature pertaining
to Web-based research methods. Rather, it should serve as an introduction
to these issues, with a useful bibliography included.
|
1) Advantages to Web-based
research.
|
7) Issues of privacy, anonymity, and confidentiality.
|
1) Advantages to Web-based research
Conducting a Web-based research project has a number of advantages to laboratory
work (as opposed to simply using the internet/server to house your research
materials; see here). The most salient
advantages include access to a wider range of participants than a lab study
usually allows (see
here for issues regarding similarity of lab and Web-based samples),
savings on time (e.g., no data entry) and financial resources (e.g., postage,
copying, stationary, etc.), and the possibility of interactive survey materials
(Schmidt, 1997). Depending on the type of project, the possible advantages
will vary. However, these advantages are tempered w/ a healthy list
of disadvantages as well.
Using Web-based storehouses for research projects (such as the Web Experiment
Psychology Lab or PsychExps) also allows easy dissemination of materials
for other researchers.
*
Further reading:
Reips, U.-D. (2000). The web experiment method: Advantages, disadvantages,
and solutions. In M. H. Birnbaum (Ed.), Psychological Experiments on
the Internet (pp. 89-114). San Diego, CA: Academic Press.
Schmidt, W. C. (1997). World-Wide Web survey research: Benefits,
potential problems, and solutions. Behavior Research Methods, Instruments,
and Computers, 29, 274-279.
Stanton, J. M. (1998). An empirical assessment of data collection
using the Internet. Personnel Psychology, 51, 709-725.
2) Disadvantages to Web-based research
Disadvantages to Web-based research frequently involve a lose of control
by the experimenter. For example, when conducting Web-based research,
the experimenter is unavailable to answer questions, deal with concerns,
or troubleshoot problems. While the dynamic nature of the internet
allows communication between researcher and respondent, help may not be
available in a timely manner. Also, integrity of the data becomes
a bigger issue, with increased possibility of multiple submissions, incomplete
responses, and data security.
*
Further reading:
Reips, U.-D. (2000). The web experiment method: Advantages, disadvantages,
and solutions. In M. H. Birnbaum (Ed.), Psychological Experiments on
the Internet (pp. 89-114). San Diego, CA: Academic Press.
Schmidt, W. C. (1997). World-Wide Web survey research: Benefits,
potential problems, and solutions. Behavior Research Methods, Instruments,
and Computers, 29, 274-279.
Stanton, J. M. (1998). An empirical assessment of data collection
using the Internet. Personnel Psychology, 51, 709-725.
3) What kinds of studies can you do
on-line?
The variety of tools now available allow researchers freedom in devising
and constructing web experiments. Surveys can be easily constructed
and implemented with a variety of formats (e.g., SurveyWiz,
WWW
Survey Assistant). WEXTOR
allows for complex experimental designs involving between-, within-, and
quasi-experimental variables. With WEXTOR, you use web pages to present
stimuli and collect dependent measures. Using PsychExps
and the Authorware software, the sky is practically the limit. Authorware
can create complex surveys (with skip patterns and real-time data presentation
to the participant), record latencies in milliseconds, present a variety
of graphic and audio stimuli, and do real-time data checking.
4) Distributing your instrument
Common methods for disseminating surveys are embedding the survey in an
email message or including the necessary URL in an email message (see
below for response rate information). If your study is posted
in a research web site (e.g., PsychExps or the Web Experiment Psychology
Lab), you will get some traffic just from random visitors. These
sites frequently report up-to-date statistics on visitors to the site.
As of November, 2000, the WEPL was linked from approximately 1,000 other
web pages and averaged over 4,500 visitors per month (Reips, 2001).
It is not uncommon to email notices to relevant listservs, though it is
advisable to check on the listserv's policy or check with the moderator
first.
There are definite cost advantages to to Web-based distribution rather
than other methods (e.g., saving on postage and copying, saving interviewer
costs). Businesses are increasingly using Local Area Networks (LANs)
to collect evaluations and surveys (Thompson, Surface, Martin, & Sanders,
2003). This advantages must be tempered with the disadvantages of
the noncoverage problem and reduced response rates to Web-based instruments.
While there's no question more and more people are jumping online, this
sample is not representative of the general public.
* Further reading:
Reips, U.-D. (2002). Internet-based psychological experimenting:
Five dos and five don'ts. Social Science Computer Review, 20, 241-249.
5) Housing your lab study online
Though you may not want to distribute your survey/experiment online for
the whole world to complete, you can still house your instrument online
with several advantages. First, it centralizes your data collection
instrument and storage, eliminating the loading and troubleshooting of
problems on multiple machines. Also, it simplifies data collection
if the same study is being conducted at multiple locations (e.g., other
lab rooms or other institutions). For example, when collaborating
with a colleague at another institution, data collected with the same instrument
is stored in the same place. Similarly, it eliminates the need for
purchasing multiple licenses of the same software package for loading on
multiple computers. Disadvantages include the possibility of server
malfunctions (e.g., if the server goes down, it's a global data collection
problem rather than a localized problem) and a lack of control should server
problems occur.
*
Further reading:
McGraw, K. O., Tew, M. D., & Williams, J. E. (2000). An online psychology
laboratory. In M. H. Birnbaum (Ed.), Psychological Experiments on the
Internet (pp. 219-233). San Diego: Academic Press.
Reips, U.-R. (2001). The web experimental psychology lab: Five years
of data collection on the internet. Behavior Research Methods, Instruments,
& Computers, 33, 201-211.
6) Response rates and attrition to Web-based
research
Studies conducted to this point have shown emailed (and email announced)
surveys, show lower response rates than "snail" mail or telephone surveys
(for a review, see Cook, Heath, & Thompson, 2000). In fact, one
early study compare e-mail to "snail" mail response rates showed a drop
in response rates between 1995 and 1998. For both years, mail responses
exceeded email responses, but there was a more dramatic difference in 1998
than 1995 (Bachamann, Elfrink, & Vazzana, 1999/2000). The authors
speculate that the increased use of email for for communication, which
includes junk mail, helped lower the response rates. This is a problem
that is only increasing.
Related is the issue of drop-out rate. With no experimenter/interviewer
watching, and often little incentive, incompletion rates can be high.
While response and attribution rates are a concern, who drops out
is also a problem. Inadequate research has assessed what specific
aspects of web studies could reduce attrition and increase valid responses.
However, a meta-analysis of response rates to Web-based surveys analyzed
68 surveys (from 49 studies) and found 1) number of contacts, 2) use of
personalized contact letters, and 3) use of precontact increased response
rates. Notably, offering incentives tended to decrease response rates
(Cook, Heath, & Thompson, 2000).
*
Further reading:
Bachmann, D., Elfrink, J., & Vazzana, G. (1999/2000). E-mail and snail
mail face off in a rematch. Marketing Research, 11, 10-15.
Cook, C., Heath, F., & Thompson, R. L. (2000). A meta-analysis of response
rates in web- and internet-based surveys. Educational and Psychological
Measurement, 60, 821-836.
Knapp, F., & Heidingsfelder, M. (2001). Drop-out analysis: Effects
of the survey design. In U.-D. Reips, & M. Bosnjak (Eds.), Dimensions
of Internet Science (pp. 221-230). Lengerich: Pabst Science Publishers.
Truell, A. D., Bartlett, J. E., & Alexander, M. W. (2002). Response
rate, speed, and completeness: A comparison of Internet-based and mail
surveys. Behavior Research Methods, Instruments, & Computers, 34,
46-49.
Welker, M. (2001). E-mail surveys: Non-response figures reflected.
In U.-D. Reips, & M. Bosnjak (Eds.), Dimensions of Internet Science
(pp.
231-237). Lengerich: Pabst Science Publishers.