Scanning questionnaires: Optical Mark Recognition software
P.Callow
Apologies if this is somewhat off-topic ... perhaps any reply
should be sent directly to me, rather than to the list.
A researcher responsible for large scale surveys has asked whether I can
recommend Remark Office OMR software (as distributed by SPSS UK Ltd) for
scanning questionnaires. However, although I have seen it demonstrated I
have no direct experience of it, or indeed of its competitors). The
benefits of such software are obvious enough (and the manufacturers are at
pains to point them out anyway). But what about the disadvantages? E.g. is
the success rate high enough?
Is anyone on this list able to advise, on Remark itself or more generally
on scanning questionnaires?
Apologies for cross-postings.
-----------------------------------------------------------------------------
Paul Callow, M.A., Ph.D. Phone 01223-335468 (direct)
University of Cambridge Computing Service 01223-334600 (switchboard)
New Museums Site, Pembroke St, Fax 01223-334679
Cambridge CB2 3QG E-mail P.Ca...@ucs.cam.ac.uk
Steve Palmer
Paul:
We've been doing large-scale surveys for a number of years utilizing an
NCS OpScan scanner and their ScanTools for Windows software (they have a
number of versions, DOS, UNIX, etc.). We've had great luck with them (we
seriously researched a number of companies before purchase). Our surveys
run the gamut between 1 page and 20 pages. The scanners available today
can read pre-printed codes (where did the survey come from, etc.), print
ID numbers on the survey as it is being scanned, and output data directly
to SPSS, Access, etc..
The forms themselves are highly customizable, easy to use, and quite
cheap when done in large numbers. Companies like NCS, Dun & Bradstreet,
Scantron, etc. can either print forms for you or you can design and print
your own. Some of the disadvantages we've seen are:
If people use pen (although the scanner can read it) you're in for a lot
of "editing" due to changes made by the respondents - the software holds
mismarked surveys in queue until they pass an edit check. This is a
manual operation.
Depending on the population being surveyed (sophistication of the
audience), you often have other editing issues (people putting checkmarks
in the bubbles, etc).
There isn't the natural visual checking done by a competent data entry
person. We have seen surveys come through with response sets like
1,2,3,4,5,4,3,2,1 throughout the entire survey, pictures drawn with the
bubbles, etc.. A good data entry clerk will pull those surveys to the
side to see if a respondent was "just playing games" and if we want to
add the data to our analyses. I typically check each survey for variance
to see if someone answered all 1's or the like, but some patterns are
much harder to catch in SPSS - or your preferred system.
Some people find the "bubble" forms too like taking a test and would
prefer the old paper and pencil method. We've had some clients that
piloted a scannable survey, only to come back and ask for a more
traditional version.
A good scanner and software are expensive, but we've found that our
system has paid for itself over and over again. We also have been able
to substantially lower the per-survey costs paid by our clients.
If you have more specific questions, please send along an e-mail.
Steve Palmer
Cambria Consulting, Inc.
spa...@cambriaconsulting.com
----------
From: P.Callow[SMTP:P.Ca...@ucs.cam.ac.uk]
Sent: Wednesday, March 19, 1997 4:48 PM
To: stevep; Multiple recipients of list SPSSX-L
Subject: Scanning questionnaires: Optical Mark Recognition software
Kevin Hoffman
Like Steve, I have been using an NCS OpScan for OMR scanning. Although
I am very happy with the accuracy and reliability of NCS system, the
flexibility and cost associated with questionnaire development could be
better. For example, NCS only guarantees reliability and accuracy if
you use their paper (which include specific timing marks necessary for
the scanning process). According to NCS, even if you purchase their
software for form development you will also need to continue using NCS
paper. Also, beyond using NCS software and paper, NCS recommends that
they also print the forms.
In addition, for small to medium scale surveys (e.g., 100 - 5000) I have
found it to be rather expensive to develop customized questionnaires for
the NCS OpScan. One solution has been to use a generic scannable answer
sheet (developed and printed by NCS) where a survey respondent would
receive a separate questionnaire and response sheet (i.e., scannable
answer sheet). Although this is not the most desirable format, it works
well for some surveys where you tend to see a high response rate.
However, I have seen response rates on more lengthy surveys decline when
compared to a "customized" questionnaire.
Another product I have been experimenting with is Teleform--A discussion
that Susan Elgie began a couple weeks ago in this group. After
purchasing the product, attending their training workshop in San Diego,
and purchasing the necessary hardware, I have mixed reviews. Teleform
has allowed me to customize survey questionnaires and print (using my
laser printer) on regular paper. Further, I have been very pleased with
Teleform's OMR capabilities (note, my experiences with OCR are not as
favorable). However, for multi-page questionnaires, where you don't ask
the respondent to include a unique ID on each page, I have run into a
number of problems (e.g., tracking multiple pages from each respondent
within an "automated" scanning process). Although Teleform has a "form
merge" feature where you can merge data into your questionnaire (e.g.,
name of respondent, ID#, unique ID, etc.), this process is very time
consuming and requires extensive technical resources (far more than a
similar mail merge in MS Word, for instance).
Overall, even with the advances in recent computing technology, I don't
believe there currently exists an ideal automated data entry system that
inexpensively does it all. However, these technologies can assist the
process.
Kevin Hoffman
Research & Development Coordinator
NEFE
Denver, CO
k...@nefe.org
davi...@freenet.edmonton.ab.ca
At my place of work, we are heavy users of NCS OMR equipment, and have
been for more than 15 years. We currently use four OpScan 21 scanners,
which we use about eight weeks a year. During those weeks we scan more
than a million sheets. NCS prints most of our forms; they have a minimum
run of 2000 copies. Obviously, for small questionnaires, they are not a
practical solution.
But our systems analyst has developed programs that generate customized
(in our case, one copy of each) scanner sheets that are reliably scannable
when printed on regular heavy bond paper on a Hewlett-Packard Laserjet
3Si. They use only one side of the sheet, as front/back registration is
insufficiently precise on the printer. Timing marks and "skunk" (form ID)
marks are printed, as well as the bubble fields. This approach could be
practical for even the smallest surveys.
E-mail me if you need more details, or want to contact our systems
analyst. (Last time I got him an external contract (for just such a
requirement), he bought me lunch!)
--
David Wasserman (davi...@freenet.edmonton.ab.ca)
Curmudgeon-At-Large (DWass...@edc.gov.ab.ca)
Home page http://www.freenet.edmonton.ab.ca/~davidwss
"The older I get, the more value I place on experience."
Mike Lane
Kevin Hoffman <K...@NEFE.ORG> wrote:
>Like Steve, I have been using an NCS OpScan for OMR scanning. Although
>I am very happy with the accuracy and reliability of NCS system, the
>flexibility and cost associated with questionnaire development could be
>better. For example, NCS only guarantees reliability and accuracy if
>you use their paper (which include specific timing marks necessary for
>the scanning process). According to NCS, even if you purchase their
>software for form development you will also need to continue using NCS
>paper. Also, beyond using NCS software and paper, NCS recommends that
>they also print the forms.
>
I have been using the NCS scanner as well and find it to be quite
reliable. Folded surveys can be a pain in the neck but overall I
would take the accuracy of the scanner over the accuracy of a data
entry person anyday of the week (twice on friday afternoons =)
>In addition, for small to medium scale surveys (e.g., 100 - 5000) I have
>found it to be rather expensive to develop customized questionnaires for
>the NCS OpScan. One solution has been to use a generic scannable answer
>sheet (developed and printed by NCS) where a survey respondent would
>receive a separate questionnaire and response sheet (i.e., scannable
>answer sheet). Although this is not the most desirable format, it works
>well for some surveys where you tend to see a high response rate.
>However, I have seen response rates on more lengthy surveys decline when
>compared to a "customized" questionnaire.
I use Bubble Publishing by SDI to create my own forms. I am getting
quite good and can create a handsome preslugged and text merged
multi-page form in about a day. (Assuming the layout and methodology
of the form is well layed out in advance.) BP allows for inserting
graphics as well. It is a pretty powerful software package, but it is
not for the PC novice, IMO, it can be quite buggy and needs to be
finessed by an experienced user many times to get premium results.
Now for printing up the form you have two options. If you are
preslugging or text merging you need to print each sheet individually
on a laser printer. This generally prints very reliably spaced sheets
and even on 20 lb paper generally works pretty well. If you are not
preslugging or test merging then you will probably need to have a good
relationship with your print shop.
My experience with organizational print shops is that if you give them
VERY clear instructions and make it well known that 1/16th of an inch
can make all the forms useless--you will get good results. In any
case, forget trying to copy the foms on a regular copier--its not
going to be even close to accurate enough.
>
>Another product I have been experimenting with is Teleform--A discussion
>that Susan Elgie began a couple weeks ago in this group. After
>purchasing the product, attending their training workshop in San Diego,
>and purchasing the necessary hardware, I have mixed reviews. Teleform
>has allowed me to customize survey questionnaires and print (using my
>laser printer) on regular paper. Further, I have been very pleased with
>Teleform's OMR capabilities (note, my experiences with OCR are not as
>favorable). However, for multi-page questionnaires, where you don't ask
>the respondent to include a unique ID on each page, I have run into a
>number of problems (e.g., tracking multiple pages from each respondent
>within an "automated" scanning process). Although Teleform has a "form
>merge" feature where you can merge data into your questionnaire (e.g.,
>name of respondent, ID#, unique ID, etc.), this process is very time
>consuming and requires extensive technical resources (far more than a
>similar mail merge in MS Word, for instance).
BP formshop does handle text merge and preslugging very well. Quite
easily too; all you need is a flat ascii file to do it.
Our office would be lost without BP formshop and TopScore in
conjunction with the NCS scanner can be a powerful combo. BP's
biggest drawback is its touchy nature, rub it wrong and it will lock
up your PC, blank out your screen, and close your checking account for
good measure. It knows when there is a novice at the keyboard =)
Best o luck,
Mike
Victor Trinidad
Paul,
We have the Remark Office OMR Scanning software here and although we
haven't used it yet for large jobs, I have tested it out so can report on
that.
The software if fairly easy to use, there isn't alot of extra commands
that can confuse you. The software allows you to scan a questionnare or
read it via a saved image file.
When creating a template I like to scan it into an image file by the
bundled PaperPort software with our HP scanner because for some reason,
when scanning the template directly in Remark, there would be line shifts
that I had to manually edit. Reading the template from a file didn't have
that happen.
You can create and read barcodes, bubbles(fill in the box), and text.
the program has an edit function that prompts you to make changes when it
encounters a problem such as double marked or incomplete marks.
The program reads the data and can also reports it via some different
stats.
I haven't exported it to SPSS yet but it is supposedly easy to do. if you
want to email me i can try it to test it out.
It is able to export into 12 different formats including ASCII, TXT,
Lotus v.2.0, Survey Pro, dBase, SPSS .SAV, and a few others.
Victor Trinidad
Senior Statistician
Mike Lane
Kevin Hoffman <K...@NEFE.ORG> wrote:
>Like Steve, I have been using an NCS OpScan for OMR scanning. Although
>I am very happy with the accuracy and reliability of NCS system, the
>flexibility and cost associated with questionnaire development could be
>better. For example, NCS only guarantees reliability and accuracy if
>you use their paper (which include specific timing marks necessary for
>the scanning process). According to NCS, even if you purchase their
>software for form development you will also need to continue using NCS
>paper. Also, beyond using NCS software and paper, NCS recommends that
>they also print the forms.
>
I have been using the NCS scanner as well and find it to be quite
reliable. Folded surveys can be a pain in the neck but overall I
would take the accuracy of the scanner over the accuracy of a data
entry person anyday of the week (twice on friday afternoons =)
>In addition, for small to medium scale surveys (e.g., 100 - 5000) I have
>found it to be rather expensive to develop customized questionnaires for
>the NCS OpScan. One solution has been to use a generic scannable answer
>sheet (developed and printed by NCS) where a survey respondent would
>receive a separate questionnaire and response sheet (i.e., scannable
>answer sheet). Although this is not the most desirable format, it works
>well for some surveys where you tend to see a high response rate.
>However, I have seen response rates on more lengthy surveys decline when
>compared to a "customized" questionnaire.
I use Bubble Publishing by SDI to create my own forms. I am getting
quite good and can create a handsome preslugged and text merged
multi-page form in about a day. (Assuming the layout and methodology
of the form is well layed out in advance.) BP allows for inserting
graphics as well. It is a pretty powerful software package, but it is
not for the PC novice, IMO, it can be quite buggy and needs to be
finessed by an experienced user many times to get premium results.
Now for printing up the form you have two options. If you are
preslugging or text merging you need to print each sheet individually
on a laser printer. This generally prints very reliably spaced sheets
and even on 20 lb paper generally works pretty well. If you are not
preslugging or test merging then you will probably need to have a good
relationship with your print shop.
My experience with organizational print shops is that if you give them
VERY clear instructions and make it well known that 1/16th of an inch
can make all the forms useless--you will get good results. In any
case, forget trying to copy the foms on a regular copier--its not
going to be even close to accurate enough.
>
>Another product I have been experimenting with is Teleform--A discussion
>that Susan Elgie began a couple weeks ago in this group. After
>purchasing the product, attending their training workshop in San Diego,
>and purchasing the necessary hardware, I have mixed reviews. Teleform
>has allowed me to customize survey questionnaires and print (using my
>laser printer) on regular paper. Further, I have been very pleased with
>Teleform's OMR capabilities (note, my experiences with OCR are not as
>favorable). However, for multi-page questionnaires, where you don't ask
>the respondent to include a unique ID on each page, I have run into a
>number of problems (e.g., tracking multiple pages from each respondent
>within an "automated" scanning process). Although Teleform has a "form
>merge" feature where you can merge data into your questionnaire (e.g.,
>name of respondent, ID#, unique ID, etc.), this process is very time
>consuming and requires extensive technical resources (far more than a
>similar mail merge in MS Word, for instance).
BP formshop does handle text merge and preslugging very well. Quite