v04INF3: comp.sources.reviewed - Guide for Reviewers
David Boyd
Posting-number: Volume 4, Info 3
Archive-name: rguide4
Last-modified: 12-Oct-1994
This is the third of five comp.sources.reviewed introductory messages.
This posting describes the procedures to be followed by those participating
in the actual reviews.
------------------------------------------------
Comp.Sources.Reviewed
Guidelines for Reviewers
(Version 1.4)
This document is designed to provide guidelines and suggestions for use
while reviewing software for the news group "comp.sources.reviewed".
It also provides potential submitters with information about what the
reviewers will be looking for, and may be useful to others who are
asked to evaluate software.
These guidelines may change as we gain more experience in reviewing
software, and comments are welcome.
How to become a CSR reviewer
----------------------------
- In order to become a CSR reviewer all you need to do is subscribe to the
package mailing list which will automatically mail the software to
you, and mail your review to the moderator at
comp-sourc...@sterling.com.
(See below for instructions on how to use the mail server).
- To be notified of packages available for review via E-mail subscribe to
csr-re...@ftp.sterling.com mailing list (See below for how to
subscribe).
What do the reviewers do
------------------------
- Reviewers receive a copy of the CFR either via their favorite new group
or via the standing reviewers mailing list (which all reviewers will be
given opportunity to join. No one will be automatically placed on the
list).
- Reviewers request the package from the mail-server announced in the
CFR by subscribe to the mailing-list for that packages reviewers.
- Reviewers should also retrieve a copy of the Guide for Reviewers
from the mail server.
- Reviewers should unpack the distribution, and build the software on
their platform(s). Any problems with the build should be coordinated
with Author of the package via the discussion mailing list. This is
expected to be the most likely step where the reviewer and author will
agree to patch the software in order for it to build on a new platform.
- The reviewer should note ALL problems they either built or executed the
software during the review period.
- Review the software based on the guideline below.
- Reviewers are free to run the software through any automated test
tools they feel are appropriate (CodeCenter, ObjectCenter, Purify,
Insight, Sentinal, etc).
- Reviewers are to submit their reviews to the CSR moderator by the
review deadline. At a minimum the review must state the name of the
package, the platform (hardware, os) the review was run on, and whether
the package successfully built and executed.
- Reviewers will be reminded prior to the expiration of the review period
that the reviews are due. If the reviewer is not able to submit a review
by the deadline they should notify the moderator. (Note - Work requirements
and deaths in the family are acceptable excuses ;-).
- Any questions about the software should be routed through the discussion
list provided. Any reviewer who is abusive or derogatory towards either
the author, the moderator, or other reviewers will be removed from
the discussion list and will only be permitted future reviews at the
discretion of the moderator. We are all professionals and I expect
us to behave that way.
- If the author provides a set of patches the reviewer will be asked
to apply the patches and regression test the package
- Provide you input on the suggested status of the package to the
moderator when polled.
- Bask in the knowledge that you helped to contribute to the quality of
free software on the internet.
Record Your Work
----------------
A crucial part of the review process is recording what you do. You
must make complete and accurate notes of your time spent working on the
package. Don't rely on your memory to record the problems or
suggestions you come across, because you will forget quickly as you
move on to other things. So, the first thing to do when you get a
submission is to start documenting everything that happens. Since you
will probably be working on several machines, you may have to work with
pencil and paper (gasp).
Evaluating the Format of the Submission
---------------------------------------
Was it in the form of a "shar" file, or similar packaging appropriate
for the architecture?
Did in unpack correctly?
Did it unpack in the places you expected it to?
Evaluating the Description and Purpose
--------------------------------------
Is there a README file that contains:
- the purpose and value of the software (give details here)
- the types of systems it is intended for (e.g., BSD only,
Unix and DOS, etc.)
- any dependencies in the system (e.g., must have perl to
run)
- known limitations of the software
- the authors of the software (with e-mail addresses)
- the "patchlevel" of the software (see below)
- any copying or distribution conditions
Is there a "patchlevel.h" file (or equivalent) indicating the version
number of the software?
Building the Software
---------------------
Is there an Installation document explaining how to build the software?
Are the options and conditional parts of the package (e.g., do this for
DOS, this for System V, etc.) clearly documented?
Is there a proper Makefile? (Imakefile for X software?)
Did the make run correctly?
Are building and installation two separate steps?
Did the software build on each of the architectures you were able to
test? (Give details about your testing environment.)
Testing the Software
--------------------
Did the software run on each of the systems you tested?
Did the program perform the functions it was supposed to perform?
It is not your place to fix the software you are reviewing. However, if
you find a problem with an obvious solution, make sure you document it
so the authors can make use of it.
The Documentation
-----------------
The documentation is a very important part of a submission, so it
should be reviewed carefully. The documentation may be built-in to the
program and be in the form of document files and/or man pages. It is
difficult to give simple rules for documentation, and not all programs
require the same amount of documentation, but here are some things to
consider.
General Guidelines
------------------
Is the documentation written in a form that is clear, precise, and
easy to understand?
Is each feature or function of the program documented?
Is the documentation organized? Sections and sub-sections often
make documents easier to read and understand.
Documentation should be provided in "text only" form. An author may
choose to also provide formatted manuals that use PostScript or TeX,
but a readable form should be provided.
Guidelines for Unix Documentation
---------------------------------
Is there a man page? Man pages should contain the following
sections (at least):
NAME required
SYNOPSIS required
AVAILABILITY optional
DESCRIPTION required
OPTIONS required if there are any command-line options
ENVIRONMENT required if there is provision for environment
variables
FILES required if use is made of other files
SEE ALSO optional
DIAGNOSTICS required if the program can produce debugging output
NOTES optional
BUGS required if any are known
AUTHOR optional
These sections should be complete (e.g., all the OPTIONS must be
described).
Guidelines for VMS Documentation
--------------------------------
Is there a VMS HELP file? HELP files should contain the following
sub-topics (at least):
Parameters if there are any
Command_Qualifiers if there are any
Examples if appropriate
Is there additional documentation to supplement the VMS HELP file?
Guidelines for DOS Documentation
--------------------------------
There are no standards for DOS documentation, and this makes the
review process difficult. We suggest that the information and
features described above should also be present in DOS
documentation, although they may not be in the form described
above.
Functionality and Features
--------------------------
Does the software perform some function that is valuable?
Are there obvious features or additions that would improve the package?
Overall Evaluation
------------------
What is your overall evaluation -- would you recommend it to a friend?
Would you suggest that this submission:
- be accepted for posting as is
- be accepted after minor revisions (that you have detailed)
- be returned to the author with a recommendation to make
major changes and re-submit
- be rejected
If you have problems with the package
-------------------------------------
A mailing list has been set up for each package under review. You should
subscribe to this list when you retrieve the package for review. Questions,
about the software (how to build, should it do this, etc) should be directed
to the mailing list. The list address will be of the form:
csr-<package_name>@ftp.sterling.com
Submitting a Summary
--------------------
The final step is to return a summary to the moderator at
comp-sourc...@sterling.com. Please make
sure you quote the "package name" in your summary, preferably in the
subject line.
DO NOT send your summary to the author. The moderator will collect
all the reviews, prepare a grand summary, and forward it to the author.
If you wish your review to be anonymous, please state so at the
beginning of the summary.
Your summary should provide as much detail as possible. Make sure you
give details on the kinds of machines you tested on. You may choose
to use this file as a template to record your review.
Include your reccomendation for posting in the review but in a seperate
part so that it may be easily edited out. This information
will be collected by the moderator and discussed with the author to determine
whether the package should be posted.
This summary might mention why you found the package
useful, what you liked about it, what machines you tested it on, and
what limitations you did find. Your name will not be attached to this
review summary when it is returned to the authors or posted.
Contacting the Authors
----------------------
The author of the package is automatically a member of the review discussion
list for the package. I strongly encourage all contact with the author be
done through this list.
Comments on the submission should not be posted to the discussion list, but
instead save your comments for the report you send to the moderator.
You should also ensure that your interactions with the author are
conducted in a courteous and professional manner.
It is not a good idea to suggest that authors make patches to software
during the review process. If you find a problem that the authors are
able to fix easily, document the problem and suggest that minor
revisions are needed before the submission is posted. It is important
that you are not evaluating software that has been patched, while others
are working with the original submission. If a patch is neccessary the
author should post the patch to the discussion list so that it is available
to all reviewers.
You should not send the author any comments about, or evaluation of, the
submission. That information should be sent to the moderator in your
final summary, and he will collect the comments from all the reviewers
and forward them to the author if appropriate..