UPDATED> Important news for Gopher Admins
Gleason Sackman
Date: Fri, 9 Sep 94 16:59:15 -0700
SENDER: Steven Foster <fos...@scs.unr.edu>
Subject: Important news for Gopher Admins! (repost)
Hi Gopher Admins,
Re: controlling veronica indexing.
I posted to this group on August 22, announcing a change to the procedures
for excluding gopher servers and gopher menus from the veronica index.
In brief, veronica will no longer use the "no-index" files because
it is necessary to move on to something more efficient and reliable.
Instead, veronica has a new, robust, simple, well-marshalled, extensible
control file convention.
I am now doing a veronica harvest. We have indexed 1488 gopher servers
today, of which six (6) have implemented the new control file scheme.
One site ( gopher.cc.utah.edu ) has taken advantage of the new control file
to specify that veronica can collect a pre-gathered data file rather than
"walking" the gopher - that's Jonzy. Any user of jughead 1.0.3 can easily
do that; I encourage you to do this, or to use software which I will release
in the near future to pre-build datafiles for the veronica harvester.
PLEASE USE the veronica control files to specify menus which should not
be included in the veronica index: things like local class schedules and
volatile information like usenet articles.
I am appending here the posting of Aug 22, which explains the new index-
control convention. It includes a sample "veronica.ctl" file - JUST CUT
IT OUT, edit the "Disallow" lines to exclude menus that should be excluded,
and PUT IT IN THE TOP-LEVEL MENU OF YOUR GOPHER SERVER. SIMPLE!
Thanks!
Steven Foster
------------------- original posting follows! -------------------------
August 22, 1994
Important Info for all Gopher Administrators!
RE: veronica improvements and changes.
From: Steven Foster
In order to better accomodate increasing loads, veronica will perform
several changes over the next few months. Several of these changes are
immediately important to all gopher server administrators:
1. September 1, 1994: veronica will recognize a new type of control
file for specifying gophers and gopher sub-menus to be omitted from
the index. VERONICA WILL NO LONGER RECOGNIZE THE "no-index" FILES.
Veronica will continue to recognize the "veronicaindex: yes|no"
setting in the gopherd.conf file. I am providing software to
facilitate this conversion (see below). I encourage all gopher
administrators to install a veronica control file.
2. September - November 1994. I will introduce several new tools
which allow gopher-server administrators to prepare data files for
veronica, rather than requiring veronica to tree-walk your gopher.
The first of these will be available by first week of September.
Later versions will optimize the index building and allow incremental
updates.
Jughead version 1.0.3 ( announced yesterday ) has provisions to
comply with the simplest of these file-passing conventions.
In order to use these file-passing schemes, you will need to
install a new-format veronica-control file in your gopher server.
The rest of this message addresses point #1: how to create a veronica control
file, and how to convert your "no-index" files to the new format.
A template of a "veronica.ctl" file is included at the end of this message.
You can edit it and install it in one of the appropriate path locations
mentioned below.
Alternatively ( and better! ) I will post a perl script in a separate
message. If your server runs on a Un*x host, and has perl available,
this script will read your no-index files and create an equivalent veronica
control file.
The control file may be installed at any of these seven paths in your gopher
server:
veronica.ctl
0veronica.ctl
0/veronica.ctl
0/etc/veronica.ctl
robots.txt
0robots.txt
0/robots.txt
Put it in only one place; veronica will honor the first of these files
that it finds. You may "hide" the file or the "etc" directory, but
the file does need to be readable by your gopher server. Be sure that
your server will return the file if you feed it the explicit path.
To the best of my knowledge, all the Unix, VMS, and Mac servers will
reply to at least one of these paths.
Two types of lines in the "veronica.ctl" file are immediately relevant:
1. The "veronicaindex: yes" line specifies that the gopher server
should be included in the veronica index. Change this to
"veronicaindex: no" only if you want to exclude the ENTIRE server
from the index. This replaces the previous convention of putting
a "no-index" file in the top-level menu of the server.
veronica will continue to honor the "veronicaindex: yes" and
"veronicaindex: no" lines in the gopherd.conf file.
2. The "Disallow: " lines. Each disallow line specifies a menu
in your gopher data heirarchy which should be avoided. This
REPLACES the "no-index" convention, which was unreliable and
inefficient.
For each selector specified in a Disallow line, veronica will omit
that menu, its contents, and its subdirectories from the index.
You should specify the selectors exactly as your gopher server
serves them.
Example:
Disallow: 1/Campus/Course_Schedules_Fall_94
Note that you can disallow only Gopher Type-1 items ( menus ).
You can't disallow a file or other type of data.
Note also that you don't need to disallow any submenus if you
have disallowed a parent directory.
Because of implementation constraints, veronica will honor only
the first 50 "Disallow" lines in a control file. That should be
enough for almost everyone.
The "veronica.ctl" file format is designed to be compliant with the
no-robots standard format defined in June 1994 by the WWW no-robots
working group. The "Disallow:" lines and "User-agent:" line are
shared with the robots group; the veronica-specific fields are compatible
but undefined in the no-robots standard.
More information will be forthcoming, concerning the format of prepared
datafiles for veronica's use.
This note, the "veronica.ctl" template, and the conversion program
are available via gopher from veronica.scs.unr.edu:70, in the menu
1/veronica/About
These files are also available via anonymous ftp from ftp.scs.unr.edu,
in pub/veronica.
#----------------- SAMPLE veronica.ctl FILE -------------------------
# File "veronica.ctl". Mon Aug 22 16:11:11 GMT-0700 1994
# Comment line; optional.
# Install this file at one of these paths in your gopher server's
# data heirarchy:
# veronica.ctl
# 0veronica.ctl
# 0/veronica.ctl
# 0/etc/veronica.ctl
# robots.txt
# 0robots.txt
# 0/robots.txt
# Test it, to be sure your gopher can return the file.
# Begin veronica control block. Blank line above this one demarks
# beginning of veronica control block. There must be NO BLANK LINES
# within the following set of veronica control lines. Comment lines
# are OK within the veronica control block.
#
User-agent: veronica
# "User-agent: veronica" line is REQUIRED.
#
# "Veronicaindex:" line is SUGGESTED. Normally it should be
# "Veronicaindex: yes". Set it to "Veronicaindex: no" only if you
# want to COMPLETELY turn off the veronica indexing for this gopher.
# If the "veronicaindex" line is absent, default will be "yes".
veronicaindex: yes
#
# The "Disallow:" line specifies gopher-selectors which veronica should
# avoid. For compatibility with the "no-robots" standard, at least
# one "disallow" is required, even if it is empty.
# Example:
# Disallow: 1/Campus/Course_Schedules_Fall_94
# The example tells veronica to avoid that menu, its contents and
# submenus. You can only disallow Type-1 gopher items.
# Specify the selectors exactly as your server serves them.
# Here is the required (empty) Disallow line:
Disallow:
#
#
# "Veronica-datafile:" line specifies a URL for veronica to use in
# retrieving a pre-built data file. Set this line only if you are
# running software which prepares such a file. Jughead v. 1.0.3 and
# above will create a raw datafile.
# The url must specify the "gopher" protocol method.
# Example line:
# veronica-datafile: gopher://gopher.turnip.com:70/00/etc/turnipdata
#
# "Veronica-datafile-prep:" line specifies degree of veronica-specific
# preprocessing that has been done to the veronica-datafile. Set this
# line only if you are running software which prepares datafiles for
# veronica.
# Example:
# veronica-datafile-prep: raw
#
#
# This file conforms to the robot-control-file standards of
# the WWW no-robots working group, June 30, 1994
# Blank lines mark the start and end of a control block.
#
# End of veronica control block.