Stopping veronica indexers
John Stanley
"the person to tell" was told that I did not want my unregistered
gopherspace indexed. That didn't solve it, because not only is the
space still in the veronica indexes, another indexer assaulted the
space today. (And this indexer was someone who I had already told I did
not want the space indexed. He was doing it again despite his saying he
wouldn't.)
So, how do you keep veronica indexers from assaulting your gopherspace
and indexing it even when you have told them not to?
If you think the answer is "tell Steve Foster at UNR", you are wrong.
(That is what HE thinks the answer is. He has been unable to remove my
space from his own index, much less remove it from anyone else's.
Telling Steve hasn't accomplished diddly-squat.)
If you think the answer is "put them in the access: listing", then you
need to know that the last time UNR came knocking they came in from
nscee.edu. To stop the veronicers you would need to shut out the entire
world.
I have had to move my gopherspace once because of these people who
can't seem to be bothered to ask and don't care what the answer is
anyway. I have been told that I have agreed to this indexing because I
registered the site (which is a lie, it is not registered. Well,
actually, it is NOW, because one of the veronica-heads decided to
register it for me. How kind of him. Not.)
I have been told that it is ok for these assaults to take place because
"it was done during off-peak hours". I am still trying to figure out
how someone in another state knows when my system's off-peak hours are,
and why I can't be allowed to schedule things I want done whenever I
want them done regardless of his idea of "off peak hours". Is there some
RFC I am missing that demands that systems on the network must all have
"off peak hours" at a certain time? And what happens when the EOS
satellites start dumping terrabytes of data down the pipe? Will they
stop so that veronica indexers can bash a system during "off peak
hours"?
I am told that it is ok to ignore my desires in this matter because it
is being done for a "socially useful purpose". And here I thought that
making my data and archives available to invited users was being
socially useful. I thought the data-specific, daily-updated indexes I
made available to those users was "socially useful". Apparently it
isn't. Apparently a site isn't socially useful unless every directory
on it is indexed and advertised to every horny college student looking
for pictures and head (two keywords that will point to my space,
without the slightest bit of content that horny college students are
looking for.)
But then, I am just a scientist trying to make some data available to
other scientists. I am not a COMPUTER scientist, so my opinion of what
should be done with my data and who should be invited to access it and
how my site's resources are used and advertised to the world is
irrelevant. I just own the data. I just manage the system. What do I
know?
Don't get me wrong, it isn't the users I mind. They have made very
little dent in my system resources. The horny college students take one
look at the names on the pictures and realize they aren't what they
want and they go away. It is the hard-driving automated indexing that
tries to run through every directory on my system, including the ones
that have thousands of files in them, most of which have absolutely
meaningless names (to a random human browser), which WILL NOT go away.
THAT is abuse, and the veronica indexers have been told that, and they
don't give a damn.
Is there some way to identify a veronica indexing attack, other than
noticing after the fact that one site has walked through every
directory in your space systematically? Some way of catching them when
they first start and denying them any further access? Some strange entry
in the menu that will break their indexers (in a way that will cause
them the most grief, presumably?)
John-David Childs
John Stanley <sta...@ruby.oce.orst.edu> wrote:
>Ok, I thought I had the problem solved when someone who claimed to be
>"the person to tell" was told that I did not want my unregistered
>gopherspace indexed. That didn't solve it, because not only is the
>space still in the veronica indexes, another indexer assaulted the
>space today. (And this indexer was someone who I had already told I did
>not want the space indexed. He was doing it again despite his saying he
>wouldn't.)
>
>(That is what HE thinks the answer is. He has been unable to remove my
>space from his own index, much less remove it from anyone else's.
>Telling Steve hasn't accomplished diddly-squat.)
>
>If you think the answer is "put them in the access: listing", then you
>need to know that the last time UNR came knocking they came in from
>nscee.edu. To stop the veronicers you would need to shut out the entire
>world.
>
seen posted in c.i.g. I realize you're frustrated, but....
Not to excuse them any, but the computers at UNR that do the
veronica indexing have been having problems, and they may have
moved an old site-indexing list to nscee.edu to do the index.
Also, some of the other veronica servers (like PSInet, U of Pisa, NyserNet,
and U Mich) MAY do their own indexing! It will probably take a while for
the indexing of your site to stop. It sounds like Steve Foster tried,
but maybe the communication between folks there is about as good as it
is in most overstretched, over-beauracratized Universities ;-)
Finally, you mentioned the phrase "invited users". If they're invited,
why not set up the default access to be !browse, !read, !search and then
give access as you inform different groups/people about it (or interested
folks write for permission to access it). That seems a lot simpler than
getting frustrated at horny college students and Veronica snake-indexers.
>RFC I am missing that demands that systems on the network must all have
>"off peak hours" at a certain time? And what happens when the EOS
>satellites start dumping terrabytes of data down the pipe? Will they
>stop so that veronica indexers can bash a system during "off peak
>hours"?
>
One time, Veronica indexed Healthline (well before I registered
it!!) at 10:00am on a Tuesday...a relatively busy time for Healthline
at that time!). I'll bet it didn't cause more than a 2% increase
in system load (it took all of about 20 seconds to complete).
>But then, I am just a scientist trying to make some data available to
>other scientists. I am not a COMPUTER scientist, so my opinion of what
>should be done with my data and who should be invited to access it and
>how my site's resources are used and advertised to the world is
>irrelevant. I just own the data. I just manage the system. What do I
>know?
I'm not a computer scientist either...I'm just an overworked,
underpaid economics/political science student who has spent a lot of
personal time making Healthline (and our soon-to-be announced campus CWIS)
something useful...to whomever happens to find/browse it. Being
or not being a computer scientist has absolutely nothing to do with
determining who can or can't use your resources. But in the Internet world,
complete with weeny hackers and zealous social do-gooders, managing your
data includes the responsibility to protect it from those who would
damage/publicize it, and it also takes a bit of understanding that
you aren't the only person complaining about XYZ to Steve Foster,
Fred Barrie, Paul Lindner, Mark McCahill, Daniel Torres and all the other
unsung Veronica/Gopher developers & hack artists. (Thanks guys).
>THAT is abuse, and the veronica indexers have been told that, and they
>don't give a damn.
I highly doubt this is true. In addition, I'll bet that you'll
find your Gopher listed in John Doyle's list of New Gophers (at
liberty.uc.wlu.edu). He has written a script that will find just about
every gopher on the Internet...as long as it is pointed to by a currently
indexed Gopher. For instance, long before I registered Healthline w/ UMinn,
the Cornucopia of Disability Information (CODI) had a link to Healthline
and vice versa. Because CODI was indexed by liberty, it wasn't long before
Healthline started to be indexed by liberty as well. You might want to send
do...@liberty.uc.wlu.edu a message too (but please don't flame him...).
Asbestos suit on,
--
John-David Childs
Senior Consultant, University of Montana CIS
UM Grizzly InfoDen/HealthLine Gopher Admin.
j...@selway.umt.edu, con...@lewis.umt.edu
Fred Barrie
>Ok, I thought I had the problem solved when someone who claimed to be
>"the person to tell" was told that I did not want my unregistered
>gopherspace indexed. That didn't solve it, because not only is the
>space still in the veronica indexes, another indexer assaulted the
>space today. (And this indexer was someone who I had already told I did
>not want the space indexed. He was doing it again despite his saying he
>wouldn't.)
Stanley,
As of two weeks ago you were taken off the veronica list. What
this means is that your site namely skyking.oce.orst.edu will not be indexed
by the veronica team. That did not mean that the current data base would
be deleted all across the world. Your file has been deleted at UNR and
when the network at UNR is functional again after the weekend the rest
of the veronica servers will receive the updated files without your host.
Now for the rest of the gopheradmins out there if you wish not to have your
site not indexed by veronica please mail vero...@comics.scs.unr.edu requesting
your site be taken off the list of hosts to be indexed on the next harvest.
>
>So, how do you keep veronica indexers from assaulting your gopherspace
>and indexing it even when you have told them not to?
>
>If you think the answer is "tell Steve Foster at UNR", you are wrong.
>(That is what HE thinks the answer is. He has been unable to remove my
>space from his own index, much less remove it from anyone else's.
>Telling Steve hasn't accomplished diddly-squat.)
>
>If you think the answer is "put them in the access: listing", then you
>need to know that the last time UNR came knocking they came in from
>nscee.edu. To stop the veronicers you would need to shut out the entire
>world.
I am sorry Stanley that because of the hardware problems at UNR we
went to our neighbors in the south to index the gopherspace in a timely
fashion. We learned our lesson and will never index from anywhere but
veronica.scs.unr.edu or gopher.unr.edu.
>
>I have had to move my gopherspace once because of these people who
>can't seem to be bothered to ask and don't care what the answer is
>anyway. I have been told that I have agreed to this indexing because I
>registered the site (which is a lie, it is not registered. Well,
>actually, it is NOW, because one of the veronica-heads decided to
>register it for me. How kind of him. Not.)
>
Stanley, please mail me a list of all gophers at Oregon State
that are under your administrative care or just tell me all gophers at Oregon
and you will never be bothered by my indexes.
>Is there some way to identify a veronica indexing attack, other than
>noticing after the fact that one site has walked through every
>directory in your space systematically? Some way of catching them when
>they first start and denying them any further access? Some strange entry
>in the menu that will break their indexers (in a way that will cause
>them the most grief, presumably?)
Yes, create a recursive loop back that will send the veronica harvester
screeming...
Stanley,
The real problem is that you don't want anyone to index your site
and that is fine by me. However there are more places other than out site
that can and do index your site that are not associated with veronica. The
latest "veronica attack" (maybe we got the making of a movie here) did
not come from the latest veronica index. I double checked my logs when
you requested to be removed you where.
How the real question...I know we have talked about this before should
veronica index unregistered but useful gophers or just registered gophers
or only those who reguest in writing to be indexed.
If you have any opinions please mail veroni...@comics.scs.unr.edu or
myself at bar...@comics.scs.unr.edu.
Fred Barrie
Stephen C. Trier
I know you're frustrated about Veronica, but I think the blame is being
pointed to the wrong place. By running a Gopher with open access, you
are making your information available to anyone. That includes letting
those people tell others about it, as Veronica does.
If you want to keep your information private, make it private. Gopher
provides the mechanisms you need for that. Speaking practically,
security through obscurity never works well, even when dealing with
humans instead of automatons.
Is Veronica treading on your rights? Good question. To play the
devil's advocate, I'd say that taking private data, even if it is left
unprotected, is unethical. I could also say that unless it is
explicitly said that data is public, it must be private.
However, tearing apart those (strawman?) arguments, let me point out
the absurdity inherent. Under these standards, taking data from almost
every current anonymous FTP site is unethical! (I haven't seen many
that explicitly grant permission to download their data.)
A resolution to the absurdity is to realize that there are implicit
grants of permission to access data as well as explicit grants.
Judging whether an implicit grant of permission exists is difficult.
For example, I have a friend who found copyrighted commercial software
on the anonymous FTP site of the company that sold said software. Does
that mean he had permission to download it?
While there is a grey area here, data providers can make it black and
white if they choose: Make the grant or refusal of permission
explicit. If you want your gopher to be private, the easiest method to
refuse permission is to turn off access to everyone except those few
people to whom you want to grant access. I have done exactly this with
my WAIS and FTP gateways, which are open to local users only. A more
complicated approach, which you appear to be currently pursuing, is to
post notices of limited access, then talk to people who don't
acknowledge them. However, there is very little you can do if they
don't cooperate. Conversely, if you offer open access, explicitly
state so.
So anyway, I'd say that Veronica has two options: assume an implicit
grant of permission exists on any gopher that permits access to its
host, or index only machines that give an explicit grant of
permission. Since anonymous FTP gives a generally accepted precedent
for presuming implicit permission, the current veronica policy seems
reasonable.
Stephen
"I'm not an ethicist, but I play one on c.i.gopher" ;-)
--
The opinions expressed in this message are solely those of the author. CWRU
IRIS/INS/Telecomunications is not responsible for the content of this message.
--
Stephen Trier (tr...@ins.cwru.edu - MIME OK)
Network Software Engineer
IRIS/INS/T
Case Western Reserve University
John Stanley
> Stanley,
> As of two weeks ago you were taken off the veronica list. What
>this means is that your site namely skyking.oce.orst.edu will not be indexed
>by the veronica team.
I guess my ability to locate my site in your index is just a figment of
my imagination, Barrie? No, I just looked again. Still there.
> Stanley, please mail me a list of all gophers at Oregon State
>that are under your administrative care or just tell me all gophers at Oregon
>and you will never be bothered by my indexes.
I have already mailed you concerning which gopherspace is not to be
indexed. Why isn't that sufficient? And, just for the record, the sole
specification on the limit is the host name. It doesn't matter what port
is being used, it doesn't matter what the process id for the server is,
the prohibition applies to all gopherspace on that host. (I just didn't
want to leave a hole for you to crawl through if I have to move my
gopherspace again because indexers really haven't stopped. Can you
believe it, one indexer thought that since I moved the space that it was
a completely new server and he registered it for me and went ahead and
indexed it.)
> Stanley,
>
> The real problem is that you don't want anyone to index your site
>and that is fine by me.
No, Barrie, the real problem is that nobody bothered to ask the admin
of the site and the owner of the data whether he wanted his unregistered
gopherspace indexed and advertised to the world. Had someone bothered to
ask, they would have been told "no thank you". Maybe they would have
been told "you can index these parts". There would have been no
problem. (If they hadn't just gone ahead and done it anyway, that is.)
The problem came when veronica came to dinner without an invitation and
wouldn't leave when she was asked to. The problem came when veronica
tried telling me that registered gopherspaces were fair game, as if
trying to claim that mine was registered. I am at the level of talking
to the network zone contacts for UNR just to get this far. She still
hasn't left, and I don't have the slightest belief that she will until
I actually see her ass being hit by the door on the way out.
>However there are more places other than out site
>that can and do index your site that are not associated with veronica.
According to Mr. Foster, who claims to be a member of the team, telling
him not to index my site is sufficient to keep indexers out. Is this not
true? Are there, as I pointed out at the time, really other people who
have indexing software which is not under Mr. Foster's control?
>The
>latest "veronica attack" (maybe we got the making of a movie here)
I don't know what you call it, but I call it an attack when an indexer
comes waltzing into a site that he has been told not to index and
bashes on the systems for a half hour in the middle of the day indexing
data that is of interest to perhaps half a dozen people in the world.
I call it an attack when that indexer registers someone else's
gopherspace so he can use the excuse that it is a registered space and
thus open for indexing.
>How the real question...I know we have talked about this before should
>veronica index unregistered but useful gophers or just registered gophers
>or only those who reguest in writing to be indexed.
I can accept that registering a gopherspace with UMN makes it a public,
open, general access, y'all come in and browse site (as long as the
registration is done by the admin of the site and not by some indexer
who thinks he HAS to index something). I have no problem with that.
Perhaps you have missed me saying that. Perhaps you have missed me
saying that I deliberately did NOT register my space because I
deliberately did NOT want the sort of crap that comes from being
indexed.
All I wanted is to be removed from indexing. Why do I have to contact
zone contacts to get that done?
John Stanley
> Ouch! This is one of the most visceral letters I've
>seen posted in c.i.g. I realize you're frustrated, but....
I am frustrated because talking to the people who say they are the
people to talk to isn't getting anything done. And sending mail to the
address in the veronica FAQ (gophadm@veronica...) bounces. You can't
even get mail to them to tell them their FAQ is giving baloney answers.
Just how long has the -s flag to gopherd been gone?
>folks write for permission to access it). That seems a lot simpler than
>getting frustrated at horny college students and Veronica snake-indexers.
I am not frustrated at the students. They are using the resources they
think they have a right to. It is the indexers who are at fault.
It would be an enormous task at this point to get the addresses of
everyone who has been invited to use the space. Mostly, I don't know
who they all are. I suppose that I could go back through all the logs
and find out what sites had accessed in the past and force any new
users to register, but that seems like such an awful amount of overhead
just to make sure indexers who claim they won't index again keep their
promise. I would like to think that being told once not to index was
sufficient.
And it still opens a site up to indexing. If one user at a site
registers, an indexer at that site will have free access (until the next
day when the logs show he wandered by.)
>at that time!). I'll bet it didn't cause more than a 2% increase
>in system load (it took all of about 20 seconds to complete).
The last indexing ran for half an hour and didn't get a fifth of the
way through the space. If it were only twenty seconds I probably
wouldn't even notice it. When the log runs for several pages, however,
it becomes painfully obvious. When the log runs for several pages
sequentially indexing a year and a half (one directory per day) of raw
binary files it is obvious a machine is doing it. If it hadn't stopped,
it would have wandered into the areas that have thousands of files, all
with names like "14325". What a useful index to have, huh?
>In addition, I'll bet that you'll
>find your Gopher listed in John Doyle's list of New Gophers (at
>liberty.uc.wlu.edu).
Oh, yes, Mr. Doyle and I are old friends (term used VERY loosly). He is
the indexer who registered both of my servers, and used that
registration as an excuse for bashing on my new server for half an
hour, despite being told the last time he indexed this site that it was
an abuse. It has taken getting HIS zone contact involved before he
would act to remove the indexing. His last mail said that "it would be
done today". Of course, the mail before that said it had been done.
He is the reason I came to the net with this problem. The UNR folks
haven't actively come indexing, they just haven't bothered to remove
indexes they are supposed to. It's Mr. Doyle who is actively indexing
without regard for previous requests not to.
>Healthline started to be indexed by liberty as well. You might want to send
>do...@liberty.uc.wlu.edu a message too (but please don't flame him...).
The first time I told him it was an abuse I didn't flame him, and he
promised it wouldn't happen again. I said "someone at your site is
doing this, it is an abuse, please have them stop." He said "it was me,
I won't do it again." I fucked up and left wlu.edu out of the disabled
list because of that. (The ability to access the site was used as
justification for his indexing the second time.)
No, flames came when smoke was ignored.
John Stanley
>I know you're frustrated about Veronica, but I think the blame is being
>pointed to the wrong place. By running a Gopher with open access, you
>are making your information available to anyone. That includes letting
>those people tell others about it, as Veronica does.
No, I do not agree. It is my right to decide to whom I advertise my
gopherspace, and it is my right to decide whether I will announce it to
certain audiences. It is also my right to put any usage limitations on
the invitation. Indexers ignore any such limitations and announce the
space as open public space.
I have expressed those rights by deliberately NOT registering my
gopherspace with any gopherspace registry. When I was asked if I wanted
my space registered in the local campus gopher, I decided not to,
because that could be viewed as an invitation to the general public to
come and look for whatever they wanted to look for.
>If you want to keep your information private, make it private. Gopher
>provides the mechanisms you need for that. Speaking practically,
>security through obscurity never works well, even when dealing with
>humans instead of automatons.
I am not attempting security through obscurity. I am reserving the right
to advertise my system resources to the audiences I wish to advertise
them to. It is not a case of security. Do I care if someone comes in
through a gopher index to look at a file or two? No. I care that that
gopher index is advertising my site to that person as an open system.
>However, tearing apart those (strawman?) arguments, let me point out
>the absurdity inherent. Under these standards, taking data from almost
>every current anonymous FTP site is unethical! (I haven't seen many
>that explicitly grant permission to download their data.)
I have seen many references to "this data is available via anonymous ftp
from x.y.z.". That is explicit permission. I have seen many references
to material via archie, which indicates a grant of approval to come get
it.
There is, however, a difference between J. User visiting a site and
picking up a file or two and an automated demon systematically visiting
every directory and indexing it, and then advertising that gopherspace
and the entire contents as publicly available.
>If you want your gopher to be private, the easiest method to
>refuse permission is to turn off access to everyone except those few
>people to whom you want to grant access.
This works fine on paper. Not so well in real life.
1. For a gopherspace announced to a large audience of potential users,
how do you maintain a list of every site they are at?
2. How do you specify that indexers who also happen to be at those sites
do not have access while still allowing the users?
3. How do you get the indexers who wander in through a user-hole to undo
the damage they cause? Then, do you shut off that user, or do you trust
the word of an indexer that he won't do it again? (I am batting 0 on the
latter.)
>However, there is very little you can do if they
>don't cooperate. Conversely, if you offer open access, explicitly
>state so.
In the places where I announce the availability of my gopherspace, I
also announce its intended use. Implicit in that is exclusion of other
uses. When an indexer announces the availability of my space, however,
there is nothing about intended use. It is being announced as an open
access server. It doesn't matter that I do not announce it as such.
My right to decide the appropriate uses for my resources are removed.
>So anyway, I'd say that Veronica has two options: assume an implicit
>grant of permission exists on any gopher that permits access to its
>host, or index only machines that give an explicit grant of
>permission.
There is a third option: assume that sites that have NOT registered have
taken that action for a reason and do not index such sites without
permission. Assume that sites that DO register as open access sites have
done so for a reason, and index them unless otherwise asked not to.
The assumption that granting access to a host is granting access to
index and advertise a gopherspace is specious. The access may be
granted to another user at that host for reasons entirely unrelated to
indexing. In fact, this is the case for my space. I do not grant
permission for any indexing from any site, but do allow users to acces
it.
>Since anonymous FTP gives a generally accepted precedent
>for presuming implicit permission, the current veronica policy seems
>reasonable.
Actually, archie is a good analogy to the registry. If an FTP site
appears in archie, then there is a good reason to assume that the data
listed is available to the public. If it does not, and you do not see
an invitation in another form, then there is good reason to assume that
the site is private. The difference here is, I believe, that I have not
heard of archie indexers being quite so rampant in seeking out
unregistered FTP sites to index. There is some expectation that those
sites in arcie indexes actually want to be there, where it is clear
that appearing in a gopher index is no proof that the admin wants you
in his space.
Richard W. Wiggins
> and indexing it even when you have told them not to?
>
The folks who run Veronica and other servers will have to provide you
with the answer to the question you pose. In the past, we've suggested
that the "right" way to do indexing is for sites to self-publish a
map of their own server. The indexer looks for the file, and if it
isn't there, the site doesn't get indexed. The World-Wide Web community
has been discussing the same problem and has come to the same conclusion.
However, that sort of scheme probably won't be implemented overnight.
And in the meantime, despite response time problems and problems
keeping servers stable, Veronica et al are in fact performing a
terribly useful service. I'm sure no one intends to do harm to
your Gopher or your scholarly endeavors. We've had folks walking
our tree in disruptive ways at our site, too; I sympathize.
In the immediate term perhaps we could devise a simple way for
you to "just say no"? How about if a file exists in the root
of the Gopher tree, under a name like "no-index"? Veronica and
cousins would look for that file first; if it's there, stop
immediately.
/Rich Wiggins, Gopher Coordinator, Michigan State U
Sridhar Venkataraman
| tr...@odin.ins.cwru.edu (Stephen C. Trier) writes:
| >I know you're frustrated about Veronica, but I think the blame is being
| >pointed to the wrong place. By running a Gopher with open access, you
| >are making your information available to anyone. That includes letting
| >those people tell others about it, as Veronica does.
|
| No, I do not agree. It is my right to decide to whom I advertise my
| gopherspace, and it is my right to decide whether I will announce it to
| certain audiences. It is also my right to put any usage limitations on
| the invitation. Indexers ignore any such limitations and announce the
| space as open public space.
Let me get this right first. Weren't you advertizing long ago on
sci.image.processing about the availability of archives of the group
somewhere in *.oce.orst.edu? If that is the case, there is probably no
excuse for letting out the info in the first place and complaining now that
veronica is coming after you.
I agree that it is not nice of veronica to come to you when you didn't ask
for it but you are perfectly justified in running a tcp wrapper and
rejecting all connections from veronica sites (akin to ftp/archie).
I run a server which isn't registered but available for indexing for Veronica.
So my vote goes for indexing any site which has a link on any indexed site.
Sridhar.
--
Sridhar Venkataraman ASU, Tempe, Arizona USA sri...@asuvax.eas.asu.edu
<A HREF="http://homecheese.eas.asu.edu/people/sridhar.html">Sridhar.</A>
Al
>"the person to tell" was told that I did not want my unregistered
>gopherspace indexed. That didn't solve it, because not only is the
>space still in the veronica indexes, another indexer assaulted the
>space today. (And this indexer was someone who I had already told I did
>not want the space indexed. He was doing it again despite his saying he
>wouldn't.)
>
>So, how do you keep veronica indexers from assaulting your gopherspace
>and indexing it even when you have told them not to?
>
Well, rather than argue, I modified the latest gopherd source to
include a copy of the login_access stuff from the secure login code.
That means I can edit a file in /etc with extremely flexible control
over individual hosts or entire domains to allow, or block access.
I like to provide public access, and I like a simple way to remain in
control if some folks out there become a little overzealous in bombarding
my system.
I've been running this stuf for some weeks. If anyone wants the diffs,
or source, they can email me.
Al
John Stanley
>Let me get this right first. Weren't you advertizing long ago on
>sci.image.processing about the availability of archives of the group
>somewhere in *.oce.orst.edu?
You are almost correct. I was advertising, in the newsgroup
sci.image.processing, to the users of sci.image.processing, an archive
of that group for use that group for accessing articles.
I was not advertising a general purpose free-for-all come take what you
want and look at all the rest gopherspace.
>If that is the case, there is probably no
>excuse for letting out the info in the first place and complaining now that
>veronica is coming after you.
Veronica is not a reader of sci.image.processing. Veronica is not
indexing just the sci.image.processing part of my gopherspace. Veronica
is not advertising it as just a sci.image.processing archive.
>I agree that it is not nice of veronica to come to you when you didn't ask
>for it but you are perfectly justified in running a tcp wrapper and
>rejecting all connections from veronica sites (akin to ftp/archie).
Please provide a list of all veronica sites. Include any Jughead or
other indexing sites. Keep in mind that the last indexing pass from UNR
came from nscee.edu.
Now, tell me again how easy it is to dissallow indexing sites.
>I run a server which isn't registered but available for indexing for Veronica.
That's nice.
>So my vote goes for indexing any site which has a link on any indexed site.
I would point out two tiny details to you: I have no control over who
puts links to my gopherspace in theirs, so assuming that I approve of
indexing because there exists a link somewhere is specious.
Second, it doesn't matter how you vote. I don't run my systems or
decide who authorized users are based on your votes.
Stephen C. Trier
You are really close to the point I'm trying to make. How many of
those "the material is available for anonymous FTP at x.y.z" messages
are posted by the people responsible for x.y.z? Not many. Some, yes,
but probably no more than a quarter. (I know I keep refering people to
boombox, hopf and wuarchive, but I'm not responsible for any.)
I'm willing to accept the postulate that when one grants permission for
someone else to access data, without an explicit demand for secrecy,
one also grants permission to reveal the location of that data.
Whether or not there is a solid ethical basis for this, it is the
convention of the Internet culture. Since you don't have any idea who
has permission to access your data, you cannot make any presumption of
having demanded secrecy from them. You have as much as said that you
expect word-of-mouth spread of your site's existence.
All Veronica does is to take advantage of this convention to automate
the process. Veronica permits you to better serve the internet
community by helping people who need your information find it. As
those silly GTE "Everything Pages" commercials say, "Businesses that
want to be successful advertise in it. Those who don't, don't." I
wouldn't say that your gopher can't be "successful" without being in
Veronica, but I see permitting Veronica access to be part of my
gopher's duty.
Anyway, I now know I'm not going to change your mind. I don't even
know why I'm writing this. :-) Although I think it will bring you
nothing but continued frustration as the years go by, I wish you luck
with your isolated gopher.
Stephen
Stephen C. Trier
Richard W. Wiggins <r...@mugwump.cl.msu.edu> wrote:
>How about if a file exists in the root
>of the Gopher tree, under a name like "no-index"? Veronica and
>cousins would look for that file first; if it's there, stop
>immediately.
Now that is a GOOD idea. I like it!
It would be especially effective if the gopher server authors would be
willing to put a mention of it in their documentation.
This would provide a very simple way for any gopher server to refuse
Veronica indexing without having to contact anyone. This is just what
the Gopher/Veronica/Jughead system needs.
Stephen C. Trier
solution is better than this one.
What could be done is to make the gopher "subscription only". To do
this cleanly, you need to use gn as your server or run two gopherds.
Make a minimal top-level server which is public-access and resigned to
Veronica indexing. If it is only a top-level menu, that Veronica
indexing won't be much of a problem. This server should include a
description of the rest of the server, who is welcome to subscribe,
instructions on how to subscribe, and a link to the rest of the
server.
The "rest of the server" will have access granted only to specific
machines, granted as people request it. With gn, it is a matter of
using .access. In gopherd, you would need to run a second gopher
server on a different port.
The subscription process could be done through mail, gopherd ASK
blocks, or gn forms.
Any human who wanted to use the system from a machine that ran an
indexer would be told to use a different machine.
This is the only way I can think of to have restricted public access.
Anything less is security through obscurity and worthless.
If all someone is worried about is veronica access, rather than limiting
the people who can access, Rich's no-index solution is superior.
John Franks
>
>In the immediate term perhaps we could devise a simple way for
>you to "just say no"? How about if a file exists in the root
>of the Gopher tree, under a name like "no-index"? Veronica and
>cousins would look for that file first; if it's there, stop
>immediately.
>
This is an excellent idea except the the file should perhaps be in an
invisible directory so that it doesn't show up on menus.
John Franks Dept of Math. Northwestern University
jo...@math.nwu.edu
John Doyle
respond.
I index top-level menus of gopher servers for the purpose of running a
jughead database of those menu items, and to detect the existence of new
gophers, and to detect new telnet links (which I forward for possible
inclusion in the Hytelnet database).
Last month when Mr Stanley requested that no further tree-walks of his
site be done I marked my gopher server list accordingly. I didn't think
at that time about the possibility of another server being created.
Subsequently Mr Stanley moved his server from port 70 to port 71, this was
automatically detected as a new server and indexed. When Mr Stanley
informed me this weekend that his site was still being indexed, I did what
I should have done before, and prevented the program from tree-walking any
server on his machine, as opposed to just those servers presently known.
The idea from Rich Wiggins that a marker in the root directory should be
used to notify programs to "descend no further", seems a good one. I will
set my program to detect a "no-index" file. However, I'm not sure that
administrators would want extraneous files cluttering their main menu.
Seems that a Gopher+ marker might be a cleaner way of handling it. As
this issue seems to generate excessive emotional heat it would be good to
solve it as soon as possible.
John Doyle. Washington & Lee University. doy...@liberty.uc.wlu.edu
John Stanley
>You are really close to the point I'm trying to make. How many of
>those "the material is available for anonymous FTP at x.y.z" messages
>are posted by the people responsible for x.y.z? Not many.
Under archie, all of them.
>All Veronica does is to take advantage of this convention to automate
>the process.
In the process of automating the "convention", veronica has decided
that every file in every directory needs to be able to be looked up by
name by every person on the Internet. Even files that NOBODY in their
right mind would ever look up, but are there in case you are already in
the space and need it. If it's there, it gets indexed. And it uses
someone else's computer cycles in the process.
>Veronica permits you to better serve the internet community
>by helping people who need your information find it.
People who need my information are told once a month how to find it.
They are told about the already existing content index, which sure beats
trying to use veronica on the titles for the files I have. Do you think
there is some value in being able to look for a file named "17245"?
Can you decide if you want to know what's in that file based on that
title?
>As
>those silly GTE "Everything Pages" commercials say, "Businesses that
>want to be successful advertise in it. Those who don't, don't."
I don't measure success of a gopher in the number of connections from
horny picture-seeking grad students it gets. In any case, my gopherspace
is not a business, it is a sideline. Nobody pays me to run one. If my
gopherspace fails because nobody uses it (hardly likely, since I use it
and I am the one who decides if it is a success or not) that is fine.
>I
>wouldn't say that your gopher can't be "successful" without being in
>Veronica, but I see permitting Veronica access to be part of my
>gopher's duty.
Your gopher is not my gopher. You can assign whatever duties to your
gopher you want. To mine, I have assigned serving humans and not robot
auto-indexers. If I recall correctly, when you got indexed, it took
twenty seconds. If that was all it took then I wouldn't find it so
objectionable, either.
>nothing but continued frustration as the years go by, I wish you luck
>with your isolated gopher.
Are we at the point where we know that keeping abusive robo-indexers
out is a matter of "luck"?
If by "luck" you mean continued successful use, then it isn't a matter
of luck. My gopherspace will be successful because I define success as
making looking up information for me and the users I am paid to support
easier.
I thought I was doing a socially useful thing by allowing other people
(people, not automated indexers) to look things up too. If it isn't
socially useful unless I let scavengers wander automatedly through every
nook and cranny, then I guess it won't be socially useful.
Stephen C. Trier
John Doyle <doy...@liberty.uc.wlu.edu> wrote:
>Seems that a Gopher+ marker might be a cleaner way of handling it. As
>this issue seems to generate excessive emotional heat it would be good to
>solve it as soon as possible.
Make it something plain gopher users can use, please. With "no-index",
I could just drop it in my root "menu" file (remember, not everyone
runs gopherd) and be done with it. If it were gopher+-only, only
gopher+ people could use it.
Arjan de Vet
>In the immediate term perhaps we could devise a simple way for
>you to "just say no"? How about if a file exists in the root
>of the Gopher tree, under a name like "no-index"? Veronica and
>cousins would look for that file first; if it's there, stop
>immediately.
Why only in the root? I would some directories being indexed but others
not (local stuff in Dutch for instance). I would suggest a .noindex file
for each directory directory.
Arjan
--
Arjan de Vet -*-*-*-*-*-*-*-*-*-*-*-*-*-*- Eindhoven University of Technology
Internet : de...@win.tue.nl -*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*- the Netherlands
X.400 : c=nl;admd=400net;prmd=surf;o=tue;ou=win;s=devet -*-*-*-*-*-*-*-*-*
Fred Barrie
>
>The idea from Rich Wiggins that a marker in the root directory should be
>used to notify programs to "descend no further", seems a good one. I will
>set my program to detect a "no-index" file. However, I'm not sure that
>administrators would want extraneous files cluttering their main menu.
>Seems that a Gopher+ marker might be a cleaner way of handling it. As
>this issue seems to generate excessive emotional heat it would be good to
>solve it as soon as possible.
>
I agree with John, the veronica indexer will now support the "no-index"
file.
For those who would not like to have "no-index" not show up on your
menus you can move the information from say your "About this gopher" into
the file no-index and create a .link or .cap file to mask the no-index file
to "About this gopher" or whatever you want and as soon as the veronica
indexer sees this file it will stop indexing at that level...this would
imply that if you don't want top level indexes put it on the top if you
don't want xyz level not to be indexed put it there.
This will be added to the FAQ...
Fred
Richard W. Wiggins
> I agree with John, the veronica indexer will now support the "no-index"
> file.
>
> For those who would not like to have "no-index" not show up on your
> menus you can move the information from say your "About this gopher" into
> the file no-index and create a .link or .cap file to mask the no-index file
> to "About this gopher" or whatever you want and as soon as the veronica
> indexer sees this file it will stop indexing at that level...this would
> imply that if you don't want top level indexes put it on the top if you
> don't want xyz level not to be indexed put it there.
Folks who feel strongly about not being indexed might want to leave
that file in place with a title like:
This Gopher server [folder] is not indexed externally
..with text in that file describing the fact that that's the policy of
the administrator. This would let users of the service, who presumably
arrived due to private invitation, know that they won't be able to
look up things via the external indexer.
That may seem excessive, but there have been many times that I've
fired up Archie to locate a file *within an FTP site that I'm already
connected to*. External indexers are a very powerful part of information
discovery on the net, in my opinion.
If you don't want to be indexed but you don't want to make this statement,
then use Fred's trick -- use "no-index" as your About file.
In any event, it sounds like a consensus has been reached. I agree that
a Gopher+ marker would be superior, but the need is apparently immediate
at least for some folks.
/rich
Paul Lindner
>In article <259itv$10...@msuinfo.cl.msu.edu> r...@mugwump.cl.msu.edu (Richard W. Wiggins) writes:
>>
>>In the immediate term perhaps we could devise a simple way for
>>you to "just say no"? How about if a file exists in the root
>>of the Gopher tree, under a name like "no-index"? Veronica and
>>cousins would look for that file first; if it's there, stop
>>immediately.
>>
>This is an excellent idea except the the file should perhaps be in an
>invisible directory so that it doesn't show up on menus.
Get patchlevel #6 of the Unix gopher code.
I pulled a veronica block out of hat just for this fun stuff. It's in
the topmost item-info block
You can have:
+VERONICA:
treewalk: no
or
+VERONICA:
treewalk: yes
The default is "yes" nyahh nyahh!
--
| Paul Lindner | lin...@boombox.micro.umn.edu | Slipping into madness
| | Computer & Information Services | is good for the sake
| Gophermaster | University of Minnesota | of comparision.
///// / / / /////// / / / / / / / / //// / / / / / / / /
Nick Andrew
>[ Article crossposted from comp.infosystems.gopher ]
>[ Author was John Stanley ]
>[ Posted on 22 Aug 1993 02:51:26 GMT ]
>So, how do you keep veronica indexers from assaulting your gopherspace
>and indexing it even when you have told them not to?
This is a little late, but the paradigm is wrong.
The server should be generating the indexes. And the server should
keep a list of Veronica sites to update with the listing.
That way you as a server can choose whether or not you are listed, can
index your gopherspace with the frequency appropriate. There's no need
to index yourself 10 times if you can index once and send to 10 sites.
The second point about sending the indexes direct to the Veronica servers
allows you to control their access. If a weekly update is required you
can send weekly.
FTP sites should do this too - if they want to be listed in archie then
they should make the index themselves (most already do - lslr.Z) and
send it (or the updates) to the appropriate archie servers.
Nick.
--
Kralizec Dialup Unix (Public Access) Data: +61-2-837-1183, 14400 24hrs 8N1
Zeta Microcomputer Software Data: +61-2-837-1868, 2400 24hrs 8N1
P.O. Box 177, Riverstone NSW 2765 Plan: To beat Gnuchess 4.1 !
Larry W. Virden
the relevant current info is this:
Veronica currently indexes all the gophers it can find, whether the
person who owns it wants it indexed or not.
There may be no way to avoid the initial indexing.
After an owner of a site has decided, for what ever reason, they no
longer want their files to be indexed, there should be some mechanism
that can be put into place that causes the following to happen:
1. All indexes for that site to be removed from all Veronica (and
similar) databases.
2. All future indexing to cease.
Could this not be achieved by adding some code to Veronica (Jughead, etc.)
which look to see if a file called 'Hey! Stop looking in here Veronica!'
or some such nonsense (naming conventions as necessary) is present - and
if it is, indexing cease and all associated indexes removed?
--
:s
:s Larry W. Virden INET: lvi...@cas.org
:s Personal: 674 Falls Place, Reynoldsburg, OH 43068-1614
Reinier Post
>Barrie) writes:
>> I agree with John, the veronica indexer will now support the "no-index"
>> file.
[comments deleted]
Please name the file '.no-index', so it will be hidden automatically!
(At least, this is what happens with Unix servers and clients - I don't
know about other systems.)
--
Reinier Post rein...@win.tue.nl
Stephen C. Trier
Reinier Post <rein...@win.tue.nl> wrote:
>Please name the file '.no-index', so it will be hidden automatically!
>(At least, this is what happens with Unix servers and clients - I don't
>know about other systems.)
It would seem to me that the "file" has to be visible on the menu in
order for Veronica to see it. Making it "hidden" would also make it
useless.
BTW, a minor nit: Please don't assume U. Minn. gopherd makes up all
"Unix servers". There are several other gopher servers for Unix, and
at least one has no special handling for .-files.
John Doyle
: The server should be generating the indexes. And the server should
: keep a list of Veronica sites to update with the listing.
: That way you as a server can choose whether or not you are listed, can
: index your gopherspace with the frequency appropriate. There's no need
: to index yourself 10 times if you can index once and send to 10 sites.
Everyone agrees the server should do the indexing. Basic trouble is one of
inertia, not enough administrators would get around to doing it. Ideal
solution would be for the server code to create the index automatically at
a pre-set configurable time. Indexing program would check for the existence
of the index file (could build up experience to know what is the most probable
day and time for a particular site):
if the index file doesn't exist, indexer would treewalk the site
if the index file does exist but is empty, then indexer would go away
(treewalking would also be stopped by the "no-index" file in any branch,
or by the "veronicaindex: no" in gopherd.conf)
if the index file does exist, then there needs to be some way of knowing
when it was created, so the indexer can match with its record on the
previous contact date/time. Gopher+ is one way, and reserving the first
line of the file for such admin info would be another (indexer grabs just
the first line of the file and disconnects if it already has the file).
A mechanism would be good to send a type back to the client that says
more-or-less, 'the following object exists, but don't display it'.
John Franks
> In article <25hp39$c...@wsinis10.info.win.tue.nl>,
> Reinier Post <rein...@win.tue.nl> wrote:
> >Please name the file '.no-index', so it will be hidden automatically!
> >(At least, this is what happens with Unix servers and clients - I don't
> >know about other systems.)
>
> It would seem to me that the "file" has to be visible on the menu in
> order for Veronica to see it. Making it "hidden" would also make it
> useless.
>
The suggestion has been made that you rename some file you want to
have displayed to "no-index". For example, if you have a file called
"about.txt" with menu title "About this gopher" you can rename the
file no-index and adjust your .cap/ or .names or menu file accordingly.
This is a bit of a hack, but is easy enough to do. It can be done
at any level to stop indexing there.
I have done this for part of one of my servers which shouldn't be indexed.
Let's see if it works.
--
Stephen C. Trier
John Franks <jo...@math.nwu.edu> wrote:
>The suggestion has been made that you rename some file you want to
>have displayed to "no-index".
Ah, I see. Veronica will be looking at the selector, then, not the
title?
<F/X: light dawns above Stephen...> :-)
John Stanley
>Everyone agrees the server should do the indexing. Basic trouble is one of
>inertia, not enough administrators would get around to doing it.
Define "enough". If the value of the centralized index servers is so
great then you should have no fear of people not indexing themselves
voluntarily.
I don't agree with your inertia claim. It is trivial (for crontab-able
systems) to set up indexing on whatever schedule is necessary. Those
parts of my space that need indexing are indexed every night --
something I set up months ago and haven't touched since.
> if the index file doesn't exist, indexer would treewalk the site
No, if the index files doesn't exist, assume the admin turned off the
automated server-indexing feature because he didn't want to be indexed.
>(treewalking would also be stopped by the "no-index" file in any branch,
>or by the "veronicaindex: no" in gopherd.conf)
If you actually look at the gopher documentation, you would almost never
know that veronica existed. The only reference in the 2.04 docs to
veronica is a comment in a change-log that something was done to make
veronica indexing easier. There is no reference in 1.12S. There is
certainly nothing in there that says anything about how to prevent
veronica indexing with a "veronicaindex:" flag in the conf file. There
isn't even anything that says "if you run a gopher server, veronica will
come."
Yet the default assumption used by indexers is that everyone knows what
they are and wants them to do it. If it isn't in the documentation, how
can you assume that anyone knows it, much less everyone?
>A mechanism would be good to send a type back to the client that says
>more-or-less, 'the following object exists, but don't display it'.
If you are looking for a specific file, then you can ask for that file.
Even if it has type=X, you will get a success return. The use of the
name .no-index (a hidden file) was suggested but shot down because it
wouldn't show up in the menu -- it will show up if the indexer bothers
to ask for it. For those clients that don't hide .-files, then .no-index
can be put in the conf file to be ignored, just like dev and bin.
sea...@nwnet.net
In article <256mvf$6...@gaia.ucs.orst.edu>, <sta...@ruby.oce.orst.edu>
writes:
> Ok, I thought I had the problem solved when someone who claimed to be
> "the person to tell" was told that I did not want my unregistered
> gopherspace indexed. That didn't solve it, because not only is the
> space still in the veronica indexes, another indexer assaulted the
> space today. (And this indexer was someone who I had already told I did
> not want the space indexed. He was doing it again despite his saying he
> wouldn't.)
>
You could put your server on a non-standard port number, say
8549, and tell everyone else look there. The veronica indexers
only look at port 70 I believe.
Allen Robel
NorthWestNet
al...@nwnet.net
Richard W. Wiggins
John's point about inertia is perfectly apt. We've been arguing since
the day we learned about Veronica that the "right" way to do indexing
was to have each server offer its own index. In fact my colleague
Dennis Boone has posted prototype code to show how it could be done.
The goals are: efficient searching (ie ability to suppress indexes
of subdirectories of little broad interest), allowing subdomains to gather
indexes (ie a campus has a good index of all campus Gophers) and
efficiency (ie multiple index servers need not walk one's tree).
The idea hasn't caught on. So, so far, we're following the model of
Veronica et al walking the tree of each server. I think that's the
best model to follow, even after the day when servers offer their own
indexes. There *is* real value to Veronica and its cousins. Inertia
includes the fact that U Mn doc doesn't tell the new server admin much
about Veronica, as you point out. This stuff is evovling; it wasn't
handed down from the mount.
This group has come to consensus on a mechanism to allow a site to
suppress indexing of entire servers or parts of a server's tree.
I hope that U Mn documents the mechanism, and we can be done with this
thread. The default condition should be that your information is
available freely to all comers, and if you don't want the robotic ones
visiting, then you have a simple scheme now to tell them "please don't."
/Rich Wiggins, Gopher Coordinator, Michigan State U
In article <25inkk$6...@gaia.ucs.orst.edu> sta...@skyking.oce.orst.edu (John
John Stanley
>
>John's point about inertia is perfectly apt.
Which John? I will assume that you mean Doyle, since you are replying to
my posting.
>We've been arguing since
>the day we learned about Veronica that the "right" way to do indexing
>was to have each server offer its own index. In fact my colleague
>Dennis Boone has posted prototype code to show how it could be done.
You have provided "prototype", "how it could be done" code and you
expect each admin of a gopher server to take that and write an indexer
for his space? No wonder you think there is inertia.
If you want people to index their spaces you need to provide the actual
code to do it. If you ain't interested enough in getting an index from
me to provide me with easily compiled source, you don't have any right
to complain when I don't take my time writing code to do it for you.
>So, so far, we're following the model of
>Veronica et al walking the tree of each server. I think that's the
>best model to follow, even after the day when servers offer their own
>indexes.
You think you should be bashing on someone's system to generate an
index even after the admin has bashed on his own system to create the
index you want to create? What an incredible waste of resources. You
must be kidding, right? A server admin takes his time to create an index
for you, but you want to create your own anyway?
You know, after seeing all the hassle it is to get out of an index, and
now this "index it anyway" statement, I get the feeling that
cs people doing this are afraid that they are going to lose their jobs
if they don't show how many sites and how big an index they can make.
Not a fact, just a feeling.
>Inertia
>includes the fact that U Mn doc doesn't tell the new server admin much
>about Veronica, as you point out.
It doesn't tell the server admin ANYTHING about veronica. That is about
as "not much" as you can get. (We know there was a change to make it
easier, but what it IS and what it DOES is a big fat secret. That it may
bash on your server for long periods of time during the middle of the
day isn't even hinted at.)
>I hope that U Mn documents the mechanism,
So do I. That does not mean that individual communications should be
ignored, however. Telling an indexer by mail to stop should become an
accepted way of getting them to stop.
It is interesting to note, though, that the most convenient way of
documenting veronica (copying the veronica FAQ into the doc directory
before packaging up a distribution) is not being done.
>The default condition should be that your information is
>available freely to all comers, and if you don't want the robotic ones
>visiting, then you have a simple scheme now to tell them "please don't."
We do NOT have a simple scheme NOW to tell them not to index. The
address listed in the FAQ to say "don't" doesn't work, and talking to
them directly doesn't work.
At the point where veronica/et.al. obeys .no-index files and the
existance of such is fully and clearly explained in the gopher
documentation (so that there is some reason to believe that gopher
admins will know that such an option exists), I will agree with you.
Until the fact that veronica/et. al. indexers exist and will come to
visit you without invitation and beat on your computers AND will
advertise your space as an open public space without a hint of
appropriate use limitations is fully documented in the gopher docs, the
default action should be NOT indexing anything that isn't registered.
Michael Morse
>suppress indexing of entire servers or parts of a server's tree.
>I hope that U Mn documents the mechanism, and we can be done with this
>thread. The default condition should be that your information is
>available freely to all comers, and if you don't want the robotic ones
>visiting, then you have a simple scheme now to tell them "please don't."
Your use of the word "consensus" is a little broad. First, you are only
hearing from a few people, compared to the number of Gopher administrators.
Also, at least one loud voice doesn't like the mechanism, so I can't see how
you think there is consensus. How many others aren't writing because their
viewpoint is already well expressed?
My view: Nobody should run any robotic action on another's CPU, no matter
how trivial the number of cycles, or the frequency of use, and no matter
what the benefits are for the common good, without the express, written
permission of the administrator of that machine. Your opinion, and
consensus you reach with others is irrelevant, because those cycles belong
to somebody else. Because the administrator allows one particular use
(human beings accessing the data with a gopher client) does not mean that
other use is permitted (indexing). Also, just because you can do it,
doesn't mean it's permitted.
I could go on and on with examples of this type of behavior in the past,
and the problems it has caused system administrators. To me, this is just
network etiquette. If it ain't yours, and you weren't invited in, stay out.
The default should be *no* indexing. I vote for a system where the system
administrator has to take some explicit action to allow indexers to come in.
--Mike
Michael Morse Internet: mmo...@nsf.gov
National Science Foundation BITNET: mmorse@NSF
1800 G St. N.W. Room 401 Telephone: (202) 357-7659
Washington, D.C. 20550 FAX: (202) 357-7663
Nigel Metheringham
>In article <25ifs4$e...@news.acns.nwu.edu>,
>John Franks <jo...@math.nwu.edu> wrote:
>>The suggestion has been made that you rename some file you want to
>>have displayed to "no-index".
>Ah, I see. Veronica will be looking at the selector, then, not the
>title?
Actually, all you need is a file called .no-index in a directory
tree which you don't want indexing. Then contents could either by
null, or a comment, say
# I don't want Veronica to index me
This won't appear in the menu - it will actually be treated as a
link file, but since there are no links in there it should be
ignored. However, if you send a specific request to your gopher for
that file, so a selector of
0/.no-index
Gopher will send it quite happily. If it ain't there Veronic gets
an error. So Veronica can test presence or absence of that file and
index accordingly.
This also shows up the fact that despite you asking gopherd to
ignore some items, you can get anything out of a gopher tree whether
or not the gophermeister wanted it available (unless he protects it
using Unix protections). So in my case the system should ignore any
usr directories, but a requestor of
1/usr
gives me the directory listing....
Nigel.
--
# Nigel Metheringham -- (NeXT) EMail: nig...@ohm.york.ac.uk #
# System Administrator, Electronics Dept, University of York #
# York YO1 5DD. Phone: +44 904 432374, Fax: +44 904 432335 #
Mitra
: We do NOT have a simple scheme NOW to tell them not to index. The
: address listed in the FAQ to say "don't" doesn't work, and talking to
: them directly doesn't work.
Why do you persist in this flame that talking to them doesnt work,
the Veronica team stated in this group that they didnt index you after you
requested not to be indexed, and the person who did index you has stated
so.
- Mitra
Richard W. Wiggins
>
> Your use of the word "consensus" is a little broad. First, you are only
> hearing from a few people, compared to the number of Gopher administrators.
> Also, at least one loud voice doesn't like the mechanism, so I can't see how
> you think there is consensus. How many others aren't writing because their
> viewpoint is already well expressed?
>
> My view: Nobody should run any robotic action on another's CPU, no matter
> how trivial the number of cycles, or the frequency of use, and no matter
> what the benefits are for the common good, without the express, written
> permission of the administrator of that machine. Your opinion, and
> consensus you reach with others is irrelevant, because those cycles belong
> to somebody else. Because the administrator allows one particular use
> (human beings accessing the data with a gopher client) does not mean that
> other use is permitted (indexing). Also, just because you can do it,
> doesn't mean it's permitted.
>
> National Science Foundation BITNET: mmorse@NSF
> 1800 G St. N.W. Room 401 Telephone: (202) 357-7659
> Washington, D.C. 20550 FAX: (202) 357-7663
I don't know how many folks aren't writing because their views have
been adequately expressed. I do hear only one loud voice, who has
expressed his views amply, and now one other. So far I don't hear
a chorus.
My view: You need to think through what you're proposing as Netiquette.
We're going to see all sorts of information agents evolve in the next
few years. If all of them begin walking the net, *that's* when the
trouble begins. Several months ago someone posted code that would
walk around and find "new" things on the net. You multiply that by
a bunch of individuals, and you've got trouble.
The way to avoid that is with a relatively small number of relatively
well-known indexes. The question is what it means to be a part of
Gopherspace and whether the complainant in this case is, or is not,
part of that cooperative community. He says he deliberately didn't
register his Gopher, and he doesn't want any external links. So that
might lead to a model where "If it's not registered in the list of all
Gophers, don't index it." I think that'd be a shame; a lot of Gophers
are open to all comers but haven't been registered officially.
If you think the "default" should be "no-index" then let's urge those
who provide Gopher servers to ship a starter Gopher tree with that
file (or Gopher+ marker) present. The instructions tell the new Gopher
admin to remove the file if they are willing to be indexed. You've
got your "default."
If you don't think consensus has been reached, then please, do a survey
that polls the silent majority. This strikes me as a minor issue in the grand
scheme of things, amenable to a simple accommodation.
In any event, if "nobody should run any robotic action" that consumes
another's CPU time, you've just defined away Listservs and automatic
mail responders. And Mark Lottor better not do any more hosts reachable
surveys.... It's a question of degree, not of kind.
/RIch Wiggins, Gopher Coordinator, Michigan State U
John Franks
>
> Actually, all you need is a file called .no-index in a directory
> tree which you don't want indexing. Then contents could either by
> null, or a comment, say
> # I don't want Veronica to index me
>
> This won't appear in the menu - it will actually be treated as a
> link file, but since there are no links in there it should be
> ignored. However, if you send a specific request to your gopher for
> that file, so a selector of
> 0/.no-index
>
> Gopher will send it quite happily. If it ain't there Veronic gets
> an error. So Veronica can test presence or absence of that file and
> index accordingly.
This will work for the Univ of Minn gopherd server but not for all
servers. In particular it won't work with the GN server.
>
> This also shows up the fact that despite you asking gopherd to
> ignore some items, you can get anything out of a gopher tree whether
> or not the gophermeister wanted it available.
This is *exactly* the reason that the GN server won't allow the scheme
you describe above. I would personally find it objectionable if any file
in my tree with read permission by the gopher daemon were accessible to
anyone who can find it whether or not I intended it to be made available.
GN will only make available files which are put in menus.
John Stanley
I continue to say that talking to them didn't work because it did not.
Them saying that they stopped is not the same as them actually having
stopped. The UNR index didn't actually go away until last Sunday, and
the University of Pisa is still indexing. Today. AFTER indexing has
stopped (according to the veronica team).
Don't take my word for it: use the keywords "pictures" and "yaquina".
And, of course, let's not forget the trigger that made me bring this
public: the indexer who beat on my systems for half a hour indexing
what he said he wasn't going to index again.
I no longer judge indexers by what they say they will do, only by what
they actually do. They say lots of things. If you choose to believe what
they say without checking, that's your option. You have the ability to
check for yourself.
sumo Kindersley
>been adequately expressed. I do hear only one loud voice, who has
>expressed his views amply, and now one other. So far I don't hear
>a chorus.
i already sent my vote by e-mail to a "person-to-tell", but i guess
the side of "no `right' to index" needs more public support. there is
so much traffic here, i initially decided not to post. i think the "one
loud voice" had a right to be upset, and has, under stress, remained a
rational voice.
i do not administer our gopher site, although i am becoming involved in
content/format. as a net/gopher/veronica/etc. *user* i am VERY UNHAPPY
at the thought that my searches might be giving me pointers to sites
that do not wish to be so searched/indexed. i am VERY HAPPY that so
many valuable public archives are so superbly maintained and indexed.
i agree with others' opinion that by having an accessible site one is
implicitly making one's data publicly consumable. however, not registering
[i still can't believe someone registered someone else's site! jeez!] and
not being indexed should be a siteadmin's right, IMHO. i personally think
no-index should be the default; BUT! i grok the other side, as long as it
is made clear how to prevent indexing.
i see this SORT of problem becoming more and more troublesome in the near
future! i think we'll see more and more special-purpose, of-interest-to-
only-a-select-few, not-for-indexing sort of sites. perhaps it is good to
be discussing these things now, and developing a Netiquette for this with
which we all can live. and not just for veronica, which [IMHO] is getting
an unfair proportion of bashing. we have an incredibly evolving net-world!
sumo kindersley
su...@cs.sfu.ca
PS - one of our net support people leaves a hot kettle of water outside
her office and doesn't mind if a few of us use it. it doesn't mean we
can announce to everyone in the faculty that there is hot water for tea
available there. sure, anyone can physically come and help themselves.
she probably wouldn't tell anyone "No". but what a pain for her!
these are my opinions, not [necessarily] my colleagues' or my employers'.
Albert Lunde
>used to notify programs to "descend no further", seems a good one. I will
>set my program to detect a "no-index" file. However, I'm not sure that
>administrators would want extraneous files cluttering their main menu.
>Seems that a Gopher+ marker might be a cleaner way of handling it. As
>this issue seems to generate excessive emotional heat it would be good to
>solve it as soon as possible.
Making the "no-index" entry an "error" type or an otherwise unused type
could prevent it from being displayed in some clients.
--
Albert Lunde Albert...@nwu.edu
Steve Bacher
Please also allow the filenames NOINDEX and NO-INDEX, for the benefit
of maintainers of the MVS server.
On second thought, maybe this isn't necessary, as long as there is a
gopher menu item that looks like this:
Type=v
Name=Please don't index me
Path=no-index
Host=nowhere.noplace.com
Port=70
About that type: how about a new Gopher type "v" meaning for the eyes
of Veronica indexers only? (Is "v" taken? How about "V" then?)
Then, in addition to no-index, we could send veronica indexers other
pertinent instructions, like
Type=v
Name=Please index me fully (like prefixing the selector strings maybe?)
Path=index-fully
Host=nowhere.noplace.com
Port=70
Type=v
Name=Message to send to a veronica indexer
Path=echo:"hi there"
Host=nowhere.noplace.com
Port=70
--
Steve Bacher (Batchman) Draper Laboratory
Internet: s...@draper.com Cambridge, MA, USA
Stephen C. Trier
Nigel Metheringham <nig...@ohm.york.ac.uk> wrote:
>However, if you send a specific request to your gopher for
>that file, so a selector of
> 0/.no-index
>Gopher will send it quite happily.
Ah! That's behavior specific to U. Minn. gopherd. I wasn't aware that
it would send files not present in a cache file. No wonder the -c
option is such a big deal in that server!
For those who are curious, gn does not behave this way. No file can be
sent unless it is listed in a .cache file. Cache files are generated
by humans (via a menu compiler), so this adds a measure of safety to
the server.
One learns something every day... :-)
Peter Murray
: For those who are implementing the "{.}no-index" hack:
: Please also allow the filenames NOINDEX and NO-INDEX, for the benefit
: of maintainers of the MVS server.
: On second thought, maybe this isn't necessary, as long as there is a
: gopher menu item that looks like this:
: Type=v
: Name=Please don't index me
: Path=no-index
: Host=nowhere.noplace.com
: Port=70
Not a bad idea, except that it is still visible to the users of most gopher
clients. The only client that I know of that doesn't display lines for
unrecognized gopher types is the UMN Unix client. All of the rest display
it with some error type ("<???>", or something similar). If one of the
conditions for this Veronica indexing problem is to make it invisible to
users, this solution won't work. Unless all of the clients are modified,
of course.
(But if people would add type "v" to thier gopher clients, there are two
more I'd like to see added as well!)
Peter
--
Peter Murray, Library Systems Manager pmu...@watson.lib.muohio.edu
King Library Technical Support pemu...@miavx1.bitnet
Miami University, Oxford, Ohio W:513/529-2884
Sheryl Coppenger
>
>> My view: Nobody should run any robotic action on another's CPU, no matter
>> how trivial the number of cycles, or the frequency of use, and no matter
>> what the benefits are for the common good, without the express, written
>> permission of the administrator of that machine. Your opinion, and
>> consensus you reach with others is irrelevant, because those cycles belong
>> to somebody else. Because the administrator allows one particular use
>> (human beings accessing the data with a gopher client) does not mean that
>> other use is permitted (indexing). Also, just because you can do it,
>> doesn't mean it's permitted.
What he said.
>I don't know how many folks aren't writing because their views have
>been adequately expressed. I do hear only one loud voice, who has
>expressed his views amply, and now one other. So far I don't hear
>a chorus.
Does three make a chorus? I've been reading this thread for a while
and I've been quiet because my views were being expressed (by the one
loud voice). I'm appalled by the cavalier attitude toward invading
other people's machines unasked. You expect that kind of thing out
of a 15 year-old bozo who can't think through his actions. You don't
expect that kind of thing from people who are trying to set themselves
up as an authority.
>The way to avoid that is with a relatively small number of relatively
>well-known indexes. The question is what it means to be a part of
>Gopherspace and whether the complainant in this case is, or is not,
>part of that cooperative community. He says he deliberately didn't
>register his Gopher, and he doesn't want any external links. So that
>might lead to a model where "If it's not registered in the list of all
>Gophers, don't index it." I think that'd be a shame; a lot of Gophers
>are open to all comers but haven't been registered officially.
So whose fault is that? I have to have you stomping around in my
private, not-open-to-all-comers gopher just because somebody else
doesn't mind but was too lazy to register? Give me a break. When
I heard about Veronica I assumed that only registered gophers would
be indexed. I can see how registering a gopher might be construed
as permission. But I can't see how just installing gopher software
for a private server makes it open season on your machine.
>If you think the "default" should be "no-index" then let's urge those
>who provide Gopher servers to ship a starter Gopher tree with that
>file (or Gopher+ marker) present. The instructions tell the new Gopher
>admin to remove the file if they are willing to be indexed. You've
>got your "default."
So if a lot of people decide not to remove that marker, are you going
to decide that everybody really wants to be registered but was just
too lazy to remove the marker? Or that, yeah, maybe they really don't
want to be indexed but "it's such a shame" that you'll do it anyway? That
seems to be the logic you've used with unregistered gophers so far.
>If you don't think consensus has been reached, then please, do a survey
>that polls the silent majority. This strikes me as a minor issue in the grand
>scheme of things, amenable to a simple accommodation.
I don't feel consensus, just a certain amount of grudging acceptance
that the Veronica people will keep on doing what they are doing after
throwing a bone to the people who pitched a fit.
Over the past couple of weeks my opinion of the "cooperative gopher
community" has fallen dramatically thanks to this thread. Why can't
your hostility be toward the people who have public gophers and
don't register them rather than toward people who just want to be
left alone?
>In any event, if "nobody should run any robotic action" that consumes
>another's CPU time, you've just defined away Listservs and automatic
>mail responders. And Mark Lottor better not do any more hosts reachable
>surveys.... It's a question of degree, not of kind.
Listservs and automatic mail responders require some action by legitimate
users of the system to get them started. And so far, they have proved
fairly easy to turn off. I disagree that it's a question of degree, not of
kind. But even if it were degree, the fact that I let a user run a
10-member mailing list on my machine doesn't mean I have to let a user
run a 10000-member mailing list. What you seem to be saying is because
I let a legitimate user on my machine run a 10-member mailing list I
have to let you (not a registered user on my machine) run any damn mailing
list you want on it. I don't buy that for a minute.
--
Sheryl Coppenger SEAS Computing Facility Staff she...@seas.gwu.edu
The George Washington University (202) 994-6853
Richard W. Wiggins
> >another's CPU time, you've just defined away Listservs and automatic
> >mail responders. And Mark Lottor better not do any more hosts reachable
> >surveys.... It's a question of degree, not of kind.
>
> Listservs and automatic mail responders require some action by legitimate
> users of the system to get them started. And so far, they have proved
> fairly easy to turn off. I disagree that it's a question of degree, not of
> kind. But even if it were degree, the fact that I let a user run a
> 10-member mailing list on my machine doesn't mean I have to let a user
> run a 10000-member mailing list. What you seem to be saying is because
> I let a legitimate user on my machine run a 10-member mailing list I
> have to let you (not a registered user on my machine) run any damn mailing
> list you want on it. I don't buy that for a minute.
>
> --
>
> Sheryl Coppenger SEAS Computing Facility Staff she...@seas.gwu.edu
> The George Washington University (202) 994-6853
Huh?? I think you misunderstand the example. The point is that there's
all sorts of ways a user can consume CPU on another's server. If I run
a Listserv service, for instance, you can do an LDBASE search that chews
up a lot of my cycles. You could chew up lots of cycles on my server
with an LDBASE search, whether robot-controlled or human controlled.
The problem is the consumption of cycles, not whether the agent is
a human at a keyboard or not.
I have a user who has a script that polls our Gopher server for weather
information every hour. I don't know why he does it, but it's useful
for him. It doesn't cost us lots of cycles, so it's not a problem.
If he pointed it at Weather Underground and chewed up valuable Telnet
slots, it would be a problem.
On the other hand, we've had folks use our Usenet News gateway as if
it were an NNTP feed, without asking and certainly without permission.
That chews up *lots* of cycles. We ask 'em to stop, or nuke 'em if
we have to.
I really don't see how anyone can propose a model of "no robots pointed
at others' CPUs" We've already had examples of private Gopherspace
walkers. The WWW community has lots of web-walkers, and no Veronica;
a bunch of private walkers are going to consume lots more cycles than
a few central indexers.
What are the rules in the Archie community, anyhow? How is it decided
which anonymous FTP sites are, and are not, indexed? This is another
example of where a central index is more efficient than having every
individual poke around a bunch of FTP sites, looking for what they want.
The question, it seems to me, is whether Gopherspace is a mostly-open
or a mostly-closed community. IMHO a default of "every Gopher is an
island" is not in the interest of building a truly useful shared information
base; the question comes down to what it means to be a part of Gopherspace.
Given a reasonable mechanism for posting "no trespassing" signs it seems to
me we can have *both* an open range and fenced-in ranches.
/Rich Wiggins, Gopher Coordinator, Michigan State U
John Stanley
Richard W. Wiggins <r...@mugwump.cl.msu.edu> wrote:
>a bunch of private walkers are going to consume lots more cycles than
>a few central indexers.
1. This isn't necessarily true.
2. The cycles used by "walkers" are spread out over a long time (days,
weeks), while indexers are heavy users for the time they spend.
3. Indexers have absolutely no idea what it is they are indexing, they
just go everywhere, but users who see two titles like "Raw Articles"
and "WAIS Index to Articles" are not going to enter the raw articles
directory.
>The question, it seems to me, is whether Gopherspace is a mostly-open
>or a mostly-closed community.
Why should there be an either/or answer? Why can't some be partly open
and some fully open?
>Given a reasonable mechanism for posting "no trespassing" signs it seems to
>me we can have *both* an open range and fenced-in ranches.
Were there such a mechanism ...
Ed Krol
data from outside eyes.
The point isn't really whether or not its indexed or whether or
not its registered, the point is you have private data which
you don't want passed around publicly. Like it or not the
culture is that anything I can get to on a gopher server is
there for the looking. That may not be the legality of it but
that is the culture. Not indexing or not registering makes
it a little harder, but if one of your anointed users tells a
friend, who tells a friend, who posts it to some widely read
newsgroup, you've got no privacy.
If you really care you need to use the tools you have to restrict
access.
If you feel violated because you were snooped at they said they
would stop. So whats the big deal.
JO...@olis.lib.ox.ac.uk
stress on Gopher servers when they walk their menus. Whether this is
true or not perhaps it would be wise to modify these indexers to walk
menus of a particular server only at off peak hours. This could be done
by setting up a database of Gophers by timezone and then batching index
requests so that they are done at 04:00 local time for the server being
indexed. Of course this assumes that most traffic for a Gopher server
is local and so the server won't be busy at this time.
--
Jose Marques, VM & VSE Systems Programmer
%INCLUDE STDSCLMR
JO...@olis.lib.ox.ac.uk
su...@cs.sfu.ca (sumo Kindersley) writes:
>
>>I don't know how many folks aren't writing because their views have
>
>i see this SORT of problem becoming more and more troublesome in the near
>future! i think we'll see more and more special-purpose, of-interest-to-
>only-a-select-few, not-for-indexing sort of sites. perhaps it is good to
>be discussing these things now, and developing a Netiquette for this with
>which we all can live. and not just for veronica, which [IMHO] is getting
>an unfair proportion of bashing. we have an incredibly evolving net-world!
>
as far as I can see it has no other purpose. To say that since you are
not registered than people should not use your Gopher is silly. If you
setup a Gopher server and put data on it then you are publishing that
data and should not be suprised that people (or robots) read it. If you
don't want your data freely available don't put it on Gopher or restrict
access to it via the mechanisims built into Gopher. If you don't want
other people using your machines CPU cycles then don't run a publically
accessible Gopher server.
>sumo kindersley
>su...@cs.sfu.ca
>
>PS - one of our net support people leaves a hot kettle of water outside
>her office and doesn't mind if a few of us use it. it doesn't mean we
>can announce to everyone in the faculty that there is hot water for tea
>available there. sure, anyone can physically come and help themselves.
>she probably wouldn't tell anyone "No". but what a pain for her!
electricity she is using. Similarly if you provide a Gopher server
then you may own the computer but you do not own the Internet. To
complain about people using your server is nothing short of double
standards (especially as the Gopher server in question is archiving
USENET postings which relies for its existance on other people giving
their CPU cylces free).
>
>these are my opinions, not [necessarily] my colleagues' or my employers'.
JO...@olis.lib.ox.ac.uk
pmu...@watson.lib.muohio.edu (Peter Murray) writes:
>
>Steve Bacher (SEB...@MVS.draper.com) wrote:
>: For those who are implementing the "{.}no-index" hack:
>
>: Please also allow the filenames NOINDEX and NO-INDEX, for the benefit
>: of maintainers of the MVS server.
>
>: On second thought, maybe this isn't necessary, as long as there is a
>: gopher menu item that looks like this:
>
>: Type=v
>: Name=Please don't index me
>: Path=no-index
>: Host=nowhere.noplace.com
>: Port=70
>
>Not a bad idea, except that it is still visible to the users of most gopher
>clients. The only client that I know of that doesn't display lines for
>unrecognized gopher types is the UMN Unix client. All of the rest display
>it with some error type ("<???>", or something similar). If one of the
>conditions for this Veronica indexing problem is to make it invisible to
>users, this solution won't work. Unless all of the clients are modified,
>of course.
>
>(But if people would add type "v" to thier gopher clients, there are two
>more I'd like to see added as well!)
>
file with variable length records.
John Stanley
You seem to be very good at making decisions for other people. If they
have a gopherspace, registered or not, you have decided that being
connected to the Internet gives everyone on the Internet the right to do
whatever they want. Robert Morris might present a different opinion.
>This could be done
>by setting up a database of Gophers by timezone and then batching index
>requests so that they are done at 04:00 local time
And now you want to decide for everyone else that 4 AM is off-peak.
>indexed. Of course this assumes that most traffic for a Gopher server
>is local and so the server won't be busy at this time.
And it assumes that gopher servers have nothing else to do but be gopher
servers.
John Stanley
Ed Krol <kr...@ux1.cso.uiuc.edu> wrote:
>It sounds to me like you are more into argueing than protecting your
>data from outside eyes.
Right. Anything you say. Just how old is the article you are replying
to? I can't tell, it has expired here.
>The point isn't really whether or not its indexed or whether or
>not its registered,
Yes, that is the point.
>the point is you have private data which
>you don't want passed around publicly.
Uh, No, Ed, that can't be the point. If that were the point, I would
just make the default action !read,!browse,!search. Apparently you
can't tell what the point is.
>If you really care you need to use the tools you have to restrict
>access.
If I could use the tools I have to restrict access by indexers that are
wasting and abusing my systems resources, I would. You tell me a list
of indexer's systems to deny access to and I will put those in my conf
file (if they aren't there already). Make sure you include any systems
that they might use, even ones outside their own domains. You tell me
how doing that will stop the new indexers, or remove entries in indexes
already existing, and I will think you are a miracle worker.
>If you feel violated because you were snooped at they said they
>would stop. So whats the big deal.
The big deal is that they said they would stop, but they didn't. Should
I have to waste the time of the zone contacts at every indexing site
just to get the indexers to stop doing what they said they would stop
doing, and which they didn't have the right to do in the first place?
sumo Kindersley
>She may own the kettle but she does not own the water supply nor the
>electricity she is using. Similarly if you provide a Gopher server
this does not seem relevant. it's her kettle; she may restrict access
to what's in it. geez i feel silly arguing about boiling water.
> . . . To
>complain about people using your server is nothing short of double
>standards (especially as the Gopher server in question is archiving
>USENET postings which relies for its existance on other people giving
>their CPU cylces free).
>
>Jose Marques, VM & VSE Systems Programmer
> %INCLUDE STDSCLMR
i don't think it's fair to call it a double standard. i hope that those
who maintain the many terrific public archives or support news & the
almost unimaginable net traffic don't think that
"because we do, EVERY SITE should do as much", or
"because we advertise to all & sundry `please come access our archive
every which way you can', ALL OTHER ARCHIVES should do the same"
because i think there is a place for special purpose archives, advertised
to a small group of interested parties. and other netters/robo-harvesters
don't have an over-riding "right" to advertise them to the world at large,
just because they are `physically'[?!] available to the world at large.
sumo kindersley
su...@cs.sfu.ca
JO...@olis.lib.ox.ac.uk
sta...@skyking.oce.orst.edu (John Stanley) writes:
>
>In article <16C3CCB...@olis.lib.ox.ac.uk>, <JO...@OLIS.LIB.OX.AC.UK> wrote:
>
>You seem to be very good at making decisions for other people. If they
>have a gopherspace, registered or not, you have decided that being
>connected to the Internet gives everyone on the Internet the right to do
>whatever they want. Robert Morris might present a different opinion.
about other people using yours then that does look like double standards.
Think about where you got your Gopher software from, you did not write
it nor did you have to pay for it. I bet you even got it via anon-FTP
from another site. If you use the software and the network then the
least you can do to repay the people whose effort made it possible by
making your server available for people to use rather than trying to hide
it. The people running Veronica indexers are not criminals stealing
your CPU cycles rather they are public spirited citizens providing a
valuable service to other network users.
>>This could be done
>>by setting up a database of Gophers by timezone and then batching index
>>requests so that they are done at 04:00 local time
>
>And now you want to decide for everyone else that 4 AM is off-peak.
one period when they are under utilised.
>>indexed. Of course this assumes that most traffic for a Gopher server
>>is local and so the server won't be busy at this time.
>
>And it assumes that gopher servers have nothing else to do but be gopher
>servers.
you don't run your Gopher server on it.
--
Christopher Davis
JM> If you use the software and the network then the least you can do to
JM> repay the people whose effort made it possible by making your server
JM> available for people to use rather than trying to hide it.
He would like it to be available for people to use. The Veronica
robo-indexers are not people. There are such things as directories in
which indexing is a waste of time--for example, a directory that a gonnrp
server serves, where the titles change from day to day, and the selectors
are opaque integers.
I suppose that since olis.lib.ox.ac.uk runs a mailer, that you wouldn't
mind if I routed all my outgoing mail through it? After all, you didn't
write the mailer, you shouldn't mind the disk and CPU used to route,
process, and queue my email...
--
Christopher Davis # People on the net are always telling other people to "get
<c...@kei.com> # a life." It would be so much simpler if there were one
<c...@eff.org> # available under the GPL. "If you use this life, you must
[CKD1] MIME RIPEM # tell other people where to get a life of their own."
Mitra
: The big deal is that they said they would stop, but they didn't. Should
: I have to waste the time of the zone contacts at every indexing site
: just to get the indexers to stop doing what they said they would stop
: doing, and which they didn't have the right to do in the first place?
Once again this allegation ....
According to everyone concerned they did stop when you asked, you
asked Veronica to stop indexing you and they did, you asked another
indexer not to index your original gopher, and it only restarted when it
saw a new one on port 70.
As I understand it, the indexes were still available on the different
sites until they received the most recent updates. This is a deliberately
infrequent occurance in order to minimize the cost of doing the indexing.
The system works - if you ask either of the organized indexing projects
to stop indexing you they do
There is no way to control people unconnected
with the organized indexers accessing publicly available data, any more
than there is to stop someone new indexing all the anonymous ftp sites in
the world.
So quit bitching and moaning at people who are expending considerable
resources providing a public service.
- Mitra
Mitra
JO...@olis.lib.ox.ac.uk
c...@eff.org (Christopher Davis) writes:
>
>JM> == Jose Marques <JO...@OLIS.LIB.OX.AC.UK>
>
> JM> If you use the software and the network then the least you can do to
> JM> repay the people whose effort made it possible by making your server
> JM> available for people to use rather than trying to hide it.
>
>He would like it to be available for people to use. The Veronica
>robo-indexers are not people. There are such things as directories in
>which indexing is a waste of time--for example, a directory that a gonnrp
>server serves, where the titles change from day to day, and the selectors
>are opaque integers.
>
that some menu trees are not worth indexing, I believe that the veronica
system tries to avoid these. From what I have read in this thread the
veronica people have been quite responsible and what problems did occur
were one off.
>I suppose that since olis.lib.ox.ac.uk runs a mailer, that you wouldn't
>mind if I routed all my outgoing mail through it? After all, you didn't
>write the mailer, you shouldn't mind the disk and CPU used to route,
>process, and queue my email...
>--
Before I setup either service and made them generally available I made
sure that I had the CPU and disk space. If I had not had the CPU or
disk space then I would either have not made these services available
or I would have used whatever facilities were in the software to limit
them to local users only.
In his posting Mr Stanley mentioned Robert Morris, I assume this is a
reference to the Internet worm. In his book Clifford Stoll metions
this but he also talks about trust, I agree also that trust is a very
important thing for networks. If I set something up that is of benefit
to our users then if it is appropriate I make it available to external
users. Why do I do this? I do it because I trust other systems admins
to do the same and hence my users will benefit. Mr Stanley is breaking
this trust, he is using software and resources provided by other people
but he does not want to give anything back in return.
Why do you think the people at UMN created Gopher, because they wanted to
provide a service to their users. Why did they make it freely available,
because they felt that by doing so other systems would provide information
via Gopher which would be of benefit to their users. Try reading the
licence agreement for Gopher sometime. It states that if you use it to
provide a publically available Gopher then you don't have to pay, but if
you are running a Gopher on a private network (or if you charge) then you
do have to pay.
>Christopher Davis # People on the net are always telling other people to "get
> <c...@kei.com> # a life." It would be so much simpler if there were one
> <c...@eff.org> # available under the GPL. "If you use this life, you must
>[CKD1] MIME RIPEM # tell other people where to get a life of their own."
John Stanley
<JO...@OLIS.LIB.OX.AC.UK> wrote:
>No, but if you make use of other peoples CPU cycles and then complain
>about other people using yours then that does look like double standards.
1. I am not complaining about people using it.
2. If I am asked to stop using someone else's resources, I stop.
>Think about where you got your Gopher software from,
Ok. Then I will think about the time spent debugging it.
> The people running Veronica indexers are not criminals stealing
>your CPU cycles rather they are public spirited citizens providing a
>valuable service to other network users.
Unfortunately, the "valuable service" they are providing is bashing on
my systems indexing useless titles and advertising them as if they were
their own to decide how to advertise. It is also being provided long
after they have said they would stop.
Sure, it is valuable. But not in this case.
>>And it assumes that gopher servers have nothing else to do but be gopher
>>servers.
>If your machine is busy all the time on other work then may I suggest that
>you don't run your Gopher server on it.
The people who use and benefit from my gopherspace thank you for
deciding for me that I should close it.
John Stanley
>Robot indexers are not people but their output is used by people.
So what?
>I agree
>that some menu trees are not worth indexing, I believe that the veronica
>system tries to avoid these.
Right. Dream on.
>From what I have read in this thread the
>veronica people have been quite responsible and what problems did occur
>were one off.
If the problems had been "one off", I wouldn't have posted anything.
If the indexers had done what they said they would do, I wouldn't have
posted anything.
>>I suppose that since olis.lib.ox.ac.uk runs a mailer, that you wouldn't
>I don't run a mailer
mconnect olis.lib.ox.ac.uk
connecting to host olis.lib.ox.ac.uk (129.67.1.150), port 25
connection open
220 OLIS.LIB.OX.AC.UK running IBM VM SMTP V2R2 on Thu, 02 Sep 93
16:23:54 BST
vrfy postmaster
250 <JO...@OLIS.LIB.OX.AC.UK>
>Mr Stanley is breaking
>this trust, he is using software and resources provided by other people
>but he does not want to give anything back in return.
That's funny, I could have sworn there were people I have never met
connecting to my gopher and ftp every day. I could have sworn I spent a
lot of my own free time writing code to archive and index and interface
uncooperative systems so that people could locate image processing
information quickly, even if they didn't have Internet connectivity.
>Why do you think the people at UMN created Gopher,
This discussion has gotten terribly sidetracked. This issue isn't
gopher, it is indexers who index without permission and in spite of
being asked not to. The indexers are not gopher, and the simple proof
that is the complete lack of documentation of any indexer in any of the
documentation that comes with gopher.
>Try reading the
>licence agreement for Gopher sometime. It states that if you use it to
>provide a publically available Gopher then you don't have to pay, but if
>you are running a Gopher on a private network (or if you charge) then you
>do have to pay.
At this point, the only people who cannot access my gopherspace are
those who are unlucky enough to be where known indexers are, or those
at sites with demonstrated abusers. That is hardly "a private network",
and if someone is charging for using my gopherspace, I would like to
know about it.
John Stanley
>
>Once again this allegation ....
>According to everyone concerned they did stop when you asked,
But according to the logs I have, and the simple act of using the
indexes, they did not. Which should I believe: the logs that show I was
indexed, and the ability to use the indexes to find my site; or the
claims that they did stop?
>you asked Veronica to stop indexing you and they did,
Keywords "pictures" and "yaquina". Try Univ. Pisa. Then tell me I am no
longer indexed. Try "coke machines" at NYSERNET.
> you asked another
>indexer not to index your original gopher, and it only restarted when it
>saw a new one on port 70.
Moving a server from one port to another on a system is not creating a
new server, no matter how hard you want to twist the language. Same
system, same data, same admin. Since the notice did not indicate port
number, and the actions of an indexer are the same no matter what port
is used, if it is abuse to index one port, it is abuse to index
another.
>As I understand it, the indexes were still available on the different
>sites until they received the most recent updates. This is a deliberately
>infrequent occurance in order to minimize the cost of doing the indexing.
It has been more than a week since the UNR indexers finally stopped. You
would think that Pisa could have copied the files within that time.
Now, here is a good poser ... why is it so important to minimize the
cost to the indexers instead of minimizing the cost to the indexed?
If this minimization were a goal, then why isn't the system set up to
let the indexed do local indexing with the results sent to the global
indexes when there is a signficant change?
Not only would that plan allow people to create really useful indexes,
it would allow them to index what they wanted to be indexed, and to
decide for themselves if they wanted to be indexed at all.
Is it this last part that keeps the local indexing from being used?
Could it be the statement from someone involved that they would tree-walk
systems that provided their own indexes anyway?
>The system works - if you ask either of the organized indexing projects
>to stop indexing you they do
Again, this allegation. The facts do not match your reality.
JO...@olis.lib.ox.ac.uk
su...@cs.sfu.ca (sumo Kindersley) writes:
>
[Stuff deleted]
>this does not seem relevant. it's her kettle; she may restrict access
>to what's in it. geez i feel silly arguing about boiling water.
>
>> . . . To
[Stuff deleted]
>
>i don't think it's fair to call it a double standard. i hope that those
>who maintain the many terrific public archives or support news & the
>almost unimaginable net traffic don't think that
>
> "because we do, EVERY SITE should do as much", or
> "because we advertise to all & sundry `please come access our archive
> every which way you can', ALL OTHER ARCHIVES should do the same"
>
>because i think there is a place for special purpose archives, advertised
>to a small group of interested parties. and other netters/robo-harvesters
>don't have an over-riding "right" to advertise them to the world at large,
>just because they are `physically'[?!] available to the world at large.
>
machines resources. However, the Internet is too big for this. Last
time I heard the Internet had over a million hosts, with the period of
doubling being measured in months. It is simply not possible to check
with each admin for each site that Gopher may bring me in contact with
and even if we all did consider the load that would be imposed on these
admins dealing with all these requests. The size of the Internet means
that inders whether we like them or not are going to become a part of
the network environment, the net will be just too big to find anything
without them. Perhaps the best place to resolve these problems would
be in the software. Maybe standardised subject and access restriction
attributes could be added to Gopher servers, this would allow a client
to ignore servers that held subjects of no interest to it or that had
been restriced.
>sumo kindersley
>su...@cs.sfu.ca
>
>these are my opinions, not [necessarily] my colleagues' or my employers'.
>
JO...@olis.lib.ox.ac.uk
sta...@skyking.oce.orst.edu (John Stanley) writes:
[Stuff deleted]
>>If your machine is busy all the time on other work then may I suggest that
>>you don't run your Gopher server on it.
>
>The people who use and benefit from my gopherspace thank you for
>deciding for me that I should close it.
>
I too will have a load problem when term starts again. To solve this I have
obtained funding to get a UNIX PC to migrate our Gopher server to. Throwing
things onto a machine (no matter how valuable they are) without considering
how they will impact that system is not a very sensible way to run a system.
Christopher Davis
JM> In an ideal world I would try to obtain permission before using another
JM> machines resources. However, the Internet is too big for this.
It is, however, not too big to stop doing something when someone asks you
to stop. That is a very simple issue of ethics; if someone asks you not to
do something with/to their equipment, you should stop.
It's as I said earlier about your mailer; just because your SMTP port
accepts connections doesn't mean you're willing to be the mail router for
an entire hemisphere.
--
Michael Morse
>machines resources. However, the Internet is too big for this.
This statement seems to me to be at the crux of this discussion. Since it's
too much trouble *for you* to obtain permissions, you think it's just fine
to impose yourself on others. In other words, trouble for others is not as
important as trouble for you. I disagree. If you can't get permission,
don't do it.
Let's look back at one of the early "resource discovery" projects. I
forget the details (someone will correct me, I'm sure), but some researcher
felt it would greatly benefit the network in general if there was a study
done of which hosts provided which services. So they designed a probe that
would go around the network and bang on each well-known TCP/IP port on
every host and see if the host responded. They were very careful to use
almost *zero* cycles on each machine visited, and network traffic was
minimal. Nonetheless, some system administrators yelled, loudly. The
reason was that not only was not permission obtained, but the project wasn't
really even advertised. What happened was that quite a few system
administrators (much more conscientious than I am) noticed the probe, and,
suspecting crackers, spent a lot of their time trying to track down where it
came from. When they found the "cracker" they were really pissed. The
well-advertised retroactive rationalization from the researcher was that the
benefit to the network in general was well worth a little hassle for a few
system administrators.
I propose the following: any future versions of Gopher server software
should contain a mechanism to prevent indexers (and the default is
"no index"). I will trust writers of indexing software to honor this
mechanism. If you say that making the default "no index" will make
Veronica worthless, then I would say that you may be greatly over-
estimating its value, at least to system administrators, who will be
voting by not changing a simple switch.
--Mike
Michael Morse Internet: mmo...@nsf.gov
National Science Foundation BITNET: mmorse@NSF
1800 G St. N.W. Room 401 Telephone: (202) 357-7659
Washington, D.C. 20550 FAX: (202) 357-7663
John Stanley
Michael Morse <MMO...@nsf.gov> wrote:
>I propose the following: any future versions of Gopher server software
>should contain a mechanism to prevent indexers (and the default is
>"no index"). I will trust writers of indexing software to honor this
>mechanism.
I would go even further towards favoring the indexers. As long as the
existance of indexers and the fact that they will be indexing every part
of the new admin's space (and making that index available to everyone
on the net) is well documented in the gopher distribution docs, AND
the method to disable indexer access is well documented (which would
take about two sentences to do), then let the default be "indexing is
allowed".
>If you say that making the default "no index" will make
>Veronica worthless, then I would say that you may be greatly over-
>estimating its value, at least to system administrators, who will be
>voting by not changing a simple switch.
The value of veronica can be measured better by the occurance of those
who explicitely disable it than by measing those who have forgotten
or don't know how to enable it.
JO...@olis.lib.ox.ac.uk
sta...@skyking.oce.orst.edu (John Stanley) writes:
>
>In article <16C3D80...@olis.lib.ox.ac.uk>, <JO...@OLIS.LIB.OX.AC.UK> wrote:
>>I don't run a mailer
>
>mconnect olis.lib.ox.ac.uk
>connecting to host olis.lib.ox.ac.uk (129.67.1.150), port 25
>connection open
>220 OLIS.LIB.OX.AC.UK running IBM VM SMTP V2R2 on Thu, 02 Sep 93
> 16:23:54 BST
>vrfy postmaster
>250 <JO...@OLIS.LIB.OX.AC.UK>
>
[Stuff deleted]
>That's funny, I could have sworn there were people I have never met
>connecting to my gopher and ftp every day. I could have sworn I spent a
>lot of my own free time writing code to archive and index and interface
>uncooperative systems so that people could locate image processing
>information quickly, even if they didn't have Internet connectivity.
>
[Stuff deleted]
I still disagree with your attitude to indexers. Ok, you have lots
of menus with meaningless names like 14367 (or whatever) which are
not worth indexing. I would like to suggest (note: only suggest not
tell or enforce) an alternative where you could use the indexers to
your advantage. I assume that the menu numbers 14378 are the numbers
of the postings from the newsgroup you archive, why not instead of
using just the message number also include the subject of the message.
The menu '14378' would then become something like:
'14378: FAQ for comp.xxx.xxx, monthly posting'
The veronica indexers would then not be wasting their time, you would
also benefit your users by giving them more descriptive menus. This
should also reduce the number of 'horny' undergrads looking for gifs
since they would see at a glance that you did not hold what they were
looking for.
JO...@olis.lib.ox.ac.uk
>
> JM> In an ideal world I would try to obtain permission before using another
> JM> machines resources. However, the Internet is too big for this.
>
>It is, however, not too big to stop doing something when someone asks you
>to stop. That is a very simple issue of ethics; if someone asks you not to
>do something with/to their equipment, you should stop.
>
I understand that the veronica server admins that Mr Stanley contacted have
stopped now, despite some false starts. I do run a small home grown indexer
for the Gophers on campus only and have received no complaints.
>It's as I said earlier about your mailer; just because your SMTP port
>accepts connections doesn't mean you're willing to be the mail router for
>an entire hemisphere.
users to send and receive mail, I therefore disabled routing for external
users. My basic principle is that if something is publically available
then it is up to me to make damn sure it does not affect our system, if it
does I don't make it available. An analogy, if I leave my door open when
I go out and get burgled then the thief is commiting a crime, however, I
should not expect to claim on the insurance :-).
JO...@olis.lib.ox.ac.uk
MMO...@nsf.gov (Michael Morse) writes:
>
>>In an ideal world I would try to obtain permission before using another
>>machines resources. However, the Internet is too big for this.
>
>This statement seems to me to be at the crux of this discussion. Since it's
>too much trouble *for you* to obtain permissions, you think it's just fine
>to impose yourself on others. In other words, trouble for others is not as
>important as trouble for you. I disagree. If you can't get permission,
>don't do it.
>
much trouble *for me* to restrict services on my machine then it fine for
me to impose that load onto everybody on the net?
>Let's look back at one of the early "resource discovery" projects. I
>
But veronica is not secret.
>I propose the following: any future versions of Gopher server software
>should contain a mechanism to prevent indexers (and the default is
>"no index"). I will trust writers of indexing software to honor this
>mechanism. If you say that making the default "no index" will make
>Veronica worthless, then I would say that you may be greatly over-
>estimating its value, at least to system administrators, who will be
>voting by not changing a simple switch.
>
decision being well documented in the software.
John Stanley
>You can also put it the other way round. If as an adminsitrator it is too
>much trouble *for me* to restrict services on my machine then it fine for
>me to impose that load onto everybody on the net?
You have yet to come up with a way to differentiate indexers from normal
users, thus allowing a restriction of one without restricting the other.
You have also ignored the claims by indexers that they will restrict
themselves when asked. You don't have to do diddlysquat to restrict
yourself, and most of the net will not have to do diddlysquat to restrict
themselves, when I ask indexers to restrict themselves. "Everybody on
the net" doesn't run an indexer.
>But veronica is not secret.
Veronica is completely undocumented in the gopher distribution.
Veronica indexers do not tell the people they are indexing that they are
being indexed.
It may not be "secret", but those who do it treat it like it was (when
it comes to the systems they index).
John Stanley
>The menu '14378' would then become something like:
>
> '14378: FAQ for comp.xxx.xxx, monthly posting'
Why not? Because I don't feel like immediately doubling the number of
files I have to maintain. Because I don't feel like modifying the
gopher code to detect news articles and then have it create a title by
having to read the file.
Please do feel free to keep suggesting how I can increase the amount of
my resources I devote to this gopher thing, however.
>The veronica indexers would then not be wasting their time,
No, they would be wasting mine.
>you would
>also benefit your users by giving them more descriptive menus.
I guess giving them a WAIS index to the articles isn't good enough. Do
you really think that anyone is going to want to page through a menu of
4000 articles? Until I aggresively expired it, another of my archives
had about 10,000 articles. That's 500 pages at 20 lines per page. Will
it matter if the titles are long or short?
>This
>should also reduce the number of 'horny' undergrads looking for gifs
>since they would see at a glance that you did not hold what they were
>looking for.
Right. Those who use the keyword search for "pictures" are not going to
come looking in that part of the gopherspace because another part of the
gopherspace has longer titles.
You want more descriptive titles? Well, the titles I use are perfectly
descriptive -- in context. What are "Pictures for 1993"? 1993 Sports
Illustrated models? 1993 Playboy playmates? Well, if you knew the title
of the parent menu (something the indexers don't care to keep track of)
you would know what they were.