Maximum 279 hits returned?

[pats...@my-deja.com]

I have a Perl CGI script that uses WAIS to search a directory of some
450 files based on keywords submitted by the user. No matter how many
files actually contain the keywords, the WAIS call never returns more
than 279 hits.

I don't know much about WAIS. This is a script I inherited, and nobody I
work with seems to know much about it either. I have been searching for
documentation but have not found a solution to this problem.

I wrote a test script with keywords that occur in nearly all of the
files. Using regular expression searches it returns the full number of
them. I wrote it to store the results in hashes or arrays to test if the
limit was somehow set on the data structures, but they get all of them
also.

So it looks to me that WAIS itself is setting this limit. Here is how it
is called in the test script, which is the way it is called in the
original script:

----------------->
$waisq = "/usr/local/bin/waisq"; # point to our version of waisq

$waisct = 0;
$query = "(math mathematics research)";
open(WAISQ, "-|") || exec ($waisq, "-s", $waissource,
"-f", "-", "-S", "$waisindexes/eims.src",
"-m", "1000", "-g", $query);
while (<WAISQ>) {
if (/:headline "(.*)"/) {
$headline = $1;
if ($headline =~ /\<(.*)\>(.*)/) {
$waisct++;
$waishash{$waisct} = 1;
$jobid = $1;
$rest = $2;
$masktype = "mask";
$jobmask = &mask_unmask_Jobid($masktype,$jobid);
$headline = "$rest";
$headlines{$jobid} = $headline;
}
}
}
close(WAISQ);
print "\n WAIS count is $waisct\n";

-----------------------

From the documentation I have found so far, it seems to me that the
"-m", "1000",
part of the exec call to WAISQ should allow for up to 1000 hits.

Is an underlying mechanism set in the installation process or somewhere
that limits the returns to a maximum (in this instance anyway, 279)
aside from what the specific call says?

Or what else might be causing this problem?

Thank you for any suggestions or ideas or references.

Patricia Murray
Programmer/Analyst
American Mathematical Society

Sent via Deja.com http://www.deja.com/
Before you buy.

[Don Rolph]

When you are doing the WAISindexing, what does the printout form the
indexing process tell you regarding number of documents indexed?

WAISsearch may have built into it a default maximum documents returned (my
source showed a default max documents of 40), alternately, if WAISsearch is
being called from a script this may define the maximum number of objects
returned. I had to set my Web search script to return a maximum of 1000
documents so I could return my entire corpus if requested.

Hope this helps!

pats...@my-deja.com wrote:

--

Regards.

Don Rolph w-r...@ds.mc.ti.com WD3 MS10-13 (508)-236-1263

[Brian Borchers]

There's a builtin limit of 100K bytes on the size of the returned results.
Depending on the length of the headlines, this can limit the total number
of returned hits to as few as 200.

If you're just using wais with your own client software, you can
resolve this problem by increasing the buffer size to something larger
than 100K bytes (This needs to be done in both the freewais software
and in all of the client software that uses the wais protocol to
access the data) . However, if you've got client software that isn't
under your control, you'll find that it typically has the same
100Kbyte limit and thus won't work with your modified version of the
wais software.

--
Brian Borchers borc...@nmt.edu
Department of Mathematics http://www.nmt.edu/~borchers/
New Mexico Tech Phone: 505-835-5813
Socorro, NM 87801 FAX: 505-835-5366

[pats...@my-deja.com]

Thanks, Don. The default maximum return for our WAIS is 40, but the
script changes that maximum to 1000. A nightly cron job re-WAIS-indexes
the files. I went through last night's log, and it seems to be getting
them all.

How large is your corpus and how many hits do you get when you set your
maximum to 1000?

I see another message in this thread says the buffer size may be
limiting the hits. I'll look into that too.

Pat

In article <385A894A...@ds.mc.ti.com>,
w-r...@ds.mc.ti.com wrote:
> This is a multi-part message in MIME format.
> --------------651290A32B55C26EF01A5BB7
> Content-Type: text/plain; charset=us-ascii
> Content-Transfer-Encoding: 7bit

> --------------651290A32B55C26EF01A5BB7
> Content-Type: text/x-vcard; charset=us-ascii;
> name="w-rolph.vcf"
> Content-Transfer-Encoding: 7bit
> Content-Description: Card for Don Rolph
> Content-Disposition: attachment;
> filename="w-rolph.vcf"
>
> begin:vcard
> n:Rolph;Don
> tel;fax:508 236 3476
> tel;work:508 236 1263
> x-mozilla-html:TRUE
> org:Advanced Analysis Laboratory
> version:2.1
> email;internet:w-r...@ds.mc.ti.com
> title:Distinguished Member Tech. Staff
> adr;quoted-printable:;;MS 10-13=0D=0A34 Forest
St.;Attleboro;MA;02703;US
> fn:Don Rolph
> end:vcard
>
> --------------651290A32B55C26EF01A5BB7--

[pats...@my-deja.com]

Thank you, Brian. I will pass this information along to the sysadmin and
see if this is the case with ours and if so if he will re-set it. We
run our own copy of WAIS (freeWAIS I think) here.

Pat

P.S.
>> Department of Mathematics
Are you a mathematician? An AMS member?

In article <83e520$p3d$1...@rainbow.nmt.edu>,

[pats...@my-deja.com]

Don, just to let you know that the problem has been solved. There was an
older version of WAIS on the Web server than the one on the development
server. The system administrator copied the newer one over to the Web
server, and now it works properly.

Thanks for your suggestions,

Pat

In article <83lnt0$5qp$1...@nnrp1.deja.com>,

> > > Sent via Deja.com http://www.deja.com/
> > > Before you buy.
> >

> > --
> >
> > Regards.
> >
> > Don Rolph w-r...@ds.mc.ti.com WD3 MS10-13 (508)-236-1263
> >
> > --------------651290A32B55C26EF01A5BB7
> > Content-Type: text/x-vcard; charset=us-ascii;
> > name="w-rolph.vcf"
> > Content-Transfer-Encoding: 7bit
> > Content-Description: Card for Don Rolph
> > Content-Disposition: attachment;
> > filename="w-rolph.vcf"
> >
> > begin:vcard
> > n:Rolph;Don
> > tel;fax:508 236 3476
> > tel;work:508 236 1263
> > x-mozilla-html:TRUE
> > org:Advanced Analysis Laboratory
> > version:2.1
> > email;internet:w-r...@ds.mc.ti.com
> > title:Distinguished Member Tech. Staff
> > adr;quoted-printable:;;MS 10-13=0D=0A34 Forest
> St.;Attleboro;MA;02703;US
> > fn:Don Rolph
> > end:vcard
> >
> > --------------651290A32B55C26EF01A5BB7--
> >
> >
>

[pats...@my-deja.com]

Brian, just to let you know that the problem has been solved. There was

an older version of WAIS on the Web server than the one on the
development server. The system administrator copied the newer one over
to the Web server, and now it works properly.

Thanks for your help,

Pat

In article <83locm$68m$1...@nnrp1.deja.com>,