newbie question - indexing PDF content
202 views
Skip to first unread message
randy
Mar 27, 2012, 11:13:30 PM3/27/12
to Constellio
Hello,
So i have a brand new install and just got it running. One of my goals
it to have Constellio index all of our PDF content.
For the test i created an html page the was just a bunch of links to
some of the PDFs on our site
so the page looked like
<html>
<head><title>PDFs</title></head>
<body>
<a href="http://mydomain/pdf/Bridal1.pdf">http://mydomain/kam/pdf/
Bridal1.pdf</a><br>
<a href="http://mydomain/pdf/Bridal2.pdf">http://mydomain/kam/pdf/
Bridal2.pdf</a><br>
.... a bunch more ......
</body>
</html>
I then went and created a collection called PDFs. In that collection i
created a http-connector.
the start URL is
http://mydomain/jsp/utils/pdfDirList.jsp (this builds the HTML above)
the INCLUSION REGEXES is ^http://mydomain/jsp/utils/pdfDirList.jsp
crawl depth =4
# threads 5
when i save and look at Latest Indexed document i see
Latest traversed documents
2012-03-27 22:06:25
http://test2.unitnet.com/jsp/utils/pdfDirList.jsp
Should i not see the pdfs in these statuses?
if i try to search for a word in the PDF it does not find it
Do i have to enable something to have it index the contents of the
PDFs?
Thanks for helping the newbie
Randy
So i have a brand new install and just got it running. One of my goals
it to have Constellio index all of our PDF content.
For the test i created an html page the was just a bunch of links to
some of the PDFs on our site
so the page looked like
<html>
<head><title>PDFs</title></head>
<body>
<a href="http://mydomain/pdf/Bridal1.pdf">http://mydomain/kam/pdf/
Bridal1.pdf</a><br>
<a href="http://mydomain/pdf/Bridal2.pdf">http://mydomain/kam/pdf/
Bridal2.pdf</a><br>
.... a bunch more ......
</body>
</html>
I then went and created a collection called PDFs. In that collection i
created a http-connector.
the start URL is
http://mydomain/jsp/utils/pdfDirList.jsp (this builds the HTML above)
the INCLUSION REGEXES is ^http://mydomain/jsp/utils/pdfDirList.jsp
crawl depth =4
# threads 5
when i save and look at Latest Indexed document i see
Latest traversed documents
2012-03-27 22:06:25
http://test2.unitnet.com/jsp/utils/pdfDirList.jsp
Should i not see the pdfs in these statuses?
if i try to search for a word in the PDF it does not find it
Do i have to enable something to have it index the contents of the
PDFs?
Thanks for helping the newbie
Randy
randy
Mar 28, 2012, 9:43:48 AM3/28/12
to Constellio
Some more info...
in my http collector i point to a file that displays all the PDFs in a
directory and links to them. it does not seem to traverse these links,
because if i put in the connector a link directly to the PDF file it
indexes it and i can search for keywords in the pdf doc
So i guess the question is, what do i need to do for Constellio to
index all my PDFs in a directory on a web server?
Thanks for any help
On Mar 27, 10:13 pm, randy <randy.par...@unitnet.com> wrote:
> Hello,
> So i have a brand new install and just got it running. One of my goals
> it to have Constellio index all of our PDF content.
>
> For the test i created an html page the was just a bunch of links to
> some of the PDFs on our site
>
> so the page looked like
>
> <html>
> <head><title>PDFs</title></head>
> <body>
> <a href="http://mydomain/pdf/Bridal1.pdf">http://mydomain/kam/pdf/
> Bridal1.pdf</a><br>
> <a href="http://mydomain/pdf/Bridal2.pdf">http://mydomain/kam/pdf/
> Bridal2.pdf</a><br>
>
> .... a bunch more ......
>
> </body>
> </html>
>
> I then went and created a collection called PDFs. In that collection i
> created a http-connector.
> the start URL ishttp://mydomain/jsp/utils/pdfDirList.jsp(this builds the HTML above)
in my http collector i point to a file that displays all the PDFs in a
directory and links to them. it does not seem to traverse these links,
because if i put in the connector a link directly to the PDF file it
indexes it and i can search for keywords in the pdf doc
So i guess the question is, what do i need to do for Constellio to
index all my PDFs in a directory on a web server?
Thanks for any help
On Mar 27, 10:13 pm, randy <randy.par...@unitnet.com> wrote:
> Hello,
> So i have a brand new install and just got it running. One of my goals
> it to have Constellio index all of our PDF content.
>
> For the test i created an html page the was just a bunch of links to
> some of the PDFs on our site
>
> so the page looked like
>
> <html>
> <head><title>PDFs</title></head>
> <body>
> <a href="http://mydomain/pdf/Bridal1.pdf">http://mydomain/kam/pdf/
> Bridal1.pdf</a><br>
> <a href="http://mydomain/pdf/Bridal2.pdf">http://mydomain/kam/pdf/
> Bridal2.pdf</a><br>
>
> .... a bunch more ......
>
> </body>
> </html>
>
> I then went and created a collection called PDFs. In that collection i
> created a http-connector.
> the INCLUSION REGEXES is ^http://mydomain/jsp/utils/pdfDirList.jsp
> crawl depth =4
> # threads 5
>
> when i save and look at Latest Indexed document i see
>
> Latest traversed documents
> 2012-03-27 22:06:25http://test2.unitnet.com/jsp/utils/pdfDirList.jsp
> crawl depth =4
> # threads 5
>
> when i save and look at Latest Indexed document i see
>
> Latest traversed documents
Nicolas Bélisle
Mar 28, 2012, 12:48:52 PM3/28/12
to const...@googlegroups.com
Hi Randy,
You can enable directory listing in your webserver.
Example :
Put everything in /var/www/html/pdfs/ (for Apache 2 on Linux)
Indicate the start URL in Constellio http://domain/pdfs/ as a start url (the urls should list your PDFs)
Include pattern : http://domain/pdfs/
Regards,
Nicolas
--
Vous recevez ce message, car vous êtes abonné au groupe Google Groupes Constellio.
Pour envoyer un message à ce groupe, adressez un e-mail à const...@googlegroups.com.
Pour vous désabonner de ce groupe, envoyez un e-mail à l'adresse constellio+...@googlegroups.com.
Pour plus d'options, consultez la page de ce groupe : http://groups.google.com/group/constellio?hl=fr
--
Nicolas Bélisle
DocuLibre inc.
http://www.doculibre.com
Phone/Tél. 514.655.5185
Skype: nicolas.belisle
Rida Benjelloun
Mar 28, 2012, 1:03:13 PM3/28/12
to const...@googlegroups.com
Hi Randy,
It is also possible tu use file system connector.
Regards.
Randy Paries
Mar 28, 2012, 1:26:39 PM3/28/12
to const...@googlegroups.com
Hey Nicolas,
Thanks for the info. The problem is that I cannot enable directory listing(there is a security problem in my situation) also I cannot move the pdfs from where they are currently. I have to index probably >20K pdfs and their content. So I have to create dynamic pages that show the pdfs for a particular user.
http://mydomain/jsp/utils/pdfDirList.jsp basically is a page that is similar to the apache dir listing. Is there something special with the dir listing?
Thanks again for the help
Randy Paries
Mar 28, 2012, 1:28:23 PM3/28/12
to const...@googlegroups.com
Rida,
Thanks for responding. I was thinking of trying the file system connector, but from what I could tell you could only use the SMB URLs. Everything I have is on a centos box
Randy
From: const...@googlegroups.com [mailto:const...@googlegroups.com] On Behalf Of Rida Benjelloun
Sent: Wednesday, March 28, 2012 12:03 PM
To: const...@googlegroups.com
Subject: Re: newbie question - indexing PDF content
Hi Randy,
Nicolas Bélisle
Mar 28, 2012, 5:02:20 PM3/28/12
to const...@googlegroups.com
Randy,
In your first configuration/email, your inclusion pattern was wrong. It should have been something like :
http://mydomain/pdf
http://mydomain/pdf
If you just put : http://mydomain/jsp/utils/pdfDirList.jsp
The crawler will skip the links to your PDFs in http://mydomain/pdf/...
Regards,
Nicolas
Randy Paries
Mar 28, 2012, 5:16:02 PM3/28/12
to const...@googlegroups.com
Nicolas,
Thank you that was it
randy
Reply all
Reply to author
Forward
0 new messages