[GSoC] Search project

[Pat Tressel]

Vishrut and I are going to be meeting on freenode #sahana-eden at 0530 UTC on Tuesday, Thursday, Friday (that's 11am IST, 7:30am CEST / Stockholm, and 10:30pm Monday, Wednesday, Friday PDT).

Vishrut asked what he might look into for the first meeting, and (by no coincidence) I was about to send out suggestions for same.  ;-)

One big thing we need to handle is access permissions. Documents may be components of other resources, rather than just standing on their own.
If a document is not a component, then it will have permissions depending on things like the user's access to doc_documents, who the owner / owning org is. But if it's a component, does its access depend on that of the primary resource? This might be something to ask Dominic. (I've CC'd him, to alert him that Vishrut may be asking.)

The next question is, can we determine the permissions only from 1) the id of the document and 2) (if it's a component) the type and id of its primary resource?

Once we know whether we just need to know a few types and ids, we can decide how to store or recover that information from a record in Lucene. Lucene can have multiple fields, so we might store the id of the document that the record came from, and the type and id of its primary resource (if those items of info are sufficient for looking up the permission). Or, given the doc id, can we find out what it's a component of? Vishrut -- you might have a look at how component documents are linked to their primary resources, and whether there is a way to query what the primary resource is (that isn't grossly database-intensive).

I'll send a separate note about how to start up Lucene without spawning lots of JVMs. We'll still need to check to make sure we aren't getting more than one. And (as Vishrut mentioned on IRC tonight), we should start up Solr and try it out.

-- Pat

[Pat Tressel]

Vishrut --

The Java you're using is "OpenJDK", which appears to be an open-source re-implementation of Java.  Since it gets a segfault, ignore or get rid of it and get a real one from Oracle.  Besides, it's Java 6, and the current version is 7.  Aaaaaand, I see other people complaining about the OpenJDK JVM dying with a segfault.

https://help.ubuntu.com/community/Java

It might be that some other installed stuff is using the openjdk-6 version of Java that's currently installed.  If you installed it, then it's fine to get rid of it.  Otherwise, you'll need to make sure your JAVA_HOME points to the new one you install.

Then re-install JCC and pylucene.  This looks like a useful example:

http://freethreads.wordpress.com/2012/07/22/pylucene-installation-on-ubuntu-12-04/

-- Pat

[Vishrut Mehta]

Hello Pat,

I have tried all the day long for installing this with every version of OpenJDK ans also the java-sun, but while running the make test, I always get this:

find test -name 'test_*.py' | PYTHONPATH=/home/vishrut/pylucene-4.3.0-1/build/test xargs -t -n 1 /usr/bin/python

/usr/bin/python test/test_BinaryDocument.py 

#

# A fatal error has been detected by the Java Runtime Environment:

#

#  SIGSEGV (0xb) at pc=0x41f40867, pid=29080, tid=1078040256

#

# JRE version: 6.0_45-b06

# Java VM: Java HotSpot(TM) Client VM (20.45-b01 mixed mode, sharing linux-x86 )

# Problematic frame:

# V  [libjvm.so+0x27e867]  unsigned+0xf7

#

# An error report file with more information is saved as:

# /home/vishrut/pylucene-4.3.0-1/hs_err_pid29080.log

#

# If you would like to submit a bug report, please visit:

#   http://java.sun.com/webapps/bugreport/crash.jsp

#

xargs: /usr/bin/python: terminated by signal 6

make: *** [test] Error 125

I had planned to write some scripts and learning Pylucene, but I am not able to run the jvm only because of this error.

Hope if you could have any suggestion ?

Regards,

Vishrut Mehta

[Pat Tressel]

Vishrut --

I have tried all the day long for installing this with every version of OpenJDK ans also the java-sun, but while running the make test, I always get this:

...

# A fatal error has been detected by the Java Runtime Environment:

#

#  SIGSEGV (0xb) at pc=0x41f40867, pid=29080, tid=1078040256

#

# JRE version: 6.0_45-b06

# Java VM: Java HotSpot(TM) Client VM (20.45-b01 mixed mode, sharing linux-x86 )

# Problematic frame:

# V  [libjvm.so+0x27e867]  unsigned+0xf7

...

Hope if you could have any suggestion ?

:-(

Let's try it on some other Linux system.  Maybe it's something to do with the particular Ubuntu version.  I just asked Fran if we could install it on demo.eden.sahanafoundation.org.  That doesn't have any Java installed at the moment, so we wouldn't be interfering with anything currently using Java.

-- Pat

[Vishrut Mehta]

Hello Pat,

Updates on work till now:

1) Installed PyLucene successfully on demo.eden.sahanafoundation.org 

2) Tested PyLucene

3) Tried Apache Solr as an alternative for the project.

4) Wrote some scripts for PyLucene

I was writing some scripts for creating index in PyLucene and came to know that, it needs text to index it from the files. So, from all types of files (like .pdf, .doc, etc) we need to extract text from it using different python libraries and then index it.

So, I wanted to know, which file type we would need ? Like I know we would need PDF and Docx/Doc for sure. Which else ?

So, according to it, we need to create indexes for it.

And regarding search through indexes, I am working currently on creating index for .txt and search through it. I am facing some issues in understanding how to create different fields and what are its advantages and working of it.

Regards,

Vishrut Mehta 

[Fran Boon]

On 11 June 2013 22:08, Vishrut Mehta <vishrut....@gmail.com> wrote:
> So, I wanted to know, which file type we would need ? Like I know we would
> need PDF and Docx/Doc for sure. Which else ?

XLS is the big one missing here - whilst that is primarily for numeric
data, there are text strings in there too which we want indexing.

In general though I wouldn't worry about being exhaustive here - your
job is to build a framework where we can 'easily' plug in additional
formats in future.

F

[Pat Tressel]

1) Installed PyLucene successfully on demo.eden.sahanafoundation.org

We don't really have to have it working on our own systems. I just installed all 5GB (!) of Visual Studio Express so I can use MS C++ to compile JCC and pylucene.  (The consensus for why people are reporting actual crashes after building Lucene is that tools from python.org and pypi are built with MS C++, so building JCC & pylucene with (say) MinGW means there are incompatible runtimes.  Example was given of calling malloc from one C++ variant, and free from a different one.)

2) Tested PyLucene

Can you write up the installation process on the project wiki?

Did using the full Java-style package names in the import statements work?
 

3) Tried Apache Solr as an alternative for the project.

Apparently this is quite popular, just to avoid the pain of figuring out how to compile JCC & pylucene from sources, at least on Windows.

http://stackoverflow.com/questions/338008/windows-build-for-pylucenejcc-on-python-2-6

"I ended up using solr and interfacing it using XML/JSON." -- lazy1
"We did this too. Found it to be the path of least resistance by far!" -- gnibbler

-- Pat

[Vishrut Mehta]

On Wednesday, June 12, 2013 4:55:07 AM UTC+5, Pat Tressel wrote:

1) Installed PyLucene successfully on demo.eden.sahanafoundation.org

We don't really have to have it working on our own systems. I just installed all 5GB (!) of Visual Studio Express so I can use MS C++ to compile JCC and pylucene.  (The consensus for why people are reporting actual crashes after building Lucene is that tools from python.org and pypi are built with MS C++, so building JCC & pylucene with (say) MinGW means there are incompatible runtimes.  Example was given of calling malloc from one C++ variant, and free from a different one.)

So, you mean to tell, as it is not working on our systems, we should not use pylucene ??

 

2) Tested PyLucene

Can you write up the installation process on the project wiki?

Did using the full Java-style package names in the import statements work?

Yeah, I will write it and the import statements worked. :)

 

 

3) Tried Apache Solr as an alternative for the project.

Apparently this is quite popular, just to avoid the pain of figuring out how to compile JCC & pylucene from sources, at least on Windows.

http://stackoverflow.com/questions/338008/windows-build-for-pylucenejcc-on-python-2-6

"I ended up using solr and interfacing it using XML/JSON." -- lazy1
"We did this too. Found it to be the path of least resistance by far!" -- gnibbler

All these comments were on 2008-2009, 4 years back. So, I think in these 4 years lott of things have been done. Though we also need to try out Solr though, what are your suggestions for it. ?

Regards,

Vishrut Mehta

 

-- Pat

[Fran Boon]

On 12 June 2013 00:55, Pat Tressel <ptre...@myuw.net> wrote:
>> 1) Installed PyLucene successfully on demo.eden.sahanafoundation.org

> Can you write up the installation process on the project wiki?

Yes please:)
e.g. I need to do an install for Pootle
If the overall process isn't repeatable then it's not usable ;)

>> 3) Tried Apache Solr as an alternative for the project.

> "I ended up using solr and interfacing it using XML/JSON." -- lazy1

My interest is in having a shared Lucene install for multiple Eden instances.
This is useful for:
* Load balancing of Eden instances in a single deployment
* Offering Hosted Eden instances with full funcitonality without a
separate Lucene per install.

Does this suggest either PyLucene or SOLR as preferred?

F

[Pat Tressel]

Fran --

>> 1) Installed PyLucene successfully on demo.eden.sahanafoundation.org

> Can you write up the installation process on the project wiki?

Yes please:)

...

>> 3) Tried Apache Solr as an alternative for the project.

> "I ended up using solr and interfacing it using XML/JSON." -- lazy1

My interest is in having a shared Lucene install for multiple Eden instances.
This is useful for:
* Load balancing of Eden instances in a single deployment
* Offering Hosted Eden instances with full funcitonality without a
separate Lucene per install.

+! !

Does this suggest either PyLucene or SOLR as preferred?

With Solr, we could have only one JVM running.  But...the Eden instances would have different data, so we'd need to include the instance ID in the attributes, or otherwise keep the instance data separate.  (Haven't checked whether one Solr instance can host isolated databases.)

Vishrut --

"I ended up using solr and interfacing it using XML/JSON." -- lazy1

"We did this too. Found it to be the path of least resistance by far!" -- gnibbler

All these comments were on 2008-2009, 4 years back. So, I think in these 4 years lot of things have been done.

But not, apparently, providing better build instructions, nor installers, for Windows...

-- Pat

[Pat Tressel]

Fran --

>> 3) Tried Apache Solr as an alternative for the project.

> "I ended up using solr and interfacing it using XML/JSON." -- lazy1

My interest is in having a shared Lucene install for multiple Eden instances.
This is useful for:
* Load balancing of Eden instances in a single deployment
* Offering Hosted Eden instances with full funcitonality without a
separate Lucene per install.

Does this suggest either PyLucene or SOLR as preferred?

Not yet... 

Load balancing:
By "Eden instances in a single deployment", do you mean the separate processes that Apache / Cherokee starts?  Or do you mean load balancing across IP addresses?  (Lumping the Lucene work together is anti-load balancing, no?  If there isn't much text searching, it could still be ok.)  One could bring up multiple Solr / Lucene instances.  Or not.  So one could (or not) do load balancing for Solr / Lucene.  For pylucene / Lucene, there would be one per instance -- each would have a local JVM, which would not span instances in separate machines or VMs.

Hosting:
pylucene would likely have one JVM per Eden instance, but we might be able to get around that.  Solr could probably be set up as one instance, but we're not entirely sure yet.  Thing is, the processed data and indices for each instance would (according to the plan we came up with last night) go in each instance's eden directory.  That would be no issue for a separate instance of Lucene per each, accessed via pylucene.  But we need to find out how to have Solr isolate each instance's data.

We're checking how we can get each JVM's VMID so we can see if we really are connecting to the same one across separate (Apache / Cherokee) processes for one instance.

-- Pat

[Fran Boon]

On 13 June 2013 12:45, Pat Tressel <ptre...@myuw.net> wrote:
>> My interest is in having a shared Lucene install for multiple Eden
>> instances.
>> This is useful for:
>> * Load balancing of Eden instances in a single deployment
>> * Offering Hosted Eden instances with full funcitonality without a
>> separate Lucene per install.
>> Does this suggest either PyLucene or SOLR as preferred?
> Not yet...

> By "Eden instances in a single deployment", do you mean the separate
> processes that Apache / Cherokee starts? Or do you mean load balancing
> across IP addresses?

LB across IPs

> (Lumping the Lucene work together is anti-load
> balancing, no? If there isn't much text searching, it could still be ok.)

I would be envisaging text-searching as not being such a major activity.
LB of Lucene itself would be an entirely diff project.
I just want to ensure we don't build a 1 Eden <> 1 Lucene dependency

> One could bring up multiple Solr / Lucene instances. Or not. So one could
> (or not) do load balancing for Solr / Lucene. For pylucene / Lucene, there
> would be one per instance -- each would have a local JVM, which would not
> span instances in separate machines or VMs.

But I want to be able to have multiple Eden instances share a single
Lucene instance
Much like they can share a common DB, Memcache, GeoServer, etc

> pylucene would likely have one JVM per Eden instance, but we might be able
> to get around that.

Is PyLucene on the Eden end of the connection or the Lucene end?
I guess the former?
Seems odd that *Py*Lucene runs a JVM...I expected that PyLucene was a
Python library to allow Python apps to easily talk to Lucene.
My hope was that this comms would happen via TCP/IP

> Solr could probably be set up as one instance, but
> we're not entirely sure yet. Thing is, the processed data and indices for
> each instance would (according to the plan we came up with last night) go in
> each instance's eden directory.

Into the filesystem? Why?
Not a huge deal as the folders could be NFS'd back to a common
host...however does Eden need direct access to this? I thought that
Lucene would need the indices, etc & Eden would only ever access them
via PyLucene?

> That would be no issue for a separate instance of Lucene per each, accessed via pylucene.

Which is exactly what I *don't* want ;)
It seems like a major overhead.

> We're checking how we can get each JVM's VMID so we can see if we really are
> connecting to the same one across separate (Apache / Cherokee) processes for
> one instance.

I would really prefer to see all the tests run on 2 completely
separate machines to get the architecture right from the start.
If Lucene can run on a separate server to Eden, then the simplified
version where both run on the same server should be cake.
Whereas splitting up a common system can be painful, which is why I
would rather start with the architecture split.

F

[Pat Tressel]

Fran --

Ok, I'd better describe this more concretely:

pylucene is a helper that launches a JVM to run Lucene, does calls to it, translates back and forth between Python and Java data types. For instance, it provides Python iterators over the collections returned from search queries. If we use this, we will probably have one JVM per Web2py process (given that we always check for a running JVM before calling initVM). Downside is that JVMs use memory. Don't know how much can be shared between them.

If we use pylucene, then the Lucene files will be on the same machine as the Eden installation. For the UMD blogsearch project, we stored them right in the Web2py app directory.

Solr is a web service that runs Lucene. There are no Python helpers for this -- we would be doing Solr API queries via http(s). That's not a real issue, but just so you understand that writing the Python wrapper will be part of the project.

If we use Solr, then the Lucene files for each Eden installation / deployment will need to be distinguished somehow. We could invent separate directories for them, or we could add an instance id field to the search data. We don't know yet what we'd need to do to keep the data for different Eden installation distinct.

> One could bring up multiple Solr / Lucene instances.  Or not.  So one could
> (or not) do load balancing for Solr / Lucene.  For pylucene / Lucene, there
> would be one per instance -- each would have a local JVM, which would not
> span instances in separate machines or VMs.

But I want to be able to have multiple Eden instances share a single
Lucene instance
Much like they can share a common DB, Memcache, GeoServer, etc

In that case, you'd probably want Solr.
 

> pylucene would likely have one JVM per Eden instance, but we might be able
> to get around that.

Is PyLucene on the Eden end of the connection or the Lucene end?
I guess the former?
Seems odd that *Py*Lucene runs a JVM...I expected that PyLucene was a
Python library to allow Python apps to easily talk to Lucene.
My hope was that this comms would happen via TCP/IP

Lucene is written in Java and is executed in a JVM.  pylucene starts up the JVM, or connects to an existing one, as needed.  I haven't looked but expect it uses sockets, or a tighter connection.
 

> Solr could probably be set up as one instance, but
> we're not entirely sure yet.  Thing is, the processed data and indices for
> each instance would (according to the plan we came up with last night) go in
> each instance's eden directory.

Into the filesystem? Why?
Not a huge deal as the folders could be NFS'd back to a common
host...however does Eden need direct access to this? I thought that
Lucene would need the indices, etc & Eden would only ever access them
via PyLucene?

See above.
 

> That would be no issue for a separate instance of Lucene per each, accessed via pylucene.

Which is exactly what I *don't* want ;)
It seems like a major overhead.

Mainly memory usage.
 

> We're checking how we can get each JVM's VMID so we can see if we really are
> connecting to the same one across separate (Apache / Cherokee) processes for
> one instance.

I would really prefer to see all the tests run on 2 completely
separate machines to get the architecture right from the start.
If Lucene can run on a separate server to Eden, then the simplified
version where both run on the same server should be cake.
Whereas splitting up a common system can be painful, which is why I
would rather start with the architecture split.

The VMID thing is to make sure we know what-all either pylucene or Solr are doing behind the scenes.

It sounds like you'll be happier if we switch over to Solr sooner rather than later.  ;-) ;-) ;-)

-- Pat

[Fran Boon]

On 15 June 2013 06:18, Pat Tressel <ptre...@myuw.net> wrote:
> Solr is a web service that runs Lucene. There are no Python helpers for this
> -- we would be doing Solr API queries via http(s). That's not a real issue,
> but just so you understand that writing the Python wrapper will be part of
> the project.

Sure, doing HTTP API calls from Python shouldn't be too big a deal :)
How complex a subset of the API do we need to support?

> If we use Solr, then the Lucene files for each Eden installation /
> deployment will need to be distinguished somehow. We could invent separate
> directories for them, or we could add an instance id field to the search
> data. We don't know yet what we'd need to do to keep the data for different
> Eden installation distinct.

There are 2 scenarios:
(1) Multiple Eden's Load-balanced as part of a single deployment
- in which case they should share a common ID
(2) Separate Eden deployments sharing a common co-app
- in which case they should have separate IDs

> It sounds like you'll be happier if we switch over to Solr sooner rather
> than later. ;-) ;-) ;-)

Indeed...seems to me to be the far preferable solution :)

This is exactly why I was specifying these 'requirements' early-on in
case they factored into the decision...which it seems they very-much
do!

Thanks,
F