islandora solr views freezes

198 views
Skip to first unread message

Jennifer Eustis

unread,
Oct 23, 2015, 2:11:54 PM10/23/15
to islandora
Hi all,

We have Islandora solr views installed. When I go to create a new view, I can select the Islandora Solr with the initial configuration and click save and continue. When I try to add a field for the PID and the others I want, I get the circle of death and then the screen freezes.

Has anyone else experienced this and how did you fix it?

Thanks,
Jennifer

Jordan Dukart

unread,
Oct 23, 2015, 2:18:41 PM10/23/15
to isla...@googlegroups.com
Hey Jennifer,

Unfortunately this is a reported bug with the number of Solr fields present in your index:
https://jira.duraspace.org/browse/ISLANDORA-1392

Jordan
--
For more information about using this group, please read our Listserv Guidelines: http://islandora.ca/content/welcome-islandora-listserv
---
You received this message because you are subscribed to the Google Groups "islandora" group.
To unsubscribe from this group and stop receiving emails from it, send an email to islandora+...@googlegroups.com.
Visit this group at http://groups.google.com/group/islandora.
To view this discussion on the web visit https://groups.google.com/d/msgid/islandora/1cc7d370-9893-4cd8-a528-e27902187b82%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Peter MacDonald

unread,
Oct 23, 2015, 5:39:58 PM10/23/15
to isla...@googlegroups.com
Jennifer:

I was able to use Solr Views before I updated my Solr schema.xml (just last night) to a more recent version. I reindexed my repo over night and found that this schema.xml file creates a lot more Solr fields than I ever had before. (We have a lot of newspaper and compound objects in our repo which create a lot of nasty looking RELS-EXT and RELS-INT fields.)

The downside of this Solr upgrade is that now, like you, I cannot configure a new Solr View. I just tried this afternoon and apparently the list of Solr fields is now so long, AJAX runs out of memory. Here is the message I get every time I need to pick a new field.

An AJAX HTTP error occurred.
HTTP Result Code: 200
Debugging information follows.
Path: /admin/structure/views/ajax/add-item/essay_titles_a_z/page/field
StatusText: OK
ResponseText:
Fatal error:  Allowed memory size of 134217728 bytes exhausted (tried to allocate 3167663 bytes) in /home/dhiham/public_html/includes/common.inc on line 5994

​I've tried allocating more memory using a PHP setting (as high as 1024M), but​ still get this message.

Looks like it needs attention, as Jordan notes in his reply to you.

I'll add my comments to the Jira ticket [1] he mentioned.


--
For more information about using this group, please read our Listserv Guidelines: http://islandora.ca/content/welcome-islandora-listserv
---
You received this message because you are subscribed to the Google Groups "islandora" group.
To unsubscribe from this group and stop receiving emails from it, send an email to islandora+...@googlegroups.com.
Visit this group at http://groups.google.com/group/islandora.
To view this discussion on the web visit https://groups.google.com/d/msgid/islandora/1cc7d370-9893-4cd8-a528-e27902187b82%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.



--
Peter MacDonald,
Library Information Systems Specialist
Hamilton College Library
Clinton, New York
315 859-4493
pmacdona-hamilton (Skype)

Diego Pino

unread,
Oct 23, 2015, 6:33:27 PM10/23/15
to islandora
Hi Jennifer and Peter, auto - assigned this to me. Will have a fix for you to test (pull) and to discuss with the committers this weekend.

Best

Diego

Jennifer Eustis

unread,
Oct 24, 2015, 12:33:01 PM10/24/15
to islandora
Thanks Jordan, Peter and Diego. I also submitted a ticket here with our developer Doug. He tried the same thing you did Peter of allocating more memory. I'll reference the Jira ticket for him so he can pull that fix in when it's ready.

I just learned about the power of views at Islandora camp! 

Jennifer

Diego Pino

unread,
Oct 25, 2015, 5:01:51 PM10/25/15
to islandora
Hi Jennifer, Jordan and Peter

I have been working hard on a fix this weekend and i have found some solutions that i would like to discuss before implementing, basically because it implies a "big change".

With the current architecture is pretty easy to make Solr Views (and Views in general) go out of memory, timeout or even surpass the max number of Input variables in php. Sadly it is related on how Views module fetches and renders the fields to be used. The ideal situation would be to enable a pager in then add field popup, but looking closely at the Views module code it's a very big deal and could easily introduce a lot more problems and also does not assure a quicker load, because we would be basically altering that form (which would be constructed anyway by the module the first time).

That said: this is my proposal (i have it almost working, lots of code, i like it).
Instead of having all Solr fields as individual "fields" to be added and listed, i modified the whole mechanic to use only a few generic ones, The base ones (PID, score, islandora_manage, solr_collection_count, query_luce, query_dismax) +  a few configurable ones based on the Solr field type: Solr sortable, Solr general, Solr Date,  (we could add some more if needed).

This reduces the number of fields our modules exposes to the minimum. Each of this last fields have now an extra configuration option, the real Solr field name to be used (defaults to PID) as source. So you add a Solr general field, you edit it, you have a Luke autocomplete field and you choose which field you will be querying.  This autocompletes are modal, means "at Sortable one's edit form" you get fields that are single valued and indexed only, etc.

Pros: quick loading of fields, you don't need to remove fields and re add them if you need to change a source solr field (as before), means you can use an existing Solr view in a particular view and fine tune the source fields without having to remove and re-add fields.
Cons: you have to rebuild all your existing Solr views, because they will be no longer compatible!

Since the cons means remaking your already existing Solr views and i'm sure this will scare a lot of people!, we(or i) could make a transform routine, basically i just need to fetch the field name and put it into an option inside one of those generic Solr views fields. This could be implemented in a update function in the install file of the module, as a drush script, whatever you like more.


What do you think? It's an open discussion of course, but having a regex that removes some fields makes me also not happy, because only if you are using a very standard solr schema, it's very possible that we end stripping fields that we could need or not stripping anything. I prefer to give users the full power to decide what to use and where.

Best.

Diego

PS: copying this also in the JIRA to get some feedback.

Jennifer Eustis

unread,
Oct 26, 2015, 9:32:51 AM10/26/15
to islandora
I'm not sure how many people have implemented Solr Views. These changes are a major reworking that will affect many people. We haven't implemented Solr views here where I am. We're interested in image galleries which would use the PID to then display thumbnails based on a filter. Other ideas that we'd like to do with this is to create views that filter off of the mods:typeOfResource element or have a different presentation of collections. I would like to hear from those who've implemented Solr Views to get their opinion.

Jennifer

Amanda Lehman

unread,
Oct 28, 2015, 11:34:58 AM10/28/15
to islandora
Hi all, just want to 1+ this improvement as I love using solr views for back-end management and reporting (as I'm not savvy with the query languages just yet).  I also see a use for the views in creating reports for Islandora Scholar users who want various information about their objects in Islandora.  

That said, Diego I definitely see need to be able to customize what fields the module exposes - even if that means documenting how to customize the solr views module components to "slurp"/process extra fields.  Also, I'm totally willing to re-do my current views if it means that I can create new ones without the time out!

Glad to see this thread, thanks.

Diego Pino

unread,
Oct 28, 2015, 4:33:30 PM10/28/15
to islandora
Thanks Amanda!. Let's see how many +1 we get. Also i would love the maintainer opinion, will see if i can get this into the committers agenda so we can have a talk.

Best

Diego

Jennifer Eustis

unread,
Oct 28, 2015, 5:01:58 PM10/28/15
to islandora
I'm a 1+ ... this comes with the knowledge that we're haven't implemented solr views so nothing to redo in our case.

Jennifer

Mitchell MacKenzie

unread,
Oct 28, 2015, 5:36:44 PM10/28/15
to islandora
Hi Diego,

Sorry if I have missed it but could you give more details on the proposed user interface for defining what fields would be available to Solr views.

Are we considering whitelisting fields that should be available or filtering fields?

I think the filtering approach would be preferable for a few reasons: existing implementations wouldn't be affected so there's no need for update operations and there is no UI complexity added for users who don't suffer from this technical issue.

Thanks,
Mitch

Diego Pino

unread,
Oct 28, 2015, 5:52:39 PM10/28/15
to islandora
Hi Mitchell. The idea is to have generic View Fields for Solr instead of a field for every existing Solr field. So basically it's the same: the only difference is this:

Now: you choose a specific field: lets say mods_somefiled_s from a list (that if too large never displays), you choose, add, you configure settings. 

With the change: you choose, e.g Solr General field (the number of fields to choose is only 9 in my design but this can change if needed). You choose, add, you configure settings. One of the settings is an autocomplete (luke) that allows you to define which REAL Solr field will be tied to that views field.

That is the idea. No whitelisting, filtering in a configuration option for a basic set o fields.

What i will do is implement the changes this weekend (almost there!) in a Fork so you can try out, so we do no harm (Jared's suggestion, fine as always). If it suits people then we can go for making the change to the main or keep them in parallel.

Thanks!

Diego Pino

unread,
Nov 3, 2015, 4:23:02 PM11/3/15
to islandora
Hi, just an update. I'm almost ready with the changes and it works. Doing some code cleanup, some UI stuff and i have to deal with some strange stuff with z-index in the autocomplete. I will post here when i'm ready so you can test. I hope i can make it for friday.

Cheers!

Jennifer Eustis

unread,
Nov 11, 2015, 10:21:45 AM11/11/15
to islandora
That's awesome news Diego and thank you for all your work on this. Can't wait to try it...

Jennifer

Diego Pino

unread,
Dec 10, 2015, 9:19:28 AM12/10/15
to islandora
Dear People!

At last i have something to show to you. It was a long tedious process to get over all the issues but it's working, it's fast, it's non blocking, it's new! (and my have bugs, but i really tested a lot)
For now, until someone has something to say, ask, fix, etc i will keep this in my local branch. 


Here is my local commit https://github.com/DiegoPino/islandora_solr_views/commit/d228d64dfcb383bbb689e0dc6be34143a42ec4b8 with comments (not sure why my github client did not respect markdown…)

Since the changes are BIG-BIG i would ask you/encourage you to test first before going for it at production. There where many bugs fixed also and some new functionality is available. I will gladly help you get started/migrate stuff and if there is agreement/further fixes needed i will do a pull request against the main repo so every one can go for it.

If we never get to an agreement (this things happen), then i can refactor this to an independent module, so you can have the both side by side.

Lastly. I'm very happy on how this one worked out. Please, please try it and/or make questions.

Cheers!

Diego Pino Navarro

Jennifer Eustis

unread,
Dec 11, 2015, 12:47:18 PM12/11/15
to islandora
This is great Diego. We're working on upgrading our Islandora/Drupal at the moment. But I will definitely put this in our queue for testing on one of our vm's next year :)! I'll also mention this in the Roadmap meeting,

Thanks again for all your work on this.

Jennifer

Brandon Weigel

unread,
Apr 22, 2016, 4:05:36 PM4/22/16
to islandora
We have a number of Solr views, but if an update/migration tool were included (either with an update to core or in a separate module) I'd be very happy with Diego's fix.

Diego Pino

unread,
Apr 22, 2016, 4:29:02 PM4/22/16
to isla...@googlegroups.com
Hi Brandon,
Will see if i got some spare time to also do a migrator for existing Solr views. The issue is i feel it could end being maybe even lost energy because my pull have been sitting many months there without even a review on it’s actual functionality.
On the interns, an export of a view, some minor changes and reimport could work too. Do you have a particular complex view you could share to test an automatised procedure on this?

Thanks!

Diego Pino
On Apr 22, 2016, at 4:05 PM, Brandon Weigel <bran...@eln.bc.ca> wrote:

We have a number of Solr views, but if an update/migration tool were included (either with an update to core or in a separate module) I'd be very happy with Diego's fix.

--
For more information about using this group, please read our Listserv Guidelines: http://islandora.ca/content/welcome-islandora-listserv
---
You received this message because you are subscribed to a topic in the Google Groups "islandora" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/islandora/LVz0-JF9fxU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to islandora+...@googlegroups.com.
Visit this group at https://groups.google.com/group/islandora.
To view this discussion on the web visit https://groups.google.com/d/msgid/islandora/9a8531ea-f372-4334-93e7-5424604e484a%40googlegroups.com.

Brandon Weigel

unread,
Apr 22, 2016, 4:41:01 PM4/22/16
to dpino...@gmail.com, isla...@googlegroups.com
Sure, I can attach a couple of exports. (I really don’t have the tools/knowledge to test your pull request on my own dev box, but if you can use these to test that would be great.) 

The main one is the Collection Item Details View, created by discoverygarden for our Browse as List display. 

collection item details view.txt
collection item details view with sort.txt

Diego Pino

unread,
Apr 22, 2016, 4:46:51 PM4/22/16
to Brandon Weigel, isla...@googlegroups.com
Thanks! Will keep you updated

Best

Diego
On Apr 22, 2016, at 4:41 PM, Brandon Weigel <bran...@eln.bc.ca> wrote:

Sure, I can attach a couple of exports. (I really don’t have the tools/knowledge to test your pull request on my own dev box, but if you can use these to test that would be great.) 

The main one is the Collection Item Details View, created by discoverygarden for our Browse as List display. 

<collection item details view.txt>
<collection item details view with sort.txt>

Brandon Weigel
Librarian
BC Electronic Library Network

Peter MacDonald

unread,
Apr 22, 2016, 5:24:12 PM4/22/16
to isla...@googlegroups.com
I have been using Diego's update to the Islandora Solr Views that enables thr calling up Islandora indexes in Solr to choose for use in a view -- even if you have massive numbers of Islandora fields in Solr.

I just want everyone to know that Diego'a extra effort has restored Views as a truly useful tool for use with Islandora.

Peter MacDonald


You received this message because you are subscribed to the Google Groups "islandora" group.
To unsubscribe from this group and stop receiving emails from it, send an email to islandora+...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.
Reply all
Reply to author
Forward
0 new messages