blacklight_range_limit

[Jason V]

Hi,

I was looking into using blacklight_range_limit for one of our
projects and read the description at:
<https://github.com/projectblacklight/blacklight_range_limit>.
I was a little unclear if the plugin supports our use case, (searching
for date ranges with a record).

For example,

If a single record has a range associated with it (startDate = 1900,
endDate = 1910) and someone searches on 1905, could the plugin find a
match?

Or is the plugin functionality designed as follows:

Record 1.) date=1900
Record 2.) date=1901
Record 3.) date=1902
Record 4.) date=1903
Record 5.) date=1904

Then, someone searches for 1901-1903 and the results returned are
Records 2, 3, and 4?
(Date ranges across records)

Can someone provide some insight? Thanks

Jason

[Jonathan Rochkind]

Yes, the plugin supports searching for ranges.

The plugin pretty much supports 'integers' -- so as far as dates, if
your dates are just years, that's fine (and that's the main use case
it's intended for). If your dates need to be year/month/date, the plugin
does not support that right now.

Maybe we need some screenshots of what the plugin does, easier to
explain with a picture than with words. I'm not sure if it's installed
in our BL "demo". But you can look at our JH BL to see the range plugin
in action, do a search and look at the "Publication Year" limit on the
left, that's the range plugin at work. http://catalyst.library.jhu.edu

[Jonathan Rochkind]

Ah, sorry, I misunderstood. Thanks Sean for noting I hadn't answered
the question you really asked.

> If a single record has a range associated with it (startDate = 1900,
endDate = 1910) and someone searches on 1905, could the plugin find a
match?

There's no built in way to do that exactly, but you might be able to get
it to work this way, by indexing into a multi-valued field, where each
year in the range represented by the document is present as a value.
That will make range searches work -- but will make the 'counts' for the
bar chart or 'facets' kind of odd, as one document will show up multiple
times. It may make the bar chart/facets unusable by being so weird, I'm
not sure.

There might be a different way to handle this on the Solr/lucene end,
but I'm not sure what it is. I'm not quite sure what the bar
chart/facets _ought_ to look like in cases where each document covers a
range. If you can figure it out, and figure out how to make the range
plugin deal with it, happy to accept a pull request.

Alternately, since the range plugin is really focused on the bar
chart/facets, and it's unclear how to make them work with your data,
perhaps you should just abandon the range plugin and barcharts, and just
do simple range searches where the endpoints are entered by hand. You
don't need the range plugin to do that, most of the code in the range
plugin and it's complexity is focused on the barcharts.

I don't know if anyone has written a simpler plugin that just does range
searches by itself, perhaps because that's a fairly straightforward
thing to do implement yourself once you understand hwo Solr works and
how BL works (big 'once's, I understand; perhaps it would be useful if
someone wrote such a plugin, but I don't think anyone has).

Jonathan

[Jason V]

Thanks Jonathan for the reply! Very helpful...

This was kind of my assumption about the plug behavior and wanted to
confirm. If we move forward with this feature, I think the plan is
for us to implement this functionality (as you suggested)...

Jason Varghese
NYPL

[Jonathan Rochkind]

It'd be a worth a try with the multi-value approach, and see if that
does something useful with the bar graphs/facet segments.

The plugin doesn't take care of indexing anyway. So, at indexing, if a
document is "1900-1904" (inclusive), actually index the values
'exploded'/'flattened' into six values, 1900,1901,1902,1903,1904

The range plugin should actually then work. It's just that the
bargraph/facet segment will multi-count it. Which might actually be what
you want, I'm not sure -- I'm not sure what you'd want a
segment/facet/bargraph/histograph display to look like in this case.

Let's say you had two documents in your index, one was 1900-1904
inclusive (that's 5 years), and one was 1903 only. And let's say you
generate the facet count/histogram for 1900-1905.

You'll get a count of "1" for every year from 1900-1904, with a count of
"2" for 1903. Even though there's actually only two documents in the
range at all, there are 6 counted items in the graph.

But I'm not sure how you'd simply do a facet/histogram over the range
anyway BUT this. I have no idea if this will be confusing or match
user's expectations. But if it turns out that's actually what you WANT,
you should be able to use the range plugin while indexing into a
multivalued field to get it.

[Jason V]

Thanks Jonathan! I plan to experiment with it to see if its produces
desired results. I actually don't need the bar graphs, and may not
need date facets initially either.

The plan is to "explode" the date as you suggest. I just wasn't sure
if I it was "ok" to explode the date ranges. Its fine for small
ranges like 1900-1905, but then its possible to have 100(x)+ year
ranges too.

Wondering if issuing a query similar to the solr equivalent of "where
date < dateEnd AND date > startDate" would be "better" than exploding
out the date ranges to find a match, but I think the former requires
modifying BL search behavior. Its probably easier to explode out the
ranges....

Thanks again for your input..

Jason

> >>>> Jason- Hide quoted text -
>
> - Show quoted text -

[Jonathan Rochkind]

"Exploding" should (I believe) be okay as far as Solr/lucene performance
goes. If you have an awful lot of them, it could lead to increased solr
disk/RAM requirements, but should be fine from a performance standpoint.
My own previous investigations led me to believe that this kind of
exploding was the best approach as far as Solr/lucene was concerned, but
there are other approaches, and I could be wrong. :)

But yeah, if you don't want the bargraph style histograms displaying
distribution of things in your range, you may be better off not using
blacklight_range_plugin -- that's the main thing it adds, and most of
it's (sometimes convoluted) code is focused on that.

[Jason V]

Thanks for sharing your feedback regarding SOLR issues around
exploding the dates. I do want to incorporate the "slider" which
lets one select date ranges as a facet. I just saw your example of
how that ties in with the "bargraph style histograms". If I demo that
to the stakeholder's I'm sure they'll request it :) I'll probably
experiment with it to see how it functions with "exploded" date
approach....

Jason
NYPL

> >> - Show quoted text -- Hide quoted text -

[Jason V]

Hi Jonathan,

I was looking at your example for the ranged date search at
https://catalyst.library.jhu.edu/advanced

For publication year, you have the two text boxes to enter begin and
end date..Is that customized or can the plugin do that out of the
box. I seem to get a list of date facets....Thanks

Jason

[Jonathan Rochkind]

If you're getting the list of facets, then the javascript for advanced
search facet isn't getting triggered. The javascript adds the actual
chart, and in some cases the Javascript is needed to add the text boxes
too. What you're seeing is the non-JS degraded version.

* Are you using a recent version of BL, with the asset pipeline?
* Do you have the latest version of BL and the latest version of the
plugin (certain old versions of the plugin didn't work quite right out
of the box, and required local hacks; I'm afraid I can't personally
support those, figure/out remember what the local hacks were).
* Is the advanced search js included in your asset pipeline
application.js manifest file? (The range generator should have added
it, but double check)
* Have you configured the facets you wanted rangeified with range=>true
in the BL config?
* Have you NOT configured the range plugin specially to _disable_ the
javascript or anything like that?

It's certainly possible you're running into a bug of one kind or
another, but I did recently confirm that the _latest_ version of the
range plugin works properly as documented (for me) with the latest
version of BL. I think I had to make a few tweaks to get it there
though, a couple weeks ago.

[Jason V]

Hi Jonathan,

I think its my facet configuration in advanced_controller.rb. Works
fine after a simple search. I can see the date range UI pieces on the
facet sidebar. This is the configuration I used in
catalog_controller.rb

config.add_facet_field 'expandedDates_i', :label => 'Date', :range =>
true

Can you provide an example of the equivalent in
advanced_controller.rb?

I tried something like the following from the example at: <https://
github.com/projectblacklight/blacklight_advanced_search>

config.advanced_search => {
:form_solr_parameters => {
"facet.field" => ["expandedDates_i", "language_facet"],
"facet.limit" => -1, # return all facet values
"facet.sort" => "index" # sort by byte order of values
}
}

(I dont have the :range=>true in this example)

Thanks

Jason

[Jason V]

btw - I am using BL 3.2.2, adv search plugin 1.2.4 and bl range limit
1.2.2

[Jason V]

OK.I also see that we don't use advanced_controller.rb anymore..but
looks like I still need to configure the date facet for advanced
search in catalog_controller.rb....

can;t seem to find an example of this....

> ...
>
> read more »- Hide quoted text -

[Jonathan Rochkind]

All you got to do is put a :range => true on the date facet
configuration in catalog_controller.

Yeah, nothing 'advanced' should have anything to do with it -- wait, you
aren't talking about your range facet _showing up_ on the advanced
search input form, are you?

Yeah, I don't think that's out of the box supported yet, range_limit and
advanced_search automatically playing well together. You can hack it
together yourself. If that's what you're talking about, someone can try
to explain.

Otherwise, we can continue debugging -- do you get range facet properly
on your actual search results facets? If not, is the
blacklight_range_limit js referenced in your
app/assets/javascripts/application.js ?

[Jason V]

Yes, I was referring to getting the two input boxes to display where
you enter a start and end date on my advanced search input
form..similar to what you have next to Publication Year on:
https://catalyst.library.jhu.edu/advanced
(is that out of the box functionality or customization?)

Everything works fine on a search result..i get the graph and also i
get two boxes in which to enter a date range...

On my advanced search page, I can only manage to get a date facet that
just lists all the dates..I can't get the two input boxes to appear in
which to enter a date range....

[Jonathan Rochkind]

Yep, that feature is indeed not built in, sorry I misunderstood.

Some of us have hacked it in ourselves; I'll try to find time to write
up how I did that if nobody else gets to it first.

But basically, just look at the query parameters produced when you enter
values in the range search boxes on the ordinary facet display.

You need to make a local customizable copy of the advanced search form,
and then you need to add html inputs to it that will result in the same
query parameters that the range limit would ordinarily produce. That's
about it.

Jonathan

[Jason V]

OK cool..I looked at the output from the facets..and hacked something
together...for the advanced search form..Thanks

Jason