Umlaut 4.0 beta! Beta testers needed!

[Jonathan Rochkind]

Umlaut 4.0.0.beta2 is out! (Yeah, don't ask about beta1 :) ).

This release is mostly back-end upgrades, including:

* Support for Rails 4.x (Rails 3.2 included to make migration easier for
existing installations, but recommend starting with Rails 4.1 in new apps)

* Based on Bootstrap 3 (Rails 3 was Bootstrap 2)

* internationalization/localization support

* A more streamlined installation process with a custom installer



I would really appreciate if anyone wanted to give a beta test. It
should be pretty straightforward to test, either upgrading an existing
Umlaut or installing from scratch.

To install a new Umlaut app, see:
https://github.com/team-umlaut/umlaut/wiki/Installation

To upgrade, you only need to update your Gemfile to allow the beta:

gem "umlaut", "> 4.0.0.beta2"

And `bundle update`. And it _should_ just work. (I do recommend
upgrading to Rails 4.x too, before you go into production or as soon as
possible. But Umlaut 4.0 supports Rails 3.2 as well to ease migration).


Thanks! Please let me know if you run into any issues!

Jonathan

[Kevin Reiss]

Hi Jonathan,

Count Princeton University Library in as a beta tester for this new release.  I upgraded our application to the 4.0.0.beta3 gem and as you say everything pretty much just worked out. We are running it under Rails 4.1.5 and Ruby 2.1.2-p95.

I had some minor adjustments to make to partials that I'd overridden for the search_controller views. Thanks for all your work on this. I did run into one technical issue with a custom local service adapter we have which threw an ActiveRecord::ImplicitConnectionForbiddenError.

ActiveRecord::ImplicitConnectionForbiddenError: Implicit ActiveRecord checkout attempted when Thread :force_explicit_connections set!

The code that produced this errors was in the handle() method of the adaptor:

fulltext_options = request.service_responses.find_all {|r| r.service_type_value_name == "fulltext"}

The adaptor does some introspection on existing full-text responses so we don't get duplicate full-text links from 856 links in MARC records. I see you've blogged extensively on Active Record and Umlaut so I'm going to catch up on what you've written to understand the problem a bit better.  

We have had an instance of umlaut running as a sandbox project for quite a long time but are finally getting it into production next month (I guess you could say it has been in an extended staff preview). Barring anything that comes up I hope we'll go live with 4.0 series gem you have put together. Thanks for all your work on this project. The long gestation period has been due to staffing and administrative issues not technical hurdles..  

Kevin

[Jonathan Rochkind]

Hi Kevin!

I'm so glad to hear you guys are experimenting with Umlaut -- if/when you go live, can you make sure to let us know, and possibly list yourselves here: https://github.com/team-umlaut/umlaut/wiki/Umlaut-installationshttps://github.com/team-umlaut/umlaut/wiki/Umlaut-installations

I'm glad the upgrade went smoothly for you.

And I'm glad you got that exception! It's a good sign!  

It means that you were leaking ActiveRecord database connections before -- in Rails 3.x, leaked connections would be cleaned up for you periodically by Rails, but it was still bad practice. In Rails4, they would not be cleaned up for you, so you'd find yourself eventually with no connections available to talk to your database. This happens because Umlaut services are executed in background threads, and because of how ActiveRecord deals with threading -- if you try to do something with the database in a new thread, it'll automatically check out a connection for you, but never check it back in. 

 So Umlaut monkey-patches ActiveRecord to provide the exception you see -- that isn't a standard part of Rails (although I think it should be). 

Fortunately, the solution is pretty darn simple!  You need to wrap all ActiveRecord operations in your custom service in an ActiveRecord `with_connection` block. For instance, you _could_ do:

fulltext_options = Request.connection_pool.with_connection do 

   request.service_responses.find_all {|r| r.service_type_value_name == "fulltext"}

end

That'll work, except then when you try to _do_ something with that array of fulltext_options, you might get another implicit connection exception. You _could_ wrap your entire service exception in with_connection, and that'll work fine, but it'll hold on to an active record connection longer than it needs to, potentially reducing the performance of your app and/or requiring you to increase the size of your `pool` in database.yml. 

What I'd recommend is get the data you need out of the active records and into ordinary strings/hashes/ruby objects that aren't AR related.  And do that in a with_connection block. And then happily deal with those strings/hashes/etc to your hearts content wherever you want. 

Make sense?   If not or if you have trouble I'm happy to help and even look at your actual code. But I suspect you'll have no trouble, it's pretty straightforward, you just need to put all AR stuff within a `with_connection` block, and ideally keep the `with_connection` block small to limit the amount of time you have a connection checked out. 

Also, want to share info about what your custom service does with us, and/or even share the code?  I see you mention it's about avoiding duplicate fulltext responses from 856's -- I took a somewhat different approach to that myself, although I had the same problem to solve, we should exchange notes (in a different email thread, ha).  Also, you will be hearing from me soon to ask advice on BorrowDirect issues (which yes, will eventually turn into an umlaut service, for sure. JH is a BD member now. So we might want to collaborate on that, actually). 

Jonathan

---

From: umlaut-...@googlegroups.com [umlaut-...@googlegroups.com] on behalf of Kevin Reiss [kevin...@gmail.com]
Sent: Saturday, September 20, 2014 4:14 PM
To: umlaut-...@googlegroups.com
Subject: [umlaut] Re: Umlaut 4.0 beta! Beta testers needed!

-- 
You received this message because you are subscribed to the Google Groups "Umlaut" group.
To unsubscribe from this group and stop receiving emails from it, send an email to umlaut-softwa...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Kevin Reiss]

Thanks for the information on how to work with the connection pool Jonathan. Your suggestion worked out perfectly. My algorithm for deciding what to do with 856 fields is pretty straightforward. If the SFX knowledge base returns a full-text match for the OpenURL and the MARC 856 field indictors are set properly I just don't add the OPAC link to the fulltext responses.  

We will definitely add ourselves to the Umlaut implementors group on the github wiki. I'm also planning to get approval to post our umlaut project as a public repo on github. Our beta system is located at http://getit.princeton.edu. 

[Jonathan Rochkind]

Cool.

As far as eliminating dups between the catalog and SFX, I actually went
further/different, you may be interested in what I did, and in the tools
Umlaut gives you to do similar.

The problem I had, was sometimes SFX did not return any responses, but
that's because SFX KB knew that we didn't have coverage of the
particular article requested. (Say, we wanted 1950, but the KB knew our
full text coverage begins in 1960).

The Catalog MARC is not really capable of telling that -- it doesn't
have coverage info in machine-readable format.

So sometimes there'd be cases where SFX returned no response, and if we
just went by that to say, sure, include 856 fields, then we'd get 856's
from the catalog for something the SFX kb correctly determined we did
not have access for.

My solution? I actually sweep the SFX kb database and try to pull out
any URLs that belong to activated targets in SFX.

Then, if a 856 has a hostname matching one of those hostnames pulled
from SFX, we don't display it, regardless of whether SFX returned a
response or not. (We may only do this suppression for citations that
look like articles, not ebooks, I forget). We still get some dups and
false negatives, but it works out reasonably well.

The exact logic for doing this is probably going to end up needing to be
localized to your particular needs, I haven't tried to make it generic.
But if you want to go this route, there are some things built into
Umlaut to make it pretty easy --

If you have a direct (read-only, preferably) db connection to SFX set up
in your database.yml under "sfx_db"; then every time you run `rake
umlaut:nightly_maintenance` (which is recommended), Umlaut is already
pulling likely URLs into a local Umlaut database. In your own local
service code, you can just check `SfxUrl.sfx_controls?
"http://whatever"`, and it'll return true if it thinks this is something
SFX should know about (basically just on hostname match).

Curious to know if you end up using any of that!

Jonathan

On 10/2/14 3:57 PM, Kevin Reiss wrote:
> Thanks for the information on how to work with the connection pool
> Jonathan. Your suggestion worked out perfectly. My algorithm for
> deciding what to do with 856 fields is pretty straightforward. If the
> SFX knowledge base returns a full-text match for the OpenURL and the
> MARC 856 field indictors are set properly I just don't add the OPAC link
> to the fulltext responses.
>
> We will definitely add ourselves to the Umlaut implementors group on the
> github wiki. I'm also planning to get approval to post our umlaut
> project as a public repo on github. Our beta system is located at
> http://getit.princeton.edu.
>
> On Saturday, September 20, 2014 4:26:36 PM UTC-4, Jonathan Rochkind wrote:
>
> Hi Kevin!
>
> I'm so glad to hear you guys are experimenting with Umlaut --
> if/when you go live, can you make sure to let us know, and possibly
> list yourselves here:
> https://github.com/team-umlaut/umlaut/wiki/Umlaut-installations

> <https://www.google.com/url?q=https%3A%2F%2Fgithub.com%2Fteam-umlaut%2Fumlaut%2Fwiki%2FUmlaut-installations&sa=D&sntz=1&usg=AFQjCNEDAZWx_bRgzAcATS9SjsjSg4wwWw>https://github.com/team-umlaut/umlaut/wiki/Umlaut-installations

> ------------------------------------------------------------------------
> *From:* umlaut-...@googlegroups.com <javascript:>
> [umlaut-...@googlegroups.com <javascript:>] on behalf of Kevin Reiss
> [kevin...@gmail.com <javascript:>]
> *Sent:* Saturday, September 20, 2014 4:14 PM
> *To:* umlaut-...@googlegroups.com <javascript:>
> *Subject:* [umlaut] Re: Umlaut 4.0 beta! Beta testers needed!

> --
> You received this message because you are subscribed to the Google
> Groups "Umlaut" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to umlaut-softwa...@googlegroups.com

> <mailto:umlaut-softwa...@googlegroups.com>.