Another Link Checker

[Gavin]

Hi everyone.

While I realize SubjectsPlus already has a link checker out-of-the-box, I found I wanted to code my own so that our librarians could check their own guides. I also didn't want to check the entire guide page's links, only the links in the guide's boxes. Our university header has a LOT of links that I don't care to check!

I also love programming. ;)

You can download the tarball HERE. It has a readme.txt file in it with some basic instructions. Installation only requires plopping two files in a directory, adding one line of code to an existing file and (if needed) editing one line of the link checker code to use your campus proxy server.

I tested it for versions 1.0 and 2.0. Let me know if you have any questions.

Have a great weekend everyone.


- Gavin

[Andrew Darby]

That's great, Gavin!  Really easy to install/add.  Nice seeing it broken down box by box.  I want to test it a bit more, but can we add this to 2.0?

- Gavin

--
You received this message because you are subscribed to the Google Groups "SubjectsPlus" group.
To unsubscribe from this group and stop receiving emails from it, send an email to subjectsplus...@googlegroups.com.
To post to this group, send email to subjec...@googlegroups.com.
Visit this group at http://groups.google.com/group/subjectsplus?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Gavin Spomer]

Sure, I would love for you to add my code to yours; nice to know it worked on someone else's system! :) Thanks for testing it. I told our library faculty here about it yesterday and not a single one of them have tried it out yet. Sheesh!

Yes, need to test a lot more and maybe consider some other features/functionality as well as other improvements. There were some nice features of Ron's (?) link checker that might be nice, like the emailing of the output. Perhaps I'll actually add some more comments to my code too, LOL.

I already found something I missed, but I'm working on it: I wasn't extracting/checking links from record descriptions. I may have that done this afternoon.

[Gavin Spomer]

Okay, new version now extracts/checks links in record descriptions and I changed the ORDER BY clause for the "All Items By Source" so that it checks links in the same order that a guide displays them. The tarball has been updated with the newer link_checker.php.

[Paul Hoffman]

On Fri, Apr 05, 2013 at 11:05:48AM -0700, Gavin wrote:
> Hi everyone.
>
> While I realize SubjectsPlus already has a link checker out-of-the-box, I
> found I wanted to code my own so that our librarians could check their own
> guides. I also didn't want to check the entire guide page's links, only the
> links in the guide's boxes. Our university header has a LOT of links that I
> don't care to check!
>
> I also love programming. ;)
>

> You can download the tarball HERE<http://www.lib.cwu.edu/~spomerg/link_checker.tar.gz>.

I think I've mentioned this on this list before, so please forgive me
for beating a dead horse, but *please* revise your script to randomize
the order in which URLs are checked, pause between URLs, or take other
measures to lessen the chance that it will hammer web servers out there
-- especially when, some day, someone's SP has grown to hundreds or
thousands of links. Our OPAC has been brought down by link checkers
before and I'm sure we're not the only ones who have suffered.

Paul.

--
Paul Hoffman <nku...@nkuitse.com>

[Gavin Spomer]

Paul,

When you say "it will hammer web servers out there", are you talking about the web server the link checker (SubjectsPlus) is running on, the web servers of the links that are being checked or both? Since my code checks one link after another, not simultaneously, I don't think it's really hammering anything per say. It just takes time.

I'm happy to make a special version for you that will randomize the order in which links are checked and/or pause between checks, if you think it will help. Honestly, I don't think it will.

- Gavin

[Paul Hoffman]

On Tue, Apr 09, 2013 at 02:17:38PM -0700, Gavin Spomer wrote:
> When you say "it will hammer web servers out there", are you talking about
> the web server the link checker (SubjectsPlus) is running on, the web
> servers of the links that are being checked or both?

The web servers of the links that are being checked.

> Since my code checks
> one link after another, not simultaneously, I don't think it's really
> hammering anything per say. It just takes time.

You're right; that was a gross exaggeration -- I'm sorry.

> I'm happy to make a special version for you that will randomize the order
> in which links are checked and/or pause between checks, if you think it
> will help. Honestly, I don't think it will.

Thanks. I wouldn't use a link checker in SP, so there's no need, but I
appreciate the offer.

Here's why I have a knee-jerk reaction to link checkers, if anyone is
interested in my tale of woe -- if 20 people in a classroom click on a
link to a record in our OPAC within a span of a couple of seconds, our
OPAC goes down. It's ridiculous but true. A link checker checking
links in SP to records in our OPAC has the same effect -- if it checks
20 in a row. If it randomizes all the links it checks, then the 20 are
likely to be spread out more and don't bring down our OPAC.

Bonus points to anyone who can identify the OPAC vendor. :-)

[Andrew Darby]

One thing you might run into on your own server's end is exceeding PHP's maximum execution time if the script takes too long to run.  The default is 30 seconds:

http://php.net/manual/en/function.set-time-limit.php

The world doesn't end if you hit that, you just get a fatal error which might be a bit disturbing to someone trying to check their links.  Setting a small pause at some interval would avoid this. 

It would be nice to merge Ron and Gavin's link checkers into one before adding to the next SP release.

Andrew

p.s. I've run into the same problem as Paul when I was trying to screen-scrape the OPAC to get "real-time" availability of a set of items (DVDs), each scraping attempt started a new session on the server, and eventually it rolled over and died.

[Gavin Spomer]

This is why it's so important to have other people (that didn't write the code) test the software and offer feedback from their experiences. Thank you!

What OPAC are you using? We use III. Yuck. We're moving to a shared ILS system in about a year or so. I'm looking forward to it... I think.

[Gavin Spomer]

Thanks for the heads up on the time limit. I wonder why I don't get the error message then? My php.ini has the default of 30 seconds, but I know some of the guides take longer than 30 seconds. I wonder if the output buffering calls void the time limit?

How would you like our our link checkers merged? Do you want to do this yourself, since it's your baby? ;)

- Gavin