500 errors from the Annotator service

[Benjamin Good]

I've been experiencing a large number of HTTP 500 errors from the Annotator service over the past few days.  

Repeating exactly the same request sometimes produces an error and sometimes does not..  My client currently tries a request up to ten times before quitting - sometimes the first one works, sometimes the third, sometimes the 5th and sometimes never..

What going on?

thanks

-Ben

[Benjamin Good]

For the last three weeks or so, the Annotator has become very unreliable.  It is reporting HTTP 500 errors for a large proportion of requests.  Repeating identical requests sometimes succeeds and sometimes do not.

These problems appear to coincide with the introduction of the new authentication policy.  

Please let us know what is going on.  I have a live service that depends on the Annotator...

thanks

-Ben

On Fri, Jun 10, 2011 at 12:55 PM, Taciana Gomes <taciana.a...@gmail.com> wrote:

I did make a client with c#

---------- Forwarded message ----------
From: Taciana Gomes <taciana.a...@gmail.com>
Date: 10 June 2011 16:46
Subject: Re: [ncbo-annotator-users] 500 errors from the Annotator service
To: ben.mcg...@gmail.com


Hi, I have this problem too.
I dont know what can be..

--
____________________________________________
Taciana Alessandra Gomes Cruz
MSc in Computer Science
Federal University of Pernambuco – UFPE/Brazil
Fone: (19) 8185-8516

--
____________________________________________
Taciana Alessandra Gomes Cruz
MSc in Computer Science
Federal University of Pernambuco – UFPE/Brazil
Fone: (19) 8185-8516

[Trish Whetzel]

Hi Ben, 

Thanks for sending this email to the support mailing list. We'll look into this issue. 

Trish 

_______________________________________________
bioontology-support mailing list
bioontolo...@lists.stanford.edu
https://mailman.stanford.edu/mailman/listinfo/bioontology-support

[Benjamin Good]

Hi Ray,

I had failures last night and this morning.

The client source code that accesses the service can be found here:

http://code.google.com/p/genewiki/source/browse/java/Miner/src/org/gnf/ncbo/web/AnnotatorClient.java

If you want me to make you an executable for testing let me know.  (That code isn't really ready for primetime).. Since the problem was also observed by some one running a c# client I hope its not a problem with my implementation.

If I repeatedly run the main method in that class (or execute the request from other resources that depend on it) sometimes it works and sometimes it doesn't (responding with a 500).  I'm sending pretty small chunks of text (typically less than a sentence) at irregular intervals.  If it fails, I wait 2 seconds and try it again.  If it fails 10 times the client quits.  That retry loop was sufficient to overcome similar behavior prior to this update for tens of thousands of requests sent right after each other.  

params:

params.put("isVirtualOntologyId", "true");

params.put("rqnum", "0");

params.put("textToAnnotate", text2annotate);

params.put("filterNumber", "true");

params.put("minTermSize", "3");

params.put("withSynonyms", "false");

params.put("longestOnly", "true");

params.put("wholeWordOnly", "true"); 

params.put("stopWords", "protein, gene");

params.put("withDefaultStopWords", "true");

params.put("scored", "true");

params.put("mappingTypes", "null");

ontologiesToKeepInResult (virtual ids for GO, DO, FMA, PRO, OMIM)

Let me know if thats enough to get you going.  If not I will try to code something that consistently reproduces your error (difficult because, from my end, there is significant randomness on the server side).

thanks

-Ben

On Thu, Jun 23, 2011 at 11:09 AM, Ray Fergerson <ray.fe...@stanford.edu> wrote:

Ben,

 

Do you have a reproducible test case for this? Certainly the entire system behavior was erratic before last week for several weeks. We believe that this problem has been fixed though. If you are still seeing failures then we would like to look into it. It will be difficult to do so though without a test showing failures. We haven’t had other complaints that I am aware of since we pushed out the patch last week.

 

Even if it is not a single call that fails, perhaps you have a script that we could execute that would reliably show failures. Also it would be very useful to know how many calls you are making in some unit of time. Also relevant are the parameters you are passing to the annotator and the type and size of text.

 

Ray

[Benjamin Good]

Ray,

Correct, only one thread.  (I'd happily launch an armada for some jobs if that was fine with you, but I assume that would not be well-received).

-Ben

On Thu, Jun 23, 2011 at 12:02 PM, Ray Fergerson <ray.fe...@stanford.edu> wrote:

Ben,

 

Thanks for the helpful information. Just confirming that you are making a call and waiting for a return before you make another call. You don’t have 1000 threads beating on the system at the same time. Correct?

 

Note, as a side matter unrelated to this instability, I believe that the longestOnly parameter only works correctly if you keep all ontologies. (I also thought though that we detected this combination of parameters now and blocked the call. Obviously I am wrong about this latter.) This limitation is really a design problem and is unlikely to get addressed anytime soon. The “longest only” logic is implemented by a separate process that knows nothing about ontologies. The filtering of results on ontologies is done after this matching. I suggest turning off the longest only parameter and doing the logic yourself on the returned values.

 

Ray

[Benjamin Good]

Hi Ray,

I haven't run this more than a couple times by hand.  I just ran it and the service failed 3 times before succeeding.  I ran it again and it worked the first time.  Ran it again and it failed once before succeeding.

If it would help you I can set up a loop and count failures per request..  Just let me know.

FYI, Aside from occasional large batch jobs, my main current use for the annotator is in support of a service that tweets about gene wiki updates and uses the annotator to summarize added text (see twitter/genewikipulse ).  That one hits the annotator highly intermittently with short blasts of short strings.

-Ben

On Thu, Jun 23, 2011 at 12:07 PM, Ray Fergerson <ray.fe...@stanford.edu> wrote:

Sorry, one more thing. I did look at your code and it seems straightforward. Just want to confirm that when you compile and run this on your system (which just annotates the same text over and offer) you can reproduce this problem. How often do you see failures? Daily? Hourly?

 

ray

 

From: Benjamin Good [mailto:ben.mcg...@gmail.com]

Sent: Thursday, June 23, 2011 11:32 AM
To: Ray Fergerson

Cc: sup...@bioontology.org; annotato...@googlegroups.com; Taciana Gomes

[Trish Whetzel]

Hi Ben, 

I've been able to reproduce the 500 error response. When setting the longestOnly parameter to true, occasionally the Annotator returns the 500 error response and occasionally the Annotator returns results. 

Trish 

[Benjamin Good]

Hi Trish,

Taking out that parameter from my request appears to solve the problem on my end.  

thanks!

-Ben