Slow server response when using INCEpTION

[Jan Matti Dollbaum]

Dear INCEpTION team,

 

we are using a server-based version of INCEpTION to annotate documents from eight accounts (three admins and five annotators). Despite using Chrome, all annotators and one admin have experienced problems with long loading times when switching between documents and when selecting a new layer from the dropdown menu, and also occasional crashes of the site. We were wondering whether there is a common set of solutions we could try to improve performance. We have currently about 800 documents loaded with most being locked for annotators but all being open to admins. Is that a potential source of the problem?

 

Many thanks for any help!

Jan

[Richard Eckart de Castilho]

Hi Jan,

> On 7. Jun 2021, at 21:34, Jan Matti Dollbaum <janmatti...@gmail.com> wrote:
>
> we are using a server-based version of INCEpTION to annotate documents from eight accounts (three admins and five annotators). Despite using Chrome, all annotators and one admin have experienced problems with long loading times when switching between documents and when selecting a new layer from the dropdown menu, and also occasional crashes of the site. We were wondering whether there is a common set of solutions we could try to improve performance. We have currently about 800 documents loaded with most being locked for annotators but all being open to admins. Is that a potential source of the problem?

I assume by "layer dropdown menu", you mean the dropdown on the top-right part of the annotation page. After it has been loaded, it should be a client-side-only thing to open it and select something. That may indicate that the issue is a client-side issue and not a server-side issue.

The number and size of documents that are viable correlates to the memory size available to INCEpTION. So far, we do not have a good way of auto-tuning that. So if you notice that your JVM is running out of heap space, then you'd have to either reduce cache sizes or increase heap memory for the JVM - but as I said before, we may rather be looking at a client-side thing.

Calculating the layout takes most time on the client - that is why INCEpTION uses a paging mechanism to show only a limited number of lines at a time. This setting can be made in the "preferences" (cogwheel) dialog on the annotation page.

How much lines have you configured to be visible at a time?

Best,

-- Richard

[Jan Matti Dollbaum]

Hi Richard,

thanks for these ideas. The number of displayed lines varies for our annotators, but my guess is that most will display about 30 because that is the average length of our documents. I will ask the annotators to reduce that number and see whether performance improves.

Many thanks!
Jan

--
You received this message because you are subscribed to the Google Groups "inception-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to inception-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/inception-users/E1C66883-559D-421C-ADA6-24E12A107294%40gmail.com.

[Richard Eckart de Castilho]

Hi,

> On 7. Jun 2021, at 22:24, Jan Matti Dollbaum <janmatti...@gmail.com> wrote:
>
> thanks for these ideas. The number of displayed lines varies for our annotators, but my guess is that most will display about 30 because that is the average length of our documents. I will ask the annotators to reduce that number and see whether performance improves.

If you documents contain long sentences, I would recommend trying the "brat @ 120" visualization mode - however, that may
require upgrading to the recently released INCEpTION 0.19.5 which fixes a bug in that visualization mode.

Best,

-- Richard

[Jan Matti Dollbaum]

We use this setting (sorry, should have been clearer). Perhaps, then, we should try upgrading.

Thanks!
Jan

> On 7. Jun 2021, at 22:32, Richard Eckart de Castilho <richard...@gmail.com> wrote:
>
> Hi,

> --
> You received this message because you are subscribed to the Google Groups "inception-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to inception-use...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/inception-users/8796D7AB-0268-4D1B-84A3-35F1EE985F3D%40gmail.com.

[Richard Eckart de Castilho]

On 7. Jun 2021, at 22:56, Jan Matti Dollbaum <janmatti...@gmail.com> wrote:
>
> We use this setting (sorry, should have been clearer). Perhaps, then, we should try upgrading.

The fixed bug relates to a failure in rendering documents containing consecutive blank lines.
If your documents do not have consecutive blank lines, you should be ok. Upgrading to the
latest bugfix release is still not a bad idea.

If you want, you could also create an account for me on your system and privately mail me the
access, then I could have a first hand look at your project and see if I notice anything
out of the ordinary. I'm quite interested in learning about and hopefully eliminating an
performance issues. Infos from user projects has been quite useful for this task in the past.

Best,

-- Richard

[Kyle Williams]

On Monday, 7 June 2021 at 12:34:17 UTC-7 janmatti...@gmail.com wrote:

we are using a server-based version of INCEpTION to annotate documents from eight accounts (three admins and five annotators). Despite using Chrome, all annotators and one admin have experienced problems with long loading times when switching between documents and when selecting a new layer from the dropdown menu, and also occasional crashes of the site. We were wondering whether there is a common set of solutions we could try to improve performance. We have currently about 800 documents loaded with most being locked for annotators but all being open to admins. Is that a potential source of the problem?

Our annotators ran into the same issue. They were loading documents with a few hundred lines and it was taking about 15 second to annotate each span.

I asked them to reduce the number of visible lines to around 20 and it seems to be much better.

Our documents are of the plan-text type and not one sentence per line. That being said, most lines are fairly short with 5-10 words. @Richard, would we still benefit from upgrading to 0.19.5 and trying the brat @ 120 visualization mode?

[Richard Eckart de Castilho]

On 8. Jun 2021, at 01:56, Kyle Williams <kylemark...@gmail.com> wrote:
>
> Our annotators ran into the same issue. They were loading documents with a few hundred lines and it was taking about 15 second to annotate each span.
> I asked them to reduce the number of visible lines to around 20 and it seems to be much better.

The brat-based visualization is very powerful, but to my experience, it does not scale too well. We added a paging mechanism early
on (originally in WebAnno)) to mitigate the problem. That said, for a given amount of text an annotations, the visualization should
be pretty much as fast as the original brat (unless you use RTL mode which is slower).

I usually configure the number of visible lines such that no scroll bar appears but most of the screen is filled with text/annotations,
which usually tends to be around 5 - 15 depending on the annotation task.

One drawback of the pagination is currently that annotations crossing the page boundary are not rendered in the main screen.
Rendering partial/clipped spans is probably one of the last features that will make it into the upcoming feature release.
Not sure yet about rendering page-crossing relations / slot arcs.

> Our documents are of the plan-text type and not one sentence per line. That being said, most lines are fairly short with 5-10 words. @Richard, would we still benefit from upgrading to 0.19.5 and trying the brat @ 120 visualization mode?

The "brat @ 120" mode word-wraps at 120 characters. It is useful for situations where you do not care much about sentence boundaries
but have rather longish sentences. Brat is unable to wrap the text inside an annotation. So if you have "one sentence per row" and
an annotation on the entire sentence, then you'll get a horizontal scrollbar. By pre-wrapping the text at 120 characters before
handing it over to brat, we can provide more "break points" which makes working with longer annotation spans much more viable.
If any of the above applies to you, then trying the pre-wrapping "brat @ 120" mode could be useful to you.

Best,

-- Richard

[Richard Eckart de Castilho]

Jan, Kyle,

I really wonder what is making things slow for you. I have tried myself

using a document with quite dense and complex annotation (POS + Dependencies

from the GUM corpus). Then I set the number of visible lines to 30 (which

already created a pretty long vertical scrollbar). Then I started adding

named entity annotations to random words. The delay between actions was

roughly 3-4 seconds.

Admittedly, I tried only 1 document, not 800 - but this if this is  about

client-side stuff, then 1 document should be sufficient to get a measure.

I used the browser's profile and timings we collect in the backend to generate

the attached measurements with 30 visible lines. Note that of the 7500ms, most

is idleing because after starting profiling it took me a moment to actually 

start creating the annotation and afterwards, I turned off the profiler.

But this is all still way off the 12 seconds or more of delay that you report.

Cheers,

-- Richard

[Richard Eckart de Castilho]

Hi,

Jan and colleagues were so kind to supply me with access to their server, so I could try to reproduce the slowness.

When switching between documents, I get a 2-3 second delay from their server. INCEpTION performs an internal garbage collection on the annotations and other stuff when switching between documents and that takes a bit of additional time.

When paging within a document using "brat@120" and even with 30 visible rows, I get a 250ms server roundtrip time (that includes network lag) and client-side rendering in Chrome takes an additional ~200ms. Feels snazzy to me.

I get zero delay when opening the layer dropdown - it opens immediately.

When I create an annotation, it takes about 2 seconds in total for the annotation to be saved server-side and the document to be re-rendered browser site. That might be worth digging a bit more into.

If anybody wants to get a bit more insight on timings, you can add the line

debug.sendServerSideTimings=true

to the `settings.properties` file and restart INCEpTION. After that, if you open the Web Developer Console in Chrome and click on a network request, there should be additional information in the "Server timings" section of the request details.

-- Richard

[Richard Eckart de Castilho]

Hi all,

further analysis indicates that the "open document" dialog on the annotation page can become
very slow if there is a large number of documents in a project. Fortunately it was easily
resolved and we will do a new bugfix release containing the fix soon.

https://github.com/inception-project/inception/issues/2365

Not that this slowness was specific to opening/using the "open document" dialog, not to
the switching between documents using the prev/next document buttons and also not related
to the number of rows visible in the browser.

If you have any further pointers towards slowness, please let us know.

-- Richard

[Jan Matti Dollbaum]

Hi Richard,

Many thanks! We will check back with the annotators tomorrow, I will let you know whether this was part of the problem.

Cheers,
Jan

> On 10. Jun 2021, at 22:15, Richard Eckart de Castilho <richard...@gmail.com> wrote:
>
> Hi all,

> --
> You received this message because you are subscribed to the Google Groups "inception-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to inception-use...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/inception-users/FF6CFA0E-4C00-4D4E-A8F8-A1A5FDAB9BEB%40gmail.com.

[richard...@gmail.com]

Hi

Have you been able to obtain insight from the annotators?

— Richard

Sent from my mobile, sorry for brevity.

[Jan Matti Dollbaum]

Hi Richard,

thanks for following up. The annotators have confirmed that they use the arrows to switch between documents, but they are still experiencing the same problems, even though we have doubled the internal memory assigned to INCEpTION on the server. However, I myself do not get any of these delays. This makes us think that it is probably the soft- or hardware of the annotators that causes the problems. However, since the use of somewhat outdates tech is probably rather widespread, we were wondering whether there is any chance in upcoming releases to decrease INCEpTION's dependency on the client side?

We will also soon replace the server on which INCEpTION is running and will keep you updated on any progress we make. Many thanks again for your help so far!

Best,
Jan

We will keep you updated in case anything

--
You received this message because you are subscribed to the Google Groups "inception-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to inception-use...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/inception-users/595D2D5B-F315-4333-9EC8-95070560A5C9%40gmail.com.

[richard...@gmail.com]

Actually the trend in web development is to put more on the client and less on the server. In fact INCEpTION does use quite traditional technology and prepares a lot on the server side and offloads very little to the client.

For a web application, we need to consider not only client and server performance but also network bandwidth, latency and the volume of the data being exchanged between Browser and server. We need to balance these aspects. At the current point, doing more on the server would likely increase the network traffic and that would reduce responsiveness for people on slower network connections.

— Richard

Sent from my mobile, sorry for brevity.