Frequent timeout exception (also on demo server)

[Murilo Habermann Torquato]

Hi all,

I would like to know if someone is also facing this situation:

During the system operation a timeout occurs for several times in a day on production environments and also on test machines.

- sometimes it happens just after we logged in to the system

- sometimes it takes a long time

- sometimes it doesn't happens  (even on a long session with a lot of operations done by the user)

We got the same behaviour several times when using demo.idempiere.com, but I'm unable to get the steps to reproduce the problem (it just happens on the fly :( )

I think that i'ts a Jetty related configuration because looking at the application logs (ALL) i can see several messages like:

java.util.concurrent.TimeoutException: Idle timeout expired: 30000/30000 ms

I found some related posts on google (like http://idletimeout.jetty.codesolution.site/4941447-1203-jetty-idletimeout.html) but all unsucessfull.

I'm using: iDempiere 3.1 (updated to the last revision) + Ubuntu 14.04 + OpenJDK 7 + PostgreSQL 9.4

Any idea?

[norber...@multimageweb.com]

Hi Murilo,

i'm not expert in this topic, but i remember something from last year. i found we have in our production system in /jettyhome/etc/jetty-selector.xml  <Set name="idleTimeout">300000</Set> (i have changed this last year august, no issue with last 9 months). So try to change 30000 to 300000.

hope help.

norbert

[Murilo Habermann Torquato]

Hi Norbert,

I will try that, thank you!

[mhernandezve]

Hi Murilo,

You can set the ZK_SESSION_TIMEOUT_IN_SECONDS parameter on System Configurator. You must set the Configuration Level = System

[Peter Shepetko]

Hi!

I set the ZK_SESSION_TIMEOUT_IN_SECONDS = *** and Configuration Level = System on System Configurator  but I didn't have a result.

This worked when I set session-timeout parameter in ../WEB-INF/web.hml

---------------------------

<!-- Miscellaneous -->

<session-config>

<session-timeout>***</session-timeout>

</session-config>

--------------------------

Maybe it's bug and we can add issue to the Jira about ZK_SESSION_TIMEOUT_IN_SECONDS  parameter.

--

Best regars,
------------------
Peter Shepetko
skype:  peter.shepetko
email:  pe...@fina.net.ua

[Marco Longo]

Unfortunately we have the same problem . it worked well before 3.1 and until december (more or less ) 

it is annoying for enduser :-(   (sometimes it happens during filling a field in a form :-( )

@PShepetko  do you use 3.1 ? your solution is not working for us.

marco

[Carlos Collazos]

It happened to me sometimes while doing something on a Idempiere window (any). It's random and I could not find a pattern. The only thing that I can say is that it happens sometimes when you switch to another window (not idempiere related) then you go back (on the same minute or so) type something and the session times out. 

It's not very often but it happens. I use firefox and also chrome over linux (centos and ubuntu) as client and ubuntu as server.

I changed all the session related parameters and sometimes still I got kicked from the session while doing something even the timeout time is still on range.

---------------------------------
Carlos Collazos
coll...@gmail.com
----------------------------------

--
You received this message because you are subscribed to the Google Groups "iDempiere" group.
To unsubscribe from this group and stop receiving emails from it, send an email to idempiere+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/idempiere/87e0f60e-b6ba-46de-ad9b-739261a43ca4%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Peter Shepetko]

@Marco Longo: Yes, we migrated to 3.1 and we are having this bug sometimes too as  other members discribed. 

To view this discussion on the web visit https://groups.google.com/d/msgid/idempiere/CAEVNTDPJeiiBArvx8aMGVgcAs0O-%3D%3DGpJ0yUD8wshM%3DF4hbKJw%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

[Murilo Habermann Torquato]

Hi All,

The same is happening with 3.1 and development branch.

Even with jetty configuration and with the ZK_SESSION_TIMEOUT_IN_SECONDS defined the behaviour is still the same.

I think that some exception is breaking the session and causing the timeout. Yesterday when I was using the system in a production environment I got this error when switched from a Window to the menu search textfield:

Abr 28, 2016 11:34:32 AM org.zkoss.zk.ui.AbstractComponent service

WARN: No page is available in [Desktop z_fqy:/index.zul] because desktop was destroyed.

It is usually caused by invalidating the native session directly. If it is required, please set Attributes.RENEW_NATIVE_SESSION first.

Abr 28, 2016 11:34:32 AM org.zkoss.zk.ui.impl.UiEngineImpl handleError

SEVERE: 

java.lang.NullPointerException

at org.zkoss.zk.ui.http.ExecutionImpl.getAttribute(ExecutionImpl.java:470)

at org.adempiere.webui.adwindow.DetailPane.setStatusMessage(DetailPane.java:490)

at org.adempiere.webui.adwindow.CompositeADTabbox.setDetailPaneStatusMessage(CompositeADTabbox.java:868)

at org.adempiere.webui.adwindow.AbstractADWindowContent.dataStatusChanged(AbstractADWindowContent.java:1487)

at org.compiere.model.GridTab.fireDataStatusChanged(GridTab.java:2362)

at org.compiere.model.GridTab.setCurrentRow(GridTab.java:2609)

at org.compiere.model.GridTab.query(GridTab.java:729)

at org.adempiere.webui.adwindow.ADTabpanel.query(ADTabpanel.java:942)

at org.adempiere.webui.adwindow.CompositeADTabbox.onActivateDetail(CompositeADTabbox.java:893)

[adamt...@gmail.com]

Hi ALL,

I have a development laptop for i2.1. I might moving my development works on another new laptop. That makes me need to redo those setup steps including materializing in eclipse and thus using the development branch. Is the session timeout things happens alot because it won't be nice if it happen while our staffs are using the system.

[Angel Lara]

Hi all I'm having the same problem, we want to upgrade our system from 2.1 to 3.1 but in the test enviroment we are getting this timeout session problem in random situations. Any ideas on how to fix this? I created a JIRA ticket about this issue https://idempiere.atlassian.net/browse/IDEMPIERE-3088

[adamt...@gmail.com]

Hi ALL,

I just need to move my development to another new laptop for only i2.1 version prior to Jetty integrated. Another new staff is coming in to maintain the system and I don't want any unexpected session expired happens.

Can anyone confirm me with above requirement? I am so afraid...

On Thursday, April 14, 2016 at 10:06:53 PM UTC+8, Murilo Habermann Torquato wrote:

[Carlos Antonio Ruiz Gomez]

Can you try changing the 30000 to 300000 in the file jettyhome/etc/jetty-ssl.xml  ?



El 02/05/16 a las 18:27, adamt...@gmail.com escribió:

[Angel Lara]

I will try that and comment the behavior, i already tried changing /jetty home/etc/jetty-selector.xml to 300000 as norbert suggested but with no success.

[Hiep Lq]

sometime i also get report about this issue. 

it's hard to reproduce on development environment to investigate, because it's random happen.

my user is nice, they just re-login, and i just don't care about this issue until i have more info to reproduce

just clear something: 

ZK_SESSION_TIMEOUT_IN_SECONDS is session timeout, now setting in web.xml with default value = 60 minute, 

just use ZK_SESSION_TIMEOUT_IN_SECONDS when you want to override this value.

idleTimeout is request timeout, have two place to setting one in jetty-ssl for https and one in other file (maybe jetty.xml) for http

normal, idleTimeout can't effect to session timeout. but like Norbert said, it can help. 

i guess it relate to atmosphase, atmosphase keep a comet request with timeout = 2 minute to do zk server push.

by idleTimeout = 30s, it will crash comet request. maybe comet handle do something wrong relate httpSession and break httpSession sometime.

so i suggest to increase idleTimeout = 123000 (2,3 minute) to test.

 

--

You received this message because you are subscribed to the Google Groups "iDempiere" group.
To unsubscribe from this group and stop receiving emails from it, send an email to idempiere+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/idempiere/36996c48-e38f-4bef-a076-71c758572119%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Lê Quý Hiệp
Email: hie...@hasuvimex.vn
Skype: admin.hasuvimex

Company: Thanh Hoa Fishery Import - Export J.s.c  (Hasuvimex) DL 47

Add: Lot E, Le Mon Industrial Zone, Thanh Hoa, Vietnam

[Carlos Antonio Ruiz Gómez]

Angel?  Adam?  Murilo?

Have you tested the proposed change in jettyhome/etc/jetty-ssl.xml ?

Any feedback?

Regards,

Carlos Ruiz

[Hiep Lq]

i update my config like i said just two days ago, set both idleTimeout = 123000.

my user report didn't see "timeout exception", but need more time to confirm.

will update late.

my environment (centos 7 + java 8 + jetty 9.3.8 + http2)

--

You received this message because you are subscribed to the Google Groups "iDempiere" group.
To unsubscribe from this group and stop receiving emails from it, send an email to idempiere+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/idempiere/0982c35c-9b2e-404e-82c1-a0243bff1e8e%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Marco Longo]

I will do the same this evening and give a feedback ( debian 8 +java8 )

[Henrique A. Lavezzo]

I will test in my test environment (Ubuntu + Java 7);

I've read Jetty docs in last days, and I realize some features inside the web server that improves performance, and security of requests.

[Angel Lara]

Hi, I made the Change in jettyhome/etc/jetty-ssl.xml to 300000 a week ago and i have not received reports of timeout exception from the users since then. 

[Murilo Habermann Torquato]

Hi,

I defined idleTimeout=300000 in both jetty-selector.xml and jetty-ssl.xml and some times the timeout is still being reported by users.

as soon as i have more news about it i'll post here.. 

BR,

[Nicolas Micoud]

Hi,

Any news on this subject ?
Is the issue still there ?

Thanks,

Nicolas

[Marco Longo]

on my side unfortunately no good news.

We tried everything mentioned in this thread and also other parameters . 

Marco

[norber...@multimageweb.com]

hi,

we are running version from 2015 dec 10 dev branch - and patches not in core - and really NO ISSUE others describe.

1. I have shared - above - what i change in my config. 

2. we analyse 1 month from some servers logs - and made changes on database level - im not sure related - however performance was better - im sure it is in peer review somewhere

3. we are using jetty with haproxy - nginx and apache is not affordable for serious work as proxy. (if you are using)

norbert

[Hiep Lq]

Norbert, your solution is already in core (commit 06d716d19724).

but i think, it just should document and mark as work-around,

1. we don't actual know reason, so solution maybe not correct at all.

2. a idleTimeout too long isn't good for server.

3. i still get issue, ever change it to 300.000, seem Marco also get it.

i now try to update jetty to 9.3.10.v20160621, hope it fine.

This e-mail is confidential and may contain legally privileged information. It is intended only for the addressees and may not be reviewed or used in any way by other recipients. If you have received this e-mail in error, kindly notify us immediately by telephone or e-mail and delete the message and any attachments thereto from your system.

--
You received this message because you are subscribed to the Google Groups "iDempiere" group.
To unsubscribe from this group and stop receiving emails from it, send an email to idempiere+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/idempiere/1c410605-0f8e-440e-9838-99951745ad92%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Murilo Habermann Torquato]

Hi all,

From our side, no good news too, we are still facing this issue.

 I think that the Hiep idea of upgrading Jetty may help.

Just for reference, my team pointed me some interesting (i think related) threads:

Bug Report Jetty
https://bugs.eclipse.org/bugs/show_bug.cgi?id=435322

Timeout Digital Oceal Serving Images
https://www.digitalocean.com/community/questions/jetty-timeout-problem-when-serving-images

Jetty Jboss Bug Report
https://issues.jboss.org/browse/JBIDE-21596

Druid Timeout Problems
https://groups.google.com/forum/#!topic/druid-user/t9XHsZKkGSU

Github Jetty Bug Report
https://github.com/eclipse/jetty.project/issues/517

BR

[adamt...@gmail.com]

Hi ALL,

Are the timeout issue solved?

May be upgrade the jetty version in the development branch and let me test on another pc? Or is there any document telling how to upgrade the local jetty version after cloning?

Our server is still i2.1.

My dev pc is still i2.1

I haven't doing the latest pull and sync the coding for latest version.

There is a request from my manager that we are going to implement iDempiere for another small company owned by my boss. Can I just clone and materialized it till i2.1 in the new pc and do some implementation?

Thanks for all the testing and solving.

Adam.

[Hiep Lq]

Adam. i already help you to update jetty :)

welcome you follow this thread to test https://groups.google.com/forum/#!topic/idempiere/liMO6fjXd8E

[adamt...@gmail.com]

Hi Hiep Lq,

I have followed your link https://experimental.idempiere.org/webui/index.zul and do some testing alone. I have not yet found any time out issue. The speed is great. I guess I will give latest version a try and be tested by a few staff later.

Thanks alot.

Adam.

[Hiep Lq]

it's not easy to get this issue :)

you have to spend long time with much lucky to meet it.

just test normal function and waiting it happen or not.

--

You received this message because you are subscribed to the Google Groups "iDempiere" group.
To unsubscribe from this group and stop receiving emails from it, send an email to idempiere+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/idempiere/2481b90d-f432-439a-be1b-3cca82f2d1f4%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Hiep Lq]

hey guys. if you give me cafe, i will give you reason and solution :)

On Wednesday, July 13, 2016 at 9:14:59 AM UTC+7, Hiep Lq wrote:

it's not easy to get this issue :)

you have to spend long time with much lucky to meet it.

just test normal function and waiting it happen or not.

On Wed, Jul 13, 2016 at 9:12 AM, <adamt...@gmail.com> wrote:

Hi Hiep Lq,

I have followed your link https://experimental.idempiere.org/webui/index.zul and do some testing alone. I have not yet found any time out issue. The speed is great. I guess I will give latest version a try and be tested by a few staff later.

Thanks alot.

Adam.

On Tuesday, July 12, 2016 at 1:01:28 AM UTC+8, Hiep Lq wrote:

Adam. i already help you to update jetty :)

welcome you follow this thread to test https://groups.google.com/forum/#!topic/idempiere/liMO6fjXd8E

On Friday, July 8, 2016 at 6:28:05 AM UTC+7, adamt...@gmail.com wrote:

Hi ALL,

Are the timeout issue solved?

May be upgrade the jetty version in the development branch and let me test on another pc? Or is there any document telling how to upgrade the local jetty version after cloning?

Our server is still i2.1.

My dev pc is still i2.1

I haven't doing the latest pull and sync the coding for latest version.

There is a request from my manager that we are going to implement iDempiere for another small company owned by my boss. Can I just clone and materialized it till i2.1 in the new pc and do some implementation?

Thanks for all the testing and solving.

Adam.

--

You received this message because you are subscribed to the Google Groups "iDempiere" group.

To unsubscribe from this group and stop receiving emails from it, send an email to idempiere+unsubscribe@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/idempiere/2481b90d-f432-439a-be1b-3cca82f2d1f4%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Hiep Lq]

https://idempiere.atlassian.net/browse/IDEMPIERE-3139

[Vasanthakumar S]

Hi,

Thank you Hiep Lq. 

Great work.

Regards,

Vasanth

To unsubscribe from this group and stop receiving emails from it, send an email to idempiere+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/idempiere/2481b90d-f432-439a-be1b-3cca82f2d1f4%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[HansAnton Riess]

Congratulations