Jetty shuts down automatically after some time

[Vaibhav Dhawan]

Hello, 

I am on Dropwizard 1.0.0 , Java 1.8 and have a few services running in there. Since the services are not in production yet, hence there are periods when no one is accessing those services. 

What i notice is that Jetty shuts down automatically and gracefully after some period of inactivity. The inactivity period is not consistent and it goes down at random times. Here is an excerpt from the logs

DEBUG  [12:34:34.693] [dw-21] o.e.j.u.t.QueuedThreadPool -  ran EPC Prod/org.eclipse.jetty.io.ManagedSelector$SelectorProducer@432959e3

DEBUG  [12:57:24.162] [Thread-13] o.e.j.u.c.AbstractLifeCycle -  stopping org.eclipse.jetty.server.Server@46e3559f

DEBUG  [12:57:24.162] [Thread-13] o.e.j.s.Server -  doStop org.eclipse.jetty.server.Server@46e3559f

DEBUG  [12:57:24.164] [Thread-13] o.e.j.s.Server -  Graceful shutdown org.eclipse.jetty.server.Server@46e3559f by

DEBUG  [12:57:24.164] [Thread-13] o.e.j.u.c.AbstractLifeCycle -  stopping application@21c747a{HTTP/1.1,[http/1.1]}{0.0.0.0:8090}

.

.

.

.INFO   [12:57:24.221] [Thread-13] o.e.j.s.h.ContextHandler -  Stopped i.d.j.MutableServletContextHandler@5bfa8cc5{/,null,UNAVAILABLE}

DEBUG  [12:57:24.221] [Thread-13] o.e.j.u.c.AbstractLifeCycle -  STOPPED i.d.j.MutableServletContextHandler@5bfa8cc5{/,null,UNAVAILABLE}

This time it went down after 23 minutes of inactivity, there are times it goes doen after 6 minutes, 12 minutes etc. 

Any clues? Should i have a dummy service which simply pings every minute to keep it alive. I have used 0.9.2 for a 6 months and never faced this issue earlier. 

Any pointers are appreciated. 

Thanks

Vaibhav

[Vaibhav Dhawan]

Just to add, i am using in hibernate as well which points to a mysql server. Just thinking if there is a connection to database here as well. 

Thanks

Vaibhav

[Evan Meagher]

Hey Vaibhav. Can you provide the full log output during shutdown (perhaps as a gist) that includes anything you elided in the "..." section of your snippet, please?

--
You received this message because you are subscribed to the Google Groups "dropwizard-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dropwizard-user+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Evan Meagher

[Vaibhav Dhawan]

Hi Evan, 

Thanks for replying. Attached is the log file.No errors in here, it simply shuts down gracefully. 

Thanks

To unsubscribe from this group and stop receiving emails from it, send an email to dropwizard-us...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Evan Meagher

[Vaibhav Dhawan]

So, it has happened twice since I wrote this. Jetty simply shuts down after few minutes of inactivity. Not consistently - because I have tried to monitor it as well.

[Vaibhav Dhawan]

[Steve Kradel]

What is your service doing?  I haven't seen anything to suggest an intrinsic problem in the framework across millions of requests.

[centos@xxxxx ~]$ systemctl status dwx

● dwx.service - dwx

   Loaded: loaded (/etc/systemd/system/dwx.service; enabled; vendor preset: disabled)

   Active: active (running) since Tue 2016-08-16 22:20:14 UTC; 3 weeks 1 days ago

[Vaibhav Dhawan]

There are a few services in there. All of them perform crud operations with my sql server via hibernate.

[Vaibhav Dhawan]

Was able to get through this finally. Still not sure what caused it but started the DW process as a service on my Ubuntu and it hasnt crashed/stopped since.  

Thanks

Vaibhav 

On Saturday, 3 September 2016 13:50:30 UTC+5:30, Vaibhav Dhawan wrote:

[thej.m...@nowfloats.com]

Hi Vaibhav,

I am facing similar problem. I was running it as a background process but still it shutdown after some time. Please help how you resolved it. Thanks in advance.

Regards,

Thej

[Vaibhav Dhawan]

Hi, 

For me it was related to the user with which i was running the service. As soon as the ssh session would time out , the service would stop, even if it was started as a background process.

As soon as i shifted to root/similar user, the problem went away. 

Thanks

Vaibhav Dhawan

--
You received this message because you are subscribed to a topic in the Google Groups "dropwizard-user" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/dropwizard-user/4L-k1tgPOm0/unsubscribe.
To unsubscribe from this group and all its topics, send an email to dropwizard-user+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Thanks

Vaibhav Dhawan
+919871488896

[Graham O'Regan]

Sounds like the process is hanging up, take a look at the Upstart or systemd scripts that were provided by yammer on the mailing list (posted by ryan a few years ago). If you can’t find them then I have one kicking around that I could share.

You received this message because you are subscribed to the Google Groups "dropwizard-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dropwizard-us...@googlegroups.com.

[Thej Kamal Rao Menneni]

It helps a lot Vaibhav. Now I am running now as root and will check. 

Regards,

Thej

THEJ KAMAL RAO MENNENI | Software  Engineer|8123635251| NowFloats.com| @NowFloats

We are uncovering #deshruption across India. To know more follow @DigitalDesh

[karthik sk]

Hey i am having a similar problem. Did running it as root solve it? and Did you figure out why it is happening? Thanks in advance.

To unsubscribe from this group and all its topics, send an email to dropwizard-us...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Thanks

Vaibhav Dhawan
+919871488896

--
You received this message because you are subscribed to a topic in the Google Groups "dropwizard-user" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/dropwizard-user/4L-k1tgPOm0/unsubscribe.

To unsubscribe from this group and all its topics, send an email to dropwizard-us...@googlegroups.com.

[Thej Kamal Rao Menneni]

Hi,

Running as 'root' solved the problem. I guess session timing out after some interval. Let me know if you face any other issue.

Regards,

Thej

THEJ KAMAL RAO MENNENI | Software  Engineer|8123635251| NowFloats.com| @NowFloats

We are uncovering #deshruption across India. To know more follow @DigitalDesh

To unsubscribe from this group and all its topics, send an email to dropwizard-user+unsubscribe@googlegroups.com.

[karthik sk]

Yes It worked for me too. Thanks a lot. I am still wondering if there is any alternative solution to this, since I am not sure if running the application as root is secure. 

[Vaibhav Dhawan]

Hi, 

You can try to set this up as a service and it wont stop automatically. It should also start automatically if an when your server restarts. 

Thanks

Vaibhav

To unsubscribe from this group and all its topics, send an email to dropwizard-user+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Saumitra Bhave]

The better way would be to use nohup,

like following

nohup java -jar app.jar server app.yml > /dev/null 2> /dev/null < /dev/null &

In the above command,

nohup - Eats all the hangup commands sent via terminal or session closing, so you java command never gets them

& at the last makes sure you still run it as background process

redirect output, error and input streams to null so that services like AWS CodeDeploy dont block on commands execution.

I have been using the exact same command as above in production for more than a year now, works like a charm for me :)