how to disable connection pool

[yu zhang]

Hi,

I'm trouble shooting a program using play WS API to call a REST service from a business partner, This REST service does its API upgrade by switching the servers. It puts a newer version on a new server, then change its DNS to point the same host name to the new server IP. 

WSRequest request = WS.url(....);

This is causing problem. Every time they do this, the program still goes to the old server unless I manually restart. I'm guessing this is due to the fact play WS API is using connection pool, which keeps old connections that still has the old IP. 

So I tried to use this parameter to turn off the connection pool,

play.ws.ahc.keepAlive=false

Still doesn't work. 

Anybody can help with this? 

Thanks,

Yu

[Igmar Palsenberg]

 

I'm trouble shooting a program using play WS API to call a REST service from a business partner, This REST service does its API upgrade by switching the servers. It puts a newer version on a new server, then change its DNS to point the same host name to the new server IP. 

WSRequest request = WS.url(....);

This is causing problem. Every time they do this, the program still goes to the old server unless I manually restart. I'm guessing this is due to the fact play WS API is using connection pool, which keeps old connections that still has the old IP. 

It's caused by the JVM caching DNS lookups. You need to set networkaddress.cache.ttl to a sane value. You also might want to point out that a changed IP for every API upgrade is a) very uncommon and b) not necessary.

Igmar

[Mariot Chauvin]

Hi,

Yes this is possible. We encountered the same problem some time ago, and added configuration settings for this (https://github.com/playframework/playframework/pull/4016)

According to documentation settings are now named:

• play.ws.ahc.maximumConnectionLifetime
• play.ws.ahc.idleConnectionInPoolTimeout

If you are using play 2.3 we created a plugin to solve this issue:

https://github.com/cb372/play-configurable-ningwsplugin

In our use case we configure the pool to keep the connection only for 1 minute, as we have 1 minute TTL on our CNAME.

play.ws.ahc.maxConnectionLifetime = 1 minute

--
You received this message because you are subscribed to the Google Groups "Play Framework" group.
To unsubscribe from this group and stop receiving emails from it, send an email to play-framewor...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/play-framework/c78a9862-07aa-4cf6-b97e-2c617ae88a69%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

---

This e-mail and all attachments are confidential and may also be privileged. If you are not the named recipient, please notify the sender and delete the e-mail and all attachments immediately. Do not disclose the contents to another person. You may not use the information for any purpose, or store, or copy, it in any way.  Guardian News & Media Limited is not liable for any computer viruses or other material transmitted with or as part of this e-mail. You should employ virus checking software.

 

Guardian News & Media Limited is a member of Guardian Media Group plc. Registered Office: PO Box 68164, Kings Place, 90 York Way, London, N1P 2AP.  Registered in England Number 908396

[Mariot Chauvin]

Hi Igmar,

I just wanted to point out that it may be more common than you may think, as this is way to achieve blue/green deployment by using DNS as your router.

We have blogged how we used such a setup for upgrading elasticsearch at the guardian,

Short TTL is also a core trick (scattering) used by cloudflare to failover other backends during a DDOS attack.

 

Regards,

Mariot

 

Igmar

--
You received this message because you are subscribed to the Google Groups "Play Framework" group.

To unsubscribe from this group and stop receiving emails from it, send an email to play-framework+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/play-framework/d58a5ad2-f090-4d1a-af70-009a844d3f9a%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Igmar Palsenberg]

I just wanted to point out that it may be more common than you may think, as this is way to achieve blue/green deployment by using DNS as your router.

We have blogged how we used such a setup for upgrading elasticsearch at the guardian,

I think it's a bad way of implementing this. For a number of reasons : 

1) It assumes that the DNS server doesn't override the TTL value. That happens, in real life.

2) It assumes that the local implementation honours it (nscd didn't in the past for one)

3) It assumes it isn't cached somewhere else

In general, I avoid relying on DNS changes : It will bite you at some point. I usually recommend either an ELB that understands resource groups (Amazon's ELB does), or use nginx that can be taught to handle this. Amazon's new ELB can even switch to a different resource group based on the request path.

That also saves you the wait for a TTL to kick in in case the switch turns out bad. Or hacking a java lib because it uses a resource pool which you have no control over.

Short TTL is also a core trick (scattering) used by cloudflare to failover other backends during a DDOS attack.

Assuming clients honour that. Your average DDOS service / tool will probably not.

If you ask me, their biggest asset is the huge amounts of bandwidth they have. 

Igmar

[Mariot Chauvin]

On 29 March 2017 at 12:12, Igmar Palsenberg <ig...@palsenberg.com> wrote:

I just wanted to point out that it may be more common than you may think, as this is way to achieve blue/green deployment by using DNS as your router.

We have blogged how we used such a setup for upgrading elasticsearch at the guardian,

I think it's a bad way of implementing this. For a number of reasons : 

1) It assumes that the DNS server doesn't override the TTL value. That happens, in real life.

2) It assumes that the local implementation honours it (nscd didn't in the past for one)

3) It assumes it isn't cached somewhere else

In general, I avoid relying on DNS changes : It will bite you at some point.

This is where I disagree, TTL is part of the DNS spec and is pretty well respected everywhere.

In the past Java had been bad but this is now fixed.

 

I usually recommend either an ELB that understands resource groups (Amazon's ELB does), or use nginx that can be taught to handle this. Amazon's new ELB can even switch to a different resource group based on the request path.

But you can't simply use an ELB if your goal is balance your traffic between 2 stacks, or you have to have a stack that contains only this ELB and then have substacks that are your "real" stack but this is complicated setup compared to having route53 letting you select and switch the ELB. You can even distribute x% of your traffic to a stack or distribute it based on geolocation of the request.

I understand the benefits of the approach you mention but in practice its is painful to set up. I know that on google cloud you have an internal LB in front of your application that allow you to siwtch between versions.

 

That also saves you the wait for a TTL to kick in in case the switch turns out bad. Or hacking a java lib because it uses a resource pool which you have no control over.

It is not 'hacking' on something you don't have control over, it is configuring a setting which control exactly the resource pool behaviour. 

 

Short TTL is also a core trick (scattering) used by cloudflare to failover other backends during a DDOS attack.

Assuming clients honour that. Your average DDOS service / tool will probably not.

If you ask me, their biggest asset is the huge amounts of bandwidth they have. 

Please read the cloudflare blog article where they explain how they do it and would like to be able to set a shot TTL on the Glue record to do the same.

 

Igmar

--
You received this message because you are subscribed to the Google Groups "Play Framework" group.
To unsubscribe from this group and stop receiving emails from it, send an email to play-framework+unsubscribe@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/play-framework/6bff6b8b-70ab-4ac9-a6d3-db47e3879dda%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Igmar Palsenberg]

I just wanted to point out that it may be more common than you may think, as this is way to achieve blue/green deployment by using DNS as your router.

We have blogged how we used such a setup for upgrading elasticsearch at the guardian,

I think it's a bad way of implementing this. For a number of reasons : 

1) It assumes that the DNS server doesn't override the TTL value. That happens, in real life.

2) It assumes that the local implementation honours it (nscd didn't in the past for one)

3) It assumes it isn't cached somewhere else

In general, I avoid relying on DNS changes : It will bite you at some point.

This is where I disagree, TTL is part of the DNS spec and is pretty well respected everywhere.

In the past Java had been bad but this is now fixed.

https://docs.oracle.com/javase/8/docs/technotes/guides/net/properties.html lists : 

The default behavior is to cache forever when a security manager is installed, and to cache for an implementation specific period of time, when a security manager is not installed.

Notice the "implementation specific period". 

Of course you can override it (AWS recommends setting it to 60). I've seen multiple clients / servers with either bugs, or simply ignoring / overriding TTL records. Not to mention application doing their own caching.

I usually recommend either an ELB that understands resource groups (Amazon's ELB does), or use nginx that can be taught to handle this. Amazon's new ELB can even switch to a different resource group based on the request path.

But you can't simply use an ELB if your goal is balance your traffic between 2 stacks,

Why not ? Just attach 2 autoscaling groups.

or you have to have a stack that contains only this ELB and then have substacks that are your "real" stack but this is complicated setup compared to having route53 letting you select and switch the ELB. You can even distribute x% of your traffic to a stack or distribute it based on geolocation of the request.

That is traffic to the ELB's themselves, I'm talking about the distribution behind the ELB.

 

I understand the benefits of the approach you mention but in practice its is painful to set up. I know that on google cloud you have an internal LB in front of your application that allow you to siwtch between versions.

 

That also saves you the wait for a TTL to kick in in case the switch turns out bad. Or hacking a java lib because it uses a resource pool which you have no control over.

It is not 'hacking' on something you don't have control over, it is configuring a setting which control exactly the resource pool behaviour.

Assuming you have control over it.

 

Short TTL is also a core trick (scattering) used by cloudflare to failover other backends during a DDOS attack.

Assuming clients honour that. Your average DDOS service / tool will probably not.

If you ask me, their biggest asset is the huge amounts of bandwidth they have. 

Please read the cloudflare blog article where they explain how they do it and would like to be able to set a shot TTL on the Glue record to do the same.

That article is about TLD glue records, and about protecting DNS servers. That still doesn't resolve the issues I describe above, which are there in real life. 

Sure, in the ideal world a short TTL would be a good thing, and solve a lot of issues. But we're not living in the ideal world.

Igmar

[yu zhang]

Thanks for the help! Now I have a better understanding of this issue. I'm trying to set JVM DNS parameter, first I use this

activator -Dsun.net.inetaddr.ttl=60 -Dconfig.resource=clip-account-local.conf -Dlogger.resource=logback-local.xml run 

didn't work

Then I changed java.security in jre/lib/security, set the

networkaddress.cache.ttl=60

still didn't work

Have I missed anything? 

Thanks,

Yu

[yu zhang]

When I run with activator command, java security manager is enabled by default? So it should work if I change the setting in java.security file?

activator -Dconfig.resource=clip-account-local.conf -Dlogger.resource=logback-local.xml run

in java.security

# NOTE: setting this to anything other than the default value can have

#       serious security implications. Do not set it unless

#       you are sure you are not exposed to DNS spoofing attack.

#

networkaddress.cache.ttl=60

On Tuesday, 28 March 2017 22:17:09 UTC-7, yu zhang wrote:

[yu zhang]

This issue is finally fixed. I need to change both JVM DNS TTL and turn off play connection pool to make it work. 

networkaddress.cache.ttl=60

play.ws.ning.allowPoolingConnection=false

play.ws.ning.allowSslConnectionPool=false

play.ws.ning.maxConnectionLifeTime doesn't work. Whatever I set in configuration file, the value is still -1, which means forever. Could be a bug in play framework.

Thanks,

On Tuesday, 28 March 2017 22:17:09 UTC-7, yu zhang wrote: