Camel Failover Support for bridged topics

[Jacob Cartledge]

Hi all,

Im currently trying to create an exponential back off in activemq/camel for bridged topics as outlined on the wiki.
This seems possible through the use of failover support that is included, however i'm running in too a few problems.

When using

<property name="brokerURL" value="tcp://datafeeds.networkrail.co.uk:61619"/>

The connection is fine, but their is no exponential back off and as such results in being black listed if an iissue arises.   If i change this to the following ...

<property name="brokerURL" value="failover:(tcp://datafeeds.networkrail.co.uk:61619)?useExponentialBackOff=true&amp;maxReconnectDelay=60000&amp;reconnectDelayExponent=2.0"/>

It is my understanding that the back off will work as expected, but this method causes UnknownHostExceptions, after a successful connection.

 Successfully connected to tcp://datafeeds.networkrail.co.uk:61619 | org.apache.activemq.transport.failover.FailoverTransport | ActiveMQ Task-1
2015-11-04 13:15:25,946 | ERROR | Failed to Lookup INetAddress for URI[ nio://ec2netrailprodmaster1:61619 ] : java.net.UnknownHostException: ec2netrailprodmaster1: Name or service not known | org.apache.activemq.transport.failover.FailoverTransport | ActiveMQ Task-2
2015-11-04 13:15:41,334 | WARN  | Failed to connect to [tcp://datafeeds.networkrail.co.uk:61619] after: 10 attempt(s) continuing to retry. | org.apache.activemq.transport.failover.FailoverTransport | ActiveMQ Task-2

Does anybody have any idea as to what the cause of this is/ another solution to this problem?

Thanks

[Jacob Cartledge]

It should problably be noted that, the only reason for using failover is its in build support for exponential backoff and reconnect logic.
Their is currently no need to move to another location

[Jacob Cartledge]

I have also come across this Stack Overflow post, outlining the same problem.

http://stackoverflow.com/questions/20913984/activemq-failover-connection-not-working

[Peter Hicks]

Hi Jacob

On Wed, 4 Nov 2015 at 13:23 Jacob Cartledge <hi.i...@gmail.com> wrote:

It is my understanding that the back off will work as expected, but this method causes UnknownHostExceptions, after a successful connection.

 Successfully connected to tcp://datafeeds.networkrail.co.uk:61619 | org.apache.activemq.transport.failover.FailoverTransport | ActiveMQ Task-1
2015-11-04 13:15:25,946 | ERROR | Failed to Lookup INetAddress for URI[ nio://ec2netrailprodmaster1:61619 ] : java.net.UnknownHostException: ec2netrailprodmaster1: Name or service not known | org.apache.activemq.transport.failover.FailoverTransport | ActiveMQ Task-2
2015-11-04 13:15:41,334 | WARN  | Failed to connect to [tcp://datafeeds.networkrail.co.uk:61619] after: 10 attempt(s) continuing to retry. | org.apache.activemq.transport.failover.FailoverTransport | ActiveMQ Task-2

This is more ActiveMQ-y, but still on topic here :)

It looks like your host is trying to connect to 'ec2netrailprodmaster1' - I'm guessing that, once you connect, the datafeeds server sends back a hostname to ActiveMQ as a sort of "try to connect here".

Short-term solution - try putting an entry in your hosts file pointing at the IP address of datafeeds.networkrail.co.uk.  Long-term, I guess this is a 'bug' that needs to be addressed on the server side.

Peter

[Jacob Cartledge]

HI Peter,

Thanks! Adding this into the local hosts file works perfectly. However our system is deployed across multiple servers and as such we are looking into the possibilty of adding this into the DNS.

Another quick question is, how often does this change or is it a static IP? Also is there any method of being notified of this changing other than it failing?

Thanks Again!

[Nigel Mundy]

If you add it to your local DNS, rather than hosts, you could make it a cname to datafeeds.networkrail.co.uk

You would need to make sure your Servers resolv.conf has a search entry for yourdomain.com so that it can resolve it without being fully qualified.

Nigel

Sent from my BlackBerry 10 smartphone.

From: Jacob Cartledge Sent: Monday, 9 November 2015 16:50 To: A gathering place for the Open Rail Data community Cc: hi.i...@gmail.com Subject: Re: [openraildata-talk] Camel Failover Support for bridged topics

--
You received this message because you are subscribed to the Google Groups "A gathering place for the Open Rail Data community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to openraildata-t...@googlegroups.com.
To post to this group, send email to openrail...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[James Singleton]

The real host name for datafeeds.networkrail.co.uk is ec2-54-247-175-93.eu-west-1.compute.amazonaws.com which to me suggests that it's an AWS Elastic IP and not an Elastic Load Balancer (ELB). This means that the IP address is unlikely to change unless it is accidentally disposed of but Peter will have to confirm. Elastic IPs can be moved between instances and stay the same. ELBs are Amazon's built in load balancers which you should only address by name as they use DNS to provide Availability Zone fail-over and IP addresses may change.

One quirk you may want to watch out for is if you try to resolve ec2-54-247-175-93.eu-west-1.compute.amazonaws.com from inside the AWS EU region using Amazon's default internal DNS server. In this case it may resolve to a private class A address (10.x.x.x) and not the internet facing IP (54.x.x.x). The internal IPs can easily change e.g. when an instance is rebooted.

It doesn't look as if Route 53 (Amazon's DNS service) is being used. I don't believe you can use this for just a sub-domain so the whole of networkrail.co.uk would have to use it if this were the case.

James

[Peter Hicks]

Sure, 'datafeeds.networkrail.co.uk' is a CNAME to another host - but I wouldn't recommend anyone actually modifies their DNS to 'fudge' this... host files are a sticking plaster, DNS is making the sticking plaster opaque.  Remember that the time you're *not* around will probably be the time somebody else tries to work out what the hell you've done!

I have a meeting tomorrow to discuss open issues - I'll ensure this one gets raised and logged.  It's really easy to fix, I think.

Peter

[Jacob Cartledge]

Thanks for the information James!

I understand what your saying Peter, but for the short term it will have to do.
Would it be possible to have an update on the outcome of the meeting please? Maybe even a place I can track the issues progress?

Thanks
Jake

[Jacob Cartledge]

Hi Peter

I have a meeting tomorrow to discuss open issues - I'll ensure this one gets raised and logged.  It's really easy to fix, I think.

 

Just wondering if you have an update on the outcome of this meeting? Has a fix been proposed?

Thanks
Jake

[Peter Hicks]

Hi Jacob

I've logged it as a bug - no idea if/when it'll get fixed I'm afraid :(

Peter

--