Derby not starting up in 1527 in AWS host

91 views
Skip to first unread message

Udayakumar Pandurangan

unread,
Nov 30, 2016, 9:24:24 PM11/30/16
to Druid Development
Hi there,
       I was able to successfully use the 0.9.1.1 version in standalone mode in my MAC as a single node setup - everything worked like charm. Now I'm creating a small cluster (t2.micro) in AWS using the same version (0.9.1.1).

1 node with Coordinator & Overlord - This same node hosts zookeeper & Derby
1 node with Broker
1 node with Historical & MiddleManager

Now when I start coordinator - ideally it should start the derby and have a successful creation of tables. But somehow this is not happening.

Coordinator Logs:

2016-12-01T01:46:45,102 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:user.name=ec2-user

2016-12-01T01:46:45,102 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:user.home=/home/ec2-user

2016-12-01T01:46:45,102 INFO [main] org.apache.zookeeper.ZooKeeper - Client environment:user.dir=/home/ec2-user/druid/druid-0.9.1.1

2016-12-01T01:46:45,103 INFO [main] org.apache.zookeeper.ZooKeeper - Initiating client connection, connectString=52.53.243.72 sessionTimeout=30000 watcher=org.apache.curator.ConnectionState@4613311f

2016-12-01T01:46:45,124 INFO [main] com.metamx.common.lifecycle.Lifecycle$AnnotationBasedHandler - Invoking start method[public void io.druid.server.listener.announcer.ListenerDiscoverer.start()] on object[io.druid.server.listener.announcer.ListenerDiscoverer@1fb2d5e].

2016-12-01T01:46:45,124 INFO [main] io.druid.server.listener.announcer.ListenerDiscoverer - Started

2016-12-01T01:46:45,124 INFO [main] com.metamx.common.lifecycle.Lifecycle$AnnotationBasedHandler - Invoking start method[public void io.druid.metadata.storage.derby.DerbyMetadataStorage.start()] on object[io.druid.metadata.storage.derby.DerbyMetadataStorage@6573d2f7].

2016-12-01T01:46:45,124 INFO [main] io.druid.metadata.storage.derby.DerbyMetadataStorage - Starting Derby Metadata Storage

2016-12-01T01:46:45,133 INFO [main-SendThread(52.53.243.72:2181)] org.apache.zookeeper.ClientCnxn - Opening socket connection to server 52.53.243.72/52.53.243.72:2181. Will not attempt to authenticate using SASL (unknown error)

2016-12-01T01:47:15,317 WARN [main-SendThread(52.53.243.72:2181)] org.apache.zookeeper.ClientCnxn - Client session timed out, have not heard from server in 30195ms for sessionid 0x0

2016-12-01T01:47:15,318 INFO [main-SendThread(52.53.243.72:2181)] org.apache.zookeeper.ClientCnxn - Client session timed out, have not heard from server in 30195ms for sessionid 0x0, closing socket connection and attempting reconnect

2016-12-01T01:47:16,420 INFO [main-SendThread(52.53.243.72:2181)] org.apache.zookeeper.ClientCnxn - Opening socket connection to server 52.53.243.72/52.53.243.72:2181. Will not attempt to authenticate using SASL (unknown error)

:

:

2016-12-01T01:48:52,753 WARN [main] com.metamx.common.RetryUtils - Failed on try 1, retrying in 1,156ms.

org.skife.jdbi.v2.exceptions.UnableToObtainConnectionException: java.sql.SQLException: Cannot create PoolableConnectionFactory (java.net.ConnectException : Error connecting to server 52.53.243.72 on port 1,527 with message Connection timed out (Connection timed out).)

        at org.skife.jdbi.v2.DBI.open(DBI.java:230) ~[jdbi-2.63.1.jar:2.63.1]

        at org.skife.jdbi.v2.DBI.withHandle(DBI.java:279) ~[jdbi-2.63.1.jar:2.63.1]

        at io.druid.metadata.SQLMetadataConnector$2.call(SQLMetadataConnector.java:123) ~[druid-server-0.9.1.1.jar:0.9.1.1]

        at com.metamx.common.RetryUtils.retry(RetryUtils.java:60) [java-util-0.27.9.jar:?]

        at com.metamx.common.RetryUtils.retry(RetryUtils.java:78) [java-util-0.27.9.jar:?]

        at io.druid.metadata.SQLMetadataConnector.retryWithHandle(SQLMetadataConnector.java:127) [druid-server-0.9.1.1.jar:0.9.1.1]

        at io.druid.metadata.SQLMetadataConnector.retryWithHandle(SQLMetadataConnector.java:136) [druid-server-0.9.1.1.jar:0.9.1.1]

        at io.druid.metadata.SQLMetadataConnector.createTable(SQLMetadataConnector.java:176) [druid-server-0.9.1.1.jar:0.9.1.1]

        at io.druid.metadata.SQLMetadataConnector.createConfigTable(SQLMetadataConnector.java:294) [druid-server-0.9.1.1.jar:0.9.1.1]

        at io.druid.metadata.SQLMetadataConnector.createConfigTable(SQLMetadataConnector.java:475) [druid-server-0.9.1.1.jar:0.9.1.1]

        at io.druid.guice.JacksonConfigManagerModule$1.start(JacksonConfigManagerModule.java:58) [druid-common-0.9.1.1.jar:0.9.1.1]

        at com.metamx.common.lifecycle.Lifecycle.start(Lifecycle.java:259) [java-util-0.27.9.jar:?]

        at io.druid.guice.LifecycleModule$2.start(LifecycleModule.java:155) [druid-api-0.9.1.1.jar:0.9.1.1]

        at io.druid.cli.GuiceRunnable.initLifecycle(GuiceRunnable.java:91) [druid-services-0.9.1.1.jar:0.9.1.1]

        at io.druid.cli.ServerRunnable.run(ServerRunnable.java:40) [druid-services-0.9.1.1.jar:0.9.1.1]

        at io.druid.cli.Main.main(Main.java:105) [druid-services-0.9.1.1.jar:0.9.1.1]

Caused by: java.sql.SQLException: Cannot create PoolableConnectionFactory (java.net.ConnectException : Error connecting to server 52.53.243.72 on port 1,527 with message Connection timed out (Connection timed out).)

        at org.apache.commons.dbcp2.BasicDataSource.createPoolableConnectionFactory(BasicDataSource.java:2152) ~[commons-dbcp2-2.0.1.jar:2.0.1]

        at org.apache.commons.dbcp2.BasicDataSource.createDataSource(BasicDataSource.java:1903) ~[commons-dbcp2-2.0.1.jar:2.0.1]

        at org.apache.commons.dbcp2.BasicDataSource.getConnection(BasicDataSource.java:1413) ~[commons-dbcp2-2.0.1.jar:2.0.1]

        at org.skife.jdbi.v2.DataSourceConnectionFactory.openConnection(DataSourceConnectionFactory.java:36) ~[jdbi-2.63.1.jar:2.63.1]

        at org.skife.jdbi.v2.DBI.open(DBI.java:212) ~[jdbi-2.63.1.jar:2.63.1]

        ... 15 more

Caused by: java.sql.SQLNonTransientConnectionException: java.net.ConnectException : Error connecting to server 52.53.243.72 on port 1,527 with message Connection timed out (Connection timed out).

        at org.apache.derby.client.am.SQLExceptionFactory.getSQLException(Unknown Source) ~[derbyclient-10.11.1.1.jar:?]

        at org.apache.derby.client.am.SqlException.getSQLException(Unknown Source) ~[derbyclient-10.11.1.1.jar:?]

        at org.apache.derby.jdbc.ClientDriver.connect(Unknown Source) ~[derbyclient-10.11.1.1.jar:?]

        at org.apache.commons.dbcp2.DriverConnectionFactory.createConnection(DriverConnectionFactory.java:39) ~[commons-dbcp2-2.0.1.jar:2.0.1]

        at org.apache.commons.dbcp2.PoolableConnectionFactory.makeObject(PoolableConnectionFactory.java:205) ~[commons-dbcp2-2.0.1.jar:2.0.1]

        at org.apache.commons.dbcp2.BasicDataSource.validateConnectionFactory(BasicDataSource.java:2162) ~[commons-dbcp2-2.0.1.jar:2.0.1]

        at org.apache.commons.dbcp2.BasicDataSource.createPoolableConnectionFactory(BasicDataSource.java:2148) ~[commons-dbcp2-2.0.1.jar:2.0.1]

        at org.apache.commons.dbcp2.BasicDataSource.createDataSource(BasicDataSource.java:1903) ~[commons-dbcp2-2.0.1.jar:2.0.1]

        at org.apache.commons.dbcp2.BasicDataSource.getConnection(BasicDataSource.java:1413) ~[commons-dbcp2-2.0.1.jar:2.0.1]

        at org.skife.jdbi.v2.DataSourceConnectionFactory.openConnection(DataSourceConnectionFactory.java:36) ~[jdbi-2.63.1.jar:2.63.1]

        at org.skife.jdbi.v2.DBI.open(DBI.java:212) ~[jdbi-2.63.1.jar:2.63.1]

        ... 15 more

Caused by: org.apache.derby.client.am.DisconnectException: java.net.ConnectException : Error connecting to server 52.53.243.72 on port 1,527 with message Connection timed out (Connection timed out).

        at org.apache.derby.client.net.NetAgent.<init>(Unknown Source) ~[derbyclient-10.11.1.1.jar:?]

        at org.apache.derby.client.net.NetConnection.newAgent_(Unknown Source) ~[derbyclient-10.11.1.1.jar:?]

        at org.apache.derby.client.am.ClientConnection.<init>(Unknown Source) ~[derbyclient-10.11.1.1.jar:?]

        at org.apache.derby.client.net.NetConnection.<init>(Unknown Source) ~[derbyclient-10.11.1.1.jar:?]

        at org.apache.derby.client.net.ClientJDBCObjectFactoryImpl.newNetConnection(Unknown Source) ~[derbyclient-10.11.1.1.jar:?]

        at org.apache.derby.jdbc.ClientDriver.connect(Unknown Source) ~[derbyclient-10.11.1.1.jar:?]

        at org.apache.commons.dbcp2.DriverConnectionFactory.createConnection(DriverConnectionFactory.java:39) ~[commons-dbcp2-2.0.1.jar:2.0.1]

        at org.apache.commons.dbcp2.PoolableConnectionFactory.makeObject(PoolableConnectionFactory.java:205) ~[commons-dbcp2-2.0.1.jar:2.0.1]

        at org.apache.commons.dbcp2.BasicDataSource.validateConnectionFactory(BasicDataSource.java:2162) ~[commons-dbcp2-2.0.1.jar:2.0.1]

        at org.apache.commons.dbcp2.BasicDataSource.createPoolableConnectionFactory(BasicDataSource.java:2148) ~[commons-dbcp2-2.0.1.jar:2.0.1]

        at org.apache.commons.dbcp2.BasicDataSource.createDataSource(BasicDataSource.java:1903) ~[commons-dbcp2-2.0.1.jar:2.0.1]

        at org.apache.commons.dbcp2.BasicDataSource.getConnection(BasicDataSource.java:1413) ~[commons-dbcp2-2.0.1.jar:2.0.1]

        at org.skife.jdbi.v2.DataSourceConnectionFactory.openConnection(DataSourceConnectionFactory.java:36) ~[jdbi-2.63.1.jar:2.63.1]

        at org.skife.jdbi.v2.DBI.open(DBI.java:212) ~[jdbi-2.63.1.jar:2.63.1]

        ... 15 more



Derby logs:

[ec2-user@ip-172-31-10-212 druid-0.9.1.1]$ cat var/druid/derby.log 

Thu Dec 01 01:46:45 UTC 2016 : Could not listen on port 1527 on host 52.53.243.72:

 java.net.BindException: Cannot assign requested address (Bind failed)

An exception was thrown during network server startup. DRDA_ListenPort.S:Could not listen on port 1527 on host 52.53.243.72:

 java.net.BindException: Cannot assign requested address (Bind failed)

java.lang.reflect.InvocationTargetException

        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

        at java.lang.reflect.Method.invoke(Method.java:498)

        at org.apache.derby.iapi.jdbc.DRDAServerStarter.run(Unknown Source)

        at java.lang.Thread.run(Thread.java:745)

Caused by: java.lang.Exception: DRDA_ListenPort.S:Could not listen on port 1527 on host 52.53.243.72:

 java.net.BindException: Cannot assign requested address (Bind failed)

        at org.apache.derby.impl.drda.NetworkServerControlImpl.consolePropertyMessageWork(Unknown Source)

        at org.apache.derby.impl.drda.NetworkServerControlImpl.consolePropertyMessage(Unknown Source)

        at org.apache.derby.impl.drda.NetworkServerControlImpl.blockingStart(Unknown Source)

        ... 6 more



Here is the Coordinator runtime config & _common config

Common Config

druid.zk.service.host=52.53.243.72

druid.zk.paths.base=/druid


#

# Metadata storage

#


# For Derby server on your Druid Coordinator (only viable in a cluster with a single Coordinator, no fail-over):

druid.metadata.storage.type=derby

druid.metadata.storage.connector.connectURI=jdbc:derby://52.53.243.72:1527/var/druid/metadata.db;create=true

druid.metadata.storage.connector.host=52.53.243.72

druid.metadata.storage.connector.port=1527


# For MySQL:

#druid.metadata.storage.type=mysql

#druid.metadata.storage.connector.connectURI=jdbc:mysql://54.183.230.200:3306/druid

#druid.metadata.storage.connector.user=druid

#druid.metadata.storage.connector.password=druid


# For PostgreSQL (make sure to additionally include the Postgres extension):

#druid.metadata.storage.type=postgresql

#druid.metadata.storage.connector.connectURI=jdbc:postgresql://db.example.com:5432/druid

#druid.metadata.storage.connector.user=...

#druid.metadata.storage.connector.password=...


#

# Deep storage

#


# For local disk (only viable in a cluster if this is a network mount):

#druid.storage.type=local

#druid.storage.storageDirectory=var/druid/segments


# For HDFS (make sure to include the HDFS extension and that your Hadoop config files in the cp):

#druid.storage.type=hdfs

#druid.storage.storageDirectory=/druid/segments



Coordinator Config:

[ec2-user@ip-172-31-10-212 druid-0.9.1.1]$ cat conf-aws-edit/druid/coordinator/runtime.properties 

druid.service=druid/coordinator

druid.port=8081


druid.coordinator.startDelay=PT30S

druid.coordinator.period=PT30S



Any help on this front will be greatly be appreciated!

Thanks,
Uday.

Udayakumar Pandurangan

unread,
Dec 1, 2016, 1:53:22 PM12/1/16
to Druid Development
Any help on this is greatly appreciated!

Thanks,
Uday.

Udayakumar Pandurangan

unread,
Dec 1, 2016, 7:43:16 PM12/1/16
to Druid Development
Found the answer for this -->

I was using the public IP for the configs from AWS. You need to use the local IPs across the cluster configs.

Thanks,
Uday.



On Wednesday, November 30, 2016 at 6:24:24 PM UTC-8, Udayakumar Pandurangan wrote:
Reply all
Reply to author
Forward
0 new messages