platform broker not found for: hadoop

[austin solomon]

Hi,

I have setup an Hadoop 2.7.1 environment in my server with java version "1.7.0_91"
and I have installed the Lingual by using the command

curl http://files.cascading.org/lingual/1.2/lingual-client/install-lingual-client.sh | bash

and did all the first steps given in the Lingual User Guide (http://docs.cascading.org/lingual/1.2/)
everything is working (both local & hadoop) fine in lingual shell, I can query the files locally as well as hadoop.

However when i tried to run the jdbc-example.java in hadoop platform 

                 Class.forName( "cascading.lingual.jdbc.Driver" );
                  Connection connection = DriverManager.getConnection(
                  "jdbc:lingual:hadoop;schemas=user/root/employees");
                  Statement statement = connection.createStatement();

                  ResultSet resultSet =
                    statement.executeQuery("SELECT * FROM EMPLOYEES.TITLES" );

I am getting this ERROR

2016-01-02 17:20:55,304 WARN  [main] util.Version (Version.java:getVersionProperties(156)) - unable to load version information
2016-01-02 17:20:56,336 INFO  [main] jdbc.LingualConnection (LingualConnection.java:initialize(107)) - using platform: hadoop
2016-01-02 17:20:56,337 INFO  [main] jdbc.LingualConnection (LingualConnection.java:initialize(118)) - using default schema
2016-01-02 17:20:56,455 INFO  [main] platform.PlatformBrokerFactory (PlatformBrokerFactory.java:addPlatform(135)) - installing platform: local
Exception in thread "main" java.lang.IllegalArgumentException: platform broker not found for: hadoop
	at cascading.lingual.platform.PlatformBrokerFactory.getPlatformBroker(PlatformBrokerFactory.java:82)
	at cascading.lingual.platform.PlatformBrokerFactory.createPlatformBroker(PlatformBrokerFactory.java:65)
	at cascading.lingual.jdbc.LingualConnection.initialize(LingualConnection.java:121)
	at cascading.lingual.jdbc.LingualConnection.<init>(LingualConnection.java:82)
	at SC.<init>(Unknown Source)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
	at cascading.lingual.jdbc.JaninoFactory.create(JaninoFactory.java:156)
	at cascading.lingual.jdbc.JaninoFactory.createConnection(JaninoFactory.java:48)
	at cascading.lingual.jdbc.Driver.connect(Driver.java:172)
	at java.sql.DriverManager.getConnection(DriverManager.java:664)
	at java.sql.DriverManager.getConnection(DriverManager.java:270)
	at cascading.lingual.examples.foodmart.JdbcExample.run(JdbcExample.java:68)
	at cascading.lingual.examples.foodmart.JdbcExample.main(JdbcExample.java:35)


I have set the hadoop related properties in my classpath as given below

export HADOOP_HOME=/home/tg1/hadoop-2.7.1
export HADOOP_INSTALL=$HADOOP_HOME
export HADOOP_MAPRED_HOME=$HADOOP_HOME
export HADOOP_COMMON_HOME=$HADOOP_HOME
export HADOOP_HDFS_HOME=$HADOOP_HOME
export HADOOP_YARN_HOME=$HADOOP_HOME
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
export HADOOP_CONF_DIR=/home/tg1/hadoop-2.7.1/etc/hadoop/
export PATH=$PATH:$HADOOP_HOME/sbin:$HADOOP_HOME/bin

Please help me to figure out whats wrong in this?


Thanks

[Andre Kelpe]

Hadoop 2.7.1 is yarn based and therefore you have to use the
hadoop2-mr1 platform for lingual. If you change the JDBC URL to

"jdbc:lingual:hadoop2-mr1;schemas=user/root/employees"


it should work as expected.

- André

> --
> You received this message because you are subscribed to the Google Groups
> "Lingual User" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to lingual-user...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.



--
André Kelpe
an...@concurrentinc.com
http://concurrentinc.com

[austin solomon]

Hi André,

I changed the driver to hadoop2-mr1 as you said, I'm still getting the same error.

I have put some sys.out to some of the properties and the error is like the following below

2016-01-04 17:38:30,735 WARN  [main] util.Version (Version.java:getVersionProperties(156)) - unable to load version information
2016-01-04 17:38:31,121 INFO  [main] jdbc.LingualConnection (LingualConnection.java:initialize(107)) - using platform: hadoop2-mr1
2016-01-04 17:38:31,122 INFO  [main] jdbc.LingualConnection (LingualConnection.java:initialize(118)) - using default schema
======PLATFORM_CLASSNAME======cascading.lingual.platform.local.LocalPlatformBroker
2016-01-04 17:38:31,140 INFO  [main] platform.PlatformBrokerFactory (PlatformBrokerFactory.java:addPlatform(138)) - installing platform: local
======PLATFORM_NAME======local
======BROKER_NAME======cascading.lingual.platform.local.LocalPlatformBroker@7276c8cd
======PLATFORM======hadoop2-mr1
Exception in thread "main" java.lang.IllegalArgumentException: platform broker not found for: hadoop2-mr1
    at cascading.lingual.platform.PlatformBrokerFactory.getPlatformBroker(PlatformBrokerFactory.java:84)

    at cascading.lingual.platform.PlatformBrokerFactory.createPlatformBroker(PlatformBrokerFactory.java:65)
    at cascading.lingual.jdbc.LingualConnection.initialize(LingualConnection.java:121)
    at cascading.lingual.jdbc.LingualConnection.<init>(LingualConnection.java:82)
    at SC.<init>(Unknown Source)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
    at cascading.lingual.jdbc.JaninoFactory.create(JaninoFactory.java:156)
    at cascading.lingual.jdbc.JaninoFactory.createConnection(JaninoFactory.java:48)
    at cascading.lingual.jdbc.Driver.connect(Driver.java:172)
    at java.sql.DriverManager.getConnection(DriverManager.java:664)
    at java.sql.DriverManager.getConnection(DriverManager.java:270)
    at cascading.lingual.examples.foodmart.JdbcExample.run(JdbcExample.java:68)
    at cascading.lingual.examples.foodmart.JdbcExample.main(JdbcExample.java:35)

 

  It is taking Platform_name and broker class as "local".
  Have i missed any properties?

[Andre Kelpe]

You are using the wrong jdbc driver jar: We publish one jdbc driver
per platform (not like hive, where it is all in one). This is the
correct jar you need:
http://conjars.org/repo/cascading/lingual-hadoop2-mr1/1.2.1/lingual-hadoop2-mr1-1.2.1-jdbc.jar

That is a fatjar with all of the things you need, except the hadoop
dependencies, which you have to have yourself.

- André


On Mon, Jan 4, 2016 at 1:13 PM, austin solomon