Error connecting to Hive w/ Presto 0.141

1,427 views
Skip to first unread message

Dan Young

unread,
Mar 8, 2016, 7:44:13 AM3/8/16
to Presto

OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=128m; support was removed in 8.0

OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=128m; support was removed in 8.0

Logging initialized using configuration in file:/etc/hive/conf.dist/hive-log4j.properties

hive> show databases;

OK

default

Time taken: 0.734 seconds, Fetched: 1 row(s)

hive> 





Trying to connect Presto 0.141 and I'm getting the following error when I run the presto-cli:


[hadoop@ip-10-199-1-142 ~]$ ./presto-cli --catalog hive --schema default

presto:default> show tables

Query 20160308_124112_00002_gx4jg failed: line 1:1: Schema 'default' does not exist

show tables


presto:default> 




In the server logs:


2016-03-08T12:36:57.690Z DEBUG query-execution-0 com.facebook.presto.execution.QueryStateMachine Query 20160308_123656_00000_gx4jg is PLANNING

2016-03-08T12:37:00.508Z DEBUG query-execution-0 com.facebook.presto.execution.QueryStateMachine Query 20160308_123656_00000_gx4jg is STARTING

2016-03-08T12:37:00.536Z DEBUG query-execution-1 com.facebook.presto.execution.StageStateMachine Stage 20160308_123656_00000_gx4jg.0 is SCHEDULING

2016-03-08T12:37:00.695Z DEBUG query-execution-1 com.facebook.presto.execution.StageStateMachine Stage 20160308_123656_00000_gx4jg.0 is SCHEDULED

2016-03-08T12:37:00.698Z DEBUG query-execution-2 com.facebook.presto.execution.QueryStateMachine Query 20160308_123656_00000_gx4jg is RUNNING

2016-03-08T12:37:01.151Z DEBUG query-execution-0 com.facebook.presto.execution.StageStateMachine Stage 20160308_123656_00000_gx4jg.0 is RUNNING

2016-03-08T12:37:02.686Z WARN UpdateResponseHandler-20160308_123656_00000_gx4jg.0.0-121 com.facebook.presto.server.HttpRemoteTask Error updating task 20160308_123656_00000_gx4jg.0.0: java.util.concurrent.TimeoutException: Idle timeout 2000ms: http://10.199.1.249:8080/v1/task/20160308_123656_00000_gx4jg.0.0

2016-03-08T12:37:04.212Z DEBUG query-execution-0 com.facebook.presto.execution.StageStateMachine Stage 20160308_123656_00000_gx4jg.0 is FINISHED

2016-03-08T12:37:04.214Z DEBUG query-execution-2 com.facebook.presto.execution.QueryStateMachine Query 20160308_123656_00000_gx4jg is FINISHING

2016-03-08T12:37:04.223Z DEBUG query-execution-2 com.facebook.presto.execution.QueryStateMachine Query 20160308_123656_00000_gx4jg is FINISHED

2016-03-08T12:37:04.386Z INFO query-execution-2 com.facebook.presto.event.query.QueryMonitor TIMELINE: Query 20160308_123656_00000_gx4jg :: Transaction:[29e38ccd-ff75-4b7c-b37f-c9a2cc1b6fae] :: elapsed 6605.00ms :: planning 2.89s :: scheduling 3491.00ms :: running 162.00ms :: finishing 62.00ms :: begin 2016-03-08T12:36:57.617Z :: end 2016-03-08T12:37:04.222Z

2016-03-08T12:37:04.454Z DEBUG query-execution-2 com.facebook.presto.execution.QueryStateMachine Query 20160308_123704_00001_gx4jg is PLANNING

2016-03-08T12:37:04.458Z DEBUG query-execution-2 com.facebook.presto.execution.QueryStateMachine Query 20160308_123704_00001_gx4jg is FAILED

2016-03-08T12:37:04.457Z ERROR Query-20160308_123704_00001_gx4jg-116 com.facebook.presto.execution.QueryStateMachine Query 20160308_123704_00001_gx4jg failed

com.facebook.presto.sql.analyzer.SemanticException: line 1:24: Catalog hive does not exist

at com.facebook.presto.sql.analyzer.StatementAnalyzer.visitTable(StatementAnalyzer.java:879)

at com.facebook.presto.sql.analyzer.StatementAnalyzer.visitTable(StatementAnalyzer.java:215)

at com.facebook.presto.sql.tree.Table.accept(Table.java:50)

at com.facebook.presto.sql.tree.AstVisitor.process(AstVisitor.java:22)

at com.facebook.presto.sql.analyzer.StatementAnalyzer.analyzeFrom(StatementAnalyzer.java:1664)

at com.facebook.presto.sql.analyzer.StatementAnalyzer.visitQuerySpecification(StatementAnalyzer.java:994)

at com.facebook.presto.sql.analyzer.StatementAnalyzer.visitQuerySpecification(StatementAnalyzer.java:215)

at com.facebook.presto.sql.tree.QuerySpecification.accept(QuerySpecification.java:125)

at com.facebook.presto.sql.tree.AstVisitor.process(AstVisitor.java:22)

at com.facebook.presto.sql.analyzer.StatementAnalyzer.visitQuery(StatementAnalyzer.java:785)

at com.facebook.presto.sql.analyzer.StatementAnalyzer.visitQuery(StatementAnalyzer.java:215)

at com.facebook.presto.sql.tree.Query.accept(Query.java:103)

at com.facebook.presto.sql.tree.AstVisitor.process(AstVisitor.java:22)

at com.facebook.presto.sql.analyzer.Analyzer.analyze(Analyzer.java:60)

at com.facebook.presto.execution.SqlQueryExecution.doAnalyzeQuery(SqlQueryExecution.java:258)

at com.facebook.presto.execution.SqlQueryExecution.analyzeQuery(SqlQueryExecution.java:244)

at com.facebook.presto.execution.SqlQueryExecution.start(SqlQueryExecution.java:208)

at com.facebook.presto.execution.QueuedExecution.lambda$start$1(QueuedExecution.java:68)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

at java.lang.Thread.run(Thread.java:745)



2016-03-08T12:37:04.481Z INFO query-execution-2 com.facebook.presto.event.query.QueryMonitor TIMELINE: Query 20160308_123704_00001_gx4jg :: Transaction:[233e4e9b-8379-4d0d-8abd-f47b93c1f3c9] :: elapsed 3.00ms :: planning 2.59ms :: scheduling 0.00ms :: running 0.00ms :: finishing 0.00ms :: begin 2016-03-08T12:37:04.451Z :: end 2016-03-08T12:37:04.454Z


hive.properties:


hive.s3.connect-timeout=2m

hive.s3.max-backoff-time=10m

hive.s3.max-error-retries=50

hive.metastore-refresh-interval=2m

hive.s3.max-connections=500

hive.s3.max-client-retries=50

connector.name=hive-hadoop2

hive.s3.socket-timeout=2m

hive.metastore.uri=thrift://10.199.1.142:9083

hive.metastore-cache-ttl=2m

hive.s3.staging-directory=/mnt/tmp/

hive.s3.use-instance-credentials=true

hive.parquet.use-column-names=true

hive.allow-drop-table=true

hive.config.resources=/etc/hadoop/conf/core-site.xml,/etc/hadoop/conf/hdfs-site.xml



I've tried with and without the hive.config.resources in the hive.properties


This is running on Hadoop 2.7.1 Hive 1.0.0.  Any insight?


Regards,


Dano


Dan Young

unread,
Mar 8, 2016, 9:51:38 AM3/8/16
to Presto
I think this has something to do with the emr release label 4.3.0.  I've tried a few older versions that we/are working on emr release 4.2.0 with the same error....I'll keep digging into it and see if I can figure out what's going on.....

Regards,

Dano

--
You received this message because you are subscribed to the Google Groups "Presto" group.
To unsubscribe from this group and stop receiving emails from it, send an email to presto-users...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Kamil Bajda-Pawlikowski

unread,
Mar 8, 2016, 11:52:56 AM3/8/16
to Presto

> com.facebook.presto.sql.analyzer.SemanticException: line 1:24: Catalog hive does not exist

Are you sure your Presto is configured properly? You need to have "hive" connector/catalog configured to be able to query from it.

Dan Young

unread,
Mar 8, 2016, 12:13:16 PM3/8/16
to Presto

It's the same config that we use on emr-4.2.0 with Presto, which works fine, albeit that version is 0.132

Rechecking the configs to see if something with my ansible scripts...

Regards

Dano

Dan Young

unread,
Mar 8, 2016, 1:39:02 PM3/8/16
to presto...@googlegroups.com
This is my hive.properties located in the <software-install-dir>/etc/catalog directory

still having issues...hmm

hive.s3.connect-timeout=2m

hive.s3.max-backoff-time=10m

hive.s3.max-error-retries=50

hive.metastore-refresh-interval=2m

hive.s3.max-connections=500

hive.s3.max-client-retries=50

connector.name=hive-hadoop2

hive.s3.socket-timeout=2m

hive.metastore.uri=thrift://10.199.1.142:9083

hive.metastore-cache-ttl=2m

hive.s3.staging-directory=/mnt/tmp/

hive.s3.use-instance-credentials=true

hive.parquet.use-column-names=true

hive.allow-drop-table=true

hive.config.resources=/etc/hadoop/conf/core-site.xml,/etc/hadoop/conf/hdfs-site.xml



Kamil Bajda-Pawlikowski

unread,
Mar 8, 2016, 2:16:09 PM3/8/16
to Presto
Is your hive.properties file readable by Presto process? 
Are you able to query from any other connectors such as "tpch", "system", "jmx" etc?

Finally, have you restarted Presto after recent configuration changes?

Dan Young

unread,
Mar 8, 2016, 2:25:44 PM3/8/16
to Presto

I can query the system....the user (hadoop) can read the files on the filesystem (read-only though) let me check the JMX...

yes I restart the cluster after each config change via ansible...


[hadoop@ip-10-199-1-122 ~]$ ./presto-cli 

presto> select * from system.runtime.nodes;

  node_id   |         http_uri         | node_version | coordinator | state  

------------+--------------------------+--------------+-------------+--------

 i-11ce3f8a | http://10.199.1.52:8080  | 0.141        | false       | active 

 i-10ce3f8b | http://10.199.1.69:8080  | 0.141        | false       | active 

 i-7c31c0e7 | http://10.199.1.145:8080 | 0.141        | false       | active 

 i-cd33c256 | http://10.199.1.119:8080 | 0.141        | false       | active 

 i-1b33c280 | http://10.199.1.122:8080 | 0.141        | true        | active 

(5 rows)


Query 20160308_191951_00001_e8vha, FINISHED, 2 nodes

Splits: 2 total, 2 done (100.00%)

0:00 [5 rows, 228B] [10 rows/s, 487B/s]


presto> 



Here's the stack trace, etc...


https://www.refheap.com/325985f0e469b6e665374ff39


Dan Young

unread,
Mar 8, 2016, 2:28:33 PM3/8/16
to presto...@googlegroups.com
Can't seem to read JMX catalog either...

presto:default> SELECT node, vmname, vmversion

             -> FROM jmx.jmx."java.lang:type=runtime";

Query 20160308_192713_00008_e8vha failed: line 2:6: Catalog jmx does not exist

com.facebook.presto.sql.analyzer.SemanticException: line 2:6: Catalog jmx does not exist


my jmx.properties:


connector.name=jmx




On Tue, Mar 8, 2016 at 12:16 PM Kamil Bajda-Pawlikowski <kba...@gmail.com> wrote:

Kamil Bajda-Pawlikowski

unread,
Mar 8, 2016, 2:32:13 PM3/8/16
to Presto

Try starting CLI without catalog/schema: ./presto-cli
Then run "SHOW CATALOGS;". Does "hive" come up?
If so, then run "USE hive;" and then "SHOW SCHEMAS;".
What is your output? If it does contain "default" then run "USE default;" and then "SHOW TABLES;".

Dan Young

unread,
Mar 8, 2016, 2:35:33 PM3/8/16
to Presto
I only see the system catalog....hmm some reason my catalogs are not being picked up...let me do some additional snooping.

Dan Young

unread,
Mar 8, 2016, 3:25:01 PM3/8/16
to presto...@googlegroups.com
Any ideas on why these catalog wouldn't be getting registered?  the hive.properties in in the etc/catalog directory.... very puzzling...

Nezih Yigitbasi

unread,
Mar 8, 2016, 3:32:45 PM3/8/16
to presto...@googlegroups.com
Can you share the presto startup logs just in case there is something interesting?

Dan Young

unread,
Mar 8, 2016, 3:40:26 PM3/8/16
to presto...@googlegroups.com
see attached...
server.log.gz
launcher.log.gz

Dan Young

unread,
Mar 8, 2016, 3:42:18 PM3/8/16
to presto...@googlegroups.com
that one ERROR, the No factory for connector jmx .... I tried removing that connector too, deleted the etc/catalog/jmx.properties .... and cluster restart each time...

On Tue, Mar 8, 2016 at 1:32 PM 'Nezih Yigitbasi' via Presto <presto...@googlegroups.com> wrote:

Dan Young

unread,
Mar 8, 2016, 5:10:26 PM3/8/16
to Presto
Ok,

So I tried 0.141 on emr-4.2.0 and am getting a bit further.  I have/had the jmx and hive defined in the etc/catalog..... I see this in the logs now:

2016-03-08T21:58:05.428Z ERROR main com.facebook.presto.server.PrestoServer No factory for connector hive-hadoop2

java.lang.IllegalArgumentException: No factory for connector hive-hadoop2

at com.google.common.base.Preconditions.checkArgument(Preconditions.java:145)

at com.facebook.presto.connector.ConnectorManager.createConnection(ConnectorManager.java:147)

at com.facebook.presto.metadata.CatalogManager.loadCatalog(CatalogManager.java:99)

at com.facebook.presto.metadata.CatalogManager.loadCatalogs(CatalogManager.java:77)

at com.facebook.presto.server.PrestoServer.run(PrestoServer.java:132)

at com.facebook.presto.server.PrestoServer.main(PrestoServer.java:70)


There was/is a similar issue/error when I had the JMX catalog defined.  


2016-03-08T21:53:23.901Z        ERROR   main    com.facebook.presto.server.PrestoServer No factory for connector jmx

java.lang.IllegalArgumentException: No factory for connector jmx

        at com.google.common.base.Preconditions.checkArgument(Preconditions.java:145)

        at com.facebook.presto.connector.ConnectorManager.createConnection(ConnectorManager.java:147)

        at com.facebook.presto.metadata.CatalogManager.loadCatalog(CatalogManager.java:99)

        at com.facebook.presto.metadata.CatalogManager.loadCatalogs(CatalogManager.java:77)

        at com.facebook.presto.server.PrestoServer.run(PrestoServer.java:132)

        at com.facebook.presto.server.PrestoServer.main(PrestoServer.java:70)



I believe I saw on the mailing list about potential issue with OpenJDK??? and to try the Oracle JDK.  Could this be what we're seeing here? Note that we're running Presto 0.132 on this same emr release, emr-4.2.0 and it seems to be working with the OpenJDK.


Also, note, I'm seeing entries like this in the server.log which I was NOT seeing in the emr-4.3.0 release



2016-03-08T21:56:10.167Z        DEBUG   main    com.facebook.presto.server.PluginManager        Classpath for hive-hadoop2:

2016-03-08T21:56:10.167Z        DEBUG   main    com.facebook.presto.server.PluginManager            /mnt/var/log/presto/plugin/hive-hadoop2/annotations-2.0.3.jar

2016-03-08T21:56:10.168Z        DEBUG   main    com.facebook.presto.server.PluginManager            /mnt/var/log/presto/plugin/hive-hadoop2/aopalliance-1.0.jar

2016-03-08T21:56:10.168Z        DEBUG   main    com.facebook.presto.server.PluginManager            /mnt/var/log/presto/plugin/hive-hadoop2/aws-java-sdk-core-1.9.40.jar

2016-03-08T21:56:10.168Z        DEBUG   main    com.facebook.presto.server.PluginManager            /mnt/var/log/presto/plugin/hive-hadoop2/aws-java-sdk-kms-1.9.40.jar

2016-03-08T21:56:10.168Z        DEBUG   main    com.facebook.presto.server.PluginManager            /mnt/var/log/presto/plugin/hive-hadoop2/aws-java-sdk-s3-1.9.40.jar

2016-03-08T21:56:10.168Z        DEBUG   main    com.facebook.presto.server.PluginManager            /mnt/var/log/presto/plugin/hive-hadoop2/bootstrap-0.124.jar

2016-03-08T21:56:10.174Z        DEBUG   main    com.facebook.presto.server.PluginManager            /mnt/var/log/presto/plugin/hive-hadoop2/bval-core-0.5.jar

2016-03-08T21:56:10.174Z        DEBUG   main    com.facebook.presto.server.PluginManager            /mnt/var/log/presto/plugin/hive-hadoop2/bval-jsr303-0.5.jar

2016-03-08T21:56:10.174Z        DEBUG   main    com.facebook.presto.server.PluginManager            /mnt/var/log/presto/plugin/hive-hadoop2/cglib-nodep-2.2.2.jar

2016-03-08T21:56:10.175Z        DEBUG   main    com.facebook.presto.server.PluginManager            /mnt/var/log/presto/plugin/hive-hadoop2/commons-beanutils-core-1.8.3.jar

2016-03-08T21:56:10.175Z        DEBUG   main    com.facebook.presto.server.PluginManager            /mnt/var/log/presto/plugin/hive-hadoop2/commons-codec-1.9.jar

2016-03-08T21:56:10.175Z        DEBUG   main    com.facebook.presto.server.PluginManager            /mnt/var/log/presto/plugin/hive-hadoop2/commons-lang3-3.1.jar

2016-03-08T21:56:10.175Z        DEBUG   main    com.facebook.presto.server.PluginManager            /mnt/var/log/presto/plugin/hive-hadoop2/concurrent-0.124.jar

2016-03-08T21:56:10.175Z        DEBUG   main    com.facebook.presto.server.PluginManager            /mnt/var/log/presto/plugin/hive-hadoop2/configuration-0.124.jar

2016-03-08T21:56:10.175Z        DEBUG   main    com.facebook.presto.server.PluginManager            /mnt/var/log/presto/plugin/hive-hadoop2/fastutil-6.5.9.jar

2016-03-08T21:56:10.175Z        DEBUG   main    com.facebook.presto.server.PluginManager            /mnt/var/log/presto/plugin/hive-hadoop2/guava-18.0.jar

2016-03-08T21:56:10.175Z        DEBUG   main    com.facebook.presto.server.PluginManager            /mnt/var/log/presto/plugin/hive-hadoop2/guice-4.0.jar

2016-03-08T21:56:10.176Z        DEBUG   main    com.facebook.presto.server.PluginManager            /mnt/var/log/presto/plugin/hive-hadoop2/guice-multibindings-4.0.jar

2016-03-08T21:56:10.176Z        DEBUG   main    com.facebook.presto.server.PluginManager            /mnt/var/log/presto/plugin/hive-hadoop2/hadoop-apache2-0.3.jar

2016-03-08T21:56:10.176Z        DEBUG   main    com.facebook.presto.server.PluginManager            /mnt/var/log/presto/plugin/hive-hadoop2/hadoop-lzo-0.4.19.jar

2016-03-08T21:56:10.176Z        DEBUG   main    com.facebook.presto.server.PluginManager            /mnt/var/log/presto/plugin/hive-hadoop2/hive-apache-0.16.jar

......

......

......


Dano

Nezih Yigitbasi

unread,
Mar 8, 2016, 5:23:18 PM3/8/16
to Presto

In the launcher logs you sent in your previous email I see that no classes are loaded other than the ones under /home/hadoop/.versions/presto-server-0.141/lib . This seems like an emr presto deployment issue. What’s under /home/hadoop/.versions/presto-server-0.141? Is there a plugin/ directory there? 

Nezih

​

Dan Young

unread,
Mar 8, 2016, 6:19:55 PM3/8/16
to presto...@googlegroups.com

Yes....

[hadoop@ip-10-199-1-155 plugin]$ pwd

/home/hadoop/.versions/presto-server-0.141/plugin

[hadoop@ip-10-199-1-155 plugin]$ ls 

blackhole  cassandra  example-http  hive-cdh4  hive-cdh5  hive-hadoop1  hive-hadoop2  jmx  kafka  ml  mysql  postgresql  raptor  redis  teradata-functions  tpch

[hadoop@ip-10-199-1-155 plugin]$ 

Nezih Yigitbasi

unread,
Mar 8, 2016, 6:37:47 PM3/8/16
to presto...@googlegroups.com
And are they soft links or regular files? I see that the plugin manager does a File.listFiles() call on the plugin directory and there are some reports online saying that this call may not work with soft links. 

Dan Young

unread,
Mar 8, 2016, 9:41:40 PM3/8/16
to presto...@googlegroups.com
Ok, fast forward.... I updated my ansible scripts to download and install/configure the system Java across the cluster to use the Oracle JDK (jdk-8u74-linux-x64.rpm).  Now on restart I see the catalogs being loaded and now I see them in the CLI. This is on emr-4.2.0 release label. I will try emr-4.3.0 tomorrow, but looking good.

I'm not using any of the AWS/EMR bootstrap and/or the Presto-Sandbox 0.x Applications you can fire up with EMR.  Currently 4.3.0 Sandbox only supports 0.130 anyways....

server.log

......
......

2016-03-08T23:23:37.124Z INFO main com.facebook.presto.metadata.CatalogManager -- Added catalog hive using connector hive-hadoop2 --

2016-03-08T23:23:37.126Z INFO main com.facebook.presto.security.AccessControlManager -- Loading system access control --

2016-03-08T23:23:37.126Z INFO main com.facebook.presto.security.AccessControlManager -- Loaded system access control allow-all --


in the Presto Cli.....

presto> show catalogs;

 Catalog 

---------

 hive    

 system  

 jmx     

(3 rows)


Query 20160308_232816_00000_49a9v, FINISHED, 1 node

Splits: 1 total, 1 done (100.00%)

0:03 [0 rows, 0B] [0 rows/s, 0B/s]


[hadoop@ip-10-199-1-155 ~]$ ./presto-cli --catalog hive --schema imdb

presto:imdb> show tables;

      Table      

-----------------

 aka_name        

 aka_title       

 cast_info       

 char_name       

 company_name    

 complete_cast   

 keyword         

 movie_companies 

 movie_info      

 movie_keyword   

 movie_link      

 name            

 person_info     

 title           

(14 rows)


Query 20160308_233347_00002_49a9v, FINISHED, 2 nodes

Splits: 2 total, 2 done (100.00%)

0:02 [14 rows, 339B] [5 rows/s, 139B/s]


presto:imdb>


Thanx for the help !


Regards,


Dano



On Tue, Mar 8, 2016 at 3:23 PM 'Nezih Yigitbasi' via Presto <presto...@googlegroups.com> wrote:

Dan Young

unread,
Mar 8, 2016, 9:42:24 PM3/8/16
to presto...@googlegroups.com
Soft links....
Reply all
Reply to author
Forward
0 new messages