Alluxio Hive Exceptions

[Boe H]

Hi - Trying to set up Alluxio with Hive on EMR 5.1 (latest version with Hadoop 5.7.3) using S3 as understore..but continue to get cut my knees with stumbling blocks. I've updated the alluxio-core-client jar, core-site.xml, hive-site.xml according to the public guide. I am able to run "hadoop fs" commands on alluxio:// successfully, but still am not able to get hive working with alluxio. Below are some details on the errors I'm seeing. Please let me know if you have any ideas or feedback on getting this to finally work. Can post anymore details needed. Thanks

(Test 1) Fails when executing CREATE TABLE with alluxio path

$ hive
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/lib/hive/lib/log4j-slf4j-impl-2.4.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/lib/hadoop/alluxio-core-client-1.3.0-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]

Logging initialized using configuration in file:/etc/hive/conf.dist/hive-log4j2.properties Async: false

hive> CREATE EXTERNAL TABLE test (c STRING) ROW FORMAT DELIMITED FIELDS TERMINATED BY ',' LOCATION 'alluxio://17.2.1.17:19998/test';
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. MetaException(message:Got exception: java.io.IOException No FileSystem for scheme: alluxio)
 

(Test 2) Fails when providing alluxio configuration values in the hive command with error unknown host exception for AlluxioWorker1.

hive --hiveconf fs.alluxio.impl=alluxio.hadoop.FileSystem --hiveconf fs.alluxio-ft.impl=alluxio.hadoop.FaultTolerantFileSystem --hiveconf fs.AbstractFileSystem.alluxio.impl=alluxio.hadoop.AlluxioFileSystem --hiveconf fs.defaultFS=alluxio://17.2.1.17:19998
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/lib/hive/lib/log4j-slf4j-impl-2.4.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/lib/hadoop/alluxio-core-client-1.3.0-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]

Logging initialized using configuration in file:/etc/hive/conf.dist/hive-log4j2.properties Async: false
Exception in thread "main" java.lang.RuntimeException: java.lang.RuntimeException: java.net.UnknownHostException: AlluxioWorker1: Name or service not known
        at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:604)
        at org.apache.hadoop.hive.ql.session.SessionState.beginStart(SessionState.java:518)
        at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:705)
        at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:641)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
Caused by: java.lang.RuntimeException: java.net.UnknownHostException: AlluxioWorker1: Name or service not known
        at com.google.common.base.Throwables.propagate(Throwables.java:160)
        at alluxio.util.network.NetworkAddressUtils.getRpcPortSocketAddress(NetworkAddressUtils.java:564)
        at alluxio.client.file.FileSystemWorkerClient.<init>(FileSystemWorkerClient.java:92)
        at alluxio.client.file.FileSystemContext.createWorkerClient(FileSystemContext.java:93)
        at alluxio.client.file.FileOutStream.<init>(FileOutStream.java:132)
        at alluxio.client.file.FileOutStream.<init>(FileOutStream.java:98)
        at alluxio.client.file.BaseFileSystem.createFile(BaseFileSystem.java:103)
        at alluxio.hadoop.AbstractFileSystem.create(AbstractFileSystem.java:153)
        at alluxio.hadoop.FileSystem.create(FileSystem.java:25)
        at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:915)
        at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:896)
        at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:793)
        at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:365)
        at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:338)
        at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:2017)
        at org.apache.hadoop.hive.ql.exec.tez.DagUtils.localizeResource(DagUtils.java:1005)
        at org.apache.hadoop.hive.ql.exec.tez.DagUtils.addTempResources(DagUtils.java:900)
        at org.apache.hadoop.hive.ql.exec.tez.DagUtils.localizeTempFilesFromConf(DagUtils.java:843)
        at org.apache.hadoop.hive.ql.exec.tez.TezSessionState.refreshLocalResourcesFromConf(TezSessionState.java:451)
        at org.apache.hadoop.hive.ql.exec.tez.TezSessionState.openInternal(TezSessionState.java:247)
        at org.apache.hadoop.hive.ql.exec.tez.TezSessionState.beginOpen(TezSessionState.java:217)
        at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:601)
        ... 9 more
Caused by: java.net.UnknownHostException: AlluxioWorker1: Name or service not known
        at java.net.Inet4AddressImpl.lookupAllHostAddr(Native Method)
        at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:928)
        at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1323)
        at java.net.InetAddress.getAllByName0(InetAddress.java:1276)
        at java.net.InetAddress.getAllByName(InetAddress.java:1192)
        at java.net.InetAddress.getAllByName(InetAddress.java:1126)
        at java.net.InetAddress.getByName(InetAddress.java:1076)
        at alluxio.util.network.NetworkAddressUtils.resolveHostName(NetworkAddressUtils.java:477)
        at alluxio.util.network.NetworkAddressUtils.getFqdnHost(NetworkAddressUtils.java:501)
        at alluxio.util.network.NetworkAddressUtils.getRpcPortSocketAddress(NetworkAddressUtils.java:560)
        ... 29 more

[andong....@gmail.com]

please encure alluxio can realize the scama Alluxio://,lose a few properities,not diffucult.

在 2016年11月20日星期日 UTC+8下午12:27:07，Boe H写道：

[Boe H]

Not sure I understand what you are trying to suggest? It appears hive is able to load the alluxio-core-client*jar and I'm already including properties in the core-site.xml, hive-site.xml and hive command line.   

[Calvin Jia]

Hi,

Have you taken a look at the documentation for Alluxio and Hive? 

For your specific error, could you ensure that AlluxioWorker1 is a resolvable hostname?

Hope this helps,

Calvin

[Boe H]

Yes I followed everything in that article and this one http://www.alluxio.org/docs/master/en/Running-Alluxio-on-EC2.html

From the external hive server, I can ping the private IP of AlluxioWorker1 and run mapreduce commands alluxio:// I get a list files, but gives the errors in my first post.

Thanks

[Boe H]

Dug this out error out of the hive log file. Let me know if anything stands out.

2016-11-24T09:24:40,248 ERROR [pool-5-thread-199([])]: hive.log (MetaStoreUtils.java:logAndThrowMetaException(1313)) - Got exception: java.io.IOException No FileSystem for scheme: alluxio

java.io.IOException: No FileSystem for scheme: alluxio

        at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2708) ~[hadoop-common-2.7.3-amzn-0.jar:?]
        at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2715) ~[hadoop-common-2.7.3-amzn-0.jar:?]
        at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:93) ~[hadoop-common-2.7.3-amzn-0.jar:?]
        at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2751) ~[hadoop-common-2.7.3-amzn-0.jar:?]
        at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2733) ~[hadoop-common-2.7.3-amzn-0.jar:?]
        at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:377) ~[hadoop-common-2.7.3-amzn-0.jar:?]
        at org.apache.hadoop.fs.Path.getFileSystem(Path.java:295) ~[hadoop-common-2.7.3-amzn-0.jar:?]
        at org.apache.hadoop.hive.metastore.Warehouse.getFs(Warehouse.java:104) [hive-exec-2.1.0-amzn-0.jar:2.1.0-amzn-0]
        at org.apache.hadoop.hive.metastore.Warehouse.getDnsPath(Warehouse.java:140) [hive-exec-2.1.0-amzn-0.jar:2.1.0-amzn-0]
        at org.apache.hadoop.hive.metastore.Warehouse.getDnsPath(Warehouse.java:146) [hive-exec-2.1.0-amzn-0.jar:2.1.0-amzn-0]
        at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_core(HiveMetaStore.java:1375) [hive-exec-2.1.0-amzn-0.jar:2.1.0-amzn-0]
        at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_with_environment_context(HiveMetaStore.java:1438) [hive-exec-2.1.0-amzn-0.jar:2.1.0-amzn-0]
        at sun.reflect.GeneratedMethodAccessor21.invoke(Unknown Source) ~[?:?]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_111]
        at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_111]
        at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:140) [hive-exec-2.1.0-amzn-0.jar:2.1.0-amzn-0]
        at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:99) [hive-exec-2.1.0-amzn-0.jar:2.1.0-amzn-0]
        at com.sun.proxy.$Proxy19.create_table_with_environment_context(Unknown Source) [?:?]
        at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$create_table_with_environment_context.getResult(ThriftHiveMetastore.java:10868) [hive-exec-2.1.0-amzn-0.jar:2.1.0-amzn-0]
        at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$create_table_with_environment_context.getResult(ThriftHiveMetastore.java:10852) [hive-exec-2.1.0-amzn-0.jar:2.1.0-amzn-0]
        at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39) [hive-exec-2.1.0-amzn-0.jar:2.1.0-amzn-0]
        at org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:110) [hive-exec-2.1.0-amzn-0.jar:2.1.0-amzn-0]
        at org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:106) [hive-exec-2.1.0-amzn-0.jar:2.1.0-amzn-0]
        at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_111]
        at javax.security.auth.Subject.doAs(Subject.java:422) [?:1.8.0_111]
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698) [hadoop-common-2.7.3-amzn-0.jar:?]
        at org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBasedProcessor.java:118) [hive-exec-2.1.0-amzn-0.jar:2.1.0-amzn-0]
        at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286) [hive-exec-2.1.0-amzn-0.jar:2.1.0-amzn-0]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_111]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_111]
        at java.lang.Thread.run(Thread.java:745) [?:1.8.0_111]

[Calvin Jia]

Hi,

This error from Hive means that the Alluxio client jar was not provided in the hive classpath. Could you ensure that the hive process which logged this error has the Alluxio-core-client jar in its Java classpath?

Thanks,

Calvin

[Boe H]

Confused ... as far I can tell the class path is found and there is successful communication between hive and alluxio on initialization. However when I try to run "create external table * location alluxio://*" command I still get exception "java.io.IOException No FileSystem for scheme: alluxio".

$ hive --hiveconf fs.alluxio.impl=alluxio.hadoop.FileSystem --hiveconf fs.alluxio-ft.impl=alluxio.hadoop.FaultTolerantFileSystem --hiveconf fs.AbstractFileSystem.alluxio.impl=alluxio.hadoop.AlluxioFileSystem --hiveconf fs.defaultFS=alluxio://1.2.1.10:19998

SLF4J: Class path contains multiple SLF4J bindings.

SLF4J: Found binding in [jar:file:/usr/lib/hive/lib/alluxio-core-client-1.3.0-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class]

SLF4J: Found binding in [jar:file:/usr/lib/hive/lib/log4j-slf4j-impl-2.4.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/lib/hadoop/alluxio-core-client-1.3.0-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.

SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]

Logging initialized using configuration in file:/etc/hive/conf.dist/hive-log4j2.properties Async: false

2016-11-25 10:53:46,070 INFO  [main] SessionState:

Logging initialized using configuration in file:/etc/hive/conf.dist/hive-log4j2.properties Async: false

2016-11-25 10:53:46,600 INFO  [main] hive.metastore: Trying to connect to metastore with URI thrift://ip-10-137-174-224.ec2.internal:9083
2016-11-25 10:53:46,632 INFO  [main] hive.metastore: Opened a connection to metastore, current connections: 1
2016-11-25 10:53:46,655 INFO  [main] hive.metastore: Connected to metastore.
2016-11-25 10:53:47,130 INFO  [main] logger.type: initialize(alluxio://1.2.1.10:19998, Configuration: core-default.xml, core-site.xml, mapred-default.xml, mapred-site.xml, yarn-default.xml, yarn-site.xml, hdfs-default.xml, hdfs-site.xml, org.apache.hadoop.hive.conf.LoopingByteArrayInputStream@1b266842, file:/etc/hive/conf.dist/hive-site.xml). Connecting to Alluxio: alluxio://1.2.1.10:19998
2016-11-25 10:53:47,136 INFO  [main] logger.type: alluxio://1.2.1.10:19998 alluxio://1.2.1.10:19998
2016-11-25 10:53:47,139 INFO  [main] logger.type: Loading Alluxio properties from Hadoop configuration: {alluxio.user.file.writetype.default=CACHE_THROUGH}
2016-11-25 10:53:47,221 INFO  [main] logger.type: Starting sinks with config: {}.
2016-11-25 10:53:47,221 INFO  [main] logger.type: Sinks have already been started.
2016-11-25 10:53:47,235 INFO  [main] logger.type: getFileStatus(/tmp/hive)
2016-11-25 10:53:47,263 INFO  [main] logger.type: Alluxio client (version 1.3.0) is trying to connect with FileSystemMasterClient master @ /1.2.1.10:19998
2016-11-25 10:53:47,279 INFO  [main] logger.type: Client registered with FileSystemMasterClient master @ /1.2.1.10:19998
2016-11-25 10:53:47,308 INFO  [main] logger.type: getFileStatus(/tmp/hive)
2016-11-25 10:53:47,311 INFO  [main] logger.type: getFileStatus(/tmp/hive/hadoop)
2016-11-25 10:53:47,313 INFO  [main] logger.type: getFileStatus(/tmp/hive/hadoop/545b900c-88a4-4951-b406-20e945541cf0)
2016-11-25 10:53:47,400 INFO  [main] logger.type: mkdirs(/tmp/hive/hadoop/545b900c-88a4-4951-b406-20e945541cf0, rwx------)
2016-11-25 10:53:47,520 INFO  [main] session.SessionState: Created HDFS directory: /tmp/hive/hadoop/545b900c-88a4-4951-b406-20e945541cf0
2016-11-25 10:53:47,520 INFO  [main] logger.type: getFileStatus(/tmp/hive/hadoop/545b900c-88a4-4951-b406-20e945541cf0)
2016-11-25 10:53:47,564 INFO  [main] session.SessionState: Created local directory: /mnt/tmp/hadoop/545b900c-88a4-4951-b406-20e945541cf0
2016-11-25 10:53:47,565 INFO  [main] logger.type: getFileStatus(/tmp/hive/hadoop/545b900c-88a4-4951-b406-20e945541cf0/_tmp_space.db)
2016-11-25 10:53:47,926 INFO  [main] logger.type: mkdirs(/tmp/hive/hadoop/545b900c-88a4-4951-b406-20e945541cf0/_tmp_space.db, rwx------)
2016-11-25 10:53:48,046 INFO  [main] session.SessionState: Created HDFS directory: /tmp/hive/hadoop/545b900c-88a4-4951-b406-20e945541cf0/_tmp_space.db
2016-11-25 10:53:48,047 INFO  [main] logger.type: getFileStatus(/tmp/hive/hadoop/545b900c-88a4-4951-b406-20e945541cf0/_tmp_space.db)
2016-11-25 10:53:48,052 INFO  [main] conf.HiveConf: Using the default value passed in for log id: 545b900c-88a4-4951-b406-20e945541cf0
2016-11-25 10:53:48,052 INFO  [main] session.SessionState: Updating thread name to 545b900c-88a4-4951-b406-20e945541cf0 main


hive> CREATE EXTERNAL TABLE test3 (c STRING)  LOCATION 'alluxio://1.2.1.10:19998/tmp';
2016-11-25 10:54:20,433 INFO  [545b900c-88a4-4951-b406-20e945541cf0 main] conf.HiveConf: Using the default value passed in for log id: 545b900c-88a4-4951-b406-20e945541cf0
2016-11-25 10:54:20,487 INFO  [545b900c-88a4-4951-b406-20e945541cf0 main] ql.Driver: Compiling command(queryId=hadoop_20161125105420_a9da9c3f-977d-40e7-b5cd-8f12d2831087): CREATE EXTERNAL TABLE test3 (c STRING)  LOCATION 'alluxio://1.2.1.10:19998/tmp'
2016-11-25 10:54:21,552 INFO  [545b900c-88a4-4951-b406-20e945541cf0 main] parse.CalcitePlanner: Starting Semantic Analysis
2016-11-25 10:54:21,582 INFO  [545b900c-88a4-4951-b406-20e945541cf0 main] parse.CalcitePlanner: Creating table default.test3 position=22
2016-11-25 10:54:21,598 INFO  [545b900c-88a4-4951-b406-20e945541cf0 main] logger.type: getWorkingDirectory: /
2016-11-25 10:54:21,651 INFO  [545b900c-88a4-4951-b406-20e945541cf0 main] sqlstd.SQLStdHiveAccessController: Created SQLStdHiveAccessController for session context : HiveAuthzSessionContext [sessionString=545b900c-88a4-4951-b406-20e945541cf0, clientType=HIVECLI]
2016-11-25 10:54:21,654 WARN  [545b900c-88a4-4951-b406-20e945541cf0 main] session.SessionState: METASTORE_FILTER_HOOK will be ignored, since hive.security.authorization.manager is set to instance of HiveAuthorizerFactory.
2016-11-25 10:54:21,656 INFO  [545b900c-88a4-4951-b406-20e945541cf0 main] hive.metastore: Mestastore configuration hive.metastore.filter.hook changed from org.apache.hadoop.hive.metastore.DefaultMetaStoreFilterHookImpl to org.apache.hadoop.hive.ql.security.authorization.plugin.AuthorizationMetaStoreFilterHook
2016-11-25 10:54:21,662 INFO  [545b900c-88a4-4951-b406-20e945541cf0 main] hive.metastore: Closed a connection to metastore, current connections: 0
2016-11-25 10:54:21,665 INFO  [545b900c-88a4-4951-b406-20e945541cf0 main] hive.metastore: Trying to connect to metastore with URI thrift://ip-10-137-174-224.ec2.internal:9083
2016-11-25 10:54:21,668 INFO  [545b900c-88a4-4951-b406-20e945541cf0 main] hive.metastore: Opened a connection to metastore, current connections: 1
2016-11-25 10:54:21,669 INFO  [545b900c-88a4-4951-b406-20e945541cf0 main] hive.metastore: Connected to metastore.
2016-11-25 10:54:21,696 INFO  [545b900c-88a4-4951-b406-20e945541cf0 main] ql.Driver: Semantic Analysis Completed
2016-11-25 10:54:21,709 INFO  [545b900c-88a4-4951-b406-20e945541cf0 main] ql.Driver: Returning Hive schema: Schema(fieldSchemas:null, properties:null)
2016-11-25 10:54:21,734 INFO  [545b900c-88a4-4951-b406-20e945541cf0 main] ql.Driver: Completed compiling command(queryId=hadoop_20161125105420_a9da9c3f-977d-40e7-b5cd-8f12d2831087); Time taken: 1.279 seconds
2016-11-25 10:54:21,734 INFO  [545b900c-88a4-4951-b406-20e945541cf0 main] ql.Driver: Concurrency mode is disabled, not creating a lock manager
2016-11-25 10:54:21,734 INFO  [545b900c-88a4-4951-b406-20e945541cf0 main] ql.Driver: Executing command(queryId=hadoop_20161125105420_a9da9c3f-977d-40e7-b5cd-8f12d2831087): CREATE EXTERNAL TABLE test3 (c STRING)  LOCATION 'alluxio://1.2.1.10:19998/tmp'
2016-11-25 10:54:21,750 INFO  [545b900c-88a4-4951-b406-20e945541cf0 main] ql.Driver: Starting task [Stage-0:DDL] in serial mode
2016-11-25 10:54:21,780 INFO  [545b900c-88a4-4951-b406-20e945541cf0 main] exec.DDLTask: creating table default.test3 on alluxio://1.2.1.10:19998/tmp
2016-11-25 10:54:22,192 ERROR [545b900c-88a4-4951-b406-20e945541cf0 main] exec.DDLTask: org.apache.hadoop.hive.ql.metadata.HiveException: MetaException(message:Got exception: java.io.IOException No FileSystem for scheme: alluxio)
        at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:842)
        at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:847)
        at org.apache.hadoop.hive.ql.exec.DDLTask.createTable(DDLTask.java:3979)

[Calvin Jia]

Hi,

It seems like your Hive driver has the correct configuration and jars, but when it launches a task, the task does not have the correct config or jars setup. Could you verify if create external table launches a separate process and that process has the correct configuration/jars?

Cheers,

Calvin

[Boe H]

Still stuck. Not sure what that separate process would be? This is out of the box Hive on EMR v5.1 so I assume this would be more common than not.  Has anyone gotten this to work with Hive yet?  If so, what version of EMR?

[Bin Fan]

Hi Boe,

for errors like "java.io.IOException No FileSystem for scheme: alluxio", it means alluxio:// scheme is not recognized by the application

(see more explanation in http://www.alluxio.org/docs/master/en/Debugging-Guide.html#q-why-do-i-see-exceptions-like-no-filesystem-for-scheme-alluxio)

When you create the table, MR (which is another system by itself) is used and that's probably why you see this error after communication between hive and alluxio on initialization

and not necessary pick up what configuration you passed to Hive through -hiveconf

Could you try to follow http://www.alluxio.org/docs/master/en/Running-Hive-with-Alluxio.html for the "Configure Hadoop MapReduce" part. and see if it helps?

Basically, remember Hive and Hadoop are two different projects and they don't necessarily share the configuration.

- Bin

[guoj...@gmail.com]

Hi, you need to add the alluxio-core-client jar to your hive-metastore node and then restart the hive-metastore process, for all DDL actions need througth the hive-metastore, metastore use the alluxio client to mkdirs. 

在 2016年11月25日星期五 UTC+8下午7:11:45，Boe H写道：

[Bin Fan]

Hi Boe,

does mine or Guojhltt's suggestion help solve your issue?

- Bin

[ShaoFeng Shi]

I encountered the same error, solved by restarting Hive server (after adding the alluxio jar and properties).

在 2016年11月20日星期日 UTC+8下午12:27:07，Boe H写道：

Hi - Trying to set up Alluxio with Hive on EMR 5.1 (latest version with Hadoop 5.7.3) using S3 as understore..but continue to get cut my knees with stumbling blocks. I've updated the alluxio-core-client jar, core-site.xml, hive-site.xml according to the public guide. I am able to run "hadoop fs" commands on alluxio:// successfully, but still am not able to get hive working with alluxio. Below are some details on the errors I'm seeing. Please let me know if you have any ideas or feedback on getting this to finally work. Can post anymore details needed. Thanks

[Bin Fan]

thanks Shaofeng for providing the solution

- Bin