Acid support and different version of HMS issue
star...@gmail.com
Its a great project.
But I have 2 different issues.
1. ACID support.
when I set the acid functionality on the primary meta-store with hive.txn.manager=org.apache.hadoop.hive.ql.lockmgr.DbTxnManager in hive-site.xml, the lock related error is happen.
2018-07-18 22:39:37.638 ERROR 31770 --- [pool-4-thread-1] o.a.t.ProcessFunction : Internal error processing lock
2018-07-18 22:33:06.154 ERROR 19139 --- [pool-4-thread-2] o.a.h.h.m.RetryingHMSHandler : MetaException(message:Waggle Dance: You cannot perform this operation on the virtual database 'waggle_prod_test'.)
at com.hotels.bdp.waggledance.server.ExceptionWrappingHMSHandler.invoke(ExceptionWrappingHMSHandler.java:56)
at com.sun.proxy.$Proxy137.lock(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:148)
at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:107)
at com.sun.proxy.$Proxy137.lock(Unknown Source)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$lock.getResult(ThriftHiveMetastore.java:14155)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$lock.getResult(ThriftHiveMetastore.java:14139)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
at org.apache.hadoop.hive.metastore.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:48)
at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2. Error due to the different version of metastore.
When I used this project with hive 2.1.1 on the federated meta store and 2.3.2 on the primary meta store, this error is happen while select some data.
2018-07-18 23:25:06.049 ERROR 8421 --- [pool-4-thread-3] o.a.h.h.m.RetryingHMSHandler : org.apache.thrift.TApplicationException: Invalid method name: 'get_table_req'
at org.apache.thrift.TApplicationException.read(TApplicationException.java:111)
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:79)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_table_req(ThriftHiveMetastore.java:1563)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_table_req(ThriftHiveMetastore.java:1550)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.hotels.bdp.waggledance.client.DefaultMetaStoreClientFactory$ReconnectingMetastoreClientInvocationHandler.invoke(DefaultMetaStoreClientFactory.java:67)
at com.sun.proxy.$Proxy134.get_table_req(Unknown Source)
at com.hotels.bdp.waggledance.server.FederatedHMSHandler.get_table_req_aroundBody618(FederatedHMSHandler.java:1481)
at com.hotels.bdp.waggledance.server.FederatedHMSHandler$AjcClosure619.run(FederatedHMSHandler.java:1)
at org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:149)
at com.hotels.bdp.waggledance.metrics.MonitoredAspect.monitor(MonitoredAspect.java:57)
at com.hotels.bdp.waggledance.metrics.MonitoredAspect.monitor(MonitoredAspect.java:47)
at com.hotels.bdp.waggledance.server.FederatedHMSHandler.get_table_req_aroundBody620(FederatedHMSHandler.java:1479)
at com.hotels.bdp.waggledance.server.FederatedHMSHandler$AjcClosure621.run(FederatedHMSHandler.java:1)
at org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:149)
at com.jcabi.aspects.aj.MethodLogger.wrap(MethodLogger.java:213)
at com.jcabi.aspects.aj.MethodLogger.ajc$inlineAccessMethod$com_jcabi_aspects_aj_MethodLogger$com_jcabi_aspects_aj_MethodLogger$wrap(MethodLogger.java:1)
at com.jcabi.aspects.aj.MethodLogger.wrapMethod(MethodLogger.java:169)
at com.hotels.bdp.waggledance.server.FederatedHMSHandler.get_table_req(FederatedHMSHandler.java:1479)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.hotels.bdp.waggledance.server.ExceptionWrappingHMSHandler.invoke(ExceptionWrappingHMSHandler.java:50)
at com.sun.proxy.$Proxy137.get_table_req(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:148)
at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:107)
at com.sun.proxy.$Proxy137.get_table_req(Unknown Source)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$get_table_req.getResult(ThriftHiveMetastore.java:11457)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$get_table_req.getResult(ThriftHiveMetastore.java:11441)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
at org.apache.hadoop.hive.metastore.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:48)
at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Is there any plan to solve those issues?
Adrian Woodhead
Hey there,
Thanks for reaching out and all the detail you provided on the two issues you've seen. What version of WaggleDance are you using? How are you using it? (i.e. from the Hive CLI or something else? and what version of this?)
For 1. would you be able to send us a small example test that reproduces the issue? i.e. more or less what your settings look like and what query you are trying to run that causes the issue, what execution environment you are using (e.g. Hive CLI? Beeline? something else?)
For 2. we have seen a similar issue when using certain versions of WaggleDance and different metastore versions. It can be quite tricky to get API compatibility across different versions but we do our best to try handle this. It would be good to know what version of WaggleDance you are using (and if the Hive CLI what version) and again what query you are running, and which metastore (the 2.1.1 or 2.3.2) is throwing the exception (you should be able to check by running the query against just one of them?)
Thanks,
Adrian
Adrian Woodhead
Principal Engineer
Hotels.com - An Expedia Brand
tel: +44 (0) 207 019 2648
Expedia.com - 407 St John Street, London, EC1V 4EX
Sent: 19 July 2018 00:36
To: Waggle Dance User
Subject: [waggle-dance-user] Acid support and different version of HMS issue
You received this message because you are subscribed to the Google Groups "Waggle Dance User" group.
To unsubscribe from this group and stop receiving emails from it, send an email to waggle-dance-u...@googlegroups.com.
To post to this group, send an email to waggle-d...@googlegroups.com.
Visit this group at https://groups.google.com/group/waggle-dance-user.
To view this discussion on the web, visit https://groups.google.com/d/msgid/waggle-dance-user/ba5f28b1-88c3-4099-aef0-4cfd8438fc87%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
star...@gmail.com
- waggle-dance version : 2.3.7 (also tested on 2.3.6)
- waggle-dance configuration
. database-resolution: PREFIXED
. primary-meta-store :
hive & metastore version : 2.1.0
thrift://localhost:9083
. remote-meta-store
hive & metastore version : 2.3.0
* acid enabled with "hive.txn.manager=org.apache.hadoop.hive.ql.lockmgr.DbTxnManager" and "hive.support.concurrency=true"
thrift://remote-ip:9083
1. I just executed use / show tables after executing hive-cli with 'hive --hiveconf hive.metastore.uris=thrift://localhost:48869'
- use testdb; (any db is okay. the error is happened on the db which is not only in the primary metastore but also the remote metastore)
- show tables; -> it makes the Internal error processing lock which was described before.
-> I think this is due to the missing acid configuration of remote hive.
2. The environment information written above.
I just execute the select query on the remote table. (I found this issue after turning off the acid configuration of primary meta store due to the 1st issue)
When I tested the remote meta store connection with through the hive cli, it showed the same error.
(hive --hiveconf hive.metastore.uris=thrift://remote ip:9083)
Thanks for checking the issue.
2018년 7월 20일 금요일 오후 6시 36분 18초 UTC+9, awoodhead 님의 말:
>
>
>
>
>
> Thanks for reaching out and all the detail you provided on the two issues you've seen. What version of WaggleDance are you using? How are you using it? (i.e. from the Hive CLI or something else? and what version of this?)
>
>
>
>
>
> For 1. would you be able to send us a small example test that reproduces the issue? i.e. more or less what your settings look like and what query you are trying to run that causes the issue, what execution environment
> you are using (e.g. Hive CLI? Beeline? something else?)
>
>
>
>
>
> For 2. we have seen a similar issue when using certain versions of WaggleDance and different metastore versions. It can be quite tricky to get API compatibility across different versions but we do our best to try handle
> this. It would be good to know what version of WaggleDance you are using (and if the Hive CLI what version) and again what query you are running, and which metastore (the 2.1.1 or 2.3.2) is throwing the exception (you should be able to check by running the
> query against just one of them?)
>
>
>
>
>
> Thanks,
>
>
>
>
>
> Adrian
>
>
>
>
>
>
>
>
>
>
>
>
>
> --
>
> Adrian Woodhead
>
> Principal Engineer
>
> Hotels.com - An Expedia Brand
>
> tel: +44 (0) 207 019 2648
>
> Expedia.com - 407 St John Street, London, EC1V 4EX
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
Adrian Woodhead
OK, so for 1) it sounds like you're going to try check something in the remote metastore as it might not be Waggle Dance related?
For 2) could you please create an issue in GitHub with the info provided below and the stack trace and we'll try reproduce and when we get a chance hopefully come up with a fix (we also welcome contributions if anyone else would like to have a go!)
Thanks,
Adrian
To: Waggle Dance User
Subject: Re: [waggle-dance-user] Acid support and different version of HMS issue
star...@gmail.com
2) Okay. I will create the issue for it.
Elliot West
- We can likely not permit queries that use ACID tables in different metastores as they have isolated txn id spaces. We'd probably need some sort of distributed txn manager. This could perhaps be a responsibility of Waggle Dance.
- Even if the 'remote' tables are not ACID, we might struggle as the local lock manager might still want to take out locks on tables that it doesn't know anything about. Would need to check how the lock manager treats non-ACID tables.
1) No. It should be supported between transactional(acid) metastore and non-transactional metastore. I will not change any thing in the remote hive configuration.
2) Okay. I will create the issue for it.
--
You received this message because you are subscribed to the Google Groups "Waggle Dance User" group.
To unsubscribe from this group and stop receiving emails from it, send an email to waggle-dance-u...@googlegroups.com.
To post to this group, send an email to waggle-d...@googlegroups.com.
Visit this group at https://groups.google.com/group/waggle-dance-user.
To view this discussion on the web, visit https://groups.google.com/d/msgid/waggle-dance-user/c2ef5048-6458-4226-842b-3a040edc876a%40googlegroups.com.
Mass Dosage
Hello,Regarding ACID support: This is not something we've prioritized and there are a number of considerations:
- We can likely not permit queries that use ACID tables in different metastores as they have isolated txn id spaces. We'd probably need some sort of distributed txn manager. This could perhaps be a responsibility of Waggle Dance.
- Even if the 'remote' tables are not ACID, we might struggle as the local lock manager might still want to take out locks on tables that it doesn't know anything about. Would need to check how the lock manager treats non-ACID tables.
As we are not current users of ACID it's unlikely that we'll look at this soon. However, we'd be very receptive to any ideas or input that you have which could move this forward.Elliot.
On Thu, 26 Jul 2018 at 00:11, <star...@gmail.com> wrote:
1) No. It should be supported between transactional(acid) metastore and non-transactional metastore. I will not change any thing in the remote hive configuration.
2) Okay. I will create the issue for it.
--
You received this message because you are subscribed to the Google Groups "Waggle Dance User" group.
To unsubscribe from this group and stop receiving emails from it, send an email to waggle-dance-user+unsubscribe@googlegroups.com.
To post to this group, send an email to waggle-dance-user@googlegroups.com.
Visit this group at https://groups.google.com/group/waggle-dance-user.
To view this discussion on the web, visit https://groups.google.com/d/msgid/waggle-dance-user/c2ef5048-6458-4226-842b-3a040edc876a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "Waggle Dance User" group.
To post to this group, send email to waggle-dance-user@googlegroups.com.To unsubscribe from this group and stop receiving emails from it, send an email to waggle-dance-user+unsubscribe@googlegroups.com.
Visit this group at https://groups.google.com/group/waggle-dance-user.
To view this discussion on the web, visit https://groups.google.com/d/msgid/waggle-dance-user/CAC3gpCZAaqT66buPCxD%2Ba0P-5ZBceoMAFwKoNbCo18z6--EMfw%40mail.gmail.com.