Custom Data Provider For avro file in Lingual

[Amiya Mishra]

Hi,

I want to create custom data provider for avro files in lingual catalog.

For that i have used AvroScheme() to create custom scheme. Also added required properties in provider.properties file. Which i have attached as provider.properties file.

Then i have executed following commands on lingual as:

lingual catalog --provider avroprovider --remove

lingual catalog --provider -add /home/hduser/avroprovider.jar 

lingual catalog --schema zyxf --add

lingual catalog --schema zyxf --stereotype zyxff --add --columns id,name,city -type int,string,string

lingual catalog --schema zyxf --table zyxf --stereotype zyxff -add hdfs://UbuntuD1:8020/home/hduser/part-00000.avro --protocol hdfs --format avro

lingual shell --verbose --showstacktrace

select * from "zyxf"."zyxf";

Above 6 commands works fine for me.i.e provider,schema,stereotype and table has been successfully added to lingual catalog.

There after when i want to view the data it won't works and shows the below errors as:

2016-04-19 13:07:51,069 INFO  [main] jdbc.LingualStatement (LingualStatement.java:execute(168)) - execute: select * from "zyxf"."zyxf"

{utcTimestamp=1461051462447, currentTimestamp=1461071262447, localTimestamp=1461071262447, timeZone=sun.util.calendar.ZoneInfo[id="Asia/Kolkata",offset=19800000,dstSavings=0,useDaylight=false,transitions=6,lastRule=null]}

2016-04-19 13:07:52,879 INFO  [main] provider.ProviderProxy (ProviderProxy.java:createScheme(189)) - using null to create scheme for stereotype zyxff with properties: {extensions=.avro, protocols=hdfs, fieldScale=2, fieldPrecision=10, provider=avroprovider}

2016-04-19 13:07:53,397 INFO  [main] provider.ProviderProxy (ProviderProxy.java:createTap(158)) - using null to create tap for Resource{identifier='hdfs://UbuntuD1:8020/home/hduser/part-00000.avro', protocol=hdfs, format=avro, mode=KEEP} with properties: {schemes=hdfs, provider=text}

2016-04-19 13:07:53,474 ERROR [main] jdbc.LingualStatement (LingualStatement.java:handleThrowable(219)) - failed with: exception while executing query ( null) 

Warning: exception while executing query: null: org/apache/avro/mapred/AvroInputFormat: org.apache.avro.mapred.AvroInputFormat (state=,code=0)

Error: exception while executing query (state=,code=0)

Any One can help, Its important to read only avro files ??

Thanks

Amiya Mishra

[Andre Kelpe]

Can you share the code of your Scheme somwhere? It is most likely a
problem in the life-cycle methods of the the scheme (sourceConfInit,
presentSourceFields etc. etc.).

- André

> --
> You received this message because you are subscribed to the Google Groups
> "Lingual User" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to lingual-user...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.



--
André Kelpe
an...@concurrentinc.com
http://concurrentinc.com

[Amiya Mishra]

Hi Andre 

I have solve this issue by providing Fat jar required for the provider.

I have also attached the source code, you can go through it.

Here the problem is while using Fat jar ,the dependencies it uses is quite large .Do you have any alternative to Fat jar ?? Or we have to minimize these dependencies 

Thanks

Amiya Mishra

[Andre Kelpe]

To limit the jar size make sure you only include what you really need.
Cascading + its dependecies and Hadoop + all its dependencies should
be set to provided scope in your build, so that they don't end up in
the fat jar. It should only contain avro + your code.

- André

[Amiya Mishra]

Thanks Andre for your valuable comments

Amiya Mishra