The second step is to upload the above uber JAR to the machine that runs Hive. Next, make the JAR accessible to Hive. There are several ways to do this, listed below. To verify that the JAR was properly added, run LIST JARS;
in the Hive CLI.
ADD JAR <path-to-jar>;
Disclaimer: Use it at your own risk. Any and all responsibility for any loss, damage or destruction of data or any other property which may arise from relying on this email's technical content is explicitly disclaimed. The author will in no case be liable for any monetary damages arising from such loss, damage or destruction.
I wrote an article on ORC file and storage index in Hive in Linkedin back in early 2016 and I recall there were issues reading from writing to ORC tables in Spark a while back. Sounds like the Spark handling of ORC tables is resolved.
I attach the sample PySpark code that generates random data, stores it in delta table, reads it back from HDFS directory where delta data resides and finally writes DF to ORC table in Hive
HTH
--
You received this message because you are subscribed to the Google Groups "Delta Lake Users and Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to delta-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/delta-users/683e9390-cdbc-491f-90db-517fe06c0920n%40googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/delta-users/c35c3065-11ff-4d71-a745-9234bee04cdfn%40googlegroups.com.
reason:
The orc format is a columnar storage table. You cannot import data directly from a local file. Only when the data source table is also stored in the orc format can it be loaded directly. Otherwise, the above error will be reported.
terms of settlement:
Either change the data source table to a table stored in the orc format, or create a new temporary table in the textfile format, first load the source file data into the table, and then insert the data into the orc target table from the textfile table
To view this discussion on the web visit https://groups.google.com/d/msgid/delta-users/1b8519f1-4b80-46dd-9060-7b5de14ca120n%40googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/delta-users/3c7dabd1-0549-4be2-9fd3-4ed872433af8n%40googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/delta-users/589b43f8-f75a-4965-bab2-efeb37b86352n%40googlegroups.com.
Best Regards,
--
You received this message because you are subscribed to the Google Groups "Delta Lake Users and Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to delta-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/delta-users/ca534b79-5fc4-4b37-b914-67376c163c8bn%40googlegroups.com.