read local file from edge node and write to HDFS as parquet using scalding

63 views
Skip to first unread message

sri hari kali charan Tummala

unread,
Jun 8, 2017, 6:59:33 PM6/8/17
to cascading-user
Hi There, 

I am looking for a solution to read a file in local file system (.gz file) and write to hdfs as parquet, I dont have Spark installed is there a way to do that in cascading ?

Thanks
Sri 

Chris K Wensel

unread,
Jun 8, 2017, 10:27:14 PM6/8/17
to cascadi...@googlegroups.com
See the user guide for details..

but if you do not run the Cascading Hadoop job on a cluster (but in Hadoop stand-alone mode), it will read a local file and write to a remote HDFS server.

See the Lfs Tap, it will force a Flow to run in Hadoop stand-alone mode.

That is, source from a Lfs Tap, and sink to a Hfs Tap.

ckw

--
You received this message because you are subscribed to the Google Groups "cascading-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cascading-use...@googlegroups.com.
To post to this group, send email to cascadi...@googlegroups.com.
Visit this group at https://groups.google.com/group/cascading-user.
To view this discussion on the web visit https://groups.google.com/d/msgid/cascading-user/e06b8168-9035-456c-8fc4-19bcbe7464d5%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Sri

unread,
Jun 9, 2017, 6:48:40 AM6/9/17
to cascadi...@googlegroups.com
Can I sink it as parquet ? Or avro ?

Thanks
Sri

Sent from my iPhone
You received this message because you are subscribed to a topic in the Google Groups "cascading-user" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/cascading-user/axEWRMidHtY/unsubscribe.
To unsubscribe from this group and all its topics, send an email to cascading-use...@googlegroups.com.

To post to this group, send email to cascadi...@googlegroups.com.
Visit this group at https://groups.google.com/group/cascading-user.

Chris K Wensel

unread,
Jun 9, 2017, 12:04:36 PM6/9/17
to cascadi...@googlegroups.com
should work fine. email the list if otherwise.

ckw

sri hari kali charan Tummala

unread,
Jun 9, 2017, 12:21:00 PM6/9/17
to cascadi...@googlegroups.com
ok will try , thanks do you have any parquet sink example by any chance  ?

Thanks
Sri 


On Fri, Jun 9, 2017 at 12:04 PM, Chris K Wensel <ch...@wensel.net> wrote:
should work fine. email the list if otherwise.

ckw

On Jun 9, 2017, at 3:48 AM, Sri <kali.t...@gmail.com> wrote:

Can I sink it as parquet ? Or avro ?

Thanks
Sri

Sent from my iPhone

On 8 Jun 2017, at 22:26, Chris K Wensel <ch...@wensel.net> wrote:

See the user guide for details..

but if you do not run the Cascading Hadoop job on a cluster (but in Hadoop stand-alone mode), it will read a local file and write to a remote HDFS server.

See the Lfs Tap, it will force a Flow to run in Hadoop stand-alone mode.

That is, source from a Lfs Tap, and sink to a Hfs Tap.

ckw

On Jun 8, 2017, at 3:59 PM, sri hari kali charan Tummala <kali.t...@gmail.com> wrote:

Hi There, 

I am looking for a solution to read a file in local file system (.gz file) and write to hdfs as parquet, I dont have Spark installed is there a way to do that in cascading ?

Thanks
Sri 


-- 
You received this message because you are subscribed to the Google Groups "cascading-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cascading-user+unsubscribe@googlegroups.com.
To post to this group, send email to cascading-user@googlegroups.com.


-- 
You received this message because you are subscribed to a topic in the Google Groups "cascading-user" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/cascading-user/axEWRMidHtY/unsubscribe.
To unsubscribe from this group and all its topics, send an email to cascading-user+unsubscribe@googlegroups.com.
To post to this group, send email to cascading-user@googlegroups.com.

-- 
You received this message because you are subscribed to the Google Groups "cascading-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cascading-user+unsubscribe@googlegroups.com.
To post to this group, send email to cascading-user@googlegroups.com.

--
You received this message because you are subscribed to a topic in the Google Groups "cascading-user" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/cascading-user/axEWRMidHtY/unsubscribe.
To unsubscribe from this group and all its topics, send an email to cascading-user+unsubscribe@googlegroups.com.
To post to this group, send email to cascading-user@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.



--
Thanks & Regards
Sri Tummala

Chris K Wensel

unread,
Jun 9, 2017, 12:25:51 PM6/9/17
to cascadi...@googlegroups.com
this is a great place to start


ckw

To unsubscribe from this group and stop receiving emails from it, send an email to cascading-use...@googlegroups.com.
To post to this group, send email to cascadi...@googlegroups.com.

sri hari kali charan Tummala

unread,
Jun 9, 2017, 12:45:18 PM6/9/17
to cascadi...@googlegroups.com
Thanks Man.

Thanks
Sri 

--
You received this message because you are subscribed to a topic in the Google Groups "cascading-user" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/cascading-user/axEWRMidHtY/unsubscribe.
To unsubscribe from this group and all its topics, send an email to cascading-user+unsubscribe@googlegroups.com.
To post to this group, send email to cascading-user@googlegroups.com.
Visit this group at https://groups.google.com/group/cascading-user.

For more options, visit https://groups.google.com/d/optout.
Reply all
Reply to author
Forward
0 new messages