[scalding] reading parquet gz in local mode

[Miguel Ping]

Hi,

I'm trying to read a parquet file in local mode that was written with pig, also in local mode. It seems to me that scalding doesn't support parquet local mode yet.

I'm trying to hook it up but unfortunately I don't know enough about the api to do it right. This is what I got, took it from:

case class KVParquetTextLine(override val fields: Fields = Fields.ALL, path: String) extends FixedPathParquetTuple(fields) {

  override def localScheme = new ParquetTupleScheme(fields).asInstanceOf[Scheme[_, _, _, _, _]]

}

This doesn't compile right now; can you guys give me some pointers on how to use scalding parquet support in local mode? 

I haven't tried the hdfs mode yet.

Thanks!

[Miguel Ping]

Anyone?

[Andre Kelpe]

Hi,

you have to be careful with local-mode. Local mode in cascading is not
the same as hadoop in standalone mode. Cascading does not have any of
the Hadoop deps on the classpath, when you use local mode. If you want
to use hadoop in standalone mode, you cannot use cascadings local
mode, you have to use cascading-hadoop.

Makes sense?

- André

> --
> You received this message because you are subscribed to the Google Groups
> "cascading-user" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to cascading-use...@googlegroups.com.
> To post to this group, send email to cascadi...@googlegroups.com.
> Visit this group at http://groups.google.com/group/cascading-user.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/cascading-user/7c1c6846-8811-4b92-9f17-127b7501a259%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>


--
André Kelpe
an...@concurrentinc.com
http://concurrentinc.com