Ingest file in hdfs

[Gaye Daouda]

Hi, I am not very good at english but I will try to be clear

I work in a spark-geotrellis project and I need to ingest file in hdfs. I try different solutions found here and it still not working. I use MultibandIngest of Etl package but I got this error: Unable to find input module of type 'hadoop' for format `geotiff. I don't know what to do, it takes all my time. Please help me. Thx

[Rob Emanuele]

Hi Gaye,

It seems like the assembly you are running is missing code. Can you describe how you are running the ELT? What commands are you using? What does your build.sbt look like?

On Thu, Apr 28, 2016 at 6:35 AM, Gaye Daouda <daoudae...@gmail.com> wrote:

Hi, I am not very good at english but I will try to be clear

I work in a spark-geotrellis project and I need to ingest file in hdfs. I try different solutions found here and it still not working. I use MultibandIngest of Etl package but I got this error: Unable to find input module of type 'hadoop' for format `geotiff. I don't know what to do, it takes all my time. Please help me. Thx

--
You received this message because you are subscribed to the Google Groups "geotrellis-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to geotrellis-us...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Wilder Nina Choquehuayta]

Hi, i have the same problem "Exception in thread "main" java.lang.RuntimeException: Unable to find input module of type 'hadoop' for format `geotiff". I generate the jar with ./sbt -> project spark-etl -> compile -> assembly.


the command is:

./spark-submit --class geotrellis.spark.etl.MultibandIngest --master spark://192.168.1.102:7077 --driver-memory 2G /opt/geotrellis-github/new-geotrellis/geotrellis/spark-etl/target/scala-2.10/geotrellis-spark-etl-assembly-0.10.0-SNAPSHOT.jar --input hadoop  --format geotiff --cache NONE -I path="file:///opt/spark-1.6.0-bin-hadoop2.6/datalandsat" --output hadoop -O path="hdfs://master:9000/datalandsat/" --layer nlcd-tms --crs EPSG:3857 --pyramid --layoutScheme tms

Thanks

[Wilder Nina Choquehuayta]

Well i resolved the problem changing the class MultibandIngest to SinglebandIngest.

Thanks !!!

[Rob Emanuele]

Hi Gaye, Wilder,

Multiband ingest requires the use of "multiband-geotiff" as the format, instead of "geotiff". Let me know if this helps.

Thanks,

Rob

--

Robert Emanuele, Tech Lead
Azavea |  990 Spring Garden Street, 5th Floor, Philadelphia, PA
remanuele@azavea.com  | T 215.701.7502  | Web azavea.com  |  @azavea

[Gaye Daouda]

Hi Rob, thank you very much for the response. I tried multiband-geotiff as format and it worked correctly (not yet the input file but I am working in it).

I have an other problème, I see that Etl.ingest require tiles, and I need to ingest a geotiff file in hdfs. I found functions that create RDD and then save it in hdfs but I need to ingest a geotiff file without transform it (e.g as geotiff format in hdfs). Is that possible? If yes, can you give me the readers and writers functions. Thx 

[Wilder Nina Choquehuayta]

Hi, it worked correctly !!!

Thanks a lot !!!

[Gaye Daouda]

Hi, Can someone answer to my message please?

Le jeudi 28 avril 2016 10:35:42 UTC, Gaye Daouda a écrit :

[Rob Emanuele]

Gaye,

It seems like what you're asking to do is save a binary file onto HDFS. geotrellis.raster.io.geotiff.GeoTiff has a "toByteArray" method, so once you have your array of bytes, it's simply writing binary files to HDFS. You can look for resources for reading and writing binary data to HDFS, there are plenty out there.

Hope this helps,

Rob

--

You received this message because you are subscribed to the Google Groups "geotrellis-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to geotrellis-us...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Gaye Daouda]

Hi Rob, Think you very much. Every is fine, I am reading and writing.

Le jeudi 28 avril 2016 10:35:42 UTC, Gaye Daouda a écrit :