Setting up GeoTrellis in EC2 and ingesting data

Tony Garcia

unread,

Jan 14, 2016, 6:52:18 PM1/14/16

to geotrellis-user

Hello,

I am completely new to GeoTrellis and have been following the readme instructions on how to get things set up. I am pretty sure that I have everything configured but I am having issue ingesting data.

So far what I have done is used the geotrellis-ec2-cluster vagrant file to build ec2 instances. Next I took the gt-admin project and started to follow it's instructions from that project (https://github.com/geotrellis/gt-admin) by first downloading the gt-admin and geotrellis projects to my leader node (probably shouldn't do this on the leader node). On that node that i successfully ran ./sb assembly and ./publish-local.sh from the geotrellis project. I got the web service set up by running npm install and grunt serve but when I went on to the next step and tried to run ./sbt assembly on the gt-admin project it fails. I am not sure why this is not working though it seems like it cannot find specific classes and I don't know where these should come from, any ideas?

Output of running ./sbt assembly on the gt-admin project:

ubuntu:~/gt-admin$ ls

ingest ingest-spatial.sh ingest-temporal.sh project README.md run-server.sh sbt server startup.py target viewer

ubuntu@ip-172-31-40-117:~/gt-admin$ ./sbt assembly

Detected sbt version 0.13.5

Using /home/ubuntu/.sbt/0.13.5 as sbt dir, -sbt-dir to override.

[info] Loading project definition from /home/ubuntu/gt-admin/project

[info] Set current project to gt-admin (in build file:/home/ubuntu/gt-admin/)

[info] Compiling 14 Scala sources to /home/ubuntu/gt-admin/ingest/target/scala-2.10/classes...

[error] /home/ubuntu/gt-admin/ingest/src/main/scala/geotrellis/admin/ingest/AccumuloIngestCommand.scala:35: type mismatch;

[error] found : Int(256)

[error] required: geotrellis.proj4.CRS

[error] val layoutScheme = ZoomedLayoutScheme(256)

[error] ^

[error] /home/ubuntu/gt-admin/ingest/src/main/scala/geotrellis/admin/ingest/AccumuloIngestCommand.scala:37: not found: value AccumuloRasterCatalog

[error] val catalog = AccumuloRasterCatalog()

[error] ^

[error] /home/ubuntu/gt-admin/ingest/src/main/scala/geotrellis/admin/ingest/Ingest.scala:57: wrong number of type arguments for geotrellis.spark.ingest.Tiler, should be 3

[error] (implicit tiler: Tiler[T, K]): Unit =

[error] ^

[error] /home/ubuntu/gt-admin/ingest/src/main/scala/geotrellis/admin/ingest/AccumuloPyramidCommand.scala:33: type mismatch;

[error] found : Int(256)

[error] required: geotrellis.proj4.CRS

[error] val layoutScheme = ZoomedLayoutScheme(256)

[error] ^

[error] /home/ubuntu/gt-admin/ingest/src/main/scala/geotrellis/admin/ingest/AccumuloPyramidCommand.scala:35: not found: value AccumuloRasterCatalog

[error] val catalog = AccumuloRasterCatalog()

[error] ^

[error] /home/ubuntu/gt-admin/ingest/src/main/scala/geotrellis/admin/ingest/Calculate.scala:28: not found: value HadoopRasterCatalog

[error] val catalog = HadoopRasterCatalog(new Path("hdfs://localhost/catalog"))

[error] ^

[error] /home/ubuntu/gt-admin/ingest/src/main/scala/geotrellis/admin/ingest/HadoopIngestCommand.scala:29: not found: value HadoopRasterCatalog

[error] val catalog = HadoopRasterCatalog(args.catalogPath)

[error] ^

[error] /home/ubuntu/gt-admin/ingest/src/main/scala/geotrellis/admin/ingest/HadoopIngestCommand.scala:31: not enough arguments for method apply: (crs: geotrellis.proj4.CRS, tileSize: Int, resolutionThreshold: Double)geotrellis.spark.tiling.ZoomedLayoutScheme in object ZoomedLayoutScheme.

[error] Unspecified value parameter crs.

[error] val layoutScheme = ZoomedLayoutScheme()

[error] ^

[error] /home/ubuntu/gt-admin/ingest/src/main/scala/geotrellis/admin/ingest/Ingest.scala:64: overloaded method value up with alternatives:

[error] [K](rdd: geotrellis.spark.MultiBandRasterRDD[K], layoutScheme: geotrellis.spark.tiling.LayoutScheme, zoom: Int)(implicit evidence$7: geotrellis.spark.SpatialComponent[K], implicit evidence$8: scala.reflect.ClassTag[K], implicit d: DummyImplicit)(Int, geotrellis.spark.MultiBandRasterRDD[K]) <and>

[error] [K](rdd: geotrellis.spark.RasterRDD[K], layoutScheme: geotrellis.spark.tiling.LayoutScheme, zoom: Int)(implicit evidence$5: geotrellis.spark.SpatialComponent[K], implicit evidence$6: scala.reflect.ClassTag[K])(Int, geotrellis.spark.RasterRDD[K]) <and>

[error] [K, TileType](rdd: org.apache.spark.rdd.RDD[(K, TileType)], sourceLayout: geotrellis.spark.tiling.LayoutDefinition, targetLayout: geotrellis.spark.tiling.LayoutDefinition)(implicit evidence$1: geotrellis.spark.SpatialComponent[K], implicit evidence$2: scala.reflect.ClassTag[K], implicit evidence$3: TileType => geotrellis.raster.mosaic.MergeMethods[TileType], implicit evidence$4: TileType => geotrellis.spark.ingest.CellGridPrototype[TileType])org.apache.spark.rdd.RDD[(K, TileType)]

[error] cannot be applied to (geotrellis.spark.RasterRDD[K], geotrellis.spark.tiling.LayoutLevel, geotrellis.spark.tiling.LayoutScheme)

[error] val (nextLevel, nextRdd) = Pyramid.up(rdd, level, layoutScheme)

[error] ^

[error] /home/ubuntu/gt-admin/ingest/src/main/scala/geotrellis/admin/ingest/Ingest.scala:66: type mismatch;

[error] found : nextRdd.type (with underlying type Any)

[error] required: geotrellis.spark.RasterRDD[K]

[error] (which expands to) org.apache.spark.rdd.RDD[(K, geotrellis.raster.Tile)] with geotrellis.spark.Metadata[geotrellis.spark.RasterMetaData]

[error] sinkLevels(nextRdd, nextLevel){ rdd.unpersist(blocking = false) }

[error] ^

[error] /home/ubuntu/gt-admin/ingest/src/main/scala/geotrellis/admin/ingest/Ingest.scala:73: overloaded method value fromRdd with alternatives:

[error] [K, V <: geotrellis.raster.CellGrid](rdd: org.apache.spark.rdd.RDD[(K, V)], scheme: geotrellis.spark.tiling.LayoutScheme)(implicit evidence$1: geotrellis.spark.ingest.IngestKey[K])(Int, geotrellis.spark.RasterMetaData) <and>

[error] [K, V <: geotrellis.raster.CellGrid](rdd: org.apache.spark.rdd.RDD[(K, V)], crs: geotrellis.proj4.CRS, scheme: geotrellis.spark.tiling.LayoutScheme)(getExtent: K => geotrellis.vector.Extent)(Int, geotrellis.spark.RasterMetaData) <and>

[error] [K, V <: geotrellis.raster.CellGrid](rdd: org.apache.spark.rdd.RDD[(K, V)], crs: geotrellis.proj4.CRS, layout: geotrellis.spark.tiling.LayoutDefinition)(getExtent: K => geotrellis.vector.Extent)geotrellis.spark.RasterMetaData

[error] cannot be applied to (org.apache.spark.rdd.RDD[(T, geotrellis.raster.Tile)], geotrellis.proj4.CRS, geotrellis.spark.tiling.LayoutScheme, Boolean)

[error] RasterMetaData.fromRdd(reprojectedTiles, destCRS, layoutScheme, isUniform) { key: T =>

[error] ^

[error] /home/ubuntu/gt-admin/ingest/src/main/scala/geotrellis/admin/ingest/Ingest.scala:78: type mismatch;

[error] found : Any

[error] required: geotrellis.spark.tiling.LayoutLevel

[error] sinkLevels(rasterRdd, layoutLevel){ reprojectedTiles.unpersist(blocking = false) }

[error] ^

[error] /home/ubuntu/gt-admin/ingest/src/main/scala/geotrellis/admin/ingest/NEXIngest.scala:28: wrong number of type arguments for geotrellis.spark.ingest.Tiler, should be 3

[error] implicit val tiler: Tiler[SpaceTimeInputKey, SpaceTimeKey] = {

[error] ^

[error] /home/ubuntu/gt-admin/ingest/src/main/scala/geotrellis/admin/ingest/NEXIngest.scala:33: missing arguments for method apply in object Tiler;

[error] follow this method with `_' if you want to treat it as a partially applied function

[error] Tiler(getExtent, createKey)

[error] ^

[error] /home/ubuntu/gt-admin/ingest/src/main/scala/geotrellis/admin/ingest/NEXIngest.scala:37: not enough arguments for method apply: (crs: geotrellis.proj4.CRS, tileSize: Int, resolutionThreshold: Double)geotrellis.spark.tiling.ZoomedLayoutScheme in object ZoomedLayoutScheme.

[error] Unspecified value parameter crs.

[error] val layoutScheme = ZoomedLayoutScheme()

[error] ^

[error] /home/ubuntu/gt-admin/ingest/src/main/scala/geotrellis/admin/ingest/NEXIngest.scala:40: not found: value AccumuloRasterCatalog

[error] val writer = AccumuloRasterCatalog().writer[SpaceTimeKey](ZCurveKeyIndexMethod.byYear, args.table)

[error] ^

[error] /home/ubuntu/gt-admin/ingest/src/main/scala/geotrellis/admin/ingest/NetCDFIngestCommand.scala:28: wrong number of type arguments for geotrellis.spark.ingest.Tiler, should be 3

[error] implicit val tiler: Tiler[NetCdfBand, SpaceTimeKey] = {

[error] ^

[error] /home/ubuntu/gt-admin/ingest/src/main/scala/geotrellis/admin/ingest/NetCDFIngestCommand.scala:33: missing arguments for method apply in object Tiler;

[error] follow this method with `_' if you want to treat it as a partially applied function

[error] Tiler(getExtent, createKey)

[error] ^

[error] /home/ubuntu/gt-admin/ingest/src/main/scala/geotrellis/admin/ingest/NetCDFIngestCommand.scala:39: not enough arguments for method apply: (crs: geotrellis.proj4.CRS, tileSize: Int, resolutionThreshold: Double)geotrellis.spark.tiling.ZoomedLayoutScheme in object ZoomedLayoutScheme.

[error] Unspecified value parameter crs.

[error] val layoutScheme = ZoomedLayoutScheme()

[error] ^

[error] /home/ubuntu/gt-admin/ingest/src/main/scala/geotrellis/admin/ingest/NetCDFIngestCommand.scala:41: not found: value AccumuloRasterCatalog

[error] val writer = AccumuloRasterCatalog().writer[SpaceTimeKey](ZCurveKeyIndexMethod.byYear, args.table)

[error] ^

[error] /home/ubuntu/gt-admin/ingest/src/main/scala/geotrellis/admin/ingest/NetCDFIngestHDFSCommand.scala:24: wrong number of type arguments for geotrellis.spark.ingest.Tiler, should be 3

[error] implicit val tiler: Tiler[NetCdfBand, SpaceTimeKey] = {

[error] ^

[error] /home/ubuntu/gt-admin/ingest/src/main/scala/geotrellis/admin/ingest/NetCDFIngestHDFSCommand.scala:29: missing arguments for method apply in object Tiler;

[error] follow this method with `_' if you want to treat it as a partially applied function

[error] Tiler(getExtent, createKey)

[error] ^

[error] /home/ubuntu/gt-admin/ingest/src/main/scala/geotrellis/admin/ingest/NetCDFIngestHDFSCommand.scala:32: not found: value HadoopRasterCatalog

[error] val catalog = HadoopRasterCatalog(args.catalogPath)

[error] ^

[error] /home/ubuntu/gt-admin/ingest/src/main/scala/geotrellis/admin/ingest/NetCDFIngestHDFSCommand.scala:34: not enough arguments for method apply: (crs: geotrellis.proj4.CRS, tileSize: Int, resolutionThreshold: Double)geotrellis.spark.tiling.ZoomedLayoutScheme in object ZoomedLayoutScheme.

[error] Unspecified value parameter crs.

[error] val layoutScheme = ZoomedLayoutScheme()

[error] ^

[error] /home/ubuntu/gt-admin/ingest/src/main/scala/geotrellis/admin/ingest/package.scala:4: object Reproject is not a member of package geotrellis.spark.ingest

[error] import geotrellis.spark.ingest.{Reproject, Tiler}

[error] ^

[error] /home/ubuntu/gt-admin/ingest/src/main/scala/geotrellis/admin/ingest/package.scala:29: polymorphic expression cannot be instantiated to expected type;

[error] found : [TileType](rdd: org.apache.spark.rdd.RDD[(geotrellis.vector.ProjectedExtent, TileType)], metaData: geotrellis.spark.RasterMetaData, resampleMethod: geotrellis.raster.resample.ResampleMethod)(implicit evidence$16: geotrellis.spark.SpatialComponent[geotrellis.spark.SpatialKey], implicit evidence$17: scala.reflect.ClassTag[geotrellis.spark.SpatialKey], implicit evidence$18: TileType => geotrellis.raster.mosaic.MergeMethods[TileType], implicit evidence$19: TileType => geotrellis.spark.ingest.CellGridPrototype[TileType], implicit evidence$20: scala.reflect.ClassTag[TileType])org.apache.spark.rdd.RDD[(geotrellis.spark.SpatialKey, TileType)]

[error] required: (org.apache.spark.rdd.RDD[(geotrellis.vector.ProjectedExtent, geotrellis.raster.Tile)], geotrellis.spark.RasterMetaData) => geotrellis.spark.RasterRDD[geotrellis.spark.SpatialKey]

[error] (which expands to) (org.apache.spark.rdd.RDD[(geotrellis.vector.ProjectedExtent, geotrellis.raster.Tile)], geotrellis.spark.RasterMetaData) => org.apache.spark.rdd.RDD[(geotrellis.spark.SpatialKey, geotrellis.raster.Tile)] with geotrellis.spark.Metadata[geotrellis.spark.RasterMetaData]

[error] Tiler(getExtent, createKey)

[error] ^

[error] /home/ubuntu/gt-admin/ingest/src/main/scala/geotrellis/admin/ingest/package.scala:33: not found: value Reproject

[error] def reproject(destCRS: CRS): RDD[(T, Tile)] = Reproject(rdd, destCRS)

[error] ^

[error] 27 errors found

[info] Compiling 6 Scala sources to /home/ubuntu/gt-admin/server/target/scala-2.10/classes...

[error] /home/ubuntu/gt-admin/server/src/main/scala/geotrellis/admin/server/CatalogService.scala:46: not found: value AccumuloRasterCatalog

[error] val catalog = AccumuloRasterCatalog()

[error] ^

[error] /home/ubuntu/gt-admin/server/src/main/scala/geotrellis/admin/server/CatalogService.scala:48: not found: type AccumuloLayerMetaData

[error] val metaDatas = attributeStore.readAll[AccumuloLayerMetaData]("metadata")

[error] ^

[error] /home/ubuntu/gt-admin/server/src/main/scala/geotrellis/admin/server/CatalogService.scala:53: not enough arguments for method apply: (crs: geotrellis.proj4.CRS, tileSize: Int, resolutionThreshold: Double)geotrellis.spark.tiling.ZoomedLayoutScheme in object ZoomedLayoutScheme.

[error] Unspecified value parameter crs.

[error] val layoutScheme = ZoomedLayoutScheme()

[error] ^

[error] /home/ubuntu/gt-admin/server/src/main/scala/geotrellis/admin/server/CatalogService.scala:86: value rasterMetaData is not a member of Any

[error] val rmd = meta.rasterMetaData

[error] ^

[error] /home/ubuntu/gt-admin/server/src/main/scala/geotrellis/admin/server/CatalogService.scala:176: value rasterMetaData is not a member of Nothing

[error] val md = layerMetaData.rasterMetaData

[error] ^

[error] /home/ubuntu/gt-admin/server/src/main/scala/geotrellis/admin/server/CatalogService.scala:217: not found: type RasterRDDQuery

[error] val filters = new RasterRDDQuery[SpaceTimeKey].where(Intersects(GridBounds(key.col, key.row, key.col, key.row)))

[error] ^

[error] /home/ubuntu/gt-admin/server/src/main/scala/geotrellis/admin/server/PredicateCount.scala:14: not found: value asRasterRDD

[error] asRasterRDD(rdd.metaData.copy(cellType = cellType)) {

[error] ^

[error] 7 errors found

[error] (server/compile:compile) Compilation failed

[error] (ingest/compile:compile) Compilation failed

[error] Total time: 48 s, completed Jan 14, 2016 11:49:48 PM

ubuntu:~/gt-admin$

Rob Emanuele

unread,

Jan 14, 2016, 7:05:25 PM1/14/16

to geotrel...@googlegroups.com

Hey Tony,

The gt-admin project is unfortunately a bit out of date. We plan on updating it before the release, but it's not the current tool we use for ingesting data.

You can check out https://github.com/geotrellis/geotrellis/tree/master/spark-etl for ingesting. Also it might be useful to look at https://github.com/geotrellis/geotrellis-landsat-tutorial/blob/master/src/main/scala/tutorial/TileGeoTiff.scala for how to ingest data into slippy map tiles.

We're trying to nail down the API before release and it's undergoing a lot of tweaks, so it might be worth it to pin your code down to a bintray version and stick with it for a while, like this: https://github.com/geotrellis/geotrellis-landsat-tutorial/blob/master/build.sbt#L23 and https://github.com/geotrellis/geotrellis-landsat-tutorial/blob/master/build.sbt#L33 (see https://bintray.com/azavea/geotrellis/geotrellis-spark/view for snapshot versions).

Thanks,

Rob

--
You received this message because you are subscribed to the Google Groups "geotrellis-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to geotrellis-us...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Robert Emanuele, Tech Lead
Azavea | 340 N 12th St, Ste 402, Philadelphia, PA
remanuele@azavea.com | T 215.701.7502 | Web azavea.com | @azavea

-----

We'll be moving on March 1, 2016. Please update your records with our new address:
990 Spring Garden Street, 5th Floor
Philadelphia, PA 19123
----

Tony Garcia

unread,

Jan 14, 2016, 7:09:18 PM1/14/16

to geotrellis-user

Rob,

Thanks for the quick reply. I'll give that a shot.

Tony

Reply all

Reply to author

Forward