S3 Ingestion

[kwpa...@gmail.com]

We are doing AWS S3 ingestion direcly using AWS EMR.
And we are also using S3 for deep storage
However, we can't find any information about things related to S3

Does metatron support AWS S3 Ingestion?

[kyungtaak Noh]

We are not yet supporting s3.

But with redshift(https://github.com/metatron-app/metatron-discovery/issues/787), we are ready to provide Amazon's infrastructure services.

We are considering support within the first quarter of 2019.

[jungryong]

I'm testing S3 ingestion now. Currently, metatron discovery does not support S3 ingestion, but you can use scripts to perform ingestion and register it with metatron discovery.

[Dongwoo Ha]

Wow... Good news!!

How do I use aws S3 with metatron discovery?

Could you tell me the tutorial about that?

[Seoeun Park]

We can use static-s3-firehose to ingest S3 objects.

1. Make sure to include druid-s3-extensions.

2. Set the AWS credentials. 

Sample configs (common.runtime.properties):

druid.s3.accessKey=XXXXXX
druid.s3.secretKey=XXXXXX

3. Submit a ingestion task spec.

Sample spec:

"ioConfig" : {
  ......
  "firehose" : {
    "type" : "static-s3",
    "uris" : ["s3://mt-test01/test-data/wikiticker-2015-09-12-sampled.json"]
  }  
  ......
}

Thanks.

[minhyu...@gmail.com]

1. After S3 ingestion, S3 ingestion can be visualized in Metatron Discovery.

2. To create a datasource, select Metatron Engine and select the S3 ingestion data table.