EMR ETL Job Always Fails on S3DistCp Step.

[Jake Williamson]

I'm trying to run the Snowplow EMR enricher and failing miserably. I consistently get the response: "Terminated with errorsShut down as step failed" with the job failing on the S3DistCp Step: Raw S3 -> HDFS step.

I've found people experiencing similar issues here and here (I originally commented on this page and was instructed to ask you kind folks).

These show a form of the following stderr:

Exception in thread "main" java.lang.RuntimeException: Error running job
	at com.amazon.elasticmapreduce.s3distcp.S3DistCp.run(S3DistCp.java:724)
	at com.amazon.elasticmapreduce.s3distcp.S3DistCp.run(S3DistCp.java:549)
	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
	at com.amazon.elasticmapreduce.s3distcp.Main.main(Main.java:13)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at org.apache.hadoop.util.RunJar.main(RunJar.java:187)
Caused by: org.apache.hadoop.mapred.InvalidInputException: Input path does not exist: hdfs:/tmp/87405325-f08a-49c5-ad7f-d0b4f83fb0ec/files
	at org.apache.hadoop.mapred.FileInputFormat.listStatus(FileInputFormat.java:197)
	at org.apache.hadoop.mapred.SequenceFileInputFormat.listStatus(SequenceFileInputFormat.java:40)
	at org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:208)
	at org.apache.hadoop.mapred.JobClient.writeOldSplits(JobClient.java:1044)
	at org.apache.hadoop.mapred.JobClient.writeSplits(JobClient.java:1036)
	at org.apache.hadoop.mapred.JobClient.access$700(JobClient.java:174)
	at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:952)
	at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:905)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:415)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1132)
	at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:905)
	at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:879)
	at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1316)
	at com.amazon.elasticmapreduce.s3distcp.S3DistCp.run(S3DistCp.java:706)
	... 9 more

Both of these issues were a result of having empty processing buckets.

I'm receiving this error as well, however the staging process has succeeded and I have logs in my processing bucket.

Included in the second link are several suggested things to check, and I've done those:

  1. s3n://<< bucket >>/processing/ definitely exists, is readable by your AWS creds and is definitely in us-east-1 ?

My processing bucket exists and contains logs
My processing bucket is definitely readable by my AWS credentials as it was able to be written to during staging
The bucket claims a region name of "US Standard", which, according to the documentation is essentially an alias for us-east
  2.  Your << keypair >> keypair was created in the correct region - us-east-1?

From the EC2 Management Console, I see two Key Pairs in the us-east-1 region.
One for unrelated EC2 projects, and one created specifically for use with Snowplow.

I've tried two different availability zones by setting :placement: first
to us-east-1a, then us-east-1b as my other EC2 instances use this zone 
and function properly. I didn't really expect this to have any effect, 
so I wasn't terribly disappointed when it didn't.

I've also tried different AMI versions and setting continue_on_unexpected_error to false and commenting out the error buckets with no success.

The S3DistCp error I receive differs from the one described in the link.

I receive the following:

2014-08-27T14:22:27.879Z INFO Fetching jar file.
2014-08-27T14:22:34.910Z INFO Working dir /mnt/var/lib/hadoop/steps/1
2014-08-27T14:22:34.910Z INFO Executing /usr/java/latest/bin/java -cp /home/hadoop/conf:/usr/java/latest/lib/tools.jar:/home/hadoop:/home/hadoop/hadoop-tools.jar:/home/hadoop/hadoop-tools-1.0.3.jar:/home/hadoop/hadoop-core-1.0.3.jar:/home/hadoop/hadoop-core.jar:/home/hadoop/lib/*:/home/hadoop/lib/jetty-ext/* -Xmx1000m -Dhadoop.log.dir=/mnt/var/log/hadoop/steps/1 -Dhadoop.log.file=syslog -Dhadoop.home.dir=/home/hadoop -Dhadoop.id.str=hadoop -Dhadoop.root.logger=INFO,DRFA -Djava.io.tmpdir=/mnt/var/lib/hadoop/steps/1/tmp -Djava.library.path=/home/hadoop/native/Linux-amd64-64 org.apache.hadoop.util.RunJar /home/hadoop/lib/emr-s3distcp-1.0.jar --src s3://my-company-name-processing-bucket/processing/ --dest hdfs:///local/snowplow/raw-events/ --groupBy .*\.([0-9]+-[0-9]+-[0-9]+)-[0-9]+\..* --targetSize 128 --outputCodec lzo --s3Endpoint s3.amazonaws.com
2014-08-27T14:22:46.250Z INFO Execution ended with ret val 1
2014-08-27T14:22:46.251Z WARN Step failed with bad retval
2014-08-27T14:22:52.585Z INFO Step created jobs: 

I've been very thorough in checking my config.yml for errors, but to cover all of my bases, I'll include a copy below.

I've changed my bucket names as they contain my company name, however the directory structure is correct.

:logging:
  :level: DEBUG # You can optionally switch to INFO for production
:aws:
  :access_key_id: XXXXXXXXXXXXXXXXXXX
  :secret_access_key: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
:s3:
  :region: us-east-1
  :buckets:
    :assets: s3://snowplow-hosted-assets # DO NOT CHANGE unless you are hosting the jarfiles etc yourself in your own bucket
    :log: s3://my-company-name-logging-bucket/enrichment_logs
    :raw:
      :in: s3://my-company-name-logging-bucket/logs
      :processing: s3://my-company-name-processing-bucket/processing
      :archive: s3://my-company-name-archive-bucket/raw    # e.g. s3://my-company-name-archive-bucket/raw
    :enriched:
      :good: s3://my-company-name-out-bucket/enriched/good       # e.g. s3://my-company-name-out-bucket/enriched/good
      :bad: s3://my-company-name-out-bucket/enriched/bad        # e.g. s3://my-company-name-out-bucket/enriched/bad
      :errors: s3://s3:my-company-name-out-bucket/enriched/errors # Leave blank unless :continue_on_unexpected_error: set to true below
    :shredded:
      :good: s3://my-company-name-out-bucket/shredded/good       # e.g. s3://my-company-name-out-bucket/shredded/good
      :bad: s3://my-company-name-out-bucket/shredded/bad  # e.g. s3://my-company-name-out-bucket/shredded/bad
      :errors: s3://my-company-name-out-bucket/shredded/errors # Leave blank unless :continue_on_unexpected_error: set to true below
:emr:
  :ami_version: 2.4.2      # Choose as per http://docs.aws.amazon.com/ElasticMapReduce/latest/DeveloperGuide/emr-plan-ami.html
  :region: us-east-1        # Always set this
  :placement: us-east-1a     # Set this if not running in VPC. Leave blank otherwise
  :ec2_subnet_id: # Set this if running in VPC. Leave blank otherwise
  :ec2_key_name: My Company Name SnowPlow Key
  :software:
    :hbase:                # To launch on cluster, provide version, "0.92.0", keep quotes
    :lingual:              # To launch on cluster, provide version, "1.1", keep quotes
  # Adjust your Hadoop cluster below
  :jobflow:
    :master_instance_type: m1.small
    :core_instance_count: 2
    :core_instance_type: m1.small
    :task_instance_count: 0 # Increase to use spot instances
    :task_instance_type: m1.small
    :task_instance_bid: 0.015 # In USD. Adjust bid, or leave blank for non-spot-priced (i.e. on-demand) task instances
:etl:
  :job_name: My Company Name ETL # Give your job a name
  :versions:
    :hadoop_enrich: 0.6.0  # Version of the Hadoop Enrichment process
    :hadoop_shred: 0.2.0 # Version of the Hadoop Shredding process
  :collector_format: cloudfront # Or 'clj-tomcat' for the Clojure Collector
  :continue_on_unexpected_error: true # Set to 'true' (and set :out_errors: above) if you don't want any exceptions thrown from ETL
:iglu:
  :schema: iglu:com.snowplowanalytics.iglu/resolver-config/jsonschema/1-0-0
  :data:
    :cache_size: 500
    :repositories:
      - :name: "Iglu Central"
        :priority: 0
        :vendor_prefixes:
          - com.snowplowanalytics
        :connection:
          :http:
            :uri: http://iglucentral.com

I'm unsure how to debug this. What are your thoughts/suggestions?

Thanks.

[Alex Dean]

Hi Jake,

Thanks for the detailed report. You say:

The S3DistCp error I receive differs from the one described in the link.

I receive the following:

2014-08-27T14:22:27.879Z INFO Fetching jar file.
2014-08-27T14:22:34.910Z INFO Working dir /mnt/var/lib/hadoop/steps/1
2014-08-27T14:22:34.910Z INFO Executing /usr/java/latest/bin/java -cp /home/hadoop/conf:/usr/java/latest/lib/tools.jar:/home/hadoop:/home/hadoop/hadoop-tools.jar:/home/hadoop/hadoop-tools-1.0.3.jar:/home/hadoop/hadoop-core-1.0.3.jar:/home/hadoop/hadoop-core.jar:/home/hadoop/lib/*:/home/hadoop/lib/jetty-ext/* -Xmx1000m -Dhadoop.log.dir=/mnt/var/log/hadoop/steps/1 -Dhadoop.log.file=syslog -Dhadoop.home.dir=/home/hadoop -Dhadoop.id.str=hadoop -Dhadoop.root.logger=INFO,DRFA -Djava.io.tmpdir=/mnt/var/lib/hadoop/steps/1/tmp -Djava.library.path=/home/hadoop/native/Linux-amd64-64 org.apache.hadoop.util.RunJar /home/hadoop/lib/emr-s3distcp-1.0.jar --src s3://my-company-name-processing-bucket/processing/ --dest hdfs:///local/snowplow/raw-events/ --groupBy .*\.([0-9]+-[0-9]+-[0-9]+)-[0-9]+\..* --targetSize 128 --outputCodec lzo --s3Endpoint s3.amazonaws.com
2014-08-27T14:22:46.250Z INFO Execution ended with ret val 1
2014-08-27T14:22:46.251Z WARN Step failed with bad retval
2014-08-27T14:22:52.585Z INFO Step created jobs: 

It would be good to find the underlying ERROR as this is very little information to go on. A few things to try:

1. Try running EmrEtlRunner with --debug to get more logging
2. Review your task failure logs as per https://github.com/snowplow/snowplow/wiki/Troubleshooting-jobs-on-Elastic-MapReduce#checking-the-hadoop-logs-for-errors
3. Try running EmrEtlRunner with --skip s3distcp to read/write directly to S3, skipping the S3DistCp steps

Let us know how you get on,

Alex

--
You received this message because you are subscribed to the Google Groups "Snowplow" group.
To unsubscribe from this group and stop receiving emails from it, send an email to snowplow-use...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Co-founder
Snowplow Analytics
The Roma Building, 32-38 Scrutton Street, London EC2A 4RQ, United Kingdom
+44 (0)203 589 6116
+44 7881 622 925
@alexcrdean

[Jake Williamson]

Apologies. Upon properly reviewing the logs, I do see the following:

2014-08-28 17:22:19,397 INFO com.amazon.elasticmapreduce.s3distcp.S3DistCp (main): Skipping key 'processing/' because it ends with '/'
2014-08-28 17:22:20,307 INFO com.amazon.elasticmapreduce.s3distcp.S3DistCp (main): Created 0 files to copy 0 files 
2014-08-28 17:22:20,685 INFO org.apache.hadoop.mapred.JobClient (main): Default number of map tasks: null

However, when viewing the processing bucket in the console, I do see many logs as staging finished successfully.

Skipping the s3distcp step, everything finishes successfully. (Including archiving)
I don't understand why this happens. Can you shed some light on this?

[Jake Williamson]

I forgot to include the output from --debug in the previous unsuccessful attempt.

Controller output:

2014-08-28T17:21:21.955Z INFO Fetching jar file.
2014-08-28T17:21:29.456Z INFO Working dir /mnt/var/lib/hadoop/steps/1
2014-08-28T17:21:29.457Z INFO Executing /usr/java/latest/bin/java -cp /home/hadoop/conf:/usr/java/latest/lib/tools.jar:/home/hadoop:/home/hadoop/hadoop-tools.jar:/home/hadoop/hadoop-tools-1.0.3.jar:/home/hadoop/hadoop-core-1.0.3.jar:/home/hadoop/hadoop-core.jar:/home/hadoop/lib/*:/home/hadoop/lib/jetty-ext/* -Xmx1000m -Dhadoop.log.dir=/mnt/var/log/hadoop/steps/1 -Dhadoop.log.file=syslog -Dhadoop.home.dir=/home/hadoop -Dhadoop.id.str=hadoop -Dhadoop.root.logger=INFO,DRFA -Djava.io.tmpdir=/mnt/var/lib/hadoop/steps/1/tmp -Djava.library.path=/home/hadoop/native/Linux-amd64-64 org.apache.hadoop.util.RunJar /mnt/var/lib/hadoop/steps/1/script-runner.jar s3://elasticmapreduce/libs/state-pusher/0.1/fetch
2014-08-28T17:21:46.224Z INFO Execution ended with ret val 0
2014-08-28T17:21:59.894Z INFO Step created jobs: 
2014-08-28T17:21:59.895Z INFO Step succeeded

syslog:

2014-08-28 17:21:37,567 INFO org.apache.hadoop.fs.s3native.NativeS3FileSystem (main): Opening '/libs/state-pusher/0.1/fetch' for reading

stderr:

+ /etc/init.d/hadoop-state-pusher-control stop
+ PID_FILE=/mnt/var/run/hadoop-state-pusher/hadoop-state-pusher.pid
+ LOG_FILE=/mnt/var/log/hadoop-state-pusher/hadoop-state-pusher.out
+ SVC_FILE=/mnt/var/lib/hadoop-state-pusher/run-hadoop-state-pusher
+ case $1 in
+ stop
+ echo 0
/etc/init.d/hadoop-state-pusher-control: line 35: /mnt/var/lib/hadoop-state-pusher/run-hadoop-state-pusher: No such file or directory
+ /etc/init.d/hadoop-state-pusher-control start
+ PID_FILE=/mnt/var/run/hadoop-state-pusher/hadoop-state-pusher.pid
+ LOG_FILE=/mnt/var/log/hadoop-state-pusher/hadoop-state-pusher.out
+ SVC_FILE=/mnt/var/lib/hadoop-state-pusher/run-hadoop-state-pusher
+ case $1 in
+ start
++ dirname /mnt/var/lib/hadoop-state-pusher/run-hadoop-state-pusher
+ sudo -u hadoop mkdir -p /mnt/var/lib/hadoop-state-pusher
+ echo 1
++ dirname /mnt/var/run/hadoop-state-pusher/hadoop-state-pusher.pid
+ sudo -u hadoop mkdir -p /mnt/var/run/hadoop-state-pusher
++ dirname /mnt/var/log/hadoop-state-pusher/hadoop-state-pusher.out
+ sudo -u hadoop mkdir -p /mnt/var/log/hadoop-state-pusher
+ disown %1
+ sleep 5
+ sudo -u hadoop /usr/bin/hadoop-state-pusher -server --pidfile /mnt/var/run/hadoop-state-pusher/hadoop-state-pusher.pid
+ exit 0
Command exiting with ret '0'

stdout:

Downloading 's3://elasticmapreduce/libs/state-pusher/0.1/fetch' to '/mnt/var/lib/hadoop/steps/1/.'

On Thursday, August 28, 2014 4:17:31 PM UTC-4, Jake Williamson wrote:

[Alex Dean]

Weird. Can you go to the failing job, find the failing step (the S3DistCp), and share with us the full CLI string (args etc) which it was called with?

Thanks,

Alex

--
You received this message because you are subscribed to the Google Groups "Snowplow" group.
To unsubscribe from this group and stop receiving emails from it, send an email to snowplow-use...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Alex Dean]

Also try changing all your buckets except snowplow-hosted-assets from s3:// to s3n://

Cheers,

Alex

[Jake Williamson]

The failed step was called with the following arguments;

--src s3://my-company-name-processing-bucket/processing/ --dest hdfs:///local/snowplow/raw-events/ --groupBy .*\.([0-9]+-[0-9]+-[0-9]+)-[0-9]+\..* --targetSize 128 --outputCodec lzo --s3Endpoint s3.amazonaws.com

I've also noticed that every file in my out bucket is 0 bytes in size. Does this help at all?

I've made the s3:// to s3n:// change and restarted. I will give you an update when it finishes/fails.

On Thursday, August 28, 2014 4:17:31 PM UTC-4, Jake Williamson wrote:

[Jake Williamson]

Changing s3:// to s3n:// had no effect. The job still fails on the S3DistCp step.

On Thursday, August 28, 2014 4:17:31 PM UTC-4, Jake Williamson wrote:

[Alex Dean]

I've also noticed that every file in my out bucket is 0 bytes in size. Does this help at all?

It sounds like even with --skip s3distcp, there is something wrong with your incoming event data.

Can you paste in a listing excerpt of the files that are being moved into the processing location, with their sizes?

A

--
You received this message because you are subscribed to the Google Groups "Snowplow" group.
To unsubscribe from this group and stop receiving emails from it, send an email to snowplow-use...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Jake Williamson]

I Apologize for the delayed response. I've been extremely busy with other projects.

Here is a short list of the files currently in my processing bucket. Please let me know if I can supply you with any other information.

Filename Storage Class File Size  Last Modified

logs-2014-08-27-22-21-26-658768C4CF2DDB93 Standard 27.5 KB Fri Aug 29 13:21:01 GMT-400 2014

logs-2014-08-27-22-21-27-C67B4F8B70F1F522 Standard 30.8 KB Fri Aug 29 13:21:01 GMT-400 2014

logs-2014-08-27-22-21-37-070B7FDFE6BC1831 Standard 2.2 KB Fri Aug 29 13:21:02 GMT-400 2014

logs-2014-08-27-22-21-54-EBCDD669CAFAB7FC Standard 1.7 KB Fri Aug 29 13:21:02 GMT-400 2014

logs-2014-08-27-22-21-59-9F0BBD121FE96908 Standard 704 bytes Fri Aug 29 13:21:02 GMT-400 2014

logs-2014-08-27-22-22-00-E639FFFE4D5085D6 Standard 704 bytes Fri Aug 29 13:21:04 GMT-400 2014

logs-2014-08-27-22-22-01-951E61E2FD89716D Standard 705 bytes Fri Aug 29 13:21:02 GMT-400 2014

logs-2014-08-27-22-22-14-3F2EE18FF5FD0491 Standard 21.6 KB Fri Aug 29 13:21:03 GMT-400 2014

logs-2014-08-27-22-22-15-55E2348962D1E2C5 Standard 31.3 KB Fri Aug 29 13:21:02 GMT-400 2014

logs-2014-08-27-22-22-18-32C7776C4468A10B Standard 39.9 KB Fri Aug 29 13:21:03 GMT-400 2014

logs-2014-08-27-22-22-24-F54F4AFED687BB19 Standard 25.7 KB Fri Aug 29 13:21:06 GMT-400 2014

logs-2014-08-27-22-22-27-237F7FAAFC044DC1 Standard 28.1 KB Fri Aug 29 13:21:06 GMT-400 2014

logs-2014-08-27-22-22-32-81D25E7380D90743 Standard 13 KB Fri Aug 29 13:21:06 GMT-400 2014

logs-2014-08-27-22-22-39-265B77B010143637 Standard 18 KB Fri Aug 29 13:21:06 GMT-400 2014

logs-2014-08-27-22-22-43-F0CBA3C5ECC6F21F Standard 706 bytes Fri Aug 29 13:21:06 GMT-400 2014

logs-2014-08-27-22-22-45-5D0F99ED39A02189 Standard 418 bytes Fri Aug 29 13:21:06 GMT-400 2014

logs-2014-08-27-22-22-56-A91716D14D10A16C Standard 440 bytes Fri Aug 29 13:21:07 GMT-400 2014

logs-2014-08-27-22-23-32-3D688515DA8BB002 Standard 32.1 KB Fri Aug 29 13:21:07 GMT-400 2014

logs-2014-08-27-22-24-34-FF067661A309ED44 Standard 22.4 KB Fri Aug 29 13:21:07 GMT-400 2014

logs-2014-08-27-22-26-06-2157A6DA88C86F21 Standard 704 bytes Fri Aug 29 13:21:08 GMT-400 2014

logs-2014-08-27-22-27-10-4765744C3DC357BD Standard 348 bytes Fri Aug 29 13:21:10 GMT-400 2014

logs-2014-08-27-22-27-19-AD9DC860EE281D83 Standard 348 bytes Fri Aug 29 13:21:10 GMT-400 2014

logs-2014-08-27-22-28-29-AAF88A03CC5B93F1 Standard 404 bytes Fri Aug 29 13:21:10 GMT-400 2014

logs-2014-08-27-22-29-21-92666ABECBD7B826 Standard 704 bytes Fri Aug 29 13:21:11 GMT-400 2014

logs-2014-08-27-22-29-23-4CB3047CDF54F924 Standard 704 bytes Fri Aug 29 13:21:11 GMT-400 2014

logs-2014-08-27-22-29-23-D8C94F69559D3DD9 Standard 418 bytes Fri Aug 29 13:21:11 GMT-400 2014

logs-2014-08-27-22-30-00-1F13E9785A6CB395 Standard 704 bytes Fri Aug 29 13:21:11 GMT-400 2014

logs-2014-08-27-22-30-30-D8B1252CBEA22037 Standard 335 bytes Fri Aug 29 13:21:11 GMT-400 2014

logs-2014-08-27-22-30-41-B1414BABA7A9AD9C Standard 393 bytes Fri Aug 29 13:21:12 GMT-400 2014

logs-2014-08-27-22-30-45-0BFE8F41CD15754F Standard 704 bytes Fri Aug 29 13:21:12 GMT-400 2014

logs-2014-08-27-22-31-17-66080509816B7EDC Standard 858 bytes Fri Aug 29 13:21:13 GMT-400 2014

logs-2014-08-27-22-31-22-1E3D4EF1BA5D24A5 Standard 392 bytes Fri Aug 29 13:21:14 GMT-400 2014

logs-2014-08-27-22-31-32-D317436F15D79932 Standard 704 bytes Fri Aug 29 13:21:15 GMT-400 2014

logs-2014-08-27-22-31-55-2E452FE19A643FDE Standard 705 bytes Fri Aug 29 13:21:15 GMT-400 2014

logs-2014-08-27-22-32-41-0A2E373726C4C353 Standard 751 bytes Fri Aug 29 13:21:15 GMT-400 2014

logs-2014-08-27-22-32-44-6E179EF6AA27EB3E Standard 348 bytes Fri Aug 29 13:21:16 GMT-400 2014

logs-2014-08-27-22-32-45-AEB1A47F23243BF3 Standard 442 bytes Fri Aug 29 13:21:16 GMT-400 2014

logs-2014-08-27-22-33-35-CB47FABAADAB61F0 Standard 767 bytes Fri Aug 29 13:21:16 GMT-400 2014

logs-2014-08-27-22-33-38-E56DD6E4477E423D Standard 705 bytes Fri Aug 29 13:21:16 GMT-400 2014

logs-2014-08-27-22-34-41-24B37BA6F59344EF Standard 706 bytes Fri Aug 29 13:21:16 GMT-400 2014

logs-2014-08-27-22-34-43-4B8B3AD1118DE71B Standard 426 bytes Fri Aug 29 13:21:16 GMT-400 2014

logs-2014-08-27-22-35-06-A6302439B9DA7EDF Standard 704 bytes Fri Aug 29 13:21:19 GMT-400 2014

logs-2014-08-27-22-35-37-F969BD5D2E99C14D Standard 1.3 KB Fri Aug 29 13:21:19 GMT-400 2014

logs-2014-08-27-22-35-49-69E188D08F0F3DA6 Standard 314 bytes Fri Aug 29 13:21:19 GMT-400 2014

logs-2014-08-27-22-35-52-545C906D51FD0354 Standard 347 bytes Fri Aug 29 13:21:19 GMT-400 2014

logs-2014-08-27-22-36-21-2544AAAE61EB9AB6 Standard 347 bytes Fri Aug 29 13:21:19 GMT-400 2014

logs-2014-08-27-22-36-54-CB39F65DB5465A13 Standard 312 bytes Fri Aug 29 13:21:20 GMT-400 2014

logs-2014-08-27-22-37-04-BC7A8252901351A2 Standard 1.5 KB Fri Aug 29 13:21:20 GMT-400 2014

logs-2014-08-27-22-37-10-01FA0D2E0F2308A6 Standard 1 KB Fri Aug 29 13:21:20 GMT-400 2014

logs-2014-08-27-22-37-11-B4EDF9F95B1E0224 Standard 938 bytes Fri Aug 29 13:21:20 GMT-400 2014

logs-2014-08-27-22-38-08-F88E4269BFD78D98 Standard 362 bytes Fri Aug 29 13:21:21 GMT-400 2014

logs-2014-08-27-22-38-36-C2D6722611E6FE4F Standard 312 bytes Fri Aug 29 13:21:23 GMT-400 2014

logs-2014-08-27-22-38-38-110EEB33FB1E1F68 Standard 625 bytes Fri Aug 29 13:21:23 GMT-400 2014

logs-2014-08-27-22-41-03-5CE141B9111A7560 Standard 1 KB Fri Aug 29 13:21:24 GMT-400 2014

logs-2014-08-27-22-42-51-EF0FBFF3BFED3B9E Standard 326 bytes Fri Aug 29 13:21:24 GMT-400 2014

logs-2014-08-27-22-46-07-B8054A9317E19BD1 Standard 314 bytes Fri Aug 29 13:21:24 GMT-400 2014

logs-2014-08-27-23-21-34-61AFB6E326D96566 Standard 3.8 KB Fri Aug 29 13:21:26 GMT-400 2014

logs-2014-08-27-23-21-48-AA9C53A10971C9AD Standard 6.9 KB Fri Aug 29 13:21:26 GMT-400 2014

logs-2014-08-27-23-21-51-0F6192C6E51DBB1E Standard 707 bytes Fri Aug 29 13:21:26 GMT-400 2014

logs-2014-08-27-23-21-56-57B99950206BA443 Standard 4 KB Fri Aug 29 13:21:26 GMT-400 2014

logs-2014-08-27-23-21-58-719979285C661740 Standard 704 bytes Fri Aug 29 13:21:26 GMT-400 2014

logs-2014-08-27-23-22-02-15E5CC7FEE05D34D Standard 5.8 KB Fri Aug 29 13:21:29 GMT-400 2014

logs-2014-08-27-23-22-08-6D40A491B64B1319 Standard 5.4 KB Fri Aug 29 13:21:28 GMT-400 2014

logs-2014-08-27-23-22-10-75CFDC89E261AB83 Standard 706 bytes Fri Aug 29 13:21:28 GMT-400 2014

logs-2014-08-27-23-22-12-0E04967179C39FA3 Standard 1007 bytes Fri Aug 29 13:21:29 GMT-400 2014

logs-2014-08-27-23-22-14-49A712CD5E11598D Standard 2 KB Fri Aug 29 13:21:30 GMT-400 2014

logs-2014-08-27-23-22-24-72A5BA13423542CE Standard 5.5 KB Fri Aug 29 13:21:30 GMT-400 2014

logs-2014-08-27-23-22-39-626D7810DF76DC51 Standard 5.6 KB Fri Aug 29 13:21:30 GMT-400 2014

logs-2014-08-27-23-22-45-64B96D308B9BBC20 Standard 5.4 KB Fri Aug 29 13:21:31 GMT-400 2014

logs-2014-08-27-23-22-45-CFADD891D3120CFA Standard 5.6 KB Fri Aug 29 13:21:32 GMT-400 2014

logs-2014-08-27-23-22-51-65FE101CEAB9C4D2 Standard 4.1 KB Fri Aug 29 13:21:32 GMT-400 2014

logs-2014-08-27-23-23-58-1517A9FAF6D22403 Standard 704 bytes Fri Aug 29 13:21:33 GMT-400 2014

logs-2014-08-27-23-24-30-BA7D05F065E883BD Standard 705 bytes Fri Aug 29 13:21:33 GMT-400 2014

logs-2014-08-27-23-26-47-78ABD8A310563985 Standard 704 bytes Fri Aug 29 13:21:33 GMT-400 2014

logs-2014-08-27-23-29-21-AAAEBA83AAA4217C Standard 13.3 KB Fri Aug 29 13:21:35 GMT-400 2014

logs-2014-08-27-23-29-42-AAF744C0C91AFF2D Standard 13.9 KB Fri Aug 29 13:21:35 GMT-400 2014

logs-2014-08-27-23-29-54-46930C2E0BD319E9 Standard 20.4 KB Fri Aug 29 13:21:35 GMT-400 2014

logs-2014-08-27-23-29-59-1AFE43447545291E Standard 7.5 KB Fri Aug 29 13:21:36 GMT-400 2014

logs-2014-08-27-23-30-01-1F3D63C32D57F3E1 Standard 15.7 KB Fri Aug 29 13:21:37 GMT-400 2014

logs-2014-08-27-23-30-06-B8D33169673494A3 Standard 704 bytes Fri Aug 29 13:21:38 GMT-400 2014

logs-2014-08-27-23-30-18-6B4C61E28A233504 Standard 12.2 KB Fri Aug 29 13:21:37 GMT-400 2014

logs-2014-08-27-23-30-20-1318BAFFEB611EFF Standard 1.3 KB Fri Aug 29 13:21:38 GMT-400 2014

logs-2014-08-27-23-30-25-296F07C5360E0EB7 Standard 9.4 KB Fri Aug 29 13:21:39 GMT-400 2014

logs-2014-08-27-23-30-25-8D7F7E29E3F836BE Standard 9.5 KB Fri Aug 29 13:21:40 GMT-400 2014

logs-2014-08-27-23-30-29-9BDD460D2FB9F743 Standard 6.9 KB Fri Aug 29 13:21:39 GMT-400 2014

logs-2014-08-27-23-30-38-3DC540B7474D53B3 Standard 8.7 KB Fri Aug 29 13:21:41 GMT-400 2014

logs-2014-08-27-23-30-38-9AD236B55A010877 Standard 3.8 KB Fri Aug 29 13:21:41 GMT-400 2014

logs-2014-08-27-23-30-44-A02A0A5FD9B8C796 Standard 10.8 KB Fri Aug 29 13:21:41 GMT-400 2014

logs-2014-08-27-23-30-56-BB32F5967AD7D418 Standard 1.3 KB Fri Aug 29 13:21:42 GMT-400 2014

logs-2014-08-27-23-31-04-125C779D5FB653BE Standard 11.6 KB Fri Aug 29 13:21:43 GMT-400 2014

logs-2014-08-27-23-31-06-3F4FFA676D5C7A37 Standard 704 bytes Fri Aug 29 13:21:43 GMT-400 2014

logs-2014-08-27-23-31-12-F7B7900F0A714F7B Standard 704 bytes Fri Aug 29 13:21:44 GMT-400 2014

logs-2014-08-27-23-31-27-11C4DEDFBDA79C9E Standard 1.3 KB Fri Aug 29 13:21:44 GMT-400 2014

logs-2014-08-27-23-31-28-3BE99D8C51348A98 Standard 12.5 KB Fri Aug 29 13:21:45 GMT-400 2014

logs-2014-08-27-23-31-59-FECE14BACF31A994 Standard 1.3 KB Fri Aug 29 13:21:45 GMT-400 2014

logs-2014-08-27-23-32-02-B37E8C8146BBE60E Standard 6.9 KB Fri Aug 29 13:21:45 GMT-400 2014

logs-2014-08-27-23-32-05-9357F959A36A6566 Standard 16.3 KB Fri Aug 29 13:21:46 GMT-400 2014

logs-2014-08-27-23-32-10-415FE19C02DDFE05 Standard 8.8 KB Fri Aug 29 13:21:46 GMT-400 2014

logs-2014-08-27-23-32-38-E52366454D886EF5 Standard 11.6 KB Fri Aug 29 13:21:47 GMT-400 2014

logs-2014-08-27-23-32-48-A0DEF4DFDE736A27 Standard 706 bytes Fri Aug 29 13:21:47 GMT-400 2014

logs-2014-08-27-23-33-27-985EE133BBEFF07F Standard 704 bytes Fri Aug 29 13:21:48 GMT-400 2014

logs-2014-08-27-23-33-53-FBF1BCA4E611FCB8 Standard 11.1 KB Fri Aug 29 13:21:48 GMT-400 2014

logs-2014-08-27-23-34-04-AF4A4DF10D619DD1 Standard 706 bytes Fri Aug 29 13:21:49 GMT-400 2014

logs-2014-08-27-23-34-07-6CBEBB73CC79630B Standard 10.6 KB Fri Aug 29 13:21:49 GMT-400 2014

logs-2014-08-27-23-34-19-E75306FA93385640 Standard 6.5 KB Fri Aug 29 13:21:50 GMT-400 2014

logs-2014-08-27-23-34-26-A353DA19AFF70394 Standard 12.7 KB Fri Aug 29 13:21:51 GMT-400 2014

logs-2014-08-27-23-35-05-328ADE8F1E7AF4DB Standard 706 bytes Fri Aug 29 13:21:52 GMT-400 2014

logs-2014-08-27-23-35-32-F23775242279633A Standard 8.7 KB Fri Aug 29 13:21:52 GMT-400 2014

logs-2014-08-27-23-35-41-4AC0FB912CDC4CB5 Standard 8.3 KB Fri Aug 29 13:21:52 GMT-400 2014

logs-2014-08-27-23-35-50-017B85FAE32DD800 Standard 7.5 KB Fri Aug 29 13:21:53 GMT-400 2014

logs-2014-08-27-23-36-01-71950542B18807D3 Standard 12.4 KB Fri Aug 29 13:21:53 GMT-400 2014

logs-2014-08-27-23-36-11-CA149606CC6BA9EC Standard 707 bytes Fri Aug 29 13:21:53 GMT-400 2014

logs-2014-08-27-23-36-54-1EB389DC93225648 Standard 9.1 KB Fri Aug 29 13:21:53 GMT-400 2014

logs-2014-08-27-23-36-54-816FBB0191E365B6 Standard 13.7 KB Fri Aug 29 13:21:54 GMT-400 2014

logs-2014-08-27-23-36-55-B33CC456A76DDBF0 Standard 15.2 KB Fri Aug 29 13:21:55 GMT-400 2014

logs-2014-08-27-23-37-07-1E68F20AE6BEE51A Standard 9.8 KB Fri Aug 29 13:21:56 GMT-400 2014

logs-2014-08-27-23-37-15-2536A12937E47BB2 Standard 14.9 KB Fri Aug 29 13:21:57 GMT-400 2014

logs-2014-08-27-23-37-16-CD9A7E389A34AA41 Standard 8.4 KB Fri Aug 29 13:21:57 GMT-400 2014

logs-2014-08-27-23-38-31-5A4636C51C806755 Standard 14.5 KB Fri Aug 29 13:21:57 GMT-400 2014

logs-2014-08-27-23-38-34-42BE55BB0B7FF0FE Standard 7.3 KB Fri Aug 29 13:21:57 GMT-400 2014

logs-2014-08-27-23-39-21-628290D10B595F51 Standard 11.4 KB Fri Aug 29 13:21:58 GMT-400 2014

logs-2014-08-27-23-39-27-9192C5AE8B9EE913 Standard 16.2 KB Fri Aug 29 13:21:59 GMT-400 2014

logs-2014-08-27-23-39-48-091599DAFE1B4918 Standard 7.6 KB Fri Aug 29 13:22:01 GMT-400 2014

logs-2014-08-27-23-40-14-A6A54024083A2D97 Standard 7.1 KB Fri Aug 29 13:22:00 GMT-400 2014

logs-2014-08-27-23-41-11-702673CFBEDA70DB Standard 9.8 KB Fri Aug 29 13:22:01 GMT-400 2014

logs-2014-08-27-23-41-26-AB4CF29DC2D5685C Standard 404 bytes Fri Aug 29 13:22:01 GMT-400 2014

logs-2014-08-27-23-41-38-E68653C0F95E7279 Standard 13 KB Fri Aug 29 13:22:02 GMT-400 2014

logs-2014-08-27-23-42-10-A2E9C70B6E4BDBEA Standard 8.4 KB Fri Aug 29 13:22:03 GMT-400 2014

logs-2014-08-28-00-21-22-09B3350953265F3A Standard 705 bytes Fri Aug 29 13:22:03 GMT-400 2014

logs-2014-08-28-00-21-41-C23AFF25CE2EF98E Standard 2 KB Fri Aug 29 13:22:03 GMT-400 2014

logs-2014-08-28-00-21-42-DD82302D0366AECB Standard 704 bytes Fri Aug 29 13:22:04 GMT-400 2014

logs-2014-08-28-00-21-44-7492CC87CE0AF968 Standard 704 bytes Fri Aug 29 13:22:05 GMT-400 2014

logs-2014-08-28-00-21-47-11B7424076AD13C2 Standard 704 bytes Fri Aug 29 13:22:06 GMT-400 2014

logs-2014-08-28-00-21-47-811B80D85F78C57B Standard 705 bytes Fri Aug 29 13:22:06 GMT-400 2014

logs-2014-08-28-00-22-00-03E9A6CF642F7C1A Standard 706 bytes Fri Aug 29 13:22:06 GMT-400 2014

logs-2014-08-28-00-22-06-9243205342961727 Standard 706 bytes Fri Aug 29 13:22:07 GMT-400 2014

logs-2014-08-28-00-22-07-465B9C06D8A49669 Standard 708 bytes Fri Aug 29 13:22:07 GMT-400 2014

logs-2014-08-28-00-22-23-6DF008FBD1292CB7 Standard 704 bytes Fri Aug 29 13:22:08 GMT-400 2014

logs-2014-08-28-00-22-28-528E7CFDB709D48F Standard 705 bytes Fri Aug 29 13:22:09 GMT-400 2014

logs-2014-08-28-00-22-47-4EF430F9683581AD Standard 705 bytes Fri Aug 29 13:22:09 GMT-400 2014

logs-2014-08-28-00-23-26-C732FEFF3608EE70 Standard 705 bytes Fri Aug 29 13:22:10 GMT-400 2014

logs-2014-08-28-00-24-50-D1D2FBAE65BEF84F Standard 1.3 KB Fri Aug 29 13:22:10 GMT-400 2014

logs-2014-08-28-00-25-23-4EFDE96B9342BE9A Standard 704 bytes Fri Aug 29 13:22:11 GMT-400 2014

logs-2014-08-28-00-28-27-BB71F22EFD3A6327 Standard 705 bytes Fri Aug 29 13:22:11 GMT-400 2014

logs-2014-08-28-00-29-09-D03244DE51E424FE Standard 704 bytes Fri Aug 29 13:22:11 GMT-400 2014

logs-2014-08-28-00-29-13-CD8DD6C95E7DE04A Standard 706 bytes Fri Aug 29 13:22:12 GMT-400 2014

logs-2014-08-28-00-29-55-E880D03E8422487C Standard 707 bytes Fri Aug 29 13:22:12 GMT-400 2014

logs-2014-08-28-00-29-56-23BA9F9BD97D7617 Standard 704 bytes Fri Aug 29 13:22:12 GMT-400 2014

logs-2014-08-28-00-30-09-D1B846C8EA6EB02B Standard 705 bytes Fri Aug 29 13:22:13 GMT-400 2014

logs-2014-08-28-00-30-36-18522853477941D4 Standard 705 bytes Fri Aug 29 13:22:15 GMT-400 2014

logs-2014-08-28-00-31-19-21E948AE5175A9CA Standard 705 bytes Fri Aug 29 13:22:15 GMT-400 2014

logs-2014-08-28-00-31-27-A4B817103389528F Standard 708 bytes Fri Aug 29 13:22:15 GMT-400 2014

logs-2014-08-28-00-31-29-1987AC17E67CD4B9 Standard 703 bytes Fri Aug 29 13:22:16 GMT-400 2014

logs-2014-08-28-00-31-33-77EBBFBCF9866E7F Standard 1.3 KB Fri Aug 29 13:22:16 GMT-400 2014

logs-2014-08-28-00-31-44-9C475E95DB1EBC33 Standard 707 bytes Fri Aug 29 13:22:16 GMT-400 2014

logs-2014-08-28-00-32-03-7870CE804051D5F1 Standard 706 bytes Fri Aug 29 13:22:18 GMT-400 2014

logs-2014-08-28-00-32-12-8034B6538A268A56 Standard 707 bytes Fri Aug 29 13:22:18 GMT-400 2014

logs-2014-08-28-00-33-11-DC8F67E98330095E Standard 704 bytes Fri Aug 29 13:22:18 GMT-400 2014

logs-2014-08-28-00-33-11-F489496F3DA56C6F Standard 704 bytes Fri Aug 29 13:22:19 GMT-400 2014

logs-2014-08-28-00-33-24-932E4CDE8A5E60D1 Standard 706 bytes Fri Aug 29 13:22:19 GMT-400 2014

logs-2014-08-28-00-33-30-6602DADEAAD0AAEF Standard 706 bytes Fri Aug 29 13:22:20 GMT-400 2014

logs-2014-08-28-00-33-31-AA9431FB1DE336EE Standard 706 bytes Fri Aug 29 13:22:20 GMT-400 2014

logs-2014-08-28-00-33-34-BFE7AAA0D82EFCCE Standard 706 bytes Fri Aug 29 13:22:20 GMT-400 2014

logs-2014-08-28-00-33-37-34B3B5CF3FDFB014 Standard 706 bytes Fri Aug 29 13:22:20 GMT-400 2014

logs-2014-08-28-00-34-26-456C38CED5FDA918 Standard 1.3 KB Fri Aug 29 13:22:21 GMT-400 2014

logs-2014-08-28-00-34-45-5D9A16D2F3B10216 Standard 2 KB Fri Aug 29 13:22:21 GMT-400 2014

logs-2014-08-28-00-34-56-9DD2F49C180F2986 Standard 704 bytes Fri Aug 29 13:22:22 GMT-400 2014

logs-2014-08-28-00-34-59-183DAA16FC22077D Standard 1.3 KB Fri Aug 29 13:22:23 GMT-400 2014

logs-2014-08-28-00-35-00-856465DFD48E41E6 Standard 1.3 KB Fri Aug 29 13:22:23 GMT-400 2014

logs-2014-08-28-00-35-07-A33C7F4D8AD6EAD1 Standard 707 bytes Fri Aug 29 13:22:23 GMT-400 2014

logs-2014-08-28-00-35-14-3C6DE4B73BAFD10F Standard 704 bytes Fri Aug 29 13:22:24 GMT-400 2014

logs-2014-08-28-00-35-15-4C3ECCAB19B03F23 Standard 705 bytes Fri Aug 29 13:22:24 GMT-400 2014

logs-2014-08-28-00-35-47-E9C96866893B8240 Standard 1.3 KB Fri Aug 29 13:22:24 GMT-400 2014

logs-2014-08-28-00-36-17-996F0687CCF2B63A Standard 1.3 KB Fri Aug 29 13:22:25 GMT-400 2014

logs-2014-08-28-00-37-10-75B533AD5BD40E4D Standard 708 bytes Fri Aug 29 13:22:25 GMT-400 2014

logs-2014-08-28-00-37-21-56F3526338A00118 Standard 706 bytes Fri Aug 29 13:22:26 GMT-400 2014

logs-2014-08-28-00-38-29-CA22B36818A5ED72 Standard 705 bytes Fri Aug 29 13:22:26 GMT-400 2014

logs-2014-08-28-00-38-37-48BED54933114721 Standard 703 bytes Fri Aug 29 13:22:27 GMT-400 2014

logs-2014-08-28-00-39-03-69DA50A08C68F015 Standard 1.3 KB Fri Aug 29 13:22:27 GMT-400 2014

logs-2014-08-28-00-39-21-83B1F4F0F26C3987 Standard 708 bytes Fri Aug 29 13:22:27 GMT-400 2014

logs-2014-08-28-00-40-00-CCB69797E7057381 Standard 705 bytes Fri Aug 29 13:22:29 GMT-400 2014

logs-2014-08-28-00-40-02-2544822B65B52AA0 Standard 708 bytes Fri Aug 29 13:22:29 GMT-400 2014

logs-2014-08-28-00-41-10-6A14A668461A4E74 Standard 703 bytes Fri Aug 29 13:22:29 GMT-400 2014

logs-2014-08-28-00-41-26-C03CEABA003A0AEA Standard 1.3 KB Fri Aug 29 13:22:29 GMT-400 2014

On Thursday, August 28, 2014 4:17:31 PM UTC-4, Jake Williamson wrote:

[Alex Dean]

Hi Jake,

Thanks for sharing that list. Those aren't CloudFront access logs - an example CloudFront access log would look like:

EO3VS2J9DJRJR.2014-09-04-11.OdeYmC6B.gz

Please make sure that:

:in: s3://my-company-name-logging-bucket/logs

is pointing to wherever CloudFront is storing your CloudFront access logs.

Hope this helps,

Alex

--
You received this message because you are subscribed to the Google Groups "Snowplow" group.
To unsubscribe from this group and stop receiving emails from it, send an email to snowplow-use...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Jake Williamson]

Perhaps I've misunderstood something.

I have 2 CloudFront distributions. One points to an S3 bucket containing the minified Snowplow JavaScript file, sp.js.

I created the second distribution's origin is the my S3 bucket containing the 1x1 tracking pixel, i, with logging enabled.

The log bucket is set to my-company-name-logging-bucket.s3.amazonaws.com.

Should I have done something differently? I don't see any filenames resembling the format you described.

On Thursday, August 28, 2014 4:17:31 PM UTC-4, Jake Williamson wrote:

[Alex Dean]

If your log bucket is set to s3://my-company-name-logging-bucket, then you need the configuration option:

:in: s3://my-company-name-logging-bucket

Not as you have now:

:in: s3://my-company-name-logging-bucket/logs

If that doesn't fix it, please have a careful read of this page which will help explain what CloudFront access logs are, their filenames and where you can expect to find them:

http://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/AccessLogs.html#AccessLogsFileNaming

If that still doesn't fix it, please paste in the contents of one of the (smaller) files you listed in your earlier email...

A

--
You received this message because you are subscribed to the Google Groups "Snowplow" group.
To unsubscribe from this group and stop receiving emails from it, send an email to snowplow-use...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Jake Williamson]

That did it. Thank you for your time. I apologize for having wasted it. I misread the documentation here where it reads:

Important 1: there is a bug in Hive on Amazon EMR where Hive dies if you attempt to read or write data to the root of an S3 bucket. Therefore always specify a sub-folder (e.g. /events/) for the processing and all three out bucket variables.

After reading this again it is clear that this doesn't apply to the :in: bucket.

On Thursday, August 28, 2014 4:17:31 PM UTC-4, Jake Williamson wrote:

On Thursday, August 28, 2014 4:17:31 PM UTC-4, Jake Williamson wrote:

[Alex Dean]

Hey Jake,

No worries at all - glad it's all working now!

Cheers,

Alex

--
You received this message because you are subscribed to the Google Groups "Snowplow" group.
To unsubscribe from this group and stop receiving emails from it, send an email to snowplow-use...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.