Netrc::Error when running snowplow-emr-etl-runner

[Brandon Clark]

Brief context, I'm setting up the snowplow components in individual docker containers. Everything has been going smoothly up until this point and now I'm configuring the shredder and storage loader and I've hit a wall.

I'm running the following commandline:

snowplow@1fe72d043da1:~$ /usr/local/bin/snowplow-emr-etl-runner -d --config /etc/snowplow/storage-loader.conf -r /etc/snowplow/iglu_resolver.json --skip staging,enrich,archive_raw

Initially I had an unhelpful "Unknown Error" which I worked out was due to a missing .netrc file (I'm unclear why this is a dependency). I created the file and verified the permissions are set correctly to 0600 but now I get the following stack trace:

D, [2016-03-30T15:49:39.887000 #2051450519] DEBUG -- : Initializing EMR jobflow

F, [2016-03-30T15:49:43.964000 #2051450519] FATAL -- : 

Netrc::Error (Permission bits for '/home/snowplow/.netrc' should be 0600, but are 644):

    /usr/local/bin/snowplow-emr-etl-runner!/gems/netrc-0.10.3/lib/netrc.rb:45:in `check_permissions'

    /usr/local/bin/snowplow-emr-etl-runner!/gems/netrc-0.10.3/lib/netrc.rb:52:in `read'

    /usr/local/bin/snowplow-emr-etl-runner!/gems/rest-client-1.8.0/lib/restclient/request.rb:284:in `parse_url_with_auth'

    /usr/local/bin/snowplow-emr-etl-runner!/gems/rest-client-1.8.0/lib/restclient/request.rb:175:in `execute'

    /usr/local/bin/snowplow-emr-etl-runner!/gems/rest-client-1.8.0/lib/restclient/request.rb:41:in `execute'

    /usr/local/bin/snowplow-emr-etl-runner!/gems/rest-client-1.8.0/lib/restclient.rb:69:in `post'

    /usr/local/bin/snowplow-emr-etl-runner!/gems/elasticity-6.0.7/lib/elasticity/aws_session.rb:29:in `submit'

    /usr/local/bin/snowplow-emr-etl-runner!/gems/elasticity-6.0.7/lib/elasticity/emr.rb:302:in `run_job_flow'

    /usr/local/bin/snowplow-emr-etl-runner!/gems/elasticity-6.0.7/lib/elasticity/job_flow.rb:151:in `run'

    /usr/local/bin/snowplow-emr-etl-runner!/emr-etl-runner/lib/snowplow-emr-etl-runner/emr_job.rb:445:in `run'

    /usr/local/bin/snowplow-emr-etl-runner!/gems/contracts-0.7/lib/contracts/method_reference.rb:46:in `send_to'

    /usr/local/bin/snowplow-emr-etl-runner!/gems/contracts-0.7/lib/contracts.rb:305:in `call_with'

    /usr/local/bin/snowplow-emr-etl-runner!/gems/contracts-0.7/lib/contracts/decorators.rb:159:in `common_method_added'

    /usr/local/bin/snowplow-emr-etl-runner!/emr-etl-runner/lib/snowplow-emr-etl-runner/runner.rb:68:in `run'

    /usr/local/bin/snowplow-emr-etl-runner!/gems/contracts-0.7/lib/contracts/method_reference.rb:46:in `send_to'

    /usr/local/bin/snowplow-emr-etl-runner!/gems/contracts-0.7/lib/contracts.rb:305:in `call_with'

    /usr/local/bin/snowplow-emr-etl-runner!/gems/contracts-0.7/lib/contracts/decorators.rb:159:in `common_method_added'

    file:/usr/local/bin/snowplow-emr-etl-runner!/emr-etl-runner/bin/snowplow-emr-etl-runner:39:in `(root)'

    org/jruby/RubyKernel.java:1091:in `load'

    file:/usr/local/bin/snowplow-emr-etl-runner!/META-INF/main.rb:1:in `(root)'

    org/jruby/RubyKernel.java:1072:in `require'

    file:/usr/local/bin/snowplow-emr-etl-runner!/META-INF/main.rb:1:in `(root)'

    /tmp/jruby4766577461049698189extract/jruby-stdlib-1.7.20.1.jar!/META-INF/jruby.home/lib/ruby/shared/rubygems/core_ext/kernel_require.rb:1:in `(root)'

A quick permission check shows:

snowplow@1fe72d043da1:~$ ls -la /home/snowplow/.netrc 

-rw------- 1 snowplow snowplow 0 Mar 29 22:15 /home/snowplow/.netrc

What am I doing wrong? Is this a bug? Thanks in advance.

[Fred Blundun]

Hi Brandon,

We haven't seen this before. Looks like the root cause is this check. As a sanity check, what happens if you open a Ruby repl in the container and execute the code below (copied from the netrc project)? 

(File.stat('/home/snowplow/.netrc').mode & 0777).to_s(8) Additionally, do you have a copy of the stack trace for your unknown error? Regards, Fred

--
You received this message because you are subscribed to the Google Groups "Snowplow" group.
To unsubscribe from this group and stop receiving emails from it, send an email to snowplow-use...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Brandon Clark]

Hi Fred, thanks for the quick reply

Running:

snowplow@1fe72d043da1:~$ jruby -e "(File.stat('/home/snowplow/.netrc').mode & 0777).to_s(8)"
outputs nothing

When I delete the .netrc file this is the original stack trace I got:

snowplow@1fe72d043da1:~$ /usr/local/bin/snowplow-emr-etl-runner -d --config /etc/snowplow/storage-loader.conf -r /etc/snowplow/iglu_resolver.json --skip staging,enrich,archive_raw

D, [2016-03-30T17:24:13.810000 #2051450519] DEBUG -- : Initializing EMR jobflow

F, [2016-03-30T17:24:18.203000 #2051450519] FATAL -- : 

SystemCallError (Unknown error - Unknown Error (0) - /home/snowplow/.netrc):

    org/jruby/RubyFile.java:872:in `stat'

    /usr/local/bin/snowplow-emr-etl-runner!/gems/netrc-0.10.3/lib/netrc.rb:43:in `check_permissions'

    /usr/local/bin/snowplow-emr-etl-runner!/gems/netrc-0.10.3/lib/netrc.rb:52:in `read'

    /usr/local/bin/snowplow-emr-etl-runner!/gems/rest-client-1.8.0/lib/restclient/request.rb:284:in `parse_url_with_auth'

    /usr/local/bin/snowplow-emr-etl-runner!/gems/rest-client-1.8.0/lib/restclient/request.rb:175:in `execute'

    /usr/local/bin/snowplow-emr-etl-runner!/gems/rest-client-1.8.0/lib/restclient/request.rb:41:in `execute'

    /usr/local/bin/snowplow-emr-etl-runner!/gems/rest-client-1.8.0/lib/restclient.rb:69:in `post'

    /usr/local/bin/snowplow-emr-etl-runner!/gems/elasticity-6.0.7/lib/elasticity/aws_session.rb:29:in `submit'

    /usr/local/bin/snowplow-emr-etl-runner!/gems/elasticity-6.0.7/lib/elasticity/emr.rb:302:in `run_job_flow'

    /usr/local/bin/snowplow-emr-etl-runner!/gems/elasticity-6.0.7/lib/elasticity/job_flow.rb:151:in `run'

    /usr/local/bin/snowplow-emr-etl-runner!/emr-etl-runner/lib/snowplow-emr-etl-runner/emr_job.rb:445:in `run'

    /usr/local/bin/snowplow-emr-etl-runner!/gems/contracts-0.7/lib/contracts/method_reference.rb:46:in `send_to'

    /usr/local/bin/snowplow-emr-etl-runner!/gems/contracts-0.7/lib/contracts.rb:305:in `call_with'

    /usr/local/bin/snowplow-emr-etl-runner!/gems/contracts-0.7/lib/contracts/decorators.rb:159:in `common_method_added'

    /usr/local/bin/snowplow-emr-etl-runner!/emr-etl-runner/lib/snowplow-emr-etl-runner/runner.rb:68:in `run'

    /usr/local/bin/snowplow-emr-etl-runner!/gems/contracts-0.7/lib/contracts/method_reference.rb:46:in `send_to'

    /usr/local/bin/snowplow-emr-etl-runner!/gems/contracts-0.7/lib/contracts.rb:305:in `call_with'

    /usr/local/bin/snowplow-emr-etl-runner!/gems/contracts-0.7/lib/contracts/decorators.rb:159:in `common_method_added'

    file:/usr/local/bin/snowplow-emr-etl-runner!/emr-etl-runner/bin/snowplow-emr-etl-runner:39:in `(root)'

    org/jruby/RubyKernel.java:1091:in `load'

    file:/usr/local/bin/snowplow-emr-etl-runner!/META-INF/main.rb:1:in `(root)'

    org/jruby/RubyKernel.java:1072:in `require'

    file:/usr/local/bin/snowplow-emr-etl-runner!/META-INF/main.rb:1:in `(root)'

    /tmp/jruby551494077955129196extract/jruby-stdlib-1.7.20.1.jar!/META-INF/jruby.home/lib/ruby/shared/rubygems/core_ext/kernel_require.rb:1:in `(root)'

[Fred Blundun]

Looks like supplying the script via the -e option means that return values aren't printed - try explicitly printing like this:

jruby -e "puts (File.stat('/home/snowplow/.netrc').mode & 0777).to_s(8)"

Regards,

Fred

[Brandon Clark]

Ah right. duh. Thanks for pointing that out.

So here's the process

snowplow@1fe72d043da1:~$ touch ~/.netrc

snowplow@1fe72d043da1:~$ chmod 0600 ~/.netrc

snowplow@1fe72d043da1:~$ jruby -e "puts (File.stat('/home/snowplow/.netrc').mode & 0777).to_s(8)"

644

[Fred Blundun]

There is something strange going on here. It should be possible to run snowplow-emr-etl-runner without a ~/.netrc file - I have just done so on Linux without getting a SystemCallError. If I create the file and try again, I get the same exception you initially posted. Then if I chmod 600 the file and try again, the job runs without errors.

This seems to be the relevant part of the netrc lib. I notice it has a lot of logic depending on things like the operating system and environment variables - do you see anything there which could be behaving oddly due to something about your Docker setup?

[Brandon Clark]

All the OS dependent code seems to be focused on determining the appropriate location and filename of .netrc for the respective platforms. That part seems to be working fine as far as I can tell. I did find this during my exhaustive googling (https://github.com/cyli/logstash-output-slack/issues/1). This poor soul seems to be experiencing the same problem in a different project so this leads me to believe that jruby netrc module isn't very reliable. The hack he suggested doesn't sit right with me and would be challenging any way since everything is packaged in a jar.

On a side note, running snowplow-emr-etl-runner on my host environment ran without a hitch (and no .netrc file) which tells me this has something to do with docker.

[Brandon Clark]

Another helpful bit is there is a jruby version difference between my docker container (based on ubuntu 12.04, jruby 1.5.6) versus my host environment (ubuntu 14.04, jruby 1.7.19). I'm going to attempt upgrading jruby inside the container and see if that changes anything

[Brandon Clark]

Bingo! Upgrading to the latest jruby fixed the problem. There appears to be a bug in 1.5.6 with File.stat

[Fred Blundun]

Glad you got it working! Quick question - were you using the jar downloaded from the Bintray as described here? Or did you build it yourself?

[Brandon Clark]

I downloaded this one http://dl.bintray.com/snowplow/snowplow-generic/snowplow_emr_r77_great_auk.zip

Which was linked from here: https://github.com/snowplow/snowplow/wiki/Hosted-assets

[Fred Blundun]

Thanks for letting me know. I was wondering whether a local JRuby version could interfere with a jar built using JRuby in that way - apparently it can.