Strange permissions problem

[Doug Hall]

I can ssh into my server and manually create the /tmp/<application> folder with my app/web role. However, when I try to do >cap staging git:check (which does the same thing), it attempts and fails to use my app/web role to create this folder. It succeeds with the db role, but then fails when attempting to use the /tmp/<application>/.git-ssh.sh at the end of the script. (See below) Both my app/web and db roles are members of the same (deploy) group.

I then simply manually created the /tmp/<application> folder on the server, to make sure that the folder's owner and group settings would permit executing the ./git-ssh.sh file, whomever created it. This didn't work, either. I'd appreciate any help.

I found this: <https://github.com/capistrano/capistrano/issues/687>, but it wasn't helpful. Something's got to be wrong with my ssh setup or something. Here are my ssh options: 

set :ssh_options, {

  forward_agent: true,

  port: ####

}

Yes, I've done the ssh-add on my Mac.

Thanks!

Versions:

• Ruby: 2.1.0
• Capistrano: 3.2.0
• Rake / Rails / etc : 10.2.2

Platform:

• Working on.... Mac
• Deploying to... Ubuntu 12.0.4 (LTS) server

Logs:

$ cap staging git:check --trace                                                                                                 [9:23:58]

** Invoke staging (first_time)

** Execute staging

** Invoke load:defaults (first_time)

** Execute load:defaults

** Invoke rbenv:validate (first_time)

** Execute rbenv:validate

DEBUG [fa95292a] Running /usr/bin/env [ ! -d /usr/local/bin ] on apps2.aidt.edu

DEBUG [fa95292a] Command: [ ! -d /usr/local/bin ]

DEBUG [f90faa17] Running /usr/bin/env [ ! -d /usr/local/bin ] on apps2.aidt.edu

DEBUG [f90faa17] Command: [ ! -d /usr/local/bin ]

DEBUG [fa95292a] Finished in 0.097 seconds with exit status 1 (failed).

DEBUG [f90faa17] Finished in 0.167 seconds with exit status 1 (failed).

** Invoke rbenv:map_bins (first_time)

** Execute rbenv:map_bins

** Invoke bundler:map_bins (first_time)

** Execute bundler:map_bins

** Invoke git:check (first_time)

** Invoke git:wrapper (first_time)

** Execute git:wrapper

 INFO [f6132a7c] Running /usr/bin/env mkdir -p /tmp/j3/ on apps2.aidt.edu

 INFO [e631d419] Running /usr/bin/env mkdir -p /tmp/j3/ on apps2.aidt.edu

DEBUG [f6132a7c] Command: ( RBENV_ROOT=~/.rbenv RBENV_VERSION=2.1.0 /usr/bin/env mkdir -p /tmp/j3/ )

DEBUG [e631d419] Command: ( RBENV_ROOT=~/.rbenv RBENV_VERSION=2.1.0 /usr/bin/env mkdir -p /tmp/j3/ )

 INFO [f6132a7c] Finished in 0.108 seconds with exit status 0 (successful).

DEBUG Uploading /tmp/j3/git-ssh.sh 0.0%

 INFO Uploading /tmp/j3/git-ssh.sh 100.0%

 INFO [06c59e05] Running /usr/bin/env chmod +x /tmp/j3/git-ssh.sh on apps2.aidt.edu

DEBUG [06c59e05] Command: ( RBENV_ROOT=~/.rbenv RBENV_VERSION=2.1.0 /usr/bin/env chmod +x /tmp/j3/git-ssh.sh )

 INFO [06c59e05] Finished in 0.010 seconds with exit status 0 (successful).

 INFO [e631d419] Finished in 0.173 seconds with exit status 0 (successful).

DEBUG Uploading /tmp/j3/git-ssh.sh 0.0%

cap aborted!

scp: /tmp/j3/git-ssh.sh: Permission denied

[Bruno Sutic]

Hey,

hm.. that's gotta be frustrating..

So hey - how far along did you go with this issue?

https://github.com/capistrano/capistrano/issues/687

That issue seems to indicate the problem:

- you're using 2 separate users for web/app and db group

- user 1 uploads the git-ssh.sh file and "chmod's" it.

- user 2 uploads the git-ssh.sh file. Things break apart here because file already exists (created by/for user 1). The trace provided above indicates this happens: see it fails just after trying to upload git-ssh.sh script *for the second time*.

How about you first try doing everything with 1 user and see what happens?

[Doug Hall]

I changed my db user to be the same as my app/web user, and that worked. But, this creates more questions.

I was under the impression that it was only going to try to upload it by user two, if it failed under user one. From what I could tell of my log, it failed on the first two roles (app/web) and then succeeded under the db role. 

Are you saying that, if you say to do something with all roles, then it's going to do that task with ALL roles, even if it can do it with the first one? Here is the task in question from the git.rake file:

  desc 'Upload the git wrapper script, this script guarantees that we can script git without getting an interactive prompt'

  task :wrapper do

    on release_roles :all do

      execute :mkdir, "-p", "#{fetch(:tmp_dir)}/#{fetch(:application)}/"

      upload! StringIO.new("#!/bin/sh -e\nexec /usr/bin/ssh -o PasswordAuthentication=no -o StrictHostKeyChecking=no \"$@\"\n"), "#{fetch(:tmp_dir)}/#{fetch(:application)}/git-ssh.sh"

      execute :chmod, "+x", "#{fetch(:tmp_dir)}/#{fetch(:application)}/git-ssh.sh"

    end

  end

The way I understand what you're telling me is, it's executing the task under EACH of the "release_roles". Why? It makes no freaking sense to do the same thing with multiple logins. If the app/web user has permissions to do what it needs to do, why would it possibly do the same thing with the db role?

How do I determine what the release roles are, by the way?

Thanks!

[Bruno Sutic]

Hey Doug,

I'm glad things work now.

I'm not 100% what's going on with your setup, but it seems as if you have multiple `server` declarations in you stage file (ie. config/deploy/production.rb), even though you're deploying only to 1 server.

Care to provide the config files you're using in a gist? (deploy.rb and a stage file)

Generally, if you're deploying to a single server, something like this should be fine for your stage file:

    server '<server_ip>', user: 'deploy', roles: %w{web app db}

Now, (and I'm doing a bit of guessing here) if you have something like this in your setup:

    server '<server_ip>', user: 'deploy', roles: %w{web app}

    server '<server_ip>', user: 'deploy', roles: %w{db}

even though you're deploying to the same server, with the same user - capistrano won't know that. It will assume 2 different servers are at play and will do everything twice: once for each server definition.

I hope it's kinda obvious Capistrano doesn't perform any checks if there are "duplicate" servers. That's left for us humans to do :)

About the "no release" question:

I never used that "feature", but here's the example use case in capistrano tests:

https://github.com/bruno-/capistrano/blob/tweak_revision_log/spec/integration/dsl_spec.rb#L18

If you're not explicitly specifying `no_release` option, I'd say you're fine and all your servers/roles are "release_roles".

Hope I helped and pls provide your config files if you have more questions!

Bruno

[Doug Hall]

You were close. I actually commented out my server definition, because of what the comments said. Here is my current deploy/staging.rb file, minus a few <variables>.
# ------------
set :rails_env, 'staging'  # For some reason, this is not being set automatically.

role :app, %w{<myUser>@apps2.aidt.edu}
role :web, %w{<myUser>@apps2.aidt.edu}
role :db,  %w{<myUser>@apps2.aidt.edu}

# Extended Server Syntax
# ======================
# This can be used to drop a more detailed server definition into the
# server list. The second argument is a, or duck-types, Hash and is
# used to set extended properties on the server.

# server 'example.com', user: 'deploy', roles: %w{web app}, my_property: :my_value

#--------------

As you can see, the "server" line is commented out, because I didn't need to set any server-specific variables.

Web/App and DB roles are all on the same server. My problem was that I had set the db role to the "postgres" user, because postgresql "best practices" say to do so. (This isolates the database from the web user, making it ostensibly more secure.) So now, my setup works, but is less secure. I don't really have any more questions, but maybe this clarification will help someone else. If there's an advantage to using a single "server" definition, as you prescribed, over what I did, let me know.

Thanks,
Doug

[Bruno Sutic]

Hey Doug,

I finally found some time today to do dig into capistrano souce code to check some assumptions around this. Your config helped steer the focus.

I first want to amend what I said (guessed) a couple posts above. So having the following in your config:

    server '<server_ip>', user: 'deploy', roles: %w{web app}

    server '<server_ip>', user: 'deploy', roles: %w{db}

It turns out capistrano does treat this as a *single* server (note the user and server ip are the same)!

• here's the method that adds a new server
• the above method invokes this one. It checks if the server is already added, or returns a new server.

Doug, your config:

    role :app, %w{<myUser>@apps2.aidt.edu}
    role :web, %w{<myUser>@apps2.aidt.edu}
    role :db,  %w{<myUser>@apps2.aidt.edu}

should also be recognized as a *single* server with 3 roles (app, web, db)! Tasks should not be executed redundantly 3 times.

I was playing with these configs on one of my apps and I could confirm that: even if there are multiple `server` or `role` declarations - capistrano will "merge" them if they have the same user, hostname and port.

Also, here's an idea how to use the 'postgres' user for the db role:

set the DB role to 'no_release'. That way, the db role does not have to worry about checking git strategy.
Invoking `cap staging git:check --trace` for the below setup passed for my test app. I'm able to reproduce your issue when `no_relase: true` setting is removed.

    role :app, %w{<myUser>@apps2.aidt.edu}
    role :web, %w{<myUser>@apps2.aidt.edu}

    role :db,  %w{postgres@apps2.aidt.edu}, no_release: true

Or maybe just leave things as-is since they're working LOL.

Anyway, sorry for the misinformation from the previous posts. This investigation helped me a bit to understand how capistrano works from the inside, hopefully it was informative to you too.

Bruno