Load mongodb json fixture files via capistrano : pain and suffering ...

[pixelboy]

Hi,

I must admit I'm not sure wether this issue I'm having is relative to capistrano or moreover ruby (being a beginnner in both). But here's the deal.
I'm currently working on a php application using mongodb driven data, and capistrano to deploy on each and every env and server we have.


My main problem is I just can't figure out how to have compliant deployments when it comes to our database. I've figured out fixture files, using json, would be cool enough, using the mongoimport command.

Heres the snippet I've come up with :

namespace :database do
  desc "Load 'em fixtures"
  task :dbcleanup do
    set :database_name, "application"
    set :fixtures_directory, "#{current_release}/fixtures/#{stage}"
    set :mongo_import, "mongoimport -d #{database_name} -c foo"
   
    jsonfiles = File.join("#{fixtures_directory}", "*.json")
    puts("#{jsonfiles}")

    files = File.join("#{fixtures_directory}","**","*.json")
    Dir.glob(files).each do |file|
      puts("#{file}")
    end

    logger.debug "Ended"

  end
end

Don 't bother telling me mongoimport won't work untill I've come up with a way to get the collection name, I know that.
We're not there yet ;)

Could someone tell me how come I get an output with the puts("#{jsonfiles}"), and not with the puts("#{file}") ??

Heres the output of my console.

[...]
 ** transaction: commit
  * executing `database:dbcleanup'
/home/server/www/application/staging/releases/20120224112251/fixtures/staging/*.json
  * Ended
[...]

Where am I wrong ?? Thanks a lot for the help, I'm pretty sure it's not much really.

[Lee Hambley]

The line:

     jsonfiles = File.join("#{fixtures_directory}", "*.json"

Runs on the machine from which you launch Capistrano, not on the server, so unless those files exist on your machine, that's not going to work for you. Use something like remote shell globbing. (I'd give an example, but you don't include an example of what you're trying to do here)

- Lee

--
* You received this message because you are subscribed to the Google Groups "Capistrano" group.
* To post to this group, send email to capis...@googlegroups.com
* To unsubscribe from this group, send email to capistrano+...@googlegroups.com For more options, visit this group at http://groups.google.com/group/capistrano?hl=en

[romain Simiand]

Hi there,

thanks for the info.
So if I get you well, all I need to do is make

File.join("#{fixtures_directory}", "*.json"

run on #{current_release} server.

Could I do that by simply applying the task to the web server, using :

task :dbcleanup do, :roles => :app

like in the example below ?
Or am I missing something crucial here ?

Also, here's an example of what I'm trying to do precisely



namespace :database do
  desc "Load 'em fixtures"

  task :dbcleanup, :roles => :web do
    set :database_name, "dbname"

    set :fixtures_directory, "#{current_release}/fixtures/#{stage}"

    set :mongo_import, "mongoimport -d #{database_name} -c #{file}"

    
    jsonfiles = File.join("#{fixtures_directory}", "*.json")
    puts("#{jsonfiles}")

    files = File.join("#{fixtures_directory}","**","*.json")
    Dir.glob(files).each do |file|

      run "#{mongo_import}"

    end

    logger.debug "Ended"
  end   
end

Am I clearer ?

[Lee Hambley]

I'd build this as a rake task (then it all runs on the server, or your workstation, and you can test it, etc) and then simply call it from Capistrano.

You're missing something fundamental in that anything outside of `run` happens on *your machine*, and anything in `run/capture()` runs on the server you are connected to. So, in your example, you're collecting a list of files which don't exist on your machine, and then running a task (with an empty list) on your server.

You'd be better served adding a Raketask to your application (rails uses `rake db:seed`, for loading runtime fixtures, for example) and then simply calling that when you do the deployment.

--

[romain Simiand]

If you have some simple examples / webpages, I'd be ok to read them, as I said, I'm not familiar with ruby/rails, and even less rake tasks :( and how to implement them in capistrano ...

[Donovan Bray]

I agree with lee that this would be better as a rake task. 

I've got some other lessons for you

You correctly said you would target the :app role but your snippet targets the :web role, :web. Is generally your front end lb; and your code shouldn't be on a lb. 

File.join("#{fixtures_directory}", "*.json")

Is not the same as

File.join("#{fixtures_directory}","**","*.json)

the /**/ will match any file spec to the right arbitrarily deep. Ie a recursive match. 

It's not necessary to quote a single variable inside a string to pass to a method

File.join("#{fixtures_directory}","**","*.json) == File.join(fixtures_directory,"**","*.json)

File.join in this context is actually dangerous; it's job is to normalize the filespec separator on the ruby system it is running on; if this string you are building is to be executed on the system running this ruby process then fine. But if say you were on windows where the filespec separator is a backslash but you were sending the string to be executed on a Linux box the separator would be wrong. 

jsonfiles = File.join("#{fixtures_directory}", "*.json")
    puts("#{jsonfiles}")

The above does nothing in your script; it's never used and it's output will not match what you actually glob. 

As lee mentioned before unless it's in a run/sudo/capture the methods are all running in the context of your local machine. 

You have the following set:

set :mongo_import, "mongoimport -d #{database_name} -c #{file}"
    

'file' in this case will always be nil because it doesn't have a value when ruby executes that set; caps deferred execution won't help you on this case either because it will only use the value the first time the variable is invoked. 

If you want a list of remote files you'll need to build the list remotely, something like the following could work

set :mongo_import, "mongoimport -d #{database_name} -c $(find #{fixtures_directory} -name *.json)"

Also

set :fixtures_directory, "#{current_release}/fixtures/#{stage}"

You should almost always use latest_release instead of current_release because current_release is only right during a deploy; latest_release will work both during a deploy and if the task is called outside of the deploy cycle. 

Last but not least this method will work fine while you only have one app server; the moment you have more than one it will run on each server in parallel you should protect it by using :once

you can remove all the dir settings and surrounding glob since you are building the list remotely

run "#{mongo_import}", :once => true

--

[Romain Simiand]

I'd like to thank you both for helping with this trickier than it looks issue.

And also to review this snippet :

set :mongo_import, "mongoimport -d #{database_name} -c $(find #{fixtures_directory} -name *.json)"

How can I find more input on how to use the $() ? Can I use it to isolate a more complex cli command (I believe the find command is cli command right ?)

Because basically, now, while running the following

namespace :database do
  desc "Load 'em fixtures"

  task :dbcleanup, :roles => :db do
    set :database_name, "database"
    set :fixtures_directory, "#{latest_release}/fixtures"

    set :mongo_import, "mongoimport -d #{database_name} -c $(find #{fixtures_directory} -name *.json)"

    run "#{mongo_import}", :once => true

  end
end

I get the expected behaviour in my console output, that is it errors a missing parameter, the collections' name.

Again, if you have something relative to rake tasks writing and tangling with capistrano, I'd be happy to read that, because I'm pretty sure, in my specific use case, I'm trying to do to much work in my single deploy.rb file.