If the gem is not available in the current environment, then the Rake task cannot be loaded. You can simply check the environment like this:
if RAILS_ENV == 'test'
require 'guard/jasmine/task'
Guard::JasmineTask.new
end
I for myself prefer the thor based command line utility `guard-jasmine` over the Rake task for the CI.