Ruby Compilation is failing

[Guruprakash Srinivasamurthy]

Hi Team,

We are upgrading to cloud Foundry version 197 and facing Ruby compilation error as below. Can you please help.

I suspect this to be some kind of a timeout issue. 

Error trace::

Started compiling packages > ruby/cd102a7fe440fd9eaeee99c6bc460b06884cbda6. Failed: Action Failed get_task: Task with id 4bc631cc-72ba-462e-7ab2-47c0b1f41ca3 could not be found (00:03:50)

Error 450001: Action Failed get_task: Task with id 4bc631cc-72ba-462e-7ab2-47c0b1f41ca3 could not be found

Debug Log::

E, [2015-04-24T16:43:57.760672 #4709] [task:378889] ERROR -- : Action Failed get_task: Task with id 4bc631cc-72ba-462e-7ab2-47c0b1f41ca3 could not be found

/var/vcap/packages/director/gem_home/ruby/2.1.0/gems/bosh-director-1.2732.0/lib/bosh/director/agent_client.rb:187:in `handle_method'

/var/vcap/packages/director/gem_home/ruby/2.1.0/gems/bosh-director-1.2732.0/lib/bosh/director/agent_client.rb:242:in `send_message'

/var/vcap/packages/director/gem_home/ruby/2.1.0/gems/bosh-director-1.2732.0/lib/bosh/director/agent_client.rb:61:in `block (2 levels) in <class:AgentClient>'

/var/vcap/packages/director/gem_home/ruby/2.1.0/gems/bosh-director-1.2732.0/lib/bosh/director/agent_client.rb:262:in `get_task_status'

/var/vcap/packages/director/gem_home/ruby/2.1.0/gems/bosh-director-1.2732.0/lib/bosh/director/agent_client.rb:111:in `wait_for_task'

/var/vcap/packages/director/gem_home/ruby/2.1.0/gems/bosh-director-1.2732.0/lib/bosh/director/agent_client.rb:254:in `send_long_running_message'

/var/vcap/packages/director/gem_home/ruby/2.1.0/gems/bosh-director-1.2732.0/lib/bosh/director/agent_client.rb:74:in `compile_package'

/var/vcap/packages/director/gem_home/ruby/2.1.0/gems/bosh-director-1.2732.0/lib/bosh/director/package_compiler.rb:83:in `block (2 levels) in compile_package'

/var/vcap/packages/director/gem_home/ruby/2.1.0/gems/bosh-director-1.2732.0/lib/bosh/director/package_compiler.rb:166:in `prepare_vm'

/var/vcap/packages/director/gem_home/ruby/2.1.0/gems/bosh-director-1.2732.0/lib/bosh/director/package_compiler.rb:80:in `block in compile_package'

/var/vcap/packages/director/gem_home/ruby/2.1.0/gems/bosh-director-1.2732.0/lib/bosh/director/lock_helper.rb:89:in `block in with_compile_lock'

/var/vcap/packages/director/gem_home/ruby/2.1.0/gems/bosh-director-1.2732.0/lib/bosh/director/lock.rb:58:in `lock'

/var/vcap/packages/director/gem_home/ruby/2.1.0/gems/bosh-director-1.2732.0/lib/bosh/director/lock_helper.rb:89:in `with_compile_lock'

/var/vcap/packages/director/gem_home/ruby/2.1.0/gems/bosh-director-1.2732.0/lib/bosh/director/package_compiler.rb:73:in `compile_package'

/var/vcap/packages/director/gem_home/ruby/2.1.0/gems/bosh-director-1.2732.0/lib/bosh/director/package_compiler.rb:300:in `block (2 levels) in process_task'

/var/vcap/packages/director/gem_home/ruby/2.1.0/gems/bosh-director-1.2732.0/lib/bosh/director/event_log.rb:83:in `call'

/var/vcap/packages/director/gem_home/ruby/2.1.0/gems/bosh-director-1.2732.0/lib/bosh/director/event_log.rb:83:in `advance_and_track'

/var/vcap/packages/director/gem_home/ruby/2.1.0/gems/bosh-director-1.2732.0/lib/bosh/director/event_log.rb:36:in `track'

/var/vcap/packages/director/gem_home/ruby/2.1.0/gems/bosh-director-1.2732.0/lib/bosh/director/package_compiler.rb:298:in `block in process_task'

/var/vcap/packages/director/gem_home/ruby/2.1.0/gems/bosh_common-1.2732.0/lib/common/thread_formatter.rb:46:in `with_thread_name'

/var/vcap/packages/director/gem_home/ruby/2.1.0/gems/bosh-director-1.2732.0/lib/bosh/director/package_compiler.rb:294:in `process_task'

/var/vcap/packages/director/gem_home/ruby/2.1.0/gems/bosh-director-1.2732.0/lib/bosh/director/package_compiler.rb:254:in `block (4 levels) in compile_packages'

/var/vcap/packages/director/gem_home/ruby/2.1.0/gems/bosh_common-1.2732.0/lib/common/thread_pool.rb:77:in `call'

/var/vcap/packages/director/gem_home/ruby/2.1.0/gems/bosh_common-1.2732.0/lib/common/thread_pool.rb:77:in `block (2 levels) in create_thread'

/var/vcap/packages/director/gem_home/ruby/2.1.0/gems/bosh_common-1.2732.0/lib/common/thread_pool.rb:63:in `loop'

/var/vcap/packages/director/gem_home/ruby/2.1.0/gems/bosh_common-1.2732.0/lib/common/thread_pool.rb:63:in `block in create_thread'

D, [2015-04-24T16:43:57.761414 #4709] [task:378889] DEBUG -- : (0.000138s) BEGIN

D, [2015-04-24T16:43:57.763057 #4709] [0x3fa34a528118] DEBUG -- : Lock renewal thread exiting

D, [2015-04-24T16:43:57.763646 #4709] [task:378889] DEBUG -- : (0.000637s) UPDATE "tasks" SET "state" = 'error', "timestamp" = '2015-04-24 16:43:57.760877+0000', "description" = 'create deployment', "result" = 'Action Failed get_task: Task with id 4bc631cc-72ba-462e-7ab2-47c0b1f41ca3 could not be found', "output" = '/var/vcap/store/director/tasks/378889', "user_id" = NULL, "checkpoint_time" = '2015-04-24 16:43:30.409449+0000', "type" = 'update_deployment' WHERE ("id" = 378889)

D, [2015-04-24T16:43:57.765812 #4709] [task:378889] DEBUG -- : (0.002008s) COMMIT

I, [2015-04-24T16:43:57.766070 #4709] [0x3fa348907330]  INFO -- : Task took 3 minutes 59.42649563500001 seconds to process.

Thanks

[Dmitriy Kalinin]

I would recommend getting on the compilation VMs and seeing what happens in /var/vcap/bosh/log/current. 'Task with id 4bc631cc-72ba-462e-7ab2-47c0b1f41ca3 could not be found' indicates that the Agent got restarted.

[Guruprakash Srinivasamurthy]

We got over this issue after tweaking the network config in the deployment YML. Issue was with the IP's used by compilation vms were already being occupied.

I have a followup question to understand whether if the bosh will ping/do some some sort of nmap test to see if the Ip's are available in the range for to be used for compilation vms. 

On Friday, April 24, 2015 at 9:49:40 AM UTC-7, Guruprakash Srinivasamurthy wrote: