CF custom buildpack failing to stage "Staging error: no available stagers"

[Troy Astle]

I am trying to use the following custom buildpack on my CF installation https://github.com/dmikusa-pivotal/cf-test-buildpack

This works on my one my environments:

• VSphere
• CF140
• Stemcell 0.8.0

It fails in my other environment:

• AWS
• CF144
• Stemcell 1226

In the failing environment I get the following error back from the trace:

  "code": 170001,
  "description": "Staging error: no available stagers"

Has anyone seen this before?

More detailed error logs here:

https://gist.github.com/trastle/7310869

Thanks.

Troy.

[Troy Astle]

Looks like the error is being thrown out from here:

https://github.com/cloudfoundry/cloud_controller_ng/blob/7fec12217257428eac002fe939d1e7d330fbe330/lib/cloud_controller/app_stager_task.rb#L44

No stagers in the stager pool able to handle the app.

Seems odd.

[James Bayer]

this line:

https://github.com/cloudfoundry/cloud_controller_ng/blob/7fec12217257428eac002fe939d1e7d330fbe330/lib/cloud_controller/app_stager_task.rb#L43

will try and make sure there is a stager (DEA) configured with your stack (assuming you did not set one explicitly therefore it takes the default stack of lucid64?).

a dea/stager must also have at least 1024MB of memory free in order to stage

are you sure your AWS env meets those above conditions?

To unsubscribe from this group and stop receiving emails from it, send an email to vcap-dev+u...@cloudfoundry.org.

--

Thank you,

James Bayer

[Troy Astle]

James,

Thank you.

You are correct. 

I was looking in the wrong place. 

 - My initial guess was the stack and not the RAM. After adding some debug statements into the CC the stack was fine.

 - For anyone else who hits this issue my memory_mb: property in the DEA was not set to use all of the available RAM on my AWS instance and it was hence under-utilised.

Details for anyone who finds this thread later with the same issue:

A more accurate description of the issue others may face if they have also made my error:

1) CF is deployed fine and looks good.

2) First test app stages correctly and runs

3) All sequent apps fail to stage with the error:

RESPONSE_BODY:
{
  "code": 170001,
  "description": "Staging error: no available stagers"
}

For me the problem was that I was not allowing the DEA process to utilise the full memory available to the DEA VM.

Resolved.

Thanks again.

Troy.

--
Troy.

[Dr Nic Williams]

"a dea/stager must also have at least 1024MB"

Ahh that'd explain why my m1.small which was configured with 1.5G couldn't  stage a second app even though it should have been able too. Thanks for noting that!

[Dr Nic Williams]

Troy, also make sure you set the disk attribute of the deas to be huge - ec2 instances have large ephemeral disks. Bosh-CF does not yet put in the large ec2 disk sizes. Need to fix that too.

[Nguyen Anh Tu]

Troy, James & Dr.Nic,

Could you help to explain my situation? I got the same issue "Staging error: no available stagers". Here is my DEA configs:

  dea_next:

    allow_networks:

    - 10.10.0.2/32

    deny_networks:

    - 169.254.0.0/16

    - 10.10.0.0/16

    directory_server_protocol: http

    disk_mb: 2048

    disk_overcommit_factor: 2

    evacuation_bail_out_time_in_seconds: 600

    instance_disk_inode_limit: 200000

    kernel_network_tuning_enabled: true

    logging_level: debug

    memory_mb: 8192

    memory_overcommit_factor: 3

    staging_disk_inode_limit: 200000

    staging_disk_limit_mb: 8192

    staging_memory_limit_mb: 1024

[Youzhi Zhu]

hi Nguyen Anh Tu

 I am not sure whether it's a bug, but I have met the same problem, the method to solve this problem is to create an offline buildpack.

get the buildpack package and use the "cf create-buildpack" cmmoand.

--
You received this message because you are subscribed to the Google Groups "Cloud Foundry Developers" group.
To view this discussion on the web visit https://groups.google.com/a/cloudfoundry.org/d/msgid/vcap-dev/06f5691c-8a55-4fb1-b14f-8ea4aaf6be33%40cloudfoundry.org.

[Alexander Lomov]

Hi. 

I guess it could be because your DEA has no available resource to work with. 

Could you check it?

To view this discussion on the web visit https://groups.google.com/a/cloudfoundry.org/d/msgid/vcap-dev/CAGO-E6q-JxbPqkxYHTsvBytohdSTcWq%2BV0-HgxyP64iykSA7EQ%40mail.gmail.com.