PowerDNS permanently crashing in Bosh (if hostname is used in PGSQL settings)

nobod...@gmail.com

unread,

May 26, 2014, 12:32:46 PM5/26/14

to bosh-...@cloudfoundry.org

Hi,

I'm trying to deploy Bosh from MicroBosh in AWS VPC at Sydney (bosh release 80, stemcell bosh-aws-xen-ubuntu-trusty-go_agent 2579).

Deployment fails with "bosh/0 not responding after update". In output of monit summary i find out, that PowerDNS is not running.

In /var/log/syslog every start of PowerDNS ends with such messages:

pdns[26791]: Creating backend connection for TCP

pdns[26791]: gpgsql Connection successful

pdns[26791]: About to create 3 backend threads for UDP

pdns[26791]: Got a signal 11, attempting to print trace:

pdns[26791]: [0x492180]

pdns[26791]: [0x695a60]

pdns[26791]: /lib/x86_64-linux-gnu/libnss_files.so.2(+0x4b2c) [0x7f9b491abb2c]

pdns[26791]: /lib/x86_64-linux-gnu/libnss_files.so.2(_nss_files_gethostbyname4_r+0xd3) [0x7f9b491acca3]

pdns[26791]: [0x98baa6]

pdns[26791]: [0x98d772]

pdns[26791]: [0x573bf7]

pdns[26791]: [0x569f2a]

pdns[26791]: [0x56a484]

pdns[26791]: [0x56a4a6]

pdns[26791]: [0x411e22]

pdns[26791]: [0x4126c4]

pdns[26791]: [0x40c8d0]

pdns[26791]: [0x4118e5]

pdns[26791]: [0x47c834]

pdns[26791]: [0x49abb2]

pdns[26791]: [0x458380]

pdns[26791]: [0x4cf18d]

pdns[26791]: [0x6912fd]

pdns[26791]: [0x9a8759]

Comparing MicroBosh and Bosh deployments i figured out, that PowerDNS is crashing if hostname is used as value of gpgsql-host option in file /var/vcap/jobs/powerdns/config/pdns.conf.

I.e. if "gpgsql-host: 0.bosh.default.bosh.microbosh" than PowerDNS crash with SIGSEGV. If same option with value 10.0.0.x all works fine.

In few latest Bosh releases PowerDNS comes in jobs/powerdns.tgz as deb package, which contains statically linked version of PowerDNS 3.0.1.

I tryed newer version from here http://downloads.powerdns.com/releases/deb/, and 3.1/3.2 is still crashing, but 3.3 no longer falling in same Bosh deployment (manually installed into AWS VPC instance).

Any advice with this? PowerDNS is builded without debug symbols and mentioned above backtrace don't give any clue why it's crashing with SEGV.

PS: info about my environment:

$ gem list bosh

*** LOCAL GEMS ***

bosh-bootstrap (0.12.0)

bosh-director-core (1.2576.0)

bosh-registry (1.2576.0)

bosh-stemcell (1.2576.0)

bosh_aws_cpi (1.2576.0)

bosh_cli (1.2576.0)

bosh_cli_plugin_aws (1.2576.0)

bosh_cli_plugin_micro (1.2576.0)

bosh_common (1.2576.0)

bosh_cpi (1.2576.0)

bosh_openstack_cpi (1.2576.0)

bosh_vcloud_cpi (0.5.4)

bosh_vsphere_cpi (1.2576.0)

$ cat bosh.yml

director_uuid: <UUID>

release:

version: latest

compilation:

workers: 3

network: default

reuse_compilation_vms: true

cloud_properties:

instance_type: m1.medium

availability_zone: ap-southeast-2a

update:

canaries: 1

canary_watch_time: 3000-120000

update_watch_time: 3000-120000

max_in_flight: 4

max_errors: 1

networks:

- name: elastic

type: vip

cloud_properties: {}

- name: default

type: manual

subnets:

- range: 10.0.0.0/24

static:

- 10.0.0.100 - 10.0.0.150

reserved:

- 10.0.0.2 - 10.00.0.15

dns:

- 10.0.0.6

- 8.8.8.8

cloud_properties:

subnet: <subnet-ID>

security_groups: <secgroup-ID>

resource_pools:

- name: medium

network: default

size: 1

stemcell:

version: latest

cloud_properties:

instance_type: m1.medium

availability_zone: ap-southeast-2a

jobs:

- name: bosh

template:

- powerdns

- nats

- postgres

- redis

- director

- blobstore

- registry

- health_monitor

instances: 1

resource_pool: medium

persistent_disk: 20480

networks:

- name: default

default: [dns, gateway]

- name: elastic

static_ips:

- xx.xx.xx.xx

properties:

env:

postgres: &bosh_db

user: postgres

password: postges

host: 0.bosh.default.bosh.microbosh

listen_address: 0.bosh.default.bosh.microbosh

database: bosh

dns:

address: xx.xx.xx.xx

db: *bosh_db

user: powerdns

password: powerdns

database:

webserver:

password: powerdns

replication:

basic_auth: replication:pass

user: replication

password: powerdns

recursor: zz.zz.zz.zz

redis:

address: 0.bosh.default.bosh.microbosh

password: redis

nats:

address: 0.bosh.default.bosh.microbosh

user: nats

password: nats

director:

address: 0.bosh.default.bosh.microbosh

db: *bosh_db

blobstore:

address: 0.bosh.default.bosh.microbosh

agent:

user: agent

password: agent

director:

user: director

password: director

registry:

address: 0.bosh.default.bosh.microbosh

db: *bosh_db

http:

user: registry

password: registry

hm:

http:

user: hm

password: hm

director_account:

user: admin

password: admin

event_nats_enabled: false

email_notifications: false

tsdb_enabled: false

pagerduty_enabled: false

varz_enabled: true

aws:

access_key_id: <AWS_KEY>

secret_access_key: <AWS_SECRET>

default_key_name: cf-bosh

region: ap-southeast-2

default_security_groups: <secgroup-ID>

Greg Oehmen

unread,

May 27, 2014, 1:56:27 AM5/27/14

to bosh-users

Hi "Nobody.mail":

I think you are running into an issue that is resolved when this story is completed: https://www.pivotaltracker.com/story/show/70452708

Given where that story sits in the backlog, we should actually be getting to it soon so please hang in there.

Best

Greg

Greg Oehmen

Cloud Foundry Product Manager - Bosh

Pivotal

To unsubscribe from this group and stop receiving emails from it, send an email to bosh-users+...@cloudfoundry.org.

Евгений Костенко

unread,

May 27, 2014, 8:27:35 AM5/27/14

to bosh-...@cloudfoundry.org

Thanks, Greg.

With your advice, I figured out, that stemcell/2579 with Ubuntu 10.04/lucid inside successfully deployed as Bosh and run normally.

Do i need to provide any additional info about stemcell with Ubuntu trusty inside?

For example, that PowerDNS is crashing with SIGSEGV, when hostname (not an IP address) used in PGSQL conenction address.

Or you planned to abandon from deb package with static PowerDNS in favor of compiling from sources as all other jobs?

вторник, 27 мая 2014 г., 9:56:27 UTC+4 пользователь Greg Oehmen написал:

Reply all

Reply to author

Forward