Trying to connect to postgresql with bosh-lite and cf-services-contrib

[pepone]

Hello bosh users,

since two days I'm facing a connection problem when I try to deploy an application on a local cloudfoundry running on bosh-lite warden in virtualbox. I hope you guys can help me out.

I followed instructions found on the github readme pages, bosh-lite with virtualbox and the cloudfoundry services contrib release.

Everything worked fine several times (tried different cf-releases, now I'm on the latest: cf-179), even the application deployment via the cloudfoundry cli works without problems except the connection to the database in a service.

My steps so far:

Successfully created an organisation and space via the cli (followed steps on the bosh-lite github readme).

Deployed the cf-contrib-release with the default manifest created by the template:

$ templates/make_manifest warden

$ bosh -n deploy

Then authenticated the cli:

$ cf create-service-auth-token postgresql core c1oudc0w

(I'm not sure if "core" is correct, but authentication works with this command anyway)

Created a new service:

$ cf create-service postgresql default my-datasource

and bound it to my application (java spring boot app btw):

$ cf bind-service my-application my-datasource

Everything returns OK and there is indeed created a new database on postgres_node/0 machine.

In the application I use the spring-cloudfoundry-connector to automatically receive the datasource.

When I connect to the loggregator stream of this app, I can see the logged datasource received in the app (so the environment variables seem to be set):

... url=jdbc:postgresql://10.244.1.90:5435/d7d751c4a915e4012adce1b461e65348a?user=ua6ca2d523ad2425581b17feeeddc6b92&password=p8fee01e49a934308bf31724d06e58b08; ...

But the connection itself is refused everytime inside the application:

Unable to get connection:

org.postgresql.util.PSQLException: Connection refused. Check that the hostname and port are correct and that the postmaster is accepting TCP/IP connections

When I do a manual connection via command line psql, I can login without any problems to the database given in this datasource from inside the host machine.

I found a good guide on how to connect to a warden container here and successfully connected via ssh to the warden instance where the application runs. There I cannot connect to postgres: Connection refused, like in the app itself.

So the service is actually somehow created and bound to the application (i got the right parameters as env-variables and even the datasource with the connector), I can connect to it manually from the host machine (the bosh-lite virtualbox), but inside the warden container connections are refused.

Because of the subnet of the default cf-services-contrib manifest for warden (10.244.1.x instead of 10.244.0.x) I even tried to change some service settings in the postgresql_node config. In the file:

cf-services-contrib-release/jobs/postgresql_node_ng/templates/postgresql_ctl.erb

I tried to allow connections from all hosts with:

echo "host all all 0.0.0.0/0 md5" >> $pg_hba_conf_file

instead of the other echos writing the postgresql connection settings.

But still no luck.

Could you guys give me a hint what I am missing? How does that service binding actually forward requests to and from the service, via the router_z1?

I got the impression that only one little step is missing or silently failing.

Any help or hint much appreciated!

If you need some more information about my setup just let me know.

Cheerz,

leo

[pepone]

Additional Info:

$ gem list bosh

*** LOCAL GEMS ***

bosh_cli (1.2334.0)

bosh_common (1.2334.0)

$ bosh status

Config

             /home/pepone/.bosh_config

Director

  Name       Bosh Lite Director

  URL        https://192.168.50.4:25555

  Version    1.2641.0 (13a474c4)

  User       admin

  UUID       9dd8ea1e-36ec-4f00-aa91-05ec4e7d496f

  CPI        warden

  dns        disabled

  compiled_package_cache enabled (provider: local)

  snapshots  disabled

Deployment

  Manifest   /home/pepone/workspace/bosh/bosh-lite/manifests/cf-manifest.yml

The deployment manifest of the spring boot application:

---

applications:

- name: my-application

  memory: 512M

  instances: 1

  host: application

  path: target/my-application-1.0.0-SNAPSHOT.jar

  spring_profiles: cloud

  services:

    - my-datasource

[Dr Nic Williams]

Perhaps re-post this on vcap-dev where the CF devs might have some insight into the regression you're seeing?

To unsubscribe from this group and stop receiving emails from it, send an email to bosh-users+...@cloudfoundry.org.

[pepone]

Hi Dr Nic,

Thanks for your reply, good idea. This issue might be more related to cf internal routing stuff.

Greetings

[Dr Nic Williams]

I don't think there is an facility in bosh-lite to prevent one warden container from communicating with another. So I think the issue must be in cf configuration itself? 

--

Dr Nic Williams

Stark & Wayne LLC - consultancy for Cloud Foundry users

http://drnicwilliams.com

http://starkandwayne.com

cell +1 (415) 860-2185

twitter @drnic

[Dmitriy Kalinin]

My guess would be that it's app security rules that prohibit communication from inside the container. They were recently introduced in cf-release. You can adjust them via cc.security_group_definitions property (cloud controller job).

[pepone]

Great, thank you guys for the hints! Will check that security configurations.

[pepone]

Excellent! Got it working within half an hour! 

You are completely right with the security group restrictions. In my cf-manifest.yml I configured an additional security group:

...

security_group_definitions:

   ...

   - name: services

      rules:

      - destination: 10.244.0.0/19

        protocol: all

...

Additionally in the same file I added this group to the cc default groups:

...

properties:

  cc:

   ..

   default_running_security_groups:

    - public_networks

    - dns

    - services

    default_staging_security_groups:

    - public_networks

    - dns

    - services

...

Afterwards redeploed the cf-release. Finally a group binding to the org/space where the application runs:

$ cf bind-security-group services my-org development

and voilà! The application can connect to the service without any issues.

Thank you very much for giving me the hint!

Greetings,

pepone

Am Mittwoch, 20. August 2014 23:49:02 UTC+2 schrieb Dmitriy Kalinin:

[dhof...@pivotal.io]

or on a already setup and running system:

$ echo '[{"protocol":"all","destination":"10.244.0.0/19"}]' > ./mySecurityGroup.txt

$ cf create-security-group services cf-security-groups.txt

$ cf security-groups

$ cf bind-security-group services <org> development

$ cf bind-running-security-group services

$ cf bind-staging-security-group services 

$ cf security-groups

$ cf staging-security-groups

$ cf running-security-groups