custom buildpacks and dependency libs

[mira...@gmail.com]

Good day,

I'm just curious what would be the best way to implement the following workflow.

We have ruby application which is using the gem and this gem require system lib which could be installed via apt-get install <lib> command.

So i downloaded sources, put them into S3 bucket, created simple custom buildpack, which is getting url from .vendor_urls and create .buildpacks file with two buildpacks urls:

1. my custom buildpack

2. ruby buildpack

then i run cf push app1 -b http://<multibuildpack>

The question is how to implement compile phase in custom buildpack.

Should I compile this lib from sources inside the cf on staging step or compile this lib and then put binaries to S3 bucket and then install somehow?

Currently I tried to compile from sources , but there are permission issues when running command make and make install , they are required sudo I believe.

Before did that I went through these docs:

https://devcenter.heroku.com/articles/buildpack-api#example-buildpacks

http://docs.cloudfoundry.org/concepts/architecture/warden.html

my compile file content:

#!/bin/bash

indent() {

  sed -u 's/^/       /'

}

echo "-----> Found a .vendor_urls file"

# Bail early but noisily

if [ ! -s $1/.vendor_urls ]; then

  echo ".vendor_urls empty. Skipping." | indent

  exit 0

fi

cd $1

while read url; do

  if [ -n "$url" ]; then # incase ensure_newline_at_eof_on_save is enabled

    echo Vendoring $url | indent

    curl $url -s -o - | tar xzf -

    cd cyrus-sasl-2.1.25

    ./configure --prefix=${BUILD_DIR}

    make

    make install

  fi

done < .vendor_urls

Also I'm not totally sure how to pass variables CACHE_DIR, BUILD_DIR, ENV_DIR, as I undestand my app should read my lib from env variable path.

Would be appreciated for any help.

[Daniel Mikusa]

On Mon, Jul 28, 2014 at 8:52 AM, <mira...@gmail.com> wrote:

Good day,

I'm just curious what would be the best way to implement the following workflow.

We have ruby application which is using the gem and this gem require system lib which could be installed via apt-get install <lib> command.

So i downloaded sources, put them into S3 bucket, created simple custom buildpack, which is getting url from .vendor_urls and create .buildpacks file with two buildpacks urls:

1. my custom buildpack

2. ruby buildpack

then i run cf push app1 -b http://<multibuildpack>

The question is how to implement compile phase in custom buildpack.

Should I compile this lib from sources inside the cf on staging step or compile this lib and then put binaries to S3 bucket and then install somehow?

If you have to compile, you should probably do it once, save the binaries and download them when you need them.  Compiling every time would add overhead to the staging process.  

Maybe you don't even need to compile though.  You said there's an Ubuntu deb available.  You could probably just extract the files from that and repackage them in a format friendly to your build pack.

 

Currently I tried to compile from sources , but there are permission issues when running command make and make install , they are required sudo I believe.

That won't work.  You can't get sudo or root access in staging.  Even if you could, it probably wouldn't help since the staging environment is different from the environment where your application runs.  If you were to build and install something in staging, it wouldn't be in the environment where your application runs.  Everything needed by your application at runtime needs to be placed in the build directory (i.e. what CF turns into your droplet) during staging.  

Given this, you'd need to build the library, place it in the build directory and configure whatever uses that library at runtime to look for it in the application directory, which is where your droplet gets extracted at runtime.

Dan

 

--
You received this message because you are subscribed to the Google Groups "Cloud Foundry Developers" group.
To view this discussion on the web visit https://groups.google.com/a/cloudfoundry.org/d/msgid/vcap-dev/bb6de27e-8974-4466-9e98-70130ab80f12%40cloudfoundry.org.

To unsubscribe from this group and stop receiving emails from it, send an email to vcap-dev+u...@cloudfoundry.org.

[iamflying]

Hi Dan,

regarding your explanation, "you'd need to build the library, place it in the build directory and configure whatever uses that library at runtime to look for it in the application directory, which is where your droplet gets extracted at runtime.", should I understand like this?
1. in warden run-time, everything generated in staging will be extracted under /home/vcap/app only
2. no way to install and start a software like openvpn client inside warden through customized buildpack

Actually, I have a requirement to install openvpn and configure it inside warden(I am trying through customized buildpack) so that the application inside the warden container could connect to another network through VPN.  It seems that buildpack cannot make it.

Could you share some insights how to achieve my goal?

Thanks.

[iamflying]

Pasted the answer from DAN in case someone has the same questions.

On Mon, Dec 8, 2014 at 11:27 PM, iamflying <guangc...@gmail.com> wrote:

Hi Dan,

regarding your explanation, "you'd need to build the library, place it in the build directory and configure whatever uses that library at runtime to look for it in the application directory, which is where your droplet gets extracted at runtime.", should I understand like this?
1. in warden run-time, everything generated in staging will be extracted under /home/vcap/app only

Correct.  When the compile script of your build pack runs, it's given a "build directory" as the first argument.  The build pack should put any files that the application will need at runtime into this directory.  As an example, if you're running a Java web app it would install the JVM and a container to run the app.  The system will save this directory as the "droplet".  The droplet is then extracted to `/home/vcap/app` in the runtime container.

This means that anything you install will be installed under `/home/vcap/app`.  Generally this works just fine, but you might have to set some environment variables (PATH, LD_LIBRARY_PATH, etc...) or pass extra configuration options to instruct software where to find things like binaries, libraries and configuration files since many apps assume certain default directories like `/etc/` for configuration and `/usr/lib` for libraries.

 

2. no way to install and start a software like openvpn client inside warden through customized buildpack

I can't speak for this application because I haven't tried it.  What I can tell you is that there's no way to use traditional package managers to install the software that you need.  These all require root access and you will not have that when your build pack runs or at application runtime.  

When I want to use a package from a package manager like this, here's what I'll typically do.

1.) Fire up a Ubuntu Lucid VM since this matches the current environment

2.) Install the package that I want.

3.) Locate the binaries I need.

4.) Run ldd on the binaries to get the list of required shared libraries.

5.) Copy the binaries, shared libraries and any configuration I need into a "vendor" directory.

   Ex:

       vendor/{usr,etc,lib}

6.) When I've grabbed everything, I'll zip that up and put in on an HTTP server.

7.) In the build pack, I can then download and extract the files placing them into the build directory.

8.) Adjust the release script to issue any commands that need to be run to execute this new app.

Sometimes if the package is small and doesn't have a lot of dependencies, I'll skip 1-4 and just download the Ubuntu Lucid deb file and extract its files with `ar`.  It's usually easier if you don't have a lot of dependencies or if you already know the dependencies.

The alternative to the steps above is to grab the source for your package and build it yourself.  Sometimes this is necessary as you have to recompile some apps to change the paths where they look for things like configuration files or where they write logs.  Not sure if it'll be necessary here, just mentioning it as an alternative.

 

Actually, I have a requirement to install openvpn and configure it inside warden(I am trying through customized buildpack) so that the application inside the warden container could connect to another network through VPN.  It seems that buildpack cannot make it.

You could try the steps above.  I haven't tried openvpn though, so I can't say how difficult it'll be to get running.

Dan

[iamflying]

Hi Dan,

Thank very much for your guide.

Now, I practiced all the steps with a customized java buildpack (cloned from https://github.com/cloudfoundry/java-buildpack), but I did not see the extra application started inside warden after "push app".

Below is the buildpack log. I did see the yml content from the log seems to be expected. could you please help?

---
addons: []
config_vars: {}
default_process_types:
  web: JAVA_HOME=$PWD/.java-buildpack/open_jdk_jre JAVA_OPTS="-Djava.io.tmpdir=$TMPDIR
    -XX:OnOutOfMemoryError=$PWD/.java-buildpack/open_jdk_jre/bin/killjava.sh -Xmx160M
    -Xms160M -XX:MaxMetaspaceSize=64M -XX:MetaspaceSize=64M -Xss853K -Daccess.logging.enabled=false
    -Dhttp.port=$PORT" $PWD/.java-buildpack/tomcat/bin/catalina.sh run
  vpn: /home/vcap/app/vendor/bin/start-openvpn-client.sh
  test: /home/vcap/app/vendor/bin/test.sh


Inside warden, the scripts exist there. For example, test.sh is for simple testing purpose. start-openvpn-client.sh is my script to start openvpn. I did not see either is running inside warden. what's wrong with my custom java buildpack?

vcap@186vl6tuq22:~/app$ cat vendor/bin/test.sh
#!/bin/bash

ping -i 2 localhost >> ping.log &


vcap@186vl6tuq22:~/app$ cat vendor/bin/start-openvpn-client.sh
#!/bin/bash

LD_LIBRARY_PATH=`echo $LD_LIBRARY_PATH`
VENDOR_PWD=`pwd && cd ..`
for i in $(find $VENDOR_PWD -type f -name "*.so*"); do
  lib_path="$(dirname $i)"

  if [[ "$LD_LIBRARY_PATH" != *"$lib_path"* ]]
  then
     LD_LIBRARY_PATH="$lib_path:$LD_LIBRARY_PATH"
  fi

done

export LD_LIBRARY_PATH=$LD_LIBRARY_PATH
$VENDOR_PWD/usr/sbin/openvpn --daemon --config $VENDOR_PWD/etc/openvpn/client.conf --tmp-dir $VENDOR_PWD &

[Daniel Mikusa]

On Thu, Dec 11, 2014 at 10:04 PM, iamflying <guangc...@gmail.com> wrote:

Hi Dan,

Thank very much for your guide.

Now, I practiced all the steps with a customized java buildpack (cloned from https://github.com/cloudfoundry/java-buildpack), but I did not see the extra application started inside warden after "push app".

Below is the buildpack log. I did see the yml content from the log seems to be expected. could you please help?

---
addons: []
config_vars: {}
default_process_types:
  web: JAVA_HOME=$PWD/.java-buildpack/open_jdk_jre JAVA_OPTS="-Djava.io.tmpdir=$TMPDIR
    -XX:OnOutOfMemoryError=$PWD/.java-buildpack/open_jdk_jre/bin/killjava.sh -Xmx160M
    -Xms160M -XX:MaxMetaspaceSize=64M -XX:MetaspaceSize=64M -Xss853K -Daccess.logging.enabled=false
    -Dhttp.port=$PORT" $PWD/.java-buildpack/tomcat/bin/catalina.sh run
  vpn: /home/vcap/app/vendor/bin/start-openvpn-client.sh
  test: /home/vcap/app/vendor/bin/test.sh

My understanding is that the only process type available in the "default_process_types" section is "web" [1].  One thing you could do instead is insert your commands into the "web" command before everything that's presently there.

Ex:

web: /home/vcap/app/vendor/bin/test.sh && /home/vcap/app/vendor/bin/start-openvpn-client.sh && JAVA_HOME=$PWD/.java-buildpack/open_jdk_jre JAVA_OPTS="-Djava.io.tmpdir=$TMPDIR -XX:OnOutOfMemoryError=$PWD/.java-buildpack/open_jdk_jre/bin/killjava.sh -Xmx160M -Xms160M -XX:MaxMetaspaceSize=64M -XX:MetaspaceSize=64M -Xss853K -Daccess.logging.enabled=false -Dhttp.port=$PORT" $PWD/.java-buildpack/tomcat/bin/catalina.sh run

This has a drawback though, because test.sh and start-openvpn-client.sh need to either finish or go to the background before the last process (i.e. Tomcat) will start, which means if one of them crashes the system won't notice and won't automatically restart your app.

There are ways to get around this limitation, but it's more work.  See James' post here [2], which provides an example of how to do this.

Dan

 

[1] - http://docs.cloudfoundry.org/buildpacks/custom.html#release-script

[2] - https://groups.google.com/a/cloudfoundry.org/d/msg/vcap-dev/Uq-d1G72jx0/ftAWzjI8Tv8J

[Daniel Watrous]

I created a series of posts that walk through the buildpack creation process:

http://software.danielwatrous.com/custom-buildpacks-in-cloudfoundry/

Some of those posts focus on Docker as implemented in Stackato, but the same should be possible using Warden.

Daniel

[iamflying]

Hi Dan,

Thanks for your great guide. However, I am still blocked on multiple process running inside the warden.  I got the following errors when I tried according to your reference. Could you please help check if I missed something? Thanks.

1. I got the error "failed to accept connections within health check timeout" when I combined a simple shell script with the original command in my java buildpack.  I have not experimented on other buildpack. Not sure if this is something special to java buildpack regarding to the health check.
---
addons: []
config_vars: {}
default_process_types:
  web: ! ' /home/vcap/app/vendor/bin/test.sh && JAVA_HOME=$PWD/.java-buildpack/open_jdk_jre

    JAVA_OPTS="-Djava.io.tmpdir=$TMPDIR -XX:OnOutOfMemoryError=$PWD/.java-buildpack/open_jdk_jre/bin/killjava.sh
    -Xmx160M -Xms160M -XX:MaxMetaspaceSize=64M -XX:MetaspaceSize=64M -Xss853K -Daccess.logging.enabled=false

    -Dhttp.port=$PORT" $PWD/.java-buildpack/tomcat/bin/catalina.sh run  '


2014-12-15T06:12:05.63+0000 [DEA]     ERR Instance (index 0) failed to start accepting connections
2014-12-15T06:12:05.64+0000 [API]     OUT App instance exited with guid 5874eabf-7c26-4fac-8c23-46ea1397c2b4 payload: {"cc_partition"=>"default", "droplet"=>"5874eabf-7c26-4fac-8c23-46ea1397c2b4", "version"=>"e673de3c-a4e2-4de3-85f5-c95480ed1c4f", "instance"=>"34cb92c0c08d41e8b7dba34ee88ae0b8", "index"=>0, "reason"=>"CRASHED", "exit_status"=>-1, "exit_description"=>"failed to accept connections within health check timeout", "crash_timestamp"=>1418623925}
2014-12-15T06:12:05.70+0000 [App/0]   ERR

2.  got the similar error when I experimented on James's solution.
---
addons: []
config_vars: {}
default_process_types:
  web: ! '/home/vcap/app/startAndExitOnAnyChildExit.sh ''sleep 60'' /home/vcap/app/ticket  '
 
 
2014-12-15T04:34:55.39+0000 [DEA]     OUT Starting app instance (index 0) with guid ab19c073-8fa2-4e11-97cb-8911de80c739
2014-12-15T04:34:58.68+0000 [App/0]   OUT Process "sleep 60" started with pid 33
2014-12-15T04:34:58.68+0000 [App/0]   OUT Process "/home/vcap/app/ticket" started with pid 34
2014-12-15T04:34:58.71+0000 [App/0]   OUT tick: [] -- FOO:
2014-12-15T04:34:59.71+0000 [App/0]   OUT tick: [] -- FOO:
2014-12-15T04:35:00.71+0000 [App/0]   OUT tick: [] -- FOO:
2014-12-15T04:35:01.71+0000 [App/0]   OUT tick: [] -- FOO:
2014-12-15T04:35:02.72+0000 [App/0]   OUT tick: [] -- FOO:
2014-12-15T04:35:03.72+0000 [App/0]   OUT tick: [] -- FOO:
2014-12-15T04:35:04.72+0000 [App/0]   OUT tick: [] -- FOO:
2014-12-15T04:35:05.73+0000 [App/0]   OUT tick: [] -- FOO:
2014-12-15T04:35:06.73+0000 [App/0]   OUT tick: [] -- FOO:
2014-12-15T04:35:07.74+0000 [App/0]   OUT tick: [] -- FOO:
2014-12-15T04:35:08.74+0000 [App/0]   OUT tick: [] -- FOO:
2014-12-15T04:35:09.74+0000 [App/0]   OUT tick: [] -- FOO:
2014-12-15T04:35:10.74+0000 [App/0]   OUT tick: [] -- FOO:
2014-12-15T04:35:11.75+0000 [App/0]   OUT tick: [] -- FOO:
2014-12-15T04:35:12.75+0000 [App/0]   OUT tick: [] -- FOO:
2014-12-15T04:35:13.75+0000 [App/0]   OUT tick: [] -- FOO:
2014-12-15T04:35:14.76+0000 [App/0]   OUT tick: [] -- FOO:
2014-12-15T04:35:15.76+0000 [App/0]   OUT tick: [] -- FOO:
2014-12-15T04:35:16.76+0000 [App/0]   OUT tick: [] -- FOO:
2014-12-15T04:35:17.76+0000 [App/0]   OUT tick: [] -- FOO:
2014-12-15T04:35:18.77+0000 [App/0]   OUT tick: [] -- FOO:
2014-12-15T04:35:19.77+0000 [App/0]   OUT tick: [] -- FOO:
2014-12-15T04:35:20.77+0000 [App/0]   OUT tick: [] -- FOO:
2014-12-15T04:35:21.78+0000 [App/0]   OUT tick: [] -- FOO:
2014-12-15T04:35:22.78+0000 [App/0]   OUT tick: [] -- FOO:
2014-12-15T04:35:23.78+0000 [App/0]   OUT tick: [] -- FOO:
2014-12-15T04:35:24.78+0000 [App/0]   OUT tick: [] -- FOO:
2014-12-15T04:35:25.79+0000 [App/0]   OUT tick: [] -- FOO:
2014-12-15T04:35:26.79+0000 [App/0]   OUT tick: [] -- FOO:
2014-12-15T04:35:27.79+0000 [App/0]   OUT tick: [] -- FOO:
2014-12-15T04:35:28.80+0000 [App/0]   OUT tick: [] -- FOO:
2014-12-15T04:35:29.80+0000 [App/0]   OUT tick: [] -- FOO:
2014-12-15T04:35:30.80+0000 [App/0]   OUT tick: [] -- FOO:
2014-12-15T04:35:31.80+0000 [App/0]   OUT tick: [] -- FOO:
2014-12-15T04:35:32.81+0000 [App/0]   OUT tick: [] -- FOO:
2014-12-15T04:35:33.81+0000 [App/0]   OUT tick: [] -- FOO:
2014-12-15T04:35:34.81+0000 [App/0]   OUT tick: [] -- FOO:
2014-12-15T04:35:35.82+0000 [App/0]   OUT tick: [] -- FOO:
2014-12-15T04:35:36.82+0000 [App/0]   OUT tick: [] -- FOO:
2014-12-15T04:35:37.82+0000 [App/0]   OUT tick: [] -- FOO:
2014-12-15T04:35:38.82+0000 [App/0]   OUT tick: [] -- FOO:
2014-12-15T04:35:39.83+0000 [App/0]   OUT tick: [] -- FOO:
2014-12-15T04:35:40.83+0000 [App/0]   OUT tick: [] -- FOO:
2014-12-15T04:35:41.83+0000 [App/0]   OUT tick: [] -- FOO:
2014-12-15T04:35:42.84+0000 [App/0]   OUT tick: [] -- FOO:
2014-12-15T04:35:43.84+0000 [App/0]   OUT tick: [] -- FOO:
2014-12-15T04:35:44.85+0000 [App/0]   OUT tick: [] -- FOO:
2014-12-15T04:35:45.85+0000 [App/0]   OUT tick: [] -- FOO:
2014-12-15T04:35:46.85+0000 [App/0]   OUT tick: [] -- FOO:
2014-12-15T04:35:47.86+0000 [App/0]   OUT tick: [] -- FOO:
2014-12-15T04:35:48.86+0000 [App/0]   OUT tick: [] -- FOO:
2014-12-15T04:35:49.86+0000 [App/0]   OUT tick: [] -- FOO:
2014-12-15T04:35:50.86+0000 [App/0]   OUT tick: [] -- FOO:
2014-12-15T04:35:51.87+0000 [App/0]   OUT tick: [] -- FOO:
2014-12-15T04:35:52.87+0000 [App/0]   OUT tick: [] -- FOO:
2014-12-15T04:35:53.87+0000 [App/0]   OUT tick: [] -- FOO:
2014-12-15T04:35:54.88+0000 [App/0]   OUT tick: [] -- FOO:
2014-12-15T04:35:55.88+0000 [App/0]   OUT tick: [] -- FOO:
2014-12-15T04:35:56.88+0000 [App/0]   OUT tick: [] -- FOO:
2014-12-15T04:35:57.89+0000 [App/0]   OUT tick: [] -- FOO:
2014-12-15T04:35:58.75+0000 [DEA]     ERR Instance (index 0) failed to start accepting connections
2014-12-15T04:35:58.76+0000 [API]     OUT App instance exited with guid ab19c073-8fa2-4e11-97cb-8911de80c739 payload: {"cc_partition"=>"default", "droplet"=>"ab19c073-8fa2-4e11-97cb-8911de80c739", "version"=>"b8bb2113-0fb4-4afc-a856-1c48da5fc580", "instance"=>"1c2102ff34b641fe985ee6e1d110fc18", "index"=>0, "reason"=>"CRASHED", "exit_status"=>-1, "exit_description"=>"failed to accept connections within health check timeout", "crash_timestamp"=>1418618158}
 
3. my shell was not started if I appended it after the original commands. but the java Application is running properly.  I can not understand here. Why the later shell was ignored?
---
addons: []
config_vars: {}
default_process_types:
  web: ! ' JAVA_HOME=$PWD/.java-buildpack/open_jdk_jre JAVA_OPTS="-Djava.io.tmpdir=$TMPDIR

    -XX:OnOutOfMemoryError=$PWD/.java-buildpack/open_jdk_jre/bin/killjava.sh -Xmx160M
    -Xms160M -XX:MaxMetaspaceSize=64M -XX:MetaspaceSize=64M -Xss853K -Daccess.logging.enabled=false

    -Dhttp.port=$PORT" $PWD/.java-buildpack/tomcat/bin/catalina.sh run && /home/vcap/app/startAndExitOnAnyChildExit.sh  /home/vcap/app/vendor/bin/test.sh
    /home/vcap/app/ticket  '

2014-12-15T06:40:14.81+0000 [STG]     OUT -----> Uploading droplet (50M)
2014-12-15T06:40:23.87+0000 [DEA]     OUT Starting app instance (index 0) with guid 789a411a-5e94-45c4-b768-3f243e379015
2014-12-15T06:40:28.82+0000 [App/0]   OUT [CONTAINER] org.apache.coyote.http11.Http11NioProtocol         INFO    Initializing ProtocolHandler ["http-nio-61175"]
2014-12-15T06:40:28.85+0000 [App/0]   OUT [CONTAINER] org.apache.catalina.startup.Catalina               INFO    Initialization processed in 1264 ms
2014-12-15T06:40:28.87+0000 [App/0]   OUT [CONTAINER] org.apache.catalina.core.StandardService           INFO    Starting service Catalina
2014-12-15T06:40:28.88+0000 [App/0]   OUT [CONTAINER] org.apache.catalina.core.StandardEngine            INFO    Starting Servlet Engine: Apache Tomcat/8.0.15
2014-12-15T06:40:28.94+0000 [App/0]   OUT [CONTAINER] org.apache.catalina.startup.HostConfig             INFO    Deploying web application directory /home/vcap/app/.java-buildpack/tomcat/webapps/ROOT
2014-12-15T06:40:30.09+0000 [App/0]   OUT [CONTAINER] org.apache.jasper.servlet.TldScanner               INFO    At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.
2014-12-15T06:40:30.25+0000 [App/0]   OUT [CONTAINER] org.apache.catalina.startup.HostConfig             INFO    Deployment of web application directory /home/vcap/app/.java-buildpack/tomcat/webapps/ROOT has finished in 1,311 ms
2014-12-15T06:40:30.26+0000 [App/0]   OUT [CONTAINER] org.apache.coyote.http11.Http11NioProtocol         INFO    Starting ProtocolHandler ["http-nio-61175"]
2014-12-15T06:40:30.27+0000 [App/0]   OUT [CONTAINER] org.apache.tomcat.util.net.NioSelectorPool         INFO    Using a shared selector for servlet write/read
2014-12-15T06:40:30.29+0000 [App/0]   OUT [CONTAINER] org.apache.catalina.startup.Catalina               INFO    Server startup in 1438 ms

[Daniel Watrous]

Hi,

When it says that it failed to start accepting connections, that could mean a few different things. What it's looking for is a web service on the PORT that was injected. You appear to be doing capturing this port and using it to start your service.

The methods for starting your runtime include

1) the buildpack has a default way of starting a process

2) in your application you can include a Procfile to override the buildpack. This can start multiple processes.

I would start by figuring out how to manually create a warden environment, similar to what I did for docker in Stackato. Then try running whatever you have put in your Procfile and see if it runs. In other words, you want to create a container and manually go through your buildpack steps and start the runtime to confirm your configuration.

Daniel

[James Bayer]

in order to bypass the port check, you need to make sure there are no routes bound to your app.

cf push APPNAME --no-route

is one way to accomplish this.

To view this discussion on the web visit https://groups.google.com/a/cloudfoundry.org/d/msgid/vcap-dev/915cf0d7-9558-49c3-a961-104dfb05cdba%40cloudfoundry.org.

To unsubscribe from this group and stop receiving emails from it, send an email to vcap-dev+u...@cloudfoundry.org.

--

Thank you,

James Bayer