How to handle dynamic schema names for Ebean Entities given this scenario?

[vladimir.p...@tealium.com]

Let's say we have a single database and 3 schemas with parameterized names:

• {env}_{env_id}_schema1
• {env}_{env_id}_schema2
• {env}_{env_id}_schema3

So for two developers Katie and Tom, there would be a database with 6 schemas:

• dev_katie_schema1
• dev_katie_schema2
• dev_katie_schema3
• dev_tom_schema1
• dev_tom_schema2
• dev_tom_schema3

Ebean entities leave @Table annotation attribute "schema" blank and rely on "ebean.dbSchema" in "ebean.properties" file. This results in a config file for every schema (not a big deal), but also implies 3 Ebean Database objects end up being created for the same underlying database (seems wasteful).

Is there a clever way of avoiding that? Ideally I would want a single "db connection pool" used by 3 Ebean Database objects (each having their own respective default schema based on a config file).

Note: above situation could be solved with multiple databases, where each developer gets their own dedicated db, but that's currently not an option.

[Rob Bygrave]

> Note: above situation could be solved with multiple databases, where each developer gets their own dedicated db, but that's currently not an option.

What is the database?  

Why is it not an option for the developers to have their own DB? 

You are not looking to use docker to run the database?  Obvious question I guess but this is to me is has been such a nice and significant productivity jump I've had myself that I expect everyone to be using it - we've made this so darn easy to various databases via docker I presume you are not using one of the usual databases?

 

Cheers, Rob.

--

---
You received this message because you are subscribed to the Google Groups "Ebean ORM" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ebean+un...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[vladimir.p...@tealium.com]

You are not looking to use docker to run the database?

I would prefer that personally and echo what you are saying. Docker makes it a breeze.

The challenge our team is facing is we are running a lot of our components & services with AWS. Both for "local" development and production. For example, an AWS Lambda needs to write data to a Postgres database (AWS Aurora RDS) and if Postgres is run locally in docker then it is not trivial to hit it from a cloud service. There are ways of running certain AWS services locally, but we haven't fully explored that yet (does not look trivial at first glance).

Unfortunately AWS Aurora RDS has limits (Link) for number of database instances (40 per account, but can be raised upon request). We are worried we might hit that limit, plus spinning up an AWS Service is a little trickier than a docker image (not by much though, thanks to Terraform).

I wanted to explore first if it's possible to make it work with 1 database instance and multiple groups of schemas, before trying to push for going down the multiple database instances route.

On Thursday, April 11, 2019 at 4:36:58 PM UTC-7, Rob Bygrave wrote:

> Note: above situation could be solved with multiple databases, where each developer gets their own dedicated db, but that's currently not an option.

What is the database?  

Why is it not an option for the developers to have their own DB? 

You are not looking to use docker to run the database?  Obvious question I guess but this is to me is has been such a nice and significant productivity jump I've had myself that I expect everyone to be using it - we've made this so darn easy to various databases via docker I presume you are not using one of the usual databases?

 

Cheers, Rob.

On Fri, 12 Apr 2019 at 11:15, <vladimir....@tealium.com> wrote:

Let's say we have a single database and 3 schemas with parameterized names:

• {env}_{env_id}_schema1
• {env}_{env_id}_schema2
• {env}_{env_id}_schema3

So for two developers Katie and Tom, there would be a database with 6 schemas:

• dev_katie_schema1
• dev_katie_schema2
• dev_katie_schema3
• dev_tom_schema1
• dev_tom_schema2
• dev_tom_schema3

Ebean entities leave @Table annotation attribute "schema" blank and rely on "ebean.dbSchema" in "ebean.properties" file. This results in a config file for every schema (not a big deal), but also implies 3 Ebean Database objects end up being created for the same underlying database (seems wasteful).

Is there a clever way of avoiding that? Ideally I would want a single "db connection pool" used by 3 Ebean Database objects (each having their own respective default schema based on a config file).

Note: above situation could be solved with multiple databases, where each developer gets their own dedicated db, but that's currently not an option.

--

---
You received this message because you are subscribed to the Google Groups "Ebean ORM" group.

To unsubscribe from this group and stop receiving emails from it, send an email to eb...@googlegroups.com.

[Rob Bygrave]

Ok I hear you.  There might not be a perfect solution for ... "local development and testing of AWS lambda's that use AWS RDS".

When I've done this myself I'm not looking to do development and testing in AWS. Roughly speaking, I'm looking to split the problem into 2 (with the vast majority of business logic & testing having NO dependency on AWS at all) so:

1. AWS "marshalling" some payload to call a service method (Some AWS dependency)

2. The service method that takes a payload and does stuff (No AWS dependency at all)

We can do everything from 2. down locally with no AWS dependency at all.  Hopefully this is 95%+ of the business logic.  We really don't want a AWS dependency on this.  We could even release this into our nexus as a library that is used by a lambda or a rest service etc.

We can test "marshalling" payloads locally with some AWS dependency but without needing to run AWS lambda in AWS. 

That leaves us with not much untested wrt running the lambda in AWS itself.  

I think spring have some stuff here for running lambda's locally?  Not sure maybe someone else can pitch in here? 

To me this is somewhat analogous to rest api's where the "service layer" knows nothing about the web/marshalling etc.

How does any of that sound?

Cheers, Rob.

To unsubscribe from this group and stop receiving emails from it, send an email to ebean+un...@googlegroups.com.

[vladimir.p...@tealium.com]

We really don't want a AWS dependency 

Totally agree. Baking AWS Services into our "local" development process never felt right. Some of our challenges are likely organizational and not technological. "People" like the idea when dev, qa, staging and prod environments are all the same. One source of "truth" to build them all in the same exact "cloud space", and then nobody can say "well it worked on my local".

There is https://github.com/lambci/docker-lambda which looks promising. If our team buys in, perhaps Postgres & Lambda can run in docker and call it a day.

Thank you for your feedback! :)

On Thursday, April 11, 2019 at 5:13:59 PM UTC-7, Rob Bygrave wrote:

Ok I hear you.  There might not be a perfect solution for ... "local development and testing of AWS lambda's that use AWS RDS".

When I've done this myself I'm not looking to do development and testing in AWS. Roughly speaking, I'm looking to split the problem into 2 (with the vast majority of business logic & testing having NO dependency on AWS at all) so:

1. AWS "marshalling" some payload to call a service method (Some AWS dependency)

2. The service method that takes a payload and does stuff (No AWS dependency at all)

We can do everything from 2. down locally with no AWS dependency at all.  Hopefully this is 95%+ of the business logic.  We really don't want a AWS dependency on this.  We could even release this into our nexus as a library that is used by a lambda or a rest service etc.

We can test "marshalling" payloads locally with some AWS dependency but without needing to run AWS lambda in AWS. 

That leaves us with not much untested wrt running the lambda in AWS itself.  

I think spring have some stuff here for running lambda's locally?  Not sure maybe someone else can pitch in here? 

To me this is somewhat analogous to rest api's where the "service layer" knows nothing about the web/marshalling etc.

How does any of that sound?

Cheers, Rob.

[Rob Bygrave]

> "People" like the idea when dev, qa, staging and prod environments are all the same.

Yes well to me that is the great thing about testing using docker databases.  My local environment is the same as the build server (apart from timezones usually so watch those sql test scripts for timestamp literals :)) and darn near identical to test, production etc (some variation between AWS RDS Postgres and docker Postgres of course).

From a development and testing perspective it is easy to cycle the database (recreate it completely from scratch) for testing locally, building/running locally which to me is great. 

Developers don't need to know how to install or setup Postgres (because ebean-test-config does all that for us).

I love ebean-test-config and docker testing, I've been using, tweaking and improving it for almost 2 years now (I moved it from org.avaje into io.ebean, it originally wasn't Ebean specific).  ebean-test-config only has 2 stars as a github project which I don't really understand because to me it is fundamental to improved developer productivity - I myself couldn't go back to not using it now.

For people reading this:

https://ebean.io/docs/testing/

https://github.com/ebean-orm/ebean-test-config   (only 2 github stars ?  ... to me everyone should be using this)

https://github.com/ebean-orm/ebean-test-docker  (only 1 github stars ?  ... this knows how to start and setup the various docker databases) 

Cheers, Rob.

On Fri, 12 Apr 2019 at 12:34, <vladimir.p...@tealium.com> wrote:

We really don't want a AWS dependency 

Totally agree. Baking AWS Services into our "local" development process never felt right. Some of our challenges are likely organizational and not technological. "People" like the idea when dev, qa, staging and prod environments are all the same. One source of "truth" to build them all, and then nobody can say "well it worked on my local".

There is https://github.com/lambci/docker-lambda which looks promising. If our team buys in, perhaps Postgres & Lambda can run in docker and call it a day.

Thank you for you feedback! :)

To unsubscribe from this group and stop receiving emails from it, send an email to ebean+un...@googlegroups.com.

[vladimir.p...@tealium.com]

After some tinkering, I found a decent enough solution of being able to impose a dynamic schema name prefix. Namely extending io.ebean.config.UnderscoreNamingConvention and providing that to the DatabaseConfig object.

class SchemaPrefixNamingConvention extends UnderscoreNamingConvention {

    private String schemaPrefix;

    private SchemaPrefixNamingConvention(String schemaPrefix) {

        this.schemaPrefix = schemaPrefix;

        }

    /**

     * Code is mostly duplicated from {@code AbstractNamingConvention#getTableNameFromAnnotation()}

     * but with an extra addition of prefix-ing schema names.

     */

    @Override

    protected TableName getTableNameFromAnnotation(Class<?> beanClass) {

            final Table t = AnnotationUtil.findAnnotationRecursive(beanClass, Table.class);

            if (t != null && !isEmpty(t.name())) {

                 return new TableName(quoteIdentifiers(t.catalog()), quoteIdentifiers(schemaPrefix + t.name()), quoteIdentifiers(t.name()));

            } else {

                 return null; // no annotation provided, default naming convention will kick in instead

            }

    }

}