I have started a forum topic on the subject of Simplifying Ebean Internals by Removing MapBean fuctionality and DB meta data use.
I'm considering an idea to simplify Ebean internals. This would potentially involve removing the MapBean functionality. If you are using the MapBean functionality and don't want it removed can you please let me know - thanks.
In terms of simplifying Ebean internals... the idea is to re-evaluate the use of DB meta data. The initial thoughts is that with a couple of changes ... the need for using the DB meta data is removed (and it becomes hard to justify keeping it).
The two main things required to do this (at this stage) is
1) use @NotNull and @Length validation annotations (or @Column(nullable=false...) annotation). This is generally required by Ebean to determine the join types (inner/outer). In a sense having @NotNull etc could be a good/best practice moving forward as it enables bean validation frameworks etc).
2) use naming convention for finding the Foreign Key columns (like the JPA spec but pluggable/configurable)... rather than Ebean finding these via the DB meta data. Most people are going to have a decent naming convention so this should not be that bad.
The benefit of this is that the DB meta data code can all be removed from Ebean (a reasonably complex bit of work). In the long term keeping everything as simple as possible is important which is why I'm thinking about this (and leaning towards doing this).<<
If you any thoughts on this I'd like to hear them... (I'd prefer feedback in the google group rather than the forum).
Simplifying is always a good idea and yes I think taking the
annotation route is the cleaner approach than using the DB meta data.
Regarding, MapBean, I haven't used it but I think it makes using
native SQL easier than having to work with JDBC. But if there is no
real need for it then cleaning it out would also be a good idea.
Just my 2 cents. I read your forum post on this topic, and it seems
your doing this all for DDL generation. I have been a java programmer
for 11 years 10 of which have been Java/SQL and I have never generated
my schema from my java beans. Java objects always seems to need work
around existing schemas. Put another way the Relation Database rules
the world no mater what the OO purists think. I would be hard pressed
to remove flexibility for existing schemas for DDL generation. This
is actually one of the reasons I am attracted to EBean, it seems to be
more DB centric instead of Object centric like Hibernate. Especially
in the Query language. No Restrictions and Projections and Criteria.
Good ol reliable Queries.
> and it seems your doing this all for DDL generation
Yes, that's pretty much true. A big benefit (side effect) is that the DDL generation can lead to easier testing (especially when linked to automatic loading of seed data). For example, the test suite for Ebean at the moment is based on DB specific scripts which is not great. DDL generation would mean the test suite for Ebean (and any application using Ebean) could be automated for numerous DB's. For testing purposes this could be a big win and this is the biggest motivation for me to support DDL generation.
> and I have never generated my schema from my java beans.
Neither have I actually. I personally have always favoured spending some time designing a DB with a DB design tool (which was built for the purpose) and personally would perfer to use such a DB centric design tool to generate the DDL rather than beans with annotations (which are not going to include views, triggers, stored procedures, tablespaces, extra indexes etc, common data types/lengths etc).
I was expecting a JPA/ERD orientated design tool to exist by now which would be "the source" for generating both the DDL and annotated java entity beans. I have not seen such a tool yet but perhaps one exists? If it did then I suspect it would have a number of benefits over DDL generation from annotated beans.
However, I believe there are Agile orientated developers expecting/wanting DDL generation and especially want it for quick development and automated testing purposes... as well as a starting point for production quality DDL. The generated DDL could be sufficient for some DB's (smaller, no tablespaces, only used for automated testing etc) .
> Put another way the Relational Database rules the world no mater what the OO purists think
My slightly different spin on this would be "Don't fight the Database". The harder you fight the natural characteristics of the Relational Database design you are working with... the more pain you are going to have. Generally you are going to have to live with the DB design longer than the ORM/application. If the ORM generates a ORM specific DB design that could be a problem if later you want to swap out the application with another technology.
> one of the reasons I am attracted to EBean, it seems to be
more DB centric instead of Object centric like Hibernate. Especially in the Query language. No Restrictions and Projections and Criteria. Good ol reliable Queries.
Well I agree in that I think Ebean is more DB centric and I think it will continue to be more DB centric.
Another way to look at this DB meta data issue is... Ebean currently hides the DB meta data it finds. Instead some of that meta data should possibly be on the entity beans (specifically the @NotNull and @Length annotations) with the benefit of supporting bean validation frameworks. Once you do this the value of the "hidden" DB meta data Ebean uses is reduced (to mostly automatic foreign key determination).
It is a bit more tricky with MapBean's which rely on the DB meta data. I'm not sure if they can be used without the DB meta data being available.
Can't speak for Rob, but I don't think that we do this just for DDL
generation.
Having correctly annotated beans opens up new possibilities as there
are
*) using of dynamic form generation like MetaWidget or Apache MyFaces
Orchestra's Sandbox stuff DynaForm
*) using of more sophisticated validation stuff like you can see in
JSR303 - Bean Validation
*) easer to read application as you can see in the code what kind of
database constraints are set up (e.g. NotNull, Length, etc).
*) and ok, finally allows DDL generation
For the more database centric approach of Ebean, I don't think we need
to fetch the db metadata. It slows down the first startup of Ebean and
will never be consistent after that if you change the db (it is cached
locally. I think it fetches new tables, but not table-column changes.
Correct me if I am wrong)
If you reflect those kind of things in your entity-bean, not only
Ebean can get rid of some complex code, but also the startup delay is
gone and your application will always behave how the developer wanted
it to do.
Regarding the qurey, I think, at least 95% of our queries are built
using the Criteria stuff. I fallback to hand written queries only if
there is absolutely no way around.
Finally, in the past, in our company we started a completely new
project where we just designed the entity-beans with all the
constraints and relations and let Hibernate create the database. First
for HSQLDB and close to production for Oracle then. It was a great
time-saver during prototyping.
Sure, in the end we handed those scripts over to our database
administrator to polish them. But he was quite happy about not having
to write all those basic stuff again.
And if I would start a new project with Ebean I would really miss this
feature.
This has nothing to do with OO purist. I am far from being one.
Having a good working entity-bean generator (from existing db schemas)
will then get the additional work from the developer.
Agreed:
1) Bean validation is nice.
2) Retrieving all meta data at start up seems messy.
Disagree:
1) Generation (in general) :)
So what exactly is lost from the removal of MapBean? Only thing I had
in mind where it would be use full is stored procedure calls. Which I
still think is useful.
> 2) Retrieving all meta data at start up seems messy.
The meta data is cached (so generally is not retrieved per say). However, it is fair to say that the use of the meta data during initialisation (to find foreign key columns etc) is pretty messy/complex. I was looking at it last night... and compared to just using a naming convention searching the db meta data does look messy/complex to me.
> Disagree: 1) Generation (in general) :)
No problem. Just to add the reverse engineering (from DB to java) has not had any attention for a long time and needs a re-write/re-visit and look at adding @NotNull and @Length to the generated beans.
> So what exactly is lost from the removal of MapBean?
I have used MapBeans in that past for utility type programs such as moving/merging data in a table/relational centric way. MapBeans don't require any configuration and are very low tech (which means they are really easy to use). I have not used MapBeans with stored procedures at all.
I don't know but suspect hardly anyone (besides myself) is using MapBeans. I hope the users of MapBeans speak up though because removing the DB meta data means the MapBean functionality is going to be hard to keep (and removing the DB meta data looks like it will simplify the internals reasonably significantly).
> > 2) Retrieving all meta data at start up seems messy.
> The meta data is cached (so generally is not retrieved per say). However, it is fair to say that the use of the meta data during initialisation (to find foreign key columns etc) is pretty messy/complex. I was looking at it last night... and compared to just using a naming convention searching the db meta data does look messy/complex to me.
> > Disagree: 1) Generation (in general) :)
> No problem. Just to add the reverse engineering (from DB to java) has not had any attention for a long time and needs a re-write/re-visit and look at adding @NotNull and @Length to the generated beans.
> > So what exactly is lost from the removal of MapBean?
> I have used MapBeans in that past for utility type programs such as moving/merging data in a table/relational centric way. MapBeans don't require any configuration and are very low tech (which means they are really easy to use). I have not used MapBeans with stored procedures at all.
> I don't know but suspect hardly anyone (besides myself) is using MapBeans. I hope the users of MapBeans speak up though because removing the DB meta data means the MapBean functionality is going to be hard to keep (and removing the DB meta data looks like it will simplify the internals reasonably significantly).
> > 2) Retrieving all meta data at start up seems messy.
> The meta data is cached (so generally is not retrieved per say). However, it is fair to say that the use of the meta data during initialisation (to find foreign key columns etc) is pretty messy/complex. I was looking at it last night... and compared to just using a naming convention searching the db meta data does look messy/complex to me.
> > Disagree: 1) Generation (in general) :)
> No problem. Just to add the reverse engineering (from DB to java) has not had any attention for a long time and needs a re-write/re-visit and look at adding @NotNull and @Length to the generated beans.
> > So what exactly is lost from the removal of MapBean?
> I have used MapBeans in that past for utility type programs such as moving/merging data in a table/relational centric way. MapBeans don't require any configuration and are very low tech (which means they are really easy to use). I have not used MapBeans with stored procedures at all.
> I don't know but suspect hardly anyone (besides myself) is using MapBeans. I hope the users of MapBeans speak up though because removing the DB meta data means the MapBean functionality is going to be hard to keep (and removing the DB meta data looks like it will simplify the internals reasonably significantly).
Do you use MapBean for insert update delete ... or just queries?
Queries is easy... and doesn't really need the DB meta data. I imagine it is useful to people as it provides a simple low tech way to use any SQL query so I would like to keep MapBean query functionality.
MapBean is aware of primary keys and version columns plus generated properties like "inserted timestamp", "last update timestamp". So for Insert/Update/Delete MapBeans are aware of concurrency checking and maintaining the version columns etc.
So, specifically it is the insert/update/delete of MapBeans that makes significant use of the DB meta data. One option is to remove the concurrency checking and automatic management of generated properties like "inserted timestamp" and "last update timestamp".... or just keep MapBeans as they are and make sure the DB meta data is available to support them.
> Will there be an alternative?
The query support is easy to keep. The IUD of MapBeans without db meta data is the tricky bit. If people are using MapBeans for IUD now... then that functionality should not be removed in my opinion.
> Do you use MapBean for insert update delete ... or just queries?
> Queries is easy... and doesn't really need the DB meta data. I imagine it is useful to people as it provides a simple low tech way to use any SQL query so I would like to keep MapBean query functionality.
> MapBean is aware of primary keys and version columns plus generated properties like "inserted timestamp", "last update timestamp". So for Insert/Update/Delete MapBeans are aware of concurrency checking and maintaining the version columns etc.
> So, specifically it is the insert/update/delete of MapBeans that makes significant use of the DB meta data. One option is to remove the concurrency checking and automatic management of generated properties like "inserted timestamp" and "last update timestamp".... or just keep MapBeans as they are and make sure the DB meta data is available to support them.
> > Will there be an alternative?
> The query support is easy to keep. The IUD of MapBeans without db meta data is the tricky bit. If people are using MapBeans for IUD now... then that functionality should not be removed in my opinion.
UPDATE:
========
Code is not in HEAD yet but I have working code that has removed the
dependency on DB meta data.
NOTES:
=======
- MapBean:
Changed name to SqlRow... and removed the ability to insert/update/
delete SqlRow/MapBean's. They can only be used for queries now!!
- @ManyToOne(optional=false) / @NotNull
I'm using the optional attribute on @ManyToOne to determine inner/
outer join. By default optional=true for the annotation. Alternatively
if you put a @NotNull on the property and that has the same effect as
@ManyToOne(optional=false). Previously the DB meta data checked the
nullablity of columns to automagically determine this.
- Lost the hidden/automagic @NotNull and @Length validation.
So now you need to explictly use @NotNull and @Length to get that
validation on the properties. In v1.2 these could be automatically
added from the DB meta data.
- Foreign Key naming convention
Following the JPA spec in this regard ... so for a @ManyToOne
billiingAddress...
1). the property name "billingAddress" gets converted to a DB column
name using the naming convention (in my case with underscores... this
becomes billing_address)
2). then _{primary key of target} is appended ... so you end up with
billing_address_id (where "id" was the primary key column of the
target table)
... I think this will generally work well and not using the DB meta
data for this will be fine.
- Internals much simpler
I think this has cleaned up things significantly!! Lots of stuff
removed, and the process of reading the 'deployment' information looks
a lot better/easier to follow.
MOVING FORWARD:
================
At this point I'm thinking about how to hook in DDL generation (with
an eye on reverse generation as well).
My current thinking is to provide public "Meta entity beans" for all
the Beans/Properties that can be queried out of Ebean (just like the
query statistics).
// find all the beans and their properties
List<MetaBean> beans = Ebean.find(MetaBean.class).findList();
So this information is sufficient for DDL generation... and in the
future for other code generation frameworks.
In a similar way I'm thinking of querying the Table/Column.
// find the tables/columns/key constraints in a given catalog/schema
// ... typically useful for generating java entity bean code
List<MetaTable> tables = Ebean.find(MetaTable.class).where().add
("schema like 'o%'").findList()
Using the SqlRow to replace the MapBean with select only is a great
idea. I'd doubt anyone would use the MapBean for anything else. So
thank you for preserving this feature.
Would migrating require a simple project wise direct replacment of
MapBean - SqlRow or more than that?
Speaking about ddl generation, you're planing to provide the tool or
just the meta data? If your tool, will you provide refactoring? The
refactoring part will be rather awesome.
Thanks for the hard work. I hope it pays.
On May 25, 7:54 pm, RobB <rbygr...@yahoo.com> wrote:
> UPDATE:
> ========
> Code is not in HEAD yet but I have working code that has removed the
> dependency on DB meta data.
> NOTES:
> =======
> - MapBean:
> Changed name to SqlRow... and removed the ability to insert/update/
> delete SqlRow/MapBean's. They can only be used for queries now!!
> - @ManyToOne(optional=false) / @NotNull
> I'm using the optional attribute on @ManyToOne to determine inner/
> outer join. By default optional=true for the annotation. Alternatively
> if you put a @NotNull on the property and that has the same effect as
> @ManyToOne(optional=false). Previously the DB meta data checked the
> nullablity of columns to automagically determine this.
> - Lost the hidden/automagic @NotNull and @Length validation.
> So now you need to explictly use @NotNull and @Length to get that
> validation on the properties. In v1.2 these could be automatically
> added from the DB meta data.
> - Foreign Key naming convention
> Following the JPA spec in this regard ... so for a @ManyToOne
> billiingAddress...
> 1). the property name "billingAddress" gets converted to a DB column
> name using the naming convention (in my case with underscores... this
> becomes billing_address)
> 2). then _{primary key of target} is appended ... so you end up with
> billing_address_id (where "id" was the primary key column of the
> target table)
> ... I think this will generally work well and not using the DB meta
> data for this will be fine.
> - Internals much simpler
> I think this has cleaned up things significantly!! Lots of stuff
> removed, and the process of reading the 'deployment' information looks
> a lot better/easier to follow.
> MOVING FORWARD:
> ================
> At this point I'm thinking about how to hook in DDL generation (with
> an eye on reverse generation as well).
> My current thinking is to provide public "Meta entity beans" for all
> the Beans/Properties that can be queried out of Ebean (just like the
> query statistics).
> // find all the beans and their properties
> List<MetaBean> beans = Ebean.find(MetaBean.class).findList();
> So this information is sufficient for DDL generation... and in the
> future for other code generation frameworks.
> In a similar way I'm thinking of querying the Table/Column.
> // find the tables/columns/key constraints in a given catalog/schema
> // ... typically useful for generating java entity bean code
> List<MetaTable> tables = Ebean.find(MetaTable.class).where().add
> ("schema like 'o%'").findList()
