[ANN] Java Data Objects Seminars & Workshops
Gregg Shenton
Microsystems; David Jordan of Object Identity and Steve Johnson of
PrismTech will be speaking at a series of seminars and workshops to be
held in the US and Europe during October 2001.
The seminars and workshops are targeted at architects, developers and
their managers who are seeking to learn about the benefits that the
new JDO technology can provide for their businesses.
The seminars and workshops will be held in:
Boston, MA, USA - 9th October
Dallas, TX, USA - 10th October
San Jose, CA, USA - 11th October
London, UK - 16th October
Paris, France - 18th October
For more information and registration details please visit:
http://www.prismtechnologies.com/English/News/events/jdo/index.html
Carl Rosenberger
more and more worried about all the time that people and vendors seem to
invest, trying to learn this specification, or worse, implement it.
A direct interface to store objects needs to be simpler than SQL, not more
complicated, otherwise it will never find it's place in the database world.
JDO is anything but simple and concise.
The failure of old-time object databases to win a noticeable market share
proves the necessity to introduce clean and simple concepts quite clearly:
Object database vendors tried to jump two paradigm steps at a time: storing
objects and transparent persistency. What they seem to have forgotten:
Persistency can never be truly transparent. Controlling locks efficiently is
one of the greatest challenges in enterprise data management and almost
always fine tuning has to be applied - by hand. Once you start doing this,
you are aware of the database, so why not keep things simple, for the start,
by explicitly storing objects? In doing this, the application can work with
"dirty" objects, without worries about events routed through to the database
or worse: objects getting locked.
A great part of the JDO specification was developed by the old guys from the
object database scene, the same people that used to do ODMG. The "enhancer"
concept, that many object databases use to modify application code can be
found again. People had difficulties to understand this concept and to get
it running in the past. Let's not make the same mistakes twice. Why don't we
start the JDO specification process again with a much simpler design than
current JDO?
Some technical comments referring to chapters in the JDO 1.0 Proposed Final
Draft specification.
5.4. JDO Identity
What are all these many different identity concepts for? Java objects should
be usable as they come, without special concerns. If code needs to be
reengineered by JDO experts first, where is the advantage to relational
databases? Uniquing is the one and only correct concept. It allows '==' to
be used to compare two instances of an object. Only one central cache is
needed to allow queries to decide, whether to return a reference to an
instantiated object or to instantiate a new object. Equality comparisons to
remote or distribute objects are possible with special methods or equality
queries, without exposing the internal ID.
The specification allows to change keys to "change object identity". What is
the use case in practice? This approach will kill the reference cache.
7. PersistenceCapable
JDO requests to implement the "PersistenceCapable" interface for all classes
that are to be made persistent. This requirement produces troubles and
maintenance work, it bloats application classes and thereby costs resources
and performance. How should the application programmer implement this
interface? By manual implementation? Deriving from a common base class is
not a solution, since it takes away the possibility to store objects derived
from existing JDK classes. Using a "ReferenceEnhancer" considerably slows
down the development process and user classes get bloated with database
implementation. Java is about to become the No.1 platform for mobile
development. Java Web Start might start a revolution for application
distribution. Resource consumption deserves more attention than has recently
been brought up by Sun. Large database applications also benefit from good
resource management, since cache overflow typically produces a performance
fiasco.
It is very well possible for a database engine to keep references to all
managed objects, without the necessity to modify application classes and
without wasting resources. There are existing object database engines that
prove this.
7.2 PersistenceCapable#jdoMakeDirty(String fieldName)
The JDO-Implementers do not seem to be aware of the fact that the use of
strings
- takes away type-safety during development. Errors will not be detected by
the compiler.
- cost lots of resources
- disables efficient refactoring
Strings have to be avoided at all costs.
We want to work with objects, don't we?
...and not with Strings that point to objects.
Why do objects need to be made "dirty"?
The database engine can perform comparisons.
7.3 jdoGetObjectId()
Internal IDs should not be exposed. We want to work with objects, not with
pointers. If an application programmer can not live without IDs, he can
easily build his own flavour into his objects. He is free, to use his own
system (social security number, license plate).
jdoGetTransactionalObjectId()
...funny...
exactly the same specification text as for jdoGetObjectId()
7.6 to 7.12
All these methods look very cryptic. Fields are replaced, get int values,
get "provided" get copied, in the end there are even simple-type-specific
fetch methods (i.e. fetchDoubleField). Most methods return void? Is this a
new data type in JDO?
What's all this for?
If this very weird API should serve to change class versions, it's terribly
designed.
Classes can use reflection to analyse themselves. Storing a superset of all
class versions removes the need to use int values to define Fields of
Classes.
8. JDOHelper
Many PersistenceCapable methods are listed here again, in static form. Why?
A good API provides one single path, how a problem is to be implemented. The
advantages:
- Code by different implementers uses the same methods. It remains
interchangeable and further requests for a change of the API remain similar.
- The API can be modified more easily.
8.2 PersistenceManager#makeDirty(Object, String fieldname)
Again:
If "makeDirty" is necessary, it should be makeDirty(Object).
14. Query
The Query interface is a complete disaster.
- It requires the use of String filters. Why not use API methods?
- declareImports, declareVariables, declareParameter, setOrdering, all with
the use of Strings? Why embed a string programming language into an existing
language. Who is supposed to parse all this? We do have Java. We do have
objects. Why not use them?
An alternative approach can be found here:
http://www.odbms.org/soda/
The website might not look nice.
The API is ready for first implementations.
16. EJB are a different specification. They have nothing to do with JDO. If
an EJB implementation wishes to use JDO, it may do so.
18. JDO has nothing to do with XML.
Schemas are defined by existing classes. If the classes are not present to
instantiate objects, what is the use of providing metadata?
Why can't the specification concentrate on persisting objects, solving this
theme nicely, before digging into third worlds?
Further points and questions:
The database world is still waiting for a scalable, beautiful solution to
handle locking and concurrency issues. Finally we have objects. A system
with callback events for
- requested locks
- relieved locks
- pre commit
- post commit
would provide the application programmer with an extremely scalable
environment to solve locking issues with tweezers. This would be much more
potent than current relational UnserializableException crowbars.
Why is JDO exclusively focused on Java? There are other worthwhile
programming languages (Ruby). A universal Object-Persistence-API would also
be helpful in other programming languages. The "What about C# and the
CLR?" - question will come and people will stick to SQL, since it's also
available on the Microsoft platform.
Who is supposed to implement all this? Who is supposed to understand it? The
ODMG specification failed because no vendor ever provided a complete
implementation. When is the first complete implementation of JDO expected?
The process has been extremely slow. In spite of all the big names in the
JCP, very little has developed forward.
This version 1.0 is not really it?
Noone will ever use this, except for consultants that make a living of never
finishing projects.
Sorry for these rude and harsh words.
Some sub-optimal designs like HTML, XML and EJB have been hyped to success
in the past.
Let's not waste another decade with database interfaces that don't do what
they are supposed to and what would be so easy:
store objects
Carl Rosenberger
db4o - database for objects
http://www.db4o.com
Craig Russell
respond.
Carl Rosenberger wrote:
>
> Having watched the development of the JDO specification for one year, I get
> more and more worried about all the time that people and vendors seem to
> invest, trying to learn this specification, or worse, implement it.
>
> A direct interface to store objects needs to be simpler than SQL, not more
> complicated, otherwise it will never find it's place in the database world.
> JDO is anything but simple and concise.
JDO concerns itself with two categories of class developers: those that
develop persistence-capable classes and those that develop
persistence-aware classes. It is well understood that developers who
know they are developing for databases (persistence-aware) need to deal
with some of the complexities of talking to databases: you have to know
which database you are trying to connect to, which classes you are
trying to query, etc. For these people, I won't claim that JDO is
simple, but it is not more complex than the alternatives.
Most of the database specific stuff is abstracted to things familiar to
JDBC users: ConnectionURL, ConnectionUserName, ConnectionPassword. But
JDO is arguably simpler once you have access to a database. You can
query for instances of persistence-capable classes with only the
knowledge of the Java class and fields. Contrast this with JDBC where
you have to know lots of details about the structure of the database
itself.
For persistence-capable class developers, I'd claim JDO has a
substantial advantage over any other persistence capability. You can
often implement Java persistence-capable classes without any knowledge
whatever of database. No mapping, no type conversions, no special read
methods, no special write methods.
With this level of transparency, JDO still allows for highly scalable
solutions, while solving the problem of the natural closure of data
store objects. In most databases, getting the closure of an object is
often the entire database. An employee has a reference to a department,
which has references to the company, and the company contains all of the
departments, all employees, all projects, etc. JDO allows for defining
the object model to navigate the application specific closure without
requiring application-specific data models.
>
> The failure of old-time object databases to win a noticeable market share
> proves the necessity to introduce clean and simple concepts quite clearly:
I think there are many reasons why old-time object databases failed to
win market share. Picking one is interesting but the story is much more
complex.
> Object database vendors tried to jump two paradigm steps at a time: storing
> objects and transparent persistency. What they seem to have forgotten:
> Persistency can never be truly transparent. Controlling locks efficiently is
> one of the greatest challenges in enterprise data management and almost
> always fine tuning has to be applied - by hand. Once you start doing this,
> you are aware of the database, so why not keep things simple, for the start,
> by explicitly storing objects? In doing this, the application can work with
> "dirty" objects, without worries about events routed through to the database
> or worse: objects getting locked.
Unfortunately, having the application keep track of dirty instances
detracts from one of object programming's major advantages: separation
of concerns. Once you require each persistence-capable class to track
its own state with explicit code, you pollute its methods with database
calls. And it doesn't matter if you delegate this to another
application class. You still need to insert database calls into the
class at odd places. It's really easy to miss out an update to a field
and then the application fails without anyone noticing.
>
> A great part of the JDO specification was developed by the old guys from the
> object database scene, the same people that used to do ODMG. The "enhancer"
> concept, that many object databases use to modify application code can be
> found again. People had difficulties to understand this concept and to get
> it running in the past. Let's not make the same mistakes twice. Why don't we
> start the JDO specification process again with a much simpler design than
> current JDO?
I'll admit to being one of the old guys from ODMG days, and I think we
learned a lot about what the market demands from a transparent
persistence interface. The requirements now are for scalable
implementations with binary compatibility that can take advantage of
distributed computing a la J2EE (TM) without writing yet another bunch
of single purpose code.
WRT the enhancer, have you looked at the strategy for CMP entity beans,
where the class file provided by the bean developer is analyzed and a
concrete implementation class constructed? Is there really a difference
between generating classes and enhancing them?
Another point that is often missed with a casual read of the
specification is that the JDO specification does not require a
post-processor (enhancer). A JDO implementation is free to use other
techniques, and several JDO vendors plan to do just that. Two
alternatives to the enhancer are pre-processing and direct code (either
.java or .class) generation from a business model or UML model or
whatever you are comfortable with.
>
> Some technical comments referring to chapters in the JDO 1.0 Proposed Final
> Draft specification.
>
> 5.4. JDO Identity
> What are all these many different identity concepts for? Java objects should
> be usable as they come, without special concerns. If code needs to be
> reengineered by JDO experts first, where is the advantage to relational
> databases? Uniquing is the one and only correct concept. It allows '==' to
> be used to compare two instances of an object. Only one central cache is
> needed to allow queries to decide, whether to return a reference to an
> instantiated object or to instantiate a new object. Equality comparisons to
> remote or distribute objects are possible with special methods or equality
> queries, without exposing the internal ID.
This is one of the most difficult concepts in JDO, and I apologize if it
is not yet crystal clear.
In a single cache managed by a single PersistenceManager, a user can
compare two instances using == and get the expected results. If you
query for an Employee, and navigate to its Department instance, then
query for a Department, you can compare the two Department instances
using == and they will compare == iff they represent the same datastore
instance.
The reason a special concept is used for comparing instances from
multiple caches is that in JDO, caches can be transactional. In the
same VM, you can have multiple caches in different transactions, and
each cache can maintain a different transactional view of the data store
instances. So in two different PersistenceManager caches, you are
guaranteed to have two different instances of the same data store
instance. The instances might compare equal (.equals(Object)) but that
is up to the implementation of .equals. But both transactional caches
might be making conflicting changes to the state of the instance. At
commit time (or earlier if the application chooses) the conflicts will
be resolved. Of course, if you want locking semantics to be applied,
you can also choose for the data store to enforce locking at the time
you retrieve the instance. But why require database locking to be
enforced immediately? Many applications require more concurrency.
>
> The specification allows to change keys to "change object identity". What is
> the use case in practice? This approach will kill the reference cache.
>
This is to allow users to update the primary key of a data store
instance, if there is a key field or key fields that map to key columns
in the data store.
> 7. PersistenceCapable
> JDO requests to implement the "PersistenceCapable" interface for all classes
> that are to be made persistent. This requirement produces troubles and
> maintenance work, it bloats application classes and thereby costs resources
> and performance. How should the application programmer implement this
> interface? By manual implementation? Deriving from a common base class is
> not a solution, since it takes away the possibility to store objects derived
> from existing JDK classes.
True, and this is why most users will use tools to provide this
functionality.
> Using a "ReferenceEnhancer" considerably slows
> down the development process and user classes get bloated with database
> implementation.
I don't agree. Tools will seamlessly integrate enhancement in the
development process. All that is needed is to enhance the classes prior
to testing. Check out the Transparent Persistence feature of Forte for
Java to see how easy this really is.
> Java is about to become the No.1 platform for mobile
> development. Java Web Start might start a revolution for application
> distribution. Resource consumption deserves more attention than has recently
> been brought up by Sun. Large database applications also benefit from good
> resource management, since cache overflow typically produces a performance
> fiasco.
>
> It is very well possible for a database engine to keep references to all
> managed objects, without the necessity to modify application classes and
> without wasting resources. There are existing object database engines that
> prove this.
Many of these modify the VM in order to achieve these goals. The
objective of JDO is specifically to not require a modified VM to make
the use of persistence transparent.
>
> 7.2 PersistenceCapable#jdoMakeDirty(String fieldName)
> The JDO-Implementers do not seem to be aware of the fact that the use of
> strings
> - takes away type-safety during development. Errors will not be detected by
> the compiler.
> - cost lots of resources
> - disables efficient refactoring
> Strings have to be avoided at all costs.
>
> We want to work with objects, don't we?
> ...and not with Strings that point to objects.
>
This particular hook allows dynamic tracking of dirty instances. The
only reason for this method is for Array typed fields that cannot
automatically be tracked. The alternative (disallowing Array type
fields entirely, as CMP Entity Beans have done) was not acceptable.
> Why do objects need to be made "dirty"?
> The database engine can perform comparisons.
The performance issue that automatic dynamic tracking solves is with
large caches of mostly clean instances with a few dirty ones. The JDO
implementation only has to keep track of the dirty instances, instead of
having to compare field by field a large number of instances. JDO
provides for a commit operation simply iterating the list of dirty
instances, and constructing a data store update for only those fields
that actually changed.
>
> 7.3 jdoGetObjectId()
> Internal IDs should not be exposed. We want to work with objects, not with
> pointers. If an application programmer can not live without IDs, he can
> easily build his own flavour into his objects. He is free, to use his own
> system (social security number, license plate).
>
This feature allows you to write the JDO identity of a persistence to a
file, transport it around the network, and use it later, separated by
space and time. It is modeled on the EJB Entity Identity concept, and
has been extremely popular when dealing with distributed objects.
> jdoGetTransactionalObjectId()
> ...funny...
> exactly the same specification text as for jdoGetObjectId()
>
The difference is only if the application is changing some key fields in
the transaction (for example, performing some maintenance work on an
instance to correct the key).
> 7.6 to 7.12
> All these methods look very cryptic. Fields are replaced, get int values,
> get "provided" get copied, in the end there are even simple-type-specific
> fetch methods (i.e. fetchDoubleField). Most methods return void? Is this a
> new data type in JDO?
> What's all this for?
These methods are specifically for JDO implementations to use to manage
the values of the persistent fields in persistent instances.
> If this very weird API should serve to change class versions, it's terribly
> designed.
This is not an API that any application developer will use. For the
APIs that are intended for public consumption, please refer to 8.1
through 8.4. Again, most users will never use these methods either.
>
> Classes can use reflection to analyse themselves. Storing a superset of all
> class versions removes the need to use int values to define Fields of
> Classes.
>
> 8. JDOHelper
> Many PersistenceCapable methods are listed here again, in static form. Why?
The JDO implementation needs access to the persistence capable instance
to implement the get/set fields behavior. The reason these methods are
in the static form here is to allow access to a subset of useful
information without requiring the application to be aware of the
existence of the PersistenceCapable interface.
> A good API provides one single path, how a problem is to be implemented. The
> advantages:
> - Code by different implementers uses the same methods. It remains
> interchangeable and further requests for a change of the API remain similar.
> - The API can be modified more easily.
I'd advise against any application developer ever using the
PersistenceCapable interface directly. Perhaps this should be more
explicit in the text.
>
> 8.2 PersistenceManager#makeDirty(Object, String fieldname)
> Again:
> If "makeDirty" is necessary, it should be makeDirty(Object).
The propsed change isn't sufficient. JDO tracks changes to fields not
just persistent instances.
>
> 14. Query
> The Query interface is a complete disaster.
> - It requires the use of String filters. Why not use API methods?
We feel that both a String form of query and an API are useful and
arguably necessary. There are strong arguments for both forms. For
practical purposes, only one form is implemented for JDO 1.0. We expect
an API form to be implemented by the JDO RI, but will have to wait for
JDO 1.0++ for standardization.
> - declareImports, declareVariables, declareParameter, setOrdering, all with
> the use of Strings? Why embed a string programming language into an existing
> language. Who is supposed to parse all this? We do have Java. We do have
> objects. Why not use them?
>
> An alternative approach can be found here:
> http://www.odbms.org/soda/
> The website might not look nice.
> The API is ready for first implementations.
>
> 16. EJB are a different specification. They have nothing to do with JDO. If
> an EJB implementation wishes to use JDO, it may do so.
>
We added this chapter after comments of the nature: what is the
relationship between JDO and EJB? Can I implement EJB with JDO, or JDO
with EJB? How do I use JDO in an application server?
> 18. JDO has nothing to do with XML.
> Schemas are defined by existing classes. If the classes are not present to
> instantiate objects, what is the use of providing metadata?
>
XML is a language for specifying information that cannot be discovered
by introspecting the persistence capable classes. For example, in
Department we have a Collection typed field named employees. What is
the type of the elements of this field? Most relational databases would
require that you store Employees in the elements of Collection. Until
Java implements such metadata itself, we are forced to annotate the Java
classes.
By the way, most of the information in the xml-format metadata is nicely
defaulted, so you don't need it in many cases.
> Why can't the specification concentrate on persisting objects, solving this
> theme nicely, before digging into third worlds?
>
> Further points and questions:
>
> The database world is still waiting for a scalable, beautiful solution to
> handle locking and concurrency issues. Finally we have objects. A system
> with callback events for
> - requested locks
> - relieved locks
> - pre commit
> - post commit
> would provide the application programmer with an extremely scalable
> environment to solve locking issues with tweezers. This would be much more
> potent than current relational UnserializableException crowbars.
>
> Why is JDO exclusively focused on Java? There are other worthwhile
> programming languages (Ruby). A universal Object-Persistence-API would also
> be helpful in other programming languages. The "What about C# and the
> CLR?" - question will come and people will stick to SQL, since it's also
> available on the Microsoft platform.
>
We decided to focus on Java as the platform for JDO. If we wanted to
have a universal platform we would have a much bigger task.
> Who is supposed to implement all this? Who is supposed to understand it? The
> ODMG specification failed because no vendor ever provided a complete
> implementation.
I agree that this was a strong negative for ODMG. The specification was
incrementally implemented but no vendor ever did the whole thing. One
of the nice features of the JCP process under which JDO is being
developed is that prior to publication of the JDO specification, a
reference implementation and compatibility test suite must be
developed.
JDO Reference Implementation is in pre-release form, available for free
download to anyone in the world. Prospective vendors of JDO can use the
Reference Implementation to get a head start on their own
implementations.
> When is the first complete implementation of JDO expected?
This calendar year.
> The process has been extremely slow. In spite of all the big names in the
> JCP, very little has developed forward.
Well, there are several commercial implementations currently available
(currently subsets due to the nature of a not-final specification).
>
> This version 1.0 is not really it?
Well, I'm personally focused on 1.0. Without a successful 1.0, there
won't be a 1.0++.
> Noone will ever use this, except for consultants that make a living of never
> finishing projects.
I beg to differ.
>
> Sorry for these rude and harsh words.
Why make them if you have to apologize for them? Perhaps you could just
have exercised your del key...
Craig
http://access1.sun.com/jdo
> Some sub-optimal designs like HTML, XML and EJB have been hyped to success
> in the past.
>
> Let's not waste another decade with database interfaces that don't do what
> they are supposed to and what would be so easy:
> store objects
>
> Carl Rosenberger
> db4o - database for objects
> http://www.db4o.com
--
Craig Russell 650 786-7819
Architect mailto:Craig....@eng.Sun.com
Sun Microsystems, Inc. http://access1.sun.com/jdo
Carl Rosenberger
> As the author (not the inventor) of the JDO specification, I'd like to
> respond.
Thank you for this very fast and interesting response.
> JDO concerns itself with two categories of class developers: those that
> develop persistence-capable classes and those that develop
> persistence-aware classes.
I don't envy the JDO team for trying to cope with the concerns of all
companies on the JCP list. In my opinion it would be easier to define an
interface for application programmers only.
> JDO is arguably simpler once you have access to a database. You can
> query for instances of persistence-capable classes with only the
> knowledge of the Java class and fields. Contrast this with JDBC where
> you have to know lots of details about the structure of the database
> itself.
I never claimed that JDO does not make certain things easier than JDBC. I
would be afraid though, that a very large and superb improvement is
necessary to attract relational developers. On the long run we don't want 1%
or less of the market. We at least want 50%.
Putting myself into the position of an SQL developer:
"Where is my GROUP BY sum calculation in JDO?"
> For persistence-capable class developers, I'd claim JDO has a
> substantial advantage over any other persistence capability. You can
> often implement Java persistence-capable classes without any knowledge
> whatever of database. No mapping, no type conversions, no special read
> methods, no special write methods.
Yes, we do want to store objects.
What would I do in JDO to store objects with parameterized constructors?
...something like a TreeMap that needs to be restored with its
Comparator....
> JDO allows for defining
> the object model to navigate the application specific closure without
> requiring application-specific data models.
I am not arguing against the principal of storing classes as they come.
A point that JDO did not consider yet:
What if different classes need to access the same data, let's say for
different applications or different tasks within one application? We are
considering a concept that we will call ("namespace alias", "class alias"
and "field alias").
> Unfortunately, having the application keep track of dirty instances
> detracts from one of object programming's major advantages: separation
> of concerns. Once you require each persistence-capable class to track
> its own state with explicit code, you pollute its methods with database
> calls. And it doesn't matter if you delegate this to another
> application class. You still need to insert database calls into the
> class at odd places. It's really easy to miss out an update to a field
> and then the application fails without anyone noticing.
There are two contradicting concerns to consider:
- taking away implementation work from the developer
- ensuring that lock overhead remains small.
In enterprise-scale applications the second point is always more problematic
and much more difficult to debug. Developers typically do not have testlabs
where concurrency can be tested reasonably.
Missing out the insert or update of an object is not one of those "hard to
find" bugs. An explicit setup may very well take away work from the
developer by comparing updated objects and by only locking the changed ones.
> The requirements now are for scalable
> implementations with binary compatibility that can take advantage of
> distributed computing a la J2EE (TM) without writing yet another bunch
> of single purpose code.
"Binary compatibility" is only a problem, if the specification insists upon
the necessity to implement a PersistentCapable interface.
If objects can be stored without modifying classes specifically for the
persistence system, classes are and remain binary compatible.
> WRT the enhancer, have you looked at the strategy for CMP entity beans,
> where the class file provided by the bean developer is analyzed and a
> concrete implementation class constructed? Is there really a difference
> between generating classes and enhancing them?
It is neither necessary to generate classes nor to enhance them.
A side hit:
What happens if you would want to use existing classes in two ways:
for persistent storage and as transient objects.
No concept can distiguish automatically, what the creation of an object
means (persistent or transient?). Here you already need an explicit
mechanism so let's keep the concept fot the entire system.
The enhancer bloats all field set-/get-access in a class with a call into
the database engine. Is this convenient and performant in case you want to
use a class for transient tasks. Would you suggest to create two versions of
the same class?
> Another point that is often missed with a casual read of the
> specification is that the JDO specification does not require a
> post-processor (enhancer).
Yes, I do understand that the enhancer is optional. Not using it would
require the application developer to implement PersistentCapable manually
for every single class.
> A JDO implementation is free to use other
> techniques, and several JDO vendors plan to do just that. Two
> alternatives to the enhancer are pre-processing and direct code (either
> .java or .class) generation from a business model or UML model or
> whatever you are comfortable with.
Now these are just other "enhancers".
> The reason a special concept is used for comparing instances from
> multiple caches is that in JDO, caches can be transactional. In the
> same VM, you can have multiple caches in different transactions, and
> each cache can maintain a different transactional view of the data store
> instances. So in two different PersistenceManager caches, you are
> guaranteed to have two different instances of the same data store
> instance. The instances might compare equal (.equals(Object)) but that
> is up to the implementation of .equals. But both transactional caches
> might be making conflicting changes to the state of the instance. At
> commit time (or earlier if the application chooses) the conflicts will
> be resolved. Of course, if you want locking semantics to be applied,
> you can also choose for the data store to enforce locking at the time
> you retrieve the instance. But why require database locking to be
> enforced immediately? Many applications require more concurrency.
O.K. thanks for this very nice explanation.
That leaves us with three necessary concepts:
== "identity"
equals() "equality"
dbEquals() "database equality"
I would suggest to remove the other unnecessary concepts (e.g. "Non-data
store JDO identity").
> I don't agree. Tools will seamlessly integrate enhancement in the
> development process. All that is needed is to enhance the classes prior
> to testing. Check out the Transparent Persistence feature of Forte for
> Java to see how easy this really is.
I have been evaluating Forte and Netbeans. In my opinion it they are
inferior to JBuilder and IBM's new Eclipse IDE. Debugger and speed of old MS
Visual J++ are yet unmatched. Should I switch to an IDE that makes me less
productive, because I need this new "Transparent Persistence" feature?
Here is a list of drawbacks that come with an enhancer. I have mentioned
some points previously:
- Every single variable reference in your code needs to be capsuled with a
getter/setter that is routed through the database engine.
- You loose a lot of performance.
- Code size increases considerably.
- Can code be obfuscated?
- How are locking issues handled? Would you really want every write access
to lock an attribute?
- If member attributes get activated automatically, when does automatic
deactivation take place? Your application will end up with a very huge
network of objects that can't be gc'd.
- Anything that happens transparently might be fine for someone who
understands it. People moving over from SQL will ask: "When do objects get
stored?"
- Adding and configuring the enhancer in the build script is an additional
hurdle for evaluators.
- How do you debug enhanced code? Some IDEs will even have problems to start
the debugger at all, if code changes after editing it.
- What do you do with classes that need special constructors? How would you
create a TreeMap with a comparator constructor, for instance?
- In many cases you will want parts of your library to be used for
persistent- and non-persistent tasks.
- What is the result of creating an object on it's own? Is it persistent or
transient? You will want different "root" objects in your database. Code
needs to take a decision somehow. I don't see a Java language construct to
differentiate.
> > It is very well possible for a database engine to keep references to all
> > managed objects, without the necessity to modify application classes and
> > without wasting resources. There are existing object database engines
that
> > prove this.
>
> Many of these modify the VM in order to achieve these goals. The
> objective of JDO is specifically to not require a modified VM to make
> the use of persistence transparent.
A specific VM is not necessary either. System.identityHashCode() is there to
identify objects and to map them to internal representations. This is what
our database engine uses and performance proves that this works very nicely.
> > 7.3 jdoGetObjectId()
> > Internal IDs should not be exposed. We want to work with objects, not
with
> > pointers. If an application programmer can not live without IDs, he can
> > easily build his own flavour into his objects. He is free, to use his
own
> > system (social security number, license plate).
> >
>
> This feature allows you to write the JDO identity of a persistence to a
> file, transport it around the network, and use it later, separated by
> space and time. It is modeled on the EJB Entity Identity concept, and
> has been extremely popular when dealing with distributed objects.
Does JDO have to provide a specification for every single usecase?
Again:
If a programmer wishes to have IDs, he may do so by defining his own flavour
and by querying for them. Personally I haven't come across the usecase
"carrying objects around in files" yet.
Replication is the main usecase for identifying multiple versions of
"dbEqual" objects. Replication requires a little more than an ID.
> I'd advise against any application developer ever using the
> PersistenceCapable interface directly. Perhaps this should be more
> explicit in the text.
O.K., thanks for this statement.
That does make it very clear that JDO is strongly focussed on transparent
persistency.
[Queries]
> We expect
> an API form to be implemented by the JDO RI, but will have to wait for
> JDO 1.0++ for standardization.
Interesting!
I am looking forward to it.
> XML is a language for specifying information that cannot be discovered
> by introspecting the persistence capable classes. For example, in
> Department we have a Collection typed field named employees. What is
> the type of the elements of this field? Most relational databases would
> require that you store Employees in the elements of Collection. Until
> Java implements such metadata itself, we are forced to annotate the Java
> classes.
O.K., typesafe Collections are a good example.
I also understand that some metadata is necessary to specify further
information about storage, for indices for instance. The use of XML would
not be necessary though. You could also define a Java API for configuring
the database interface. This would benefit from Java type-safety and reduce
XML typos.
> > Sorry for these rude and harsh words.
>
> Why make them if you have to apologize for them? Perhaps you could just
> have exercised your del key...
Your long answer does prove that my arguments were not entirely
out-of-this-world.
I would love to have a common specification that I could consider worthwhile
implementing for our product. As a very small party in comparison to Sun and
using Sun's beautiful Java language it might not be up to me, to use harsh
words. What other chance do I have though, to work into the direction of a
change?
Thanks again for taking your time to respond with some very interesting
comments.
Kind regards,
Carl
---
Carl Rosenberger
db4o - database for objects - http://www.db4o.com
Juha Lindström
I strongly recommend that you read the JDO specification properly
before starting to spread your lies and misunderstandings of it. From
your comments it is clear that you either haven't read it or you are
just spreading FUD in order to promote your own JDO incompatible (and
IMHO inferior) solution (db4o). JDO is the best hope for us to finally
get rid of the object-relational mapping nightmare and make object
databases get the status they deserve.
You say that old-time object databases failed because they were too
complicated. I wonder if the real reason was the lack of commitment to
common standards...
Juha Lindström
akmal @ city
> Dear Carl,
>
> I strongly recommend that you read the JDO specification properly
> before starting to spread your lies and misunderstandings of it. From
> your comments it is clear that you either haven't read it or you are
> just spreading FUD in order to promote your own JDO incompatible (and
> IMHO inferior) solution (db4o). JDO is the best hope for us to finally
> get rid of the object-relational mapping nightmare and make object
> databases get the status they deserve.
Juha, I'm sure Carl will speak for himself, but I think he has read it. In
Carl's shoes, I would get involved with the JDO initiative (as Luca has
done, for example) and try and influence it from within. If he thinks
there are things seriously wrong with the standard, he should submit his
concerns using the relevant procedure. That way it will be sure to reach
the right people.
>
> You say that old-time object databases failed because they were too
> complicated. I wonder if the real reason was the lack of commitment to
> common standards...
I think standards were part of the problem. As Craig hinted though, it was
more complicated than that. Furthermore, I kind of think of OODBs a bit
like water - they will find their own level.
>
> Juha Lindström
>
>
Greetings from London.
<akmal/>
--
[ >>> OOPSLA 2001 Workshop on "Objects, <XML> and Databases" <<< ]
[ http://www.soi.city.ac.uk/~akmal/oopsla01.dir/01-workshop.html ]
Carl Rosenberger
Dear Juha,
if you insult me for telling lies, please do point out in detail where I am
wrong. I think my comments prove that I have at least read the
specification. I would also be extremely interested in a strong standard to
get rid of the object-relational nightmare. My opinion is: The simpler a
standard would be, the more likely it is that many vendors will support it
completely.
Yes, of course I do have my own commercial interests. I guess everyone
trying to earn a living for all the years he has been sitting in front of
the computer instead of sitting at the beach has some. Progress is driven by
commercial interests.
Carl Rosenberger
> In Carl's shoes, I would get involved with the JDO initiative
> (as Luca has done, for example) and try and influence it from
> within. If he thinks there are things seriously wrong with the
> standard, he should submit his concerns using the relevant
> procedure. That way it will be sure to reach the right people.
Hi Akmal,
as I understand Sun's Java Community Process, it is not up to me to "get
involved". The formation of the JDO "expert group" has been completed a long
time ago.
More than a year ago, I tried to influence the process directly by putting
forward my worries concerning the query interface. I never received a
response or comment. Possibly I did not always choose the "right" words and
the "right" procedings.
I am not sure if my opinions would be benefitial for the current JDO expert
group.
Yes, I do think we should get rid of the "object-relational mapping
nightmare", as Juha writes.
No, I don't think the times are there for transparent persistency, and this
is where JDO is headed.
There is a missing link between the "object-relational nightmare" and
"transparent persistency":
storing objects explicitely and querying the database with an object query
API.
This is where I am headed.
akmal @ city
> akmal @ city wrote:
> > In Carl's shoes, I would get involved with the JDO initiative
> > (as Luca has done, for example) and try and influence it from
> > within. If he thinks there are things seriously wrong with the
> > standard, he should submit his concerns using the relevant
> > procedure. That way it will be sure to reach the right people.
>
> Hi Akmal,
>
> as I understand Sun's Java Community Process, it is not up to me to "get
> involved". The formation of the JDO "expert group" has been completed a long
> time ago.
Carl, this is true. However, I believe the process allows for input not
just from the expert group. Most standards provide a mechanism for public
review and feedback. Craig mentioned v1.0++. This could provide an
opportunity for you before further work is done in that direction.
>
> More than a year ago, I tried to influence the process directly by putting
> forward my worries concerning the query interface. I never received a
> response or comment. Possibly I did not always choose the "right" words and
> the "right" procedings.
Could be that the comments didn't reach the right people.
>
> I am not sure if my opinions would be benefitial for the current JDO expert
> group.
It never hurts to try IMHO.
> Yes, I do think we should get rid of the "object-relational mapping
> nightmare", as Juha writes.
> No, I don't think the times are there for transparent persistency, and this
> is where JDO is headed.
> There is a missing link between the "object-relational nightmare" and
> "transparent persistency":
> storing objects explicitely and querying the database with an object query
> API.
> This is where I am headed.
I suspect the world will go down the JDO route, simply because of the
drive by Sun, the apparent industry backing and the likely implementations
that are emerging.
I'd like to attend the Seminar in London on 16 October(?), but
unfortunately I shall be at OOPSLA.
Carl Rosenberger
> I suspect the world will go down the JDO route, simply because of the
> drive by Sun, the apparent industry backing and the likely implementations
> that are emerging.
I suspect that the world is more likely to remain with JDBC and Oracle.
An excerpt from the JDO home site
http://access1.sun.com/jdo/
"The JDO Vendor community getting on board: Sun Forte(TM) for Java(TM),
Internet Edition; Versant Judo; Object Identity, Inc.; PrismTech;
Orientechnologies; TechTrader; Poet Software; ICS Computer Services; Object
Design; TradeCityCybersoft Rexip; LIBeLIS;"
I don't see a single large relational vendor yet.
The above list contains many companies that used to claim "Yes, we support
ODMG." and never did.
Besides:
Java isn't "the world".
There are other languages that are still strong and others that will become
strong. A database standard that focusses on Java only will never be a world
database standard.
How would you migrate a JDO database application to C# ? (dotGNU ?)
akmal @ city
> akmal @ city wrote:
> > I suspect the world will go down the JDO route, simply because of the
> > drive by Sun, the apparent industry backing and the likely implementations
> > that are emerging.
>
> I suspect that the world is more likely to remain with JDBC and Oracle.
For now :) In any case, JDBC is not competing against JDO:
http://java.sun.com/products/jdbc/related.html
>
> An excerpt from the JDO home site
> http://access1.sun.com/jdo/
> "The JDO Vendor community getting on board: Sun Forte(TM) for Java(TM),
> Internet Edition; Versant Judo; Object Identity, Inc.; PrismTech;
> Orientechnologies; TechTrader; Poet Software; ICS Computer Services; Object
> Design; TradeCityCybersoft Rexip; LIBeLIS;"
>
> I don't see a single large relational vendor yet.
Give them time. Those guys tend to have other things to think about as
well, such as what their existing customers need.
>
> The above list contains many companies that used to claim "Yes, we support
> ODMG." and never did.
True. One of my big gripes about ODMG - very loose in terms of
"compliance" IMHO. A product was "complaint" if a vendor said it was
AFAIR. At least with JDO, a test suite comes as well.
>
> Besides:
> Java isn't "the world".
> There are other languages that are still strong and others that will become
> strong. A database standard that focusses on Java only will never be a world
> database standard.
Hard to say. Things change so quickly today. Maybe see these survey
results:
"Java to overtake C/C++ in 2002"
http://www.zdnet.com/zdnn/stories/news/0,4586,2804967,00.html?chkpt=zdhpnews01
It is important to Sun, because of Java (Sun does not care much for
Microsoft and hence C#). Important enough for ANSI to develop SQLj.
I'm sure there will be other standards.
>
> How would you migrate a JDO database application to C# ? (dotGNU ?)
According to the article above C# is not doing much right now (future of
course may be different). Someone may well identify a potential
opportunity and develop some tools (I wouldn't be surprised).
>
> Kind regards,
> Carl
> ---
> Carl Rosenberger
> db4o - database for objects - http://www.db4o.com
>
>
>
>
>
Cheers,
akmal @ city
[snip]
>
> >
> > The above list contains many companies that used to claim "Yes, we support
> > ODMG." and never did.
>
> True. One of my big gripes about ODMG - very loose in terms of
> "compliance" IMHO. A product was "complaint" if a vendor said it was
> AFAIR. At least with JDO, a test suite comes as well.
OOPS! "complaint" should of course read "compliant".
(I'm sure vendors have had lots of "complaints" too :) )
Carl Rosenberger
> Hard to say. Things change so quickly today. Maybe see these survey
> results:
>
> "Java to overtake C/C++ in 2002"
>
http://www.zdnet.com/zdnn/stories/news/0,4586,2804967,00.html?chkpt=zdhpnews
01
Yes, I did read that article. There was a long discussion at Slashdot about
it. At this second I can't dig a link out, since Slashdot seems to be
reorganizing some of the site.
A point people brought up against the credibility of the study:
It was initiated by IBM.
"Java to overtake C/C++" also implies that it currently is behind, by the
way.
Instead of talking about C# I should have referred to Microsoft's Common
Language Runtime. The CLR is Visual Basic also so no matter how strong C#
will be, there is a very large user base that would be worthwhile targetting
with a database interface API.
Juha Lindström
should have just corrected the points which I think are wrong. Your post
just felt so aggressive that I got carried away.
Anyway, in this post I'll try to be a better man and correct the issues
I believe you got wrong. Note that I'm not an expert on object databases
or JDO and may very well be wrong. I have just read the specification
and
this is what I found from it.
> Yes, we do want to store objects.
> What would I do in JDO to store objects with parameterized constructors?
> ...something like a TreeMap that needs to be restored with its
> Comparator....
I don't get this one. Why not just use the usual:
persistenceManager.makePersistent(myTreeMap);
If you have enhanced the treemap and the comparator classes everything
should
work just fine. Or am I missing something here? I couldn't find any
requirement that all persistent classes should have a default
constructor.
Instead I found that one gets generated by the enhancer if it doesn't
exist (20.16.4).
> "Binary compatibility" is only a problem, if the specification insists upon
> the necessity to implement a PersistentCapable interface.
>
> If objects can be stored without modifying classes specifically for the
> persistence system, classes are and remain binary compatible.
2.1.1 states that every JDO implementation is supposed to work with
classes
enhanced by the reference enhancer. So once you have compiled your
classes
and enhanced them with the reference enhancer, they are and remain
binary
compatible with different JDO implementations.
> A side hit:
> What happens if you would want to use existing classes in two ways:
> for persistent storage and as transient objects.
> No concept can distiguish automatically, what the creation of an object
> means (persistent or transient?). Here you already need an explicit
> mechanism so let's keep the concept fot the entire system.
If your persistence capable object is transient (i.e. it has no
associated
StateManager) it behaves like a "normal" transient object (5.1.1).
> The enhancer bloats all field set-/get-access in a class with a call into
> the database engine. Is this convenient and performant in case you want to
> use a class for transient tasks. Would you suggest to create two versions of
> the same class?
The overhead added by the getters and setters is not an issue. The
methods
contain an if clause that checks if the object is transient before going
into the engine (see 20.16.9-10). Like the following "getter":
X get_field_x () {
if (transient) return x;
else {
if (field_x_is_loaded()) return x;
else return load_the_field_from_the_engine();
}
}
Now how long does it take to evaluate "if (transient)"?
> - Every single variable reference in your code needs to be capsuled with a
> getter/setter that is routed through the database engine.
> - You loose a lot of performance.
See above.
> - Code size increases considerably.
Well, depends on how you define considerably. However, I agree that this
might
be a problem if the resources are very limited (like with a wireless
device
of some sort).
> - Can code be obfuscated?
I don't see any reason why you couldn't first compile your classes then
obfuscate and finally enhance them. However, the order of the steps
cannot
be changed as the ehanced classes contain references to the names of the
fields (e.g. 20.16.1). A minor inconvenience might be the need to
"obfuscate" the XML metadata as well. However, this can be automated
quite
easily.
> - How are locking issues handled? Would you really want every write access
> to lock an attribute?
JDO specification supports optimistic transactions (5.8, 13.3).
> - If member attributes get activated automatically, when does automatic
> deactivation take place? Your application will end up with a very huge
> network of objects that can't be gc'd.
First, if your application maintains a reference to every domain object
it
uses, then there is a problem with the application. I mean, you can
easily
create a network of objects that can't be gc'd even without JDO. That is
usually called a memory leak.
Second, I don't see anything in the specification that forbids you from
implementing any deactivation scheme you wish. As long as it is
transparent
to the user...:)
> - Anything that happens transparently might be fine for someone who
> understands it. People moving over from SQL will ask: "When do objects get
> stored?"
When you commit the transaction. (13.4.4)
> - Adding and configuring the enhancer in the build script is an additional
> hurdle for evaluators.
Agreed, but this shouldn't be any major obstacle.
> - How do you debug enhanced code? Some IDEs will even have problems to start
> the debugger at all, if code changes after editing it.
Just use those System.out.println:s as usual :)
The reference enhancer tries to make debugging easier by preserving
source
line numbers (20.3).
> - What do you do with classes that need special constructors? How would you
> create a TreeMap with a comparator constructor, for instance?
See above.
> - In many cases you will want parts of your library to be used for
> persistent- and non-persistent tasks.
See above.
> - What is the result of creating an object on it's own? Is it persistent or
> transient?
Transient according to the spec. (5.5.1)
> You will want different "root" objects in your database. Code
> needs to take a decision somehow. I don't see a Java language construct to
> differentiate.
You can use the query API provided by the specification (14) to find a
root
object or any other object for that matter. Agreed, I miss an easy
interface
like void setRoot(String name, Object root), Object getRoot(String name)
to
access a root object. However, implementing this interface on top of JDO
is
trivial.
Best regards,
Juha
Craig Russell
Carl Rosenberger wrote:
>
> Craig Russell wrote:
> > As the author (not the inventor) of the JDO specification, I'd like to
> > respond.
>
> Thank you for this very fast and interesting response.
>
> > JDO concerns itself with two categories of class developers: those that
> > develop persistence-capable classes and those that develop
> > persistence-aware classes.
>
> I don't envy the JDO team for trying to cope with the concerns of all
> companies on the JCP list. In my opinion it would be easier to define an
> interface for application programmers only.
>
We started down that path, and there was substantial interest in
defining a binary compatible interface, so that the developer of a
persistence-capable class could run that class on any JDO
implementation, and developers of persistence-aware classes could use
any persistence-capable class. That's the background for the detailed
PersistenceCapable contract.
> > JDO is arguably simpler once you have access to a database. You can
> > query for instances of persistence-capable classes with only the
> > knowledge of the Java class and fields. Contrast this with JDBC where
> > you have to know lots of details about the structure of the database
> > itself.
>
> I never claimed that JDO does not make certain things easier than JDBC. I
> would be afraid though, that a very large and superb improvement is
> necessary to attract relational developers. On the long run we don't want 1%
> or less of the market. We at least want 50%.
>
> Putting myself into the position of an SQL developer:
> "Where is my GROUP BY sum calculation in JDO?"
For this you will need a full query language, such as SQL or OQL.
Neither JDOQL nor EJBQL has this kind of complete processing+query
language capability as an objective.
>
> > For persistence-capable class developers, I'd claim JDO has a
> > substantial advantage over any other persistence capability. You can
> > often implement Java persistence-capable classes without any knowledge
> > whatever of database. No mapping, no type conversions, no special read
> > methods, no special write methods.
>
> Yes, we do want to store objects.
> What would I do in JDO to store objects with parameterized constructors?
> ...something like a TreeMap that needs to be restored with its
> Comparator....
>
> > JDO allows for defining
> > the object model to navigate the application specific closure without
> > requiring application-specific data models.
>
> I am not arguing against the principal of storing classes as they come.
>
> A point that JDO did not consider yet:
> What if different classes need to access the same data, let's say for
> different applications or different tasks within one application? We are
> considering a concept that we will call ("namespace alias", "class alias"
> and "field alias").
JDO permits multiple mappings of the same data to different classes,
which works as long as you don't try to instantiate multiple objects
from the same data and try to update the database from multiple such
objects.
But I might have missed your point...
>
> > Unfortunately, having the application keep track of dirty instances
> > detracts from one of object programming's major advantages: separation
> > of concerns. Once you require each persistence-capable class to track
> > its own state with explicit code, you pollute its methods with database
> > calls. And it doesn't matter if you delegate this to another
> > application class. You still need to insert database calls into the
> > class at odd places. It's really easy to miss out an update to a field
> > and then the application fails without anyone noticing.
>
> There are two contradicting concerns to consider:
> - taking away implementation work from the developer
> - ensuring that lock overhead remains small.
>
I don't consider lock overhead a conflicting concern. Developers must
consider the effect of locking when developing database access
programs. Transparent persistence isn't a magic wand, but by the same
token it doesn't prohibit deployers from adding concurrency
improvements.
JDO was designed with two built-in concurrency models: data store
transactions and optimistic transactions. Beyond these two high level
models, JDO implementations can add more features. The features are
activated either by policy (similar to the way EJB works, using a
deployment descriptor of some type to annotate database accesses) or
explicit by API. Some combination of these will be needed for more
sophisticated application concurrency models. We chose not to define
concurrency models more explicitly until we got some real world
experience.
> In enterprise-scale applications the second point is always more problematic
> and much more difficult to debug. Developers typically do not have testlabs
> where concurrency can be tested reasonably.
>
> Missing out the insert or update of an object is not one of those "hard to
> find" bugs. An explicit setup may very well take away work from the
> developer by comparing updated objects and by only locking the changed ones.
>
> > The requirements now are for scalable
> > implementations with binary compatibility that can take advantage of
> > distributed computing a la J2EE (TM) without writing yet another bunch
> > of single purpose code.
>
> "Binary compatibility" is only a problem, if the specification insists upon
> the necessity to implement a PersistentCapable interface.
>
> If objects can be stored without modifying classes specifically for the
> persistence system, classes are and remain binary compatible.
>
> > WRT the enhancer, have you looked at the strategy for CMP entity beans,
> > where the class file provided by the bean developer is analyzed and a
> > concrete implementation class constructed? Is there really a difference
> > between generating classes and enhancing them?
>
> It is neither necessary to generate classes nor to enhance them.
>
The biggest issue with this point is that databases are usually
completely connected, meaning that once you get one instance, you can
navigate to the rest of the database. If the navigation is truly
transparent, then you have to have a way to instantiate part of an
instance, which means that you have to have a way of detecting access to
a field that was not instantiated. If the navigation requires special
calls to the persistence layer, I'd claim that this is not transparent
at all, and the object model is compromised.
> A side hit:
> What happens if you would want to use existing classes in two ways:
> for persistent storage and as transient objects.
> No concept can distiguish automatically, what the creation of an object
> means (persistent or transient?). Here you already need an explicit
> mechanism so let's keep the concept fot the entire system.
>
I didn't follow your point. The class itself can be used for persistent
or transient instances, with no effect on the class. It is the programs
that access the class, known as persistence-aware in JDO, that need to
know if the instance is supposed to be persistent or not.
> The enhancer bloats all field set-/get-access in a class with a call into
> the database engine. Is this convenient and performant in case you want to
> use a class for transient tasks. Would you suggest to create two versions of
> the same class?
The call in to the database engine is only done in the case of
persistent or transactional instances, never for transient instances.
So the overhead of a field access is exactly one local method call, not
a call outside the instance.
>
> > Another point that is often missed with a casual read of the
> > specification is that the JDO specification does not require a
> > post-processor (enhancer).
>
> Yes, I do understand that the enhancer is optional. Not using it would
> require the application developer to implement PersistentCapable manually
> for every single class.
>
> > A JDO implementation is free to use other
> > techniques, and several JDO vendors plan to do just that. Two
> > alternatives to the enhancer are pre-processing and direct code (either
> > .java or .class) generation from a business model or UML model or
> > whatever you are comfortable with.
>
> Now these are just other "enhancers".
>
I'm really missing your point now...
> > The reason a special concept is used for comparing instances from
> > multiple caches is that in JDO, caches can be transactional. In the
> > same VM, you can have multiple caches in different transactions, and
> > each cache can maintain a different transactional view of the data store
> > instances. So in two different PersistenceManager caches, you are
> > guaranteed to have two different instances of the same data store
> > instance. The instances might compare equal (.equals(Object)) but that
> > is up to the implementation of .equals. But both transactional caches
> > might be making conflicting changes to the state of the instance. At
> > commit time (or earlier if the application chooses) the conflicts will
> > be resolved. Of course, if you want locking semantics to be applied,
> > you can also choose for the data store to enforce locking at the time
> > you retrieve the instance. But why require database locking to be
> > enforced immediately? Many applications require more concurrency.
>
> O.K. thanks for this very nice explanation.
>
> That leaves us with three necessary concepts:
> == "identity"
> equals() "equality"
> dbEquals() "database equality"
>
> I would suggest to remove the other unnecessary concepts (e.g. "Non-data
> store JDO identity").
This is for a special case where there is no identity possible. The
case in relational databases is non-primary-key tables, where for
performance reasons no key column is defined. This is typically for log
tables where every insert is unique without requiring an artificial key.
>
> > I don't agree. Tools will seamlessly integrate enhancement in the
> > development process. All that is needed is to enhance the classes prior
> > to testing. Check out the Transparent Persistence feature of Forte for
> > Java to see how easy this really is.
>
> I have been evaluating Forte and Netbeans. In my opinion it they are
> inferior to JBuilder and IBM's new Eclipse IDE. Debugger and speed of old MS
> Visual J++ are yet unmatched. Should I switch to an IDE that makes me less
> productive, because I need this new "Transparent Persistence" feature?
>
I was not trying to get you to switch IDE's, which for most people is
akin to switching banks (why do it if you are not forced to). My point
was just that tools will make the enhancement process painless, and of
no more concern than making sure that you compile dependent classes
before trying to execute your application.
> Here is a list of drawbacks that come with an enhancer. I have mentioned
> some points previously:
>
> - Every single variable reference in your code needs to be capsuled with a
> getter/setter that is routed through the database engine.
Please see the description of the get/set methods in Chapter 20 to see
that this is completely incorrect.
> - You loose a lot of performance.
You will have to explain this in more detail. You basically have two
choices: intercept all accesses, which means you can optimize things
like updating a small number of objects from a large result set; or
completely instantiate all objects of interest, and then at commit time
go through the entire list checking for differences.
> - Code size increases considerably.
I'm not going to debate this point. The issue is how much bigger the
code gets, compared to the functionality that it gives you. I hope you
are not suggesting that the "instance compare at commit" has no code
associated with it? So there is overhead no matter what your
persistence technique.
> - Can code be obfuscated?
I don't see any issues. If there is a problem it's either a bug in the
obfuscator or a bug in the enhancer.
> - How are locking issues handled? Would you really want every write access
> to lock an attribute?
There are many locking and concurrency policies possible, and I expect
that JDO vendors will pay attention to this issue. JDO doesn't try to
prescribe or proscribe locking policies, but just get out of the way.
> - If member attributes get activated automatically, when does automatic
> deactivation take place? Your application will end up with a very huge
> network of objects that can't be gc'd.
I don't see this as different from alternatives that bring in all
interesting objects at the beginning. This just isn't a difference with
JDO.
> - Anything that happens transparently might be fine for someone who
> understands it. People moving over from SQL will ask: "When do objects get
> stored?"
Objects get stored when they need to be (either at query time if you are
performing a query that depends on changed values, or at commit time).
> - Adding and configuring the enhancer in the build script is an additional
> hurdle for evaluators.
That's why tools will do it for you. Only folks who need more control
will add enhancement to construction makefiles.
> - How do you debug enhanced code? Some IDEs will even have problems to start
> the debugger at all, if code changes after editing it.
Enhancement doesn't alter the debugging information in a bad way. This
is a specific requirement in the JDO specification.
> - What do you do with classes that need special constructors? How would you
> create a TreeMap with a comparator constructor, for instance?
> - In many cases you will want parts of your library to be used for
> persistent- and non-persistent tasks.
Not an issue.
> - What is the result of creating an object on it's own? Is it persistent or
> transient? You will want different "root" objects in your database. Code
> needs to take a decision somehow. I don't see a Java language construct to
> differentiate.
Instances newly constructed are transient. Unless they are told to
become persistent or associated with a persistent instance they stay
transient.
>
> > > It is very well possible for a database engine to keep references to all
> > > managed objects, without the necessity to modify application classes and
> > > without wasting resources. There are existing object database engines
> that
> > > prove this.
These have varying degrees of transparency, as discussed above.
> >
> > Many of these modify the VM in order to achieve these goals. The
> > objective of JDO is specifically to not require a modified VM to make
> > the use of persistence transparent.
>
> A specific VM is not necessary either. System.identityHashCode() is there to
> identify objects and to map them to internal representations. This is what
> our database engine uses and performance proves that this works very nicely.
>
You still cannot get away from the wide connectivity issue. The closure
of persistent references from an instance are often the entire database.
We picked XML as a commonly used format for specifying nested
information with universally-understood rules. I personally hate XML
for direct editing, but with a tool that keeps you from hurting
yourself, it is pretty good.
> > > Sorry for these rude and harsh words.
> >
> > Why make them if you have to apologize for them? Perhaps you could just
> > have exercised your del key...
>
> Your long answer does prove that my arguments were not entirely
> out-of-this-world.
>
> I would love to have a common specification that I could consider worthwhile
> implementing for our product. As a very small party in comparison to Sun and
> using Sun's beautiful Java language it might not be up to me, to use harsh
> words. What other chance do I have though, to work into the direction of a
> change?
>
> Thanks again for taking your time to respond with some very interesting
> comments.
>
I hope that you would consider using parts of JDO that are of interest
and letting the rest of us know which parts you think should be
eliminated, with some analysis of the corresponding reduction in
portability.
Regards,
Craig
> Kind regards,
> Carl
> ---
> Carl Rosenberger
> db4o - database for objects - http://www.db4o.com
--
Carl Rosenberger
> Your post just felt so aggressive that I got carried away.
Sorry, indeed I am aggressive but I am open for all constructive
discussions. If you take a look at the provisional outcome:
This is the first discussion on specific technical details of the JDO spec
here for a long time. It is a great pleasure that Craig Russell took the
time to anwer all the points I brought up. I have learned a lot about JDO
from his comments.
As I understand it now, JDO's focus is on providing transparent persistency.
Extended complex query functionality is not part of the specification yet
for good. JDO is the replacement for the ODMG specification. The interface
is intended to match the concepts of the database engines of the old-time
object database vendors. If all of them provide complete support for JDO,
there indeed is a good chance that users will not be locked in with a
specific vendor.
We are headed in a different direction:
We want to attract relational guys with an explicit storage interface and
the strong query support we are working on. Additionally we want to keep our
very small memory footprint at high performance to continue to be a good
choice for mobile platforms like EPOC. Explicit activation and deactivation
of object hierarchies is intended to allow application developers full
control over memory usage. We also want to target Microsoft's CLR with our
engine and we refrain from the effort necessary to provide another enhancer
and another plugin for the development environment if we can manage without.
Approaches simply are *very* different so it is no wonder that we come up
with different arguments to back them up.
Good luck for JDO!
Concerning your points:
> > What would I do in JDO to store objects with parameterized constructors?
> > ...something like a TreeMap that needs to be restored with its
> > Comparator....
>
> I don't get this one. Why not just use the usual:
> persistenceManager.makePersistent(myTreeMap);
I don't know the specification of the enhancer exactly but I am afraid there
will be problems to choose the right constructor. Some constructors need a
persistent parameter which may differ, depending how the object looked like,
when it was stored. For many JDK classes, restoring all non-transient fields
will not work, since lots of essential internal fields are declared
transient and the initialization mechanism is dependant on the called
constructor. Believe me, we have tried it. Things get very tricky and very
awful if you use different JDKs since implementations and field names of the
same JDK classes differ.
We use a workaround to allow to define what we call "translators" for these
special cases:
public interface ObjectTranslator {
public Object onStore(ObjectContainer container, Object
applicationObject);
public void onActivate(ObjectContainer container, Object
applicationObject, Object storedObject);
public Class storedClass();
}
"Translators" for JDK2 collection classes are provided and built in, so you
don't even notice the use. A "translator" library can develop over times and
users may very well exchange implementations for exotic classes. For very
difficult cases a "serializable translator" can be used, of course taking
away the possibility to query for members.
> > "Binary compatibility" is only a problem, if the specification insists
upon
> > the necessity to implement a PersistentCapable interface.
> >
> > If objects can be stored without modifying classes specifically for the
> > persistence system, classes are and remain binary compatible.
>
> 2.1.1 states that every JDO implementation is supposed to work with
> classes
> enhanced by the reference enhancer. So once you have compiled your
> classes
> and enhanced them with the reference enhancer, they are and remain
> binary
> compatible with different JDO implementations.
Yes, I understand that high goal and it would be great if it turns out to
work.
My point was that there would be no need to discuss binary compatibility if
classes would not need to be enhanced at all. As Craig has correctly pointed
out:
Transparent persistency is only possible if you enhance classes or if you
have hooks in the VM. If JDO is headed towards transparent persistency:
Enhancement is the way to go. In this case I do think that the requirement
for binary compatibility does make sense. It cleans up with the enhancer
tools mess and if enhancers are built into IDEs (hopefully JBuilder will
follow), database vendors do not even have to write enhancers themselves.
Out of the stomach I also think that binary compatibility is a good choice
in order not to have different behaviour on different VMs.
> If your persistence capable object is transient (i.e. it has no
> associated
> StateManager) it behaves like a "normal" transient object (5.1.1).
> }
> Now how long does it take to evaluate "if (transient)"?
O.K. agreed.
That's really not a performance problem, if minimizing the slightest sources
of resource consumption is not an issue.
> > - Code size increases considerably.
>
> Well, depends on how you define considerably. However, I agree that this
> might
> be a problem if the resources are very limited (like with a wireless
> device
> of some sort).
Yes, this is where I am looking from. Future Java cards might very well also
be capable of running object databases.
> > - Can code be obfuscated?
>
> I don't see any reason why you couldn't first compile your classes then
> obfuscate and finally enhance them. However, the order of the steps
> cannot
> be changed as the ehanced classes contain references to the names of the
> fields (e.g. 20.16.1). A minor inconvenience might be the need to
> "obfuscate" the XML metadata as well. However, this can be automated
> quite easily.
It was just a question.
Probably you are right.
Will reflection remain possible, by the way?
> > - How are locking issues handled? Would you really want every write
access
> > to lock an attribute?
>
> JDO specification supports optimistic transactions (5.8, 13.3).
Craig's comment in the other mail:
"We chose not to define concurrency models more explicitly until we got some
real world experience."
Honest words and not the worst of choices.
> > - If member attributes get activated automatically, when does automatic
> > deactivation take place? Your application will end up with a very huge
> > network of objects that can't be gc'd.
>
> Second, I don't see anything in the specification that forbids you from
> implementing any deactivation scheme you wish. As long as it is
> transparent
> to the user...:)
Now how would that work?
Counting references is not possible, I am afraid.
commit() actually is the JDO concept, to hollow instances. That is a handle.
> > - What is the result of creating an object on it's own? Is it persistent
or
> > transient?
>
> Transient according to the spec. (5.5.1)
O.K., I didn't understand this in the first place.
Thanks for your comments.
I will stop yelling against JDO for the time being, now that I have realized
that we are headed in a different direction.
Brian Smith
> >
> > > A JDO implementation is free to use other
> > > techniques, and several JDO vendors plan to do just that. Two
> > > alternatives to the enhancer are pre-processing and direct code (either
> > > .java or .class) generation from a business model or UML model or
> > > whatever you are comfortable with.
> >
> > Now these are just other "enhancers".
I think that there should be a reference implementation that does
enhancement at class loading time. I think that this capability should
be a prereqisite for completion of the spec.
> > - You loose a lot of performance.
>
> You will have to explain this in more detail. You basically have two
> choices: intercept all accesses, which means you can optimize things
> like updating a small number of objects from a large result set; or
> completely instantiate all objects of interest, and then at commit time
> go through the entire list checking for differences.
Here's an argument about why the current enhancement scheme hurts
performance:
PersistenceCapable requires a lot of methods to be implemented by the
class itself. Each class enhanced by a common enhancer will have an
almost identical implementation of PersistenceCapable. Because there
are multiple implementations of these methods, the JVM will never be
able to inline calls to these methods. In order to do inlining, all
calls through PersistenceCapable would have to be handled by a single
class. This problem could be avoided if the PersistenceCapable class
had, for example, a getPersistanceInfo() method that returned an
object of class PersistanceInfo. PersistanceInfo would be an abstract
class (since PersistenceInfo currently has fields) that defines all
the methods and fields that are currently defined in
PersistanceCapable. Then, the enhancer-provider could provide a single
class that fully implements this PersitenceInfo contract, which would
greatly increase the odds of inlining by the JVM. The JDO spec could
even mandate that JDO implementations must provide a default
implementation of PersistenceInfo that is accessable from
StateManager. Then the enhancer vendor could choose to provide his own
implementation, or use the implementation provided by data-access
vendor.
This would also let an enhancer reduce some of the bloat of
PersistanceCapable classes because they wouldn't necessarily have to
define all the methods and fields.
> > - Code size increases considerably.
>
> I'm not going to debate this point. The issue is how much bigger the
> code gets, compared to the functionality that it gives you. I hope you
> are not suggesting that the "instance compare at commit" has no code
> associated with it? So there is overhead no matter what your
> persistence technique.
But the JDO specification does not seem to put any effort in reducing
the bloat of enhancement. The PersistanceCapable contract has many
methods that seem unnecessary as they are provided by StateManager. In
particular, the methods defined in section 7.1, 7.2, 7.4, 7.5, 7.7,
7.8,, 7.9, 7.10 will all delegate to the StateManager, right? So, why
not have a getStateManager() instead?. For example, instead of:
pc.jdoGetPersistenceManager()
use:
pc.getStateManager().getPersistenceManager()
Also, why are the the static fields in 7.11 defined in the
PersistanceCapable contract? You are just replicating the same symbols
with the same values in every class. Shouldn't they be defined in a
central location?
Thanks,
Brian Smith
Steven Shaw
[huge snip]
> > 18. JDO has nothing to do with XML.
> > Schemas are defined by existing classes. If the classes are not present to
> > instantiate objects, what is the use of providing metadata?
> >
>
> XML is a language for specifying information that cannot be discovered
> by introspecting the persistence capable classes. For example, in
> Department we have a Collection typed field named employees. What is
> the type of the elements of this field? Most relational databases would
> require that you store Employees in the elements of Collection. Until
> Java implements such metadata itself, we are forced to annotate the Java
> classes.
>
> By the way, most of the information in the xml-format metadata is nicely
> defaulted, so you don't need it in many cases.
Will this requirement for additional metadata go away when generics
are introduced to the Java language?