Re: [persevere] Persevere on Google App Engine

[Peter Svensson]

This is very good news. I have quite a lot to do, but a data source adapter would be something that would benefit essentially all SSJS projects, so maybe we could form a joint team? I'll help as much as I can.
Cheers,
PS

On Fri, Apr 10, 2009 at 3:57 PM, Kris Zyp <kri...@gmail.com> wrote:

With Google's announcement of support for Java in their App Engine,
Persevere now has the potential to run on GAE. I have reviewed the GAE
docs, and it doesn't look like there would be any serious problems with
running Persevere. However, GAE does prevent file write access, so
Persevere's default storage engine could not run, but within the context
GAE, I would think it would be preferable to use Google's provided data
store anyway (the BigTable database). Therefore, the main effort that
would be required would be writing a data source adapter for Google's
data store API for their database. The BigTable database would actually
fit very nicely with Persevere, since I believe it supports very dynamic
table structures with expando properties on objects. One would probably
be able to retain pretty comprehensive support for Persevere's broad
range of persistence capabilities on GAE.

Support for GAE is therefore something I would really like to see added
to Persevere. Besides noting that this is on the roadmap, I also wanted
to see if anyone was interested in working on this. Creating data source
adapters is a relatively self-contained project (in comparison to other
feature additions), so it might be a reasonable, fun project. If not, I
do still plan on adding support for GAE, albeit it will probably remain
lower priority than bug fixes/stabilization, and some of the more
immediate efforts.

Thanks,

--
Kris Zyp
SitePen
(503) 806-1841
http://sitepen.com

[Robert Thurnher]

2009/4/10 Peter Svensson <psve...@gmail.com>:

> This is very good news. I have quite a lot to do, but a data source adapter
> would be something that would benefit essentially all SSJS projects, so
> maybe we could form a joint team? I'll help as much as I can.

I'd like to contribute to that as well (as time permits).
Already did something actually pretty similar for Helma NG, i.e., a
Rhino JS layer wrapping Hibernate API for convenient use. [1]

[1] http://dev.helma.org/ng/Persistence+API

-- Robert T.

[Kevin Dangoor]

To be honest, my own interest in App Engine has waned considerably
since the initial announcement. Jonathan Ellis, after a fair bit of
experimentation, confirmed what I was thinking:

http://spyced.blogspot.com/2008/08/app-engine-conclusions.html

I'm much more bullish on the approach taken by Amazon (and Rackspace,
and every other virtualization provider). App Engine may feel like
"hey, easy scaling!", but it gives you far less control while at the
same time making simple apps harder than they need be.

Will GAE continue to get better over time? Without a doubt... but then
so will all of the virtualized server provides *and* the countless
open source projects that build useful infrastructure than can run
there. (I just added a worker queue to Bespin... all I had to do was
grab beanstalkd and the Python beanstalkc client. simple. A job queue
on GAE? now possible with the cron support, but one more thing to roll
yourself.)

Kevin

--
Kevin Dangoor

work: http://labs.mozilla.com/
email: k...@blazingthings.com
blog: http://www.BlueSkyOnMars.com

[Kris Zyp]

Robert Thurnher wrote:
> 2009/4/10 Peter Svensson <psve...@gmail.com>:
>
>> This is very good news. I have quite a lot to do, but a data source adapter
>> would be something that would benefit essentially all SSJS projects, so
>> maybe we could form a joint team? I'll help as much as I can.
>>
>
>

Good idea, perhaps we should start working on a database/datastore API,
so we could create interoperable datastore adapters that could be
utilized various frameworks. It would be great if there was a GAE
BigTable adapter that could be used by both Persevere and Helma. I think
both systems already have MySQL adapters, but shared code might have
robustness benefits. The new JavaScriptDB that I have just recently
built for Persevere might be really useful for others frameworks as
well, as it is designed specifically for high-performance scalable
storage of Rhino/JS objects.

That being said, such an endeavor is certainly non-trivial. The surface
area between JS persistence mapping mechanism and the database is
actually pretty large, and making interoperable API maybe very difficult
with the different architectures out there, I am not sure.

> I'd like to contribute to that as well (as time permits).
> Already did something actually pretty similar for Helma NG, i.e., a
> Rhino JS layer wrapping Hibernate API for convenient use. [1]
>
> [1] http://dev.helma.org/ng/Persistence+API
>
>

Very cool, sounds great.

Kris

[George Moschovitis]

> Good idea, perhaps we should start working on a database/datastore API,
> so we could create interoperable datastore adapters that could be
> utilized various frameworks.

A database API is really needed, here are my ideas:

http://github.com/gmosx/nitro/blob/63d1e31d625dcafdda337a6df92d09f1c974aed2/lib/database.js
http://github.com/gmosx/nitro/blob/63d1e31d625dcafdda337a6df92d09f1c974aed2/lib/database/rdb.js

perhaps we should also make this compatible with the Client side apis
(HTML5)


-g.

[Kris Zyp]

This looks like a reasonable API, but it highlights that we may need
multiple levels of APIs in this area (and we been thinking of different
things when we talk about database APIs). In my case, this is
insufficient, a little bit higher-level API is necessary. Persevere
utilizes data store plugin API that abstracts away the differences in
query languages. This level of abstraction is necessary for what we are
doing. Providing a data store like Google's BigTable, that exposes a
completely different query language means it would not work in place of
our JavaScriptDB data store or MySQL data store. Even the differences
between different SQL dbs are painful and lead to incompatible data
sources without further abstraction. The API that Persevere uses
standard query structure across plugins that should actually be a pretty
unifying query structure for all of us; JavaScript itself. Essentially a
query is defined by a function, of the same form as a function that is
passed to an Array.prototype.filter function. The datastore adapter can
then analyze the AST of the function and translate the conditions
defined by the AST into the appropriate query language of the underlying
data store, fully abstracting away the differences in the datastores.
This is interface currently used by Persevere to provide different
access to different data sources. Datastores can also reject queries as
something they can't understand (since it unreasonable to try to convert
any arbitrary JS AST into another native query), in which case the query
can be executed by Rhino (with the associate slower performance of O(n)
execution instead of O(log n) indexed access). (Also in reality, I pass
the AST nodes directly instead of passing a function, but using a
function would be fine, especially if it was a JS API instead of a Java
API).

So maybe we want a low-level database API and higher-level datastore
API? Or maybe what I am doing in Persevere is such completely different
architecture from what others are doing that there would not be any
benefit (at least yet) from standardizing a higher-level API that would
permit transparently interchangeable datastores (like swapping BigTable
for MySQL or JavaScriptDB with rewriting application code).

Thanks,
Kris

[Hannes Wallnoefer]

On Apr 11, 1:43 am, Kris Zyp <kris...@gmail.com> wrote:
> Robert Thurnher wrote:

> > 2009/4/10 Peter Svensson <psvens...@gmail.com>:

>
> >> This is very good news. I have quite a lot to do, but a data source adapter
> >> would be something that would benefit essentially all SSJS projects, so
> >> maybe we could form a joint team? I'll help as much as I can.
>
> Good idea, perhaps we should start working on a database/datastore API,
> so we could create interoperable datastore adapters that could be
> utilized various frameworks. It would be great if there was a GAE
> BigTable adapter that could be used by both Persevere and Helma.

I've started working on a google datastore api for Rhino/Helma NG, see
the new (third) item on this list:

http://dev.helma.org/ng/Persistence+API/

This maps entities and their properties from google's low level api to
native JS objects providing to the outside a more or less standard JS
interface (constructor and prototype chain, constructor takes property
holder object as argument).

Currently supported are primitive values and object references,
haven't done any work on collections and queries so far. And objects
have to be saved before they can be used as property values, so yes,
it's very alpha.

The module uses a few Helma-isms such as export() and include() which
are built on top of the securable modules infrastructure. It should be
pretty simple to port to other Rhino-based environments.

Hannes

[Kris Zyp]

I don't see where this does any abstraction of google's query mechanism.
Wouldn't applications still need to write GAE-specific queries? Maybe
you were aiming for a lower-level API...
Kris

[Hannes Wallnoefer]

2009/4/13 Kris Zyp <kri...@gmail.com>:

>
>
>
> Hannes Wallnoefer wrote:
>>
>> I've started working on a google datastore api for Rhino/Helma NG, see
>> the new (third) item on this list:
>>
>> http://dev.helma.org/ng/Persistence+API/
>>
>> This maps entities and their properties from google's low level api to
>> native JS objects providing to the outside a more or less standard JS
>> interface (constructor and prototype chain, constructor takes property
>> holder object as argument).
>>
>> Currently supported are primitive values and object references,
>> haven't done any work on collections and queries so far. And objects
>> have to be saved before they can be used as property values, so yes,
>> it's very alpha.
>>
>> The module uses a few Helma-isms such as export() and include() which
>> are built on top of the securable modules infrastructure. It should be
>> pretty simple to port to other Rhino-based environments.
>>
>>
> I don't see where this does any abstraction of google's query mechanism.
> Wouldn't applications still need to write GAE-specific queries? Maybe
> you were aiming for a lower-level API...

As I wrote, queries aren't supported yet, as are some other things.
Google datatore is pretty limited in what queries can do (only <, <=,
=, >=, >, and logical and), and I'll try to map that into javascript
in in a way that makes it feel natural (suggestions welcome). However,
the goal for Helma is not to abstract away from the db backend to some
common query language.

I found your account of the unified query approach in Persevere quite
interesting. Analyzing the AST tree of a function and mapping it on a
native db query is a good approach for backends that have powerful
query functionality. Maybe for Google datastore the CouchDB approach
to actually evaluate the function on the data and store the result in
an incremental index would be an interesting option.

Hannes

> Kris
>
> >
>

[Kris Zyp]

Yeah, I didn't think that was your goal, and I certainly don't think
everyone needs to do that. But this is an important functionality that
Persevere provides, so that RESTful queries can be made directly in URLs
and safely converted to the appropriate low-level storage API. So in
terms of truly interoperable data stores that can be swapped out with
other data stores, Persevere requires this type of higher-level API.

> I found your account of the unified query approach in Persevere quite
> interesting. Analyzing the AST tree of a function and mapping it on a
> native db query is a good approach for backends that have powerful
> query functionality. Maybe for Google datastore the CouchDB approach
> to actually evaluate the function on the data and store the result in
> an incremental index would be an interesting option.
>
>

Surely GAE's BigTable already has indexes though doesn't it? This is one
of the big reasons why I avoided having Persevere use CouchDB's
view-server style of indexing. Most storage engines/databases already
provide indexing. Doing indexing (possible again) at a higher level in
the application is extremely inefficient compared to doing indexing
directly at the storage level. This is why CouchDB generally is not very
competitive with traditional databases when it comes to performance,
whereas Persevere can leverage storage-level indexing performance.
Kris

[Rav]

Hi
So is there any Persevere Big Table adapter already?
Rafał

[sleepnova]

I'm very interested in this potential, too. Would any one kindly
provide more information or reference (maybe implementation detail or
src to go) so that I can investigate into?