Disapointed by the speed of the db backend ...

[manatlan]

I planned to migrate a python app ... but my tests on google engine
was really bad, compare to sqlite or mysql.
I had just make a model for "users", with 4 fields, which can have
zero-to-many marks in another model "marks" with 3 fields.
And try to get the higher, the lower and the average mark for each
user.
It is really really slow, on appspot.com, compare to a real db where
it could done in one select (select max(mark), min(mark),
avg(mark) ...)
(i filled my db with 40 users which have each one 10 marks)

- is there a trick to do this with datastore, without parsing all
data ?

ok, i'm pretty sure that a page could be served at the same speed for
a australia boy, or a french boy ... but I asked myself if a can do
better ? (index are well defined), where can I be wrong ?!

[Lee O]

I have yet to do, or even see any speed tests by a majority. I think most of us assumed this would just be very very fast...

Can we get any official word on this? Are the servers slower in the beta phase than the intended release speed?

--
Lee Olayvar
http://www.leeolayvar.com

[Brett Morgan]

Instead of calculating the results at query time, caculate them when
you are adding the records. This means that displaying the results is
just a lookup, and that the calculation costs are amortised over each
record addition.

[manatlan]

> Instead of calculating the results at query time, caculate them when
> you are adding the records.

of course I had thought at that .... and if I migrate my app, I will
need to do that ... sure.
But I simple asked why it's too slow than another db, for this kind of
needs.
and if there are tricks to replace avg(), min(), max()

[Brett Morgan]

It might look almost look like a sql db when you squint, but it's
optimised for a totally different goal. If you think that each
different entity you retrieve could be retrieving a different disk
block from a different machine in the cluster, then suddenly things
start to make sense. avg() over a column in a sql server makes sense,
because the disk accesses are pulling blocks in a row from the same
disk (hopefully), or even better, all from the same ram on the one
computer. With DataStore, which is built on top of BigTable, which is
built on top of GFS, there ain't no such promise. Each entity in
DataStore is quite possibly a different file in gfs.

So if you build things such that web requests are only ever pulling a
single entity from DataStore - by always precomputing everything -
then your app will fly on all the read requests. In fact, if that
single entity gets hot - is highly utilised across the cluster - then
it will be replicated across the cluster.

Yes, this means that everything that we think we know about building
web applications is suddenly wrong. But this is actually a good thing.
Having been on the wrong side of trying to scale up web app code, I
can honestly say it is better to push the requirements of scaling into
the face of us developers so that we do the right thing from the
beginning. It's easier to solve the issues at the start, than try and
retrofit hacks at the end of the development cycle.

brett

[flashpad]

could you re-explain this in just a little bit simpler english?

On Apr 12, 1:02 am, "Brett Morgan" <brett.mor...@gmail.com> wrote:

[dimrub]

On Apr 12, 11:02 am, "Brett Morgan" <brett.mor...@gmail.com> wrote:
> Yes, this means that everything that we think we know about building
> web applications is suddenly wrong. But this is actually a good thing.
> Having been on the wrong side of trying to scale up web app code, I
> can honestly say it is better to push the requirements of scaling into
> the face of us developers so that we do the right thing from the
> beginning. It's easier to solve the issues at the start, than try and
> retrofit hacks at the end of the development cycle.

This is all very fine, however, consider that many of the sites are
not meant (and most probably never will) serve more than a few
hundreds hits per day, and for them to be basically redeveloped using
this frame of mind is impractical. But also consider, that, according
to a small test I've run, it takes GAE almost 3 seconds to save in the
datastore a miserable 50 of dummy records (consisting of just 2 text
fields). This means, for example, that my application, that
occasionally needs to create several hundreds of records upon a single
hit, just won't work - it hits the time limit appointed to a single
hit, and just fails.

[Lee O]

..ouch, :/

[barryhunter]

This might be a very naive observation, but I perhaps wonder then if
GAE is the tool for you.

As I see it the App Engine is for applications that are meant to
scale, scale and really scale. Sounds like an application with a few
hundred hits daily could easily run on traditional hosting platforms.

It's a completely different mindset.

[Lee O]

Very true.

Myself, i use it as a nice Python host ;P

[Randy Johnson]

I don't think that is really the point.  The overall point is the speed of the Db right?  He has a couple hundred hits at the click of a button now and is complaining about the speed... What about someone who has several thousand / hundred thousand or even millions...

-Randy

[barryhunter]

Again maybe I am missing something, but the datastore isnt designed to
be super fast at the small scale, but rather handle large amounts of
data, and be distributed. (and because its distributed it can appear
very fast at large scale)

So you break down your database access into very simple processes.
Assume your database acess is VERY slow, and rethink how you do
things. (Of course the peice in the puzzle 'we' are missing is
MapReduce! - the 'processing' part of the BigTable mindset)

On Apr 12, 6:02 pm, "Randy Johnson" <program...@cfconcepts.com> wrote:
> I don't think that is really the point. The overall point is the speed of
> the Db right? He has a couple hundred hits at the click of a button now and
> is complaining about the speed... What about someone who has several
> thousand / hundred thousand or even millions...
>
> -Randy
>

> On Sat, Apr 12, 2008 at 11:06 AM, barryhunter <BarryBHun...@googlemail.com>

[DennisP]

Hmm, don't know that I'd go quite that far....scalability is important
but it would also be nice for our web pages to be fast.

It's nice that a page will be no slower with a million users than with
one user, but if "no slower" means "it will still take 4 seconds
serverside" that's a little less satisfying.

> > > > hit, and just fails.- Hide quoted text -
>
> - Show quoted text -

[Brett Morgan]

And how long does it take to save one record with two lists of 50 numbers?

[Brett Morgan]

Can you come up with some questions? I don't understand what you don't
understand. So if you can begin by telling me a bit of a story about
where you are at, understanding wise, I'll start tailoring for you.
Help me here =)

[Brett Morgan]

On Sun, Apr 13, 2008 at 3:02 AM, Randy Johnson
<progr...@cfconcepts.com> wrote:
> I don't think that is really the point. The overall point is the speed of
> the Db right? He has a couple hundred hits at the click of a button now and
> is complaining about the speed... What about someone who has several
> thousand / hundred thousand or even millions...
>
> -Randy

Remember what GFS and BigTable were originally designed for. Each
BigTable entry contained a whole web page, and all the data relating
to that web page as the various stages of the google processing
pipeline are applied to the page. So storing two numbers in a BigTable
entry is like putting a person in a 747, then complaining how long it
takes to get the person 50 feet along the ground in said 747 - it
would be quicker to get the person to walk.

The power of BigTable comes to the fore when you fill the 747 with
people, fire up the engines, and then get the aircraft to cruising
altitude. That's when you are using the tool properly.

[Brett Morgan]

On Sun, Apr 13, 2008 at 5:33 AM, barryhunter
<BarryB...@googlemail.com> wrote:
>
> Again maybe I am missing something, but the datastore isnt designed to
> be super fast at the small scale, but rather handle large amounts of
> data, and be distributed. (and because its distributed it can appear
> very fast at large scale)

The large chunks of data is the important bit. Much much larger than
traditional rdbms rows. By several hundred orders of magnitude.

> So you break down your database access into very simple processes.
> Assume your database acess is VERY slow, and rethink how you do
> things. (Of course the peice in the puzzle 'we' are missing is
> MapReduce! - the 'processing' part of the BigTable mindset)

Oh what i'd do to have access to MapReduce.

[Brett Morgan]

On Sun, Apr 13, 2008 at 8:52 AM, DennisP <DennisB...@gmail.com> wrote:
>
> Hmm, don't know that I'd go quite that far....scalability is important
> but it would also be nice for our web pages to be fast.
>
> It's nice that a page will be no slower with a million users than with
> one user, but if "no slower" means "it will still take 4 seconds
> serverside" that's a little less satisfying.

I know it's going to sound glib, but if it is taking four seconds to
render a page, you are using the tool wrong.

What we have to do here is together, as a group, start to explore all
the ways of using this toolset, and come up with best practices on how
to do things. So yes, seeing things like "it takes X seconds to save Y
records" is important. It starts to give us all a feel for how not to
do things.

But the next step is important. Exploring how to use it, instead of
getting depressed that our initial attempts are wrong. We will keep at
it until we get it.

[DennisP]

Definitely agree with that...a wiki with best practices might be a
good idea...


On Apr 12, 8:13 pm, "Brett Morgan" <brett.mor...@gmail.com> wrote:

[Brett Morgan]

Yup yup. Where can we host one? Can we build one on top of jotspot or something?

[Lee O]

Why not use an app for it. :)

(by the way, loving the discussion guys :)

[Brett Morgan]

Bwahahaha

Why didn't i see that one coming? =)

[PatrickNegri]

Heya guys.

I have did some tests about the speed issues you guys talking about.
What i suggest, at first, is to read some Search Engines paper,
regarding boolean operators and search algs.

Some things i have noticed:
* AppEngine Database model seens to be based on inverted indexes
* You only have AND operator
* Each additional query its a perfomance issue

The overall perfomance, for search, its fast, i am doing some local
tests to compare to other search engines i have writed/supported and
speed seens good. Currently doing a test with 1 million records. Will
post bench later, need some additional data yet.

One thing i must say, the appengine database model is VERY different
from conventional relational and sqls, you must think the way search
engines think.

For example, knowing that we dont have an "OR" operator, i have
implemented the following way:
1. Query DB with first term
2. Query DB with second term
3. Merge Sort the two itens based on relevancy (Do your own
calculations if necessary)

I am pretty happy with the results.

But, when you create something like 'manatlan', the speed will be a
issue, because he designed the DB as a Relational Database, with joins
and on query calculations.

In general, you can explain a query in mySQL for example and
reimplement the steps in your script. Like i said, the BigTable is a
BASE model.

The approach of Brett Morgan is the way you need to think, but, a good
approach would be something like that:
1. Ignore your USERS model for the moment, you dont need to do
any select at this table
2. Do a query in Marks model, get topK or top100K records
3. Using a user function, do a data range select, using ifs
and merges to create a result
4. Sort the result array the way you like
5. Use it to get the top 10,20,30 users from result

I would like to hear what you guys think and any other ideas.

PS: Sorry for english, Brazil here.

Patrick Negri

On Apr 12, 8:16 pm, "Brett Morgan" <brett.mor...@gmail.com> wrote:
> Yup yup. Where can we host one? Can we build one on top of jotspot or something?
>

[PatrickNegri]

Also, i must say something about Insert speed.

Yes, its slow. Very slow compared to mySql, but, you cant compare ;).

I dont know exactly how the indexes work behind the scenes, but i
hope it works the way google indexes works (Any Google Engineer here
to help us on this?).

If the indexes work like this:
* Each time a record is added, new term is weighted and added
to inverted index.
* This could explain the slow insert time, because we are
adding + indexing
* The inverted index is something like a pre-made "search".

For those that dont know what inverted indexes are:

Supose you have 2 docs.
DocID 1 => The cat is king
DocID 2 => I have bought food for my cat today.

Now, in indexes, you have some lexicon:
1. the, 2. cat, 3. is, 4. king, 5. i, 6. have, 7. bought, 8. food, 9.
for, 10. my, 11. today

Using this, we create inverted indexes for each lexicon, example:
Lexicon "cat" => TermID = 3
DocID(1),1 // DocID 1 has 1 hit for this
DocID(2),1 // DocID 2 has 1 hit for this (Also, here another question,
how the relevancy alg is calculated, i dont think AppEngine is a
exacly match of google indexes, but, if i have 2 hits, the DocID goes
up?)

Now, you guys can see why the search is faster. When you search, a
lookup for termID is done, then you can access a list of hits,
instantly. Because this its very efficient on searchers. Now imagine
the insert, its a bit tricky and i would imagine why its a bit slow.

Any Google Engineers here to answer these questions about indexes?

Regards
Patrick

[Brett Morgan]

From what i have read, i believe indexes are remade on each insert.

And no, I'm not a google engineer.

[PatrickNegri]

Indexes are updated, you mean.

Testing 100,000 records here. Insert was a hell slow.
Doing speed tests now.

[Brett Morgan]

I have no idea on how they are remade. Updated, recalc'd afresh? i have no idea.

[DennisP]

Fwiw, here are a couple wiki engines written in Python...
http://moinmo.in/
http://infomesh.net/pwyky/

Don't know how adaptable they would be, but just a quick read of the
project descriptions sounds kinda promising...there was another using
zope, I skipped that one.

I just started learning python a week ago so I'm not quite ready to
tackle it myself.

> >  Doing speed tests now.- Hide quoted text -

[PatrickNegri]

Tryied to install it, its incompatible.

I am developing a basic wiki, will post it to AEUniversity, so we can
work on it.

Regards
Patrick

On Apr 13, 2:36 am, DennisP <DennisBPeter...@gmail.com> wrote:
> Fwiw, here are a couple wiki engines written in Python...http://moinmo.in/http://infomesh.net/pwyky/

[DennisP]

yeah I didn't think either would work out of the box, just thought
moinmoin is supposed to be very modular, and the other is just pretty
small, so adapting one might be easier than starting from scratch.

but you know more about it than me

> > > - Show quoted text -- Hide quoted text -

[Hubert Chen]

I'd be 99% certain that DataStore indexes are not inverted
indexes.Inverted indexes serve a very different purpose than what a
traditional database index does. They are used for doing unstructured
text searches. DataStore is closer to a traditional database in this
respect because it is doing simple id based lookups.

[Dmitry Rubinstein]

On Sun, Apr 13, 2008 at 3:13 AM, Brett Morgan <brett....@gmail.com> wrote:

I know it's going to sound glib, but if it is taking four seconds to
render a page, you are using the tool wrong.

Or, maybe, I'm using a wrong tool. See, Google must have been in a situation, in which user's action triggers a heavy bit of data being uploaded to their servers for processing. Take Google Reader's import feature, for example (which is REALLY slow, but humor me for a moment here - it does the work, to some extent). So, they do have a way of dealing with such situation (which are immanent in many user cases). However, I don't think they externalized any method of dealing with these situations in the current release of GAE. Maybe they will add this later on, maybe not, but my point is, currently - there ain't such a thing. I could think of a solution, in which user's action causes a background process to be launched, and a way for the client page to poll for the results of the above (this is, presumably, the way Google Reader's import works). The thing is - we (those of us, who've encountered or are likely to encounter this problem) should probably get the message across somehow to the Google's engineers.

And while at that. If there is a Wiki, I would love to see there a list of missing features, together with the community's prioritization. The one in the 'issues' part of the project doesn't cut it. I have quite a list already :).
--
Vale,
Dmitry Rubinstein

[Dmitry Rubinstein]

On Sun, Apr 13, 2008 at 4:15 AM, PatrickNegri <patric...@gmail.com> wrote:

Also, i must say something about Insert speed.

Yes, its slow. Very slow compared to mySql, but, you cant compare ;).

I  dont know exactly how the indexes work behind the scenes, but i
hope it works the way google indexes works (Any Google Engineer here
to help us on this?).

Suppose then, I have a table in which all the lookups are guaranteed to be by a primary key. In this case, I would love to have a way to disable this reverse index thing, if it is indeed what it is. Perhaps, tinkering with the index.yaml can cause some boost in performance...

--
Vale,
Dmitry Rubinstein

[Brett Morgan]

On Sun, Apr 13, 2008 at 6:32 PM, Dmitry Rubinstein <dim...@gmail.com> wrote:
>
>
> On Sun, Apr 13, 2008 at 3:13 AM, Brett Morgan <brett....@gmail.com>
> wrote:
> > I know it's going to sound glib, but if it is taking four seconds to
> > render a page, you are using the tool wrong.
>
> Or, maybe, I'm using a wrong tool.

Ok, yes, valid point. I guess the question i keep forgetting to ask
people is, do you want to write code for a platform that will
distribute your code across hundreds of nodes and replicate your data
across hundreds of nodes such that your application can scale to
millions of users, without having to pony up the money and sysadmin
effort at the start of the project to install and configure those
hundreds of nodes?

Because if your intended audience can be served from a single box
sitting in a co-lo somewhere, using the techniques and languages you
already know and love, then yeah, GAE could quite possibly be the
wrong tool for your application.

I'm however doing this purely for the mental challenge. I like the
fact that this platform by the very way it is implemented, is
encouraging us developers to build code in a way that will scale. Yes,
this means that just about everything we are used to doing is going to
suck. Because everything we are used to doing is optimised for the
single sql database engine behind a single webserver app, because that
is what we have been developing on for the last decade. When you go
from single webserver to multiple servers, suddenly things get nasty.
You get session replication issues. And when you have to shard your
database you wind up with data correctness issues.

Writing for scale is hard. GAE is interesting because it makes the
scaling issues obvious because it either makes them illegal to do
(JOINs) or exceedingly slow (Pulling back hundreds of records and
computing something before rendering a single page). Because it is
obvious from the get go, we can learn how to write for scale.

[Dmitry Rubinstein]

On Sun, Apr 13, 2008 at 11:43 AM, Brett Morgan <brett....@gmail.com> wrote:

On Sun, Apr 13, 2008 at 6:32 PM, Dmitry Rubinstein <dim...@gmail.com> wrote:
>
>
> On Sun, Apr 13, 2008 at 3:13 AM, Brett Morgan <brett....@gmail.com>
> wrote:
> > I know it's going to sound glib, but if it is taking four seconds to
> > render a page, you are using the tool wrong.
>
> Or, maybe, I'm using a wrong tool.

Ok, yes, valid point.

But I am afraid, you got it wrong :). See, I don't think scaling is bad - on the contrary, it is very good. Also, GAE can solve some significant problems even for people, whose apps will never make it to the top - everyone has administrative overheads, not just the top 10 apps. The point is - not every problem can be solved by what GAE currently provides. There is no use to creative thinking, if the basic tools are missing - including ones used by Google itself elsewhere!


--
Vale,
Dmitry Rubinstein

[Brett Morgan]

Agree to disagree?

[Dmitry Rubinstein]

No problem :).  But please do let me know if you hit the same brick wall I did :)))

On Sun, Apr 13, 2008 at 11:51 AM, Brett Morgan <brett....@gmail.com> wrote:

Agree to disagree?

--
Vale,
Dmitry Rubinstein

[Brett Morgan]

Brick wall? The fact that this application environment is alpha
release is known, and stated openly by the googlers in question. They
are listening to our requests on the bug tracker. In fact, my request
for background processing is called out in their second post as one of
the things they are working on, along with support for other
languages. They said they came out early so as to get feedback from us
about what areas of functionality they needed to implement to allow us
to do what we want to do.

All I'm trying to do is to encourage people to see this as an
opportunity to understand a new tool, instead of bitching that it
doesn't handle like the old tools that they have been using up until
now. Because, in all honesty, an Su-30 makes for a horrible car, with
really bad cornering, and horrible clearence, what with the wings
hanging out the side and all. But as a jet fighter? Now that's a
different question...

[PatrickNegri]

Ok, 5:39 AM here and tests done.

My preliminar results:
100,000 - 500,000 records => Very fast search, insert is slow (i am
near convinced that we are talking about inverted indexes)

Did the test about the approach we was talking:

A recordset with "Points"
A recordset with "People"

5,000 records at People and 60,000 records at Points.

Search results => Fast
Insert time => Slow, veryyyyyyy slow :P (Maybe its because this we
have the bulk insert tool at dashboard)

WiKi is near done, will post here soon!

Patrick

[Peter Svensson]

Great work Patrick! 

Cheers,
PS

[PatrickNegri]

Any1 can contact me to help?

patrick...@hotmail.com

Want to deploy this wiki webapp before sleep.

On Apr 13, 5:45 am, "Peter Svensson" <psvens...@gmail.com> wrote:
> Great work Patrick!
>
> Cheers,
> PS
>

> On Sun, Apr 13, 2008 at 11:43 AM, PatrickNegri <patrickne...@gmail.com>

[Peter Svensson]

I can help. I think? What do you need help with?

Cheers,
PS

[PatrickNegri]

Base WiKi class done.

Will start to create the webapp soon.

Take a look, feel free to test:
http://aeuniversity.appspot.com/wk/

Patrick

On Apr 13, 6:00 am, PatrickNegri <patrickne...@gmail.com> wrote:
> Any1 can contact me to help?
>

> patrick_rne...@hotmail.com

[PatrickNegri]

Organizing the stuff, creating the wiki data itself.
Coding a way to post images, etc.

[Peter Svensson]

It works fine. Good minimum functionality, and everything.

Cheers,
PS

[PatrickNegri]

Its done!!!!!!!!!!

http://aeuniversity.appspot.com/

Now, who will help me create content?

Also, needing some image upload stuff and security (Only edit if we
authorize)

Patrick

On Apr 13, 6:44 am, "Peter Svensson" <psvens...@gmail.com> wrote:
> It works fine. Good minimum functionality, and everything.
>
> Cheers,
> PS
>

> On Sun, Apr 13, 2008 at 12:41 PM, PatrickNegri <patrickne...@gmail.com>

[Brett Morgan]

I think i need some help understand how you want us to use your wiki.
Like markup and such

=)

[PatrickNegri]

still planning to add some features like embed youtube in window
but...

the basic stuff:
{ = <ul>
* item = <li>item
[ text ] = blockquote
--- = <hr>
@title@ = <h1>
===subtitle=== <h2>
[http://www. name] => external link
{wikiobj Name} => text link (open a aewindow)
|wikiobj Name} => open a aewindow of wiki page
||wikiobj "Name" window_to_close|| => open a aewindow of wiki page,
and close window_to_close page
[IMG:url] => insert external image

On Apr 13, 9:16 am, "Brett Morgan" <brett.mor...@gmail.com> wrote:
> I think i need some help understand how you want us to use your wiki.
> Like markup and such
>
> =)
>

[PatrickNegri]

What first impressions?

Patrick

On Apr 13, 9:16 am, "Brett Morgan" <brett.mor...@gmail.com> wrote:

> I think i need some help understand how you want us to use your wiki.
> Like markup and such
>
> =)
>

[Dmitry Rubinstein]

How do I make a page appear in a main window (rather than a pop-up)?

Good work, BTW!

On Sun, Apr 13, 2008 at 4:23 PM, PatrickNegri <patric...@gmail.com> wrote:

still planning to add some features like embed youtube in window
but...

the basic stuff:
{ = <ul>
* item = <li>item
[ text ] = blockquote
--- = <hr>
@title@ = <h1>
===subtitle===  <h2>
[http://www. name] => external link
{wikiobj Name} => text link (open a aewindow)
|wikiobj Name} => open a aewindow of wiki page
||wikiobj "Name" window_to_close|| => open a aewindow of wiki page,
and close window_to_close page
[IMG:url] => insert external image

--
Vale,
Dmitry Rubinstein

[PatrickNegri]

All pages are popup dmitry.

If you want anything to appear in main window, Edit this page at main.

On Apr 13, 9:24 am, "Dmitry Rubinstein" <dim...@gmail.com> wrote:
> How do I make a page appear in a main window (rather than a pop-up)?
>
> Good work, BTW!
>

> On Sun, Apr 13, 2008 at 4:23 PM, PatrickNegri <patrickne...@gmail.com>

[DennisP]

Hmmm....so, people are talking about getting around with the lack of
joins by doing the work with writes instead of reads...seems that
kinda falls down if inserts are so slow...how slow are we talking
about?

[PatrickNegri]

Dennis, the approach i have did is better.

Also, i was testing the "reference" datatype. Its works well

Something like:

Animal:
Owner: reference do Person
Name

Person
Name

so, you can do a select to all animals
and do a Owner.Name()... without the need to do another query.

For range selects, this is what i am doing:
Do a backward query instead of forward.

Example:
Person
Name

Points
Person: reference to person
Points: integer

Supose you want to select Persons that have gaved to another person >=
30 points and <= 35 points.
Do a AllQuery in table Points, using a userfunction, go to each record
adding itens that have >=30 and <=30. (This is how mysql does for
example, internally)
After this, you will have only points that are in your range. Using
person reference, get data for the records you want.

This is how i am doing.
Patrick

[Leap]

Why not just serialize the classes into a single field while were at
it.