MicroTG2 anyone?
Mark Ramm
dependencies. The particular reason for wanting this is the Google
App Engine which restricts users to 1000 files. We might just be
able to fit TG into the environment, but we wouldn't be leaving much
room for add-on libraries or user code. :(
But I think given the single-threaded nature of the app engine
environment we can drop a lot of things, perhaps even switching to
only WebOb, and removing the Paste and Pylons stuff we're using, which
will drop our file count quite a bit. I think the modus operandi
for this project would be to start with WebOb and build up the TG2 API
from there, and I would very much like to help make this happen.
Hopefully there are some other people interested in helping to make
this happen, and we can talk about it in detail at this weekend's
WorldWide Virtual sprint.
http://docs.turbogears.org/SprintSeries2008
It's very likely that we'll be able to get TG2 working on the Google
app engine without this, but it won't be useful for people who want
*free* hosting, because it will require more files than are allowed in
the free plan.
--
Mark Ramm-Christensen
email: mark at compoundthinking dot com
blog: www.compoundthinking.com/blog
Daniel Fetchinson
I might be misunderstanding something but isn't it possible to zip a
whole package into a single egg file and thereby reducing the file
count dramatically?
Cheers,
Daniel
Ian Bicking
App Engine doesn't support zipimport.
--
Ian Bicking : ia...@colorstudy.com : http://blog.ianbicking.org
Christopher Arndt
>> I think we can make a much smaller version of TG, with fewer
>> dependencies. The particular reason for wanting this is the Google
>> App Engine which restricts users to 1000 files.
>> It's very likely that we'll be able to get TG2 working on the Google
>> app engine without this, but it won't be useful for people who want
>> *free* hosting, because it will require more files than are allowed in
>> the free plan.
>
>
> I might be misunderstanding something but isn't it possible to zip a
> whole package into a single egg file and thereby reducing the file
> count dramatically?
No, zip-file importing is not supported by GAE.
But I still don't understand why so much effort is currently put into
trying to make TG work on GAE, where it is very clear that atm many of
the underlying libraries arent't even supported (Genshi, RuleDispatch,
ORMs, other C-extensions). Shouldn't we try to get a TurboGears 2 beta
release together first that will work for normal server users?
It might be frustrating that atm Django runs on GAE and TG not, but a
"me too" attitude rarely helps with planning...
Chris
gas...@gmail.com
pylons
http://code.google.com/p/appengine-monkey/wiki/Pylons
It took me 862 files.
I think it's ok to define the core (tg.controllers, tg.decorators,
tg.defaults, tg.exceptions, tg.flash, tg.util, 6 files)
and make others(commands and templates) as 'major' extensions.
Then we don't need much modification for tg2 to work on appengine.
--
Fred
Mark Ramm
> trying to make TG work on GAE, where it is very clear that atm many of
> the underlying libraries arent't even supported (Genshi, RuleDispatch,
> ORMs, other C-extensions).
In TG2 there aren't that many other C-extensions, just RuleDispatch
and I thought there was a python implementation of everything in RD,
which is just _slow_. And even RD isn't used anywhere in the "core"
of TG it's just in ToscaWidgets and TurboJson.
Christopher Lenz has a partially-working version of genshi already.
ORM's are kind of not the point since App Engine
users should be using the DataStore.
> Shouldn't we try to get a TurboGears 2 beta
> release together first that will work for normal server users?
TG2 is pretty much ready for a beta release, I have been waiting
on Pylons to do a 1.9.7 release so that we can depend on a
released version of pylons. And 1.9.7 has been coming "any day
now" for a couple of weeks. But I actually do think they are
getting very close.
> It might be frustrating that atm Django runs on GAE and TG not, but a
> "me too" attitude rarely helps with planning...
I hear that. I'm actually not woring on the App Engine at all. But I
am trying to provide some direction and help to people who have
expressed that they are very motivated to work on this.
One of the things about open source is that you have to allow people
to work on what they are interested in and scratch their own itches.
Besides that I think working on App Engine is important for TG,
because it provides us with an answer to the "super low cost:
hosting of PHP. It's not about trying to make TG more competitive
with Django, it's about making it more competitive with PHP.
PHP is not the best language, but it's incredibly successful for 2
reasons:
1) Cheep hosting
2) relatively easy deployment.
App Engien allows us for the first time in python history, to do
*better* than PHP on both of those fronts.
Combine that with the promise of no-worries scalability, and the
"cool factor" of running you app on the same infrastructure that
runs google, and you're going to see a lot of people developing
for this system.
So, I'm convinced that this is a good thing for someone to spend
time on. And it seemed to me that there were people who were
not already working on TG2 who wanted to get involved here.
Thus, I'm trying to help.
--Mark Ramm
Mark Ramm
> pylons
>
> http://code.google.com/p/appengine-monkey/wiki/Pylons
>
> It took me 862 files.
>
> I think it's ok to define the core (tg.controllers, tg.decorators,
But what happens when you install ToscaWidgets, TurboJSON, Genshi, etc?
We're going to be pushing the limits. And since all that exists at
the moment is the free service level, we can't even tell people that
they just need to buy a little bit more space.
But, I think that running the whole of TG in App Engine is a very good
thing. And we're really close there given cmlenz's genshi work and
Ian's work on Paste/Pylons.
--Mark Ramm
Jonathan LaCour
> I think we can make a much smaller version of TG, with fewer
> dependencies. The particular reason for wanting this is the
> Google App Engine which restricts users to 1000 files. We might
> just be able to fit TG into the environment, but we wouldn't be
> leaving much room for add-on libraries or user code. :(
I love the idea of a Micro TG2, but not for the same reasons as you.
What I'd love to see is the core of TurboGears 2.0 broken down into
the smallest possible kernel of code and functionality.
I'd love to see a `TurboGears2-Core` egg that included nothing but:
* Base Pylons Dependency
* Genshi
* Object-Dispatch
That tiny core of functionality is all you really need to get
started with TurboGears, in my opinion. This is just a good idea
in general, to keep the core as small as possible, and build
in additional functionality (SQLAlchemy support, transaction
middleware, auth/auth, etc) as additional eggs that can be
installed.
For people who want just the basics, they can install just the core.
Others, who want more of the "full stack" can install an egg that
pulls all the other features (which are eggs) as dependencies.
This is all motivated by a somewhat selfish desire to have such
a thing, since I am actively developing an application on top of
TurboGears 2.0, and am finding myself to be very happy rolling my
own for things like transactions and authentication/authorization.
It'd be a shame to have to install all of these extra dependencies
if I don't need them :)
--
Jonathan LaCour
http://cleverdevil.org
Mark Ramm
<jonatha...@cleverdevil.org> wrote:
>
> Mark Ramm wrote:
>
> > I think we can make a much smaller version of TG, with fewer
> > dependencies. The particular reason for wanting this is the
> > Google App Engine which restricts users to 1000 files. We might
> > just be able to fit TG into the environment, but we wouldn't be
> > leaving much room for add-on libraries or user code. :(
>
> I love the idea of a Micro TG2, but not for the same reasons as you.
> What I'd love to see is the core of TurboGears 2.0 broken down into
> the smallest possible kernel of code and functionality.
But that's kind of opposed the "full stack" or "batteries included"
approach that TurboGears was founded on.
I'm all for a minimal alternative with a TG2 like API, but I don't
think it should be the default.
IIRC, our plan was to to release a tg-server or something egg which
has exactly enough to run the simplest possible hello world app.
This will definitely not include genshi or turboJSON, or mako, or
sqlalchemy, but MAY include FormEncode.
Making this a separate egg should not be too hard.
Determining what to call the eggs, and how to put them together,
wiithout making things complicated for new users isn't as easy. I
don't want to make people do:
from tg-server import expose, redirect
from tg import validate
and have to keep track of which thing is where... at least without
some very clear, very obvious devision between the two.
At this point it's just an itch nobody who's actively working on TG2
itself has had to scratch yet. But I think it will happen.
> This is all motivated by a somewhat selfish desire to have such
> a thing, since I am actively developing an application on top of
> TurboGears 2.0, and am finding myself to be very happy rolling my
> own for things like transactions and authentication/authorization.
> It'd be a shame to have to install all of these extra dependencies
> if I don't need them :)
Yea, that extra 2 meg of disk space is a real killer. ;)
I just want to point out that installing extra stuff does not mean
that it will be run, or take up memory in TG2 it just means that the
eggs live somewhere on disk. Everything's imported from the
template, so if you don't use them, don't import them and you should
be good to go.
Jonathan LaCour
> But that's kind of opposed the "full stack" or "batteries
> included" approach that TurboGears was founded on.
>
> I'm all for a minimal alternative with a TG2 like API, but I don't
> think it should be the default.
Cool, I agree with you here. I think the "default" should be to
install everything. I am just saying that I think that the core
of TurboGears itself should be installable separately, and that it
should *be* this "micro" framework at some level. In my opinion,
its just a nice way to keep ourselves from getting to cluttered with
"stuff" in the core.
> IIRC, our plan was to to release a tg-server or something egg
> which has exactly enough to run the simplest possible hello world
> app.
>
> This will definitely not include genshi or turboJSON, or mako, or
> sqlalchemy, but MAY include FormEncode.
>
> Making this a separate egg should not be too hard.
Cool, I'm into that.
> Determining what to call the eggs, and how to put them together,
> wiithout making things complicated for new users isn't as easy. I
> don't want to make people do:
>
> from tg-server import expose, redirect from tg import validate
>
> and have to keep track of which thing is where... at least without
> some very clear, very obvious devision between the two.
Sure, and I don't want that either. Eggs allow you to get over this
pretty easily, ensuring that everything can still live under the
"tg" namespace, even though they may live in separate eggs.
> Yea, that extra 2 meg of disk space is a real killer. ;)
>
> I just want to point out that installing extra stuff does not
> mean that it will be run, or take up memory in TG2 it just means
> that the eggs live somewhere on disk. Everything's imported from
> the template, so if you don't use them, don't import them and you
> should be good to go.
Hah, good point ;) I agree that its really not that important since
all of the dependencies are pulled automatically anyway, but I still
like the idea of having an extremely minimal core.
Jorge Vargas
in exposure, and the BigTable engine is a big win for everyone, it
will (I hope) let us break away from the relation databases that
where build to optimize space which right now is obsolete.
in fact I'm going to spend some time trying to get this working on
Saturday. with appengine-monkey most of the work is already done now
all we need is make sure it runs.
about the WebOb that will almost be a fork. so I don't like it.
Although I did read in the pylons list a post saying it could be a
possibility for pylons itself.
about the 1000 files limit I don't think this should be the way to go,
the correct solution will be to make google up the file limit (so go
star that ticket). in the meanwhile how about
a) try to refractor the projects so that we use less files? make more
modules, less packages.
b) build a tool that will do the above in some way, for example
collect all the widgets and put them into a single module.
c) make a non-setuptools distribution, each setuptools package adds 10
files, from my virtualenv tg2 jail I can count 30ish directories
that's 300 files.
d) in general cut the cruft not the tool.
anyway I guess this is item 2 in my list of things for saturday.
question does the dev environment has this 1k file limit? maybe it has
a warning....
> --Mark Ramm
>
>
>
> >
>
Daniel Fetchinson
I thought about this too. My guess is that given the 1000 file limit
on google this fact alone will motivate projects to release new
versions with less number of files. I believe this is possible in a
backward compatible way for the majority of packages.
Florent Aide
<fetch...@googlemail.com> wrote:
>
>
> >
> > about the 1000 files limit I don't think this should be the way to go,
> > the correct solution will be to make google up the file limit (so go
> > star that ticket). in the meanwhile how about
> > a) try to refractor the projects so that we use less files? make more
> > modules, less packages.
>
>
> I thought about this too. My guess is that given the 1000 file limit
> on google this fact alone will motivate projects to release new
> versions with less number of files. I believe this is possible in a
> backward compatible way for the majority of packages.
>
Guido just answered the ticket
http://code.google.com/p/googleappengine/issues/detail?id=161 saying
that we should use zipimport. This means it (zipimport) should now
work. Maybe we can now just use zipped eggs ? Anyone tested this
lately ?
Florent
UltraDaniel
I tried zipping eggs, but it didn't work. It seems like GAE does not
have zipimport.
(I run the code on my machine and zipimport works, but when I upload
the same code to GAE, it does not work.
Any suggestions?
-Daniel
mark.mch...@gmail.com
trying to use the built in zipimport?
I think guido is suggesting that we use:
http://svn.python.org/view/sandbox/trunk/import_in_py/zipimport_/