ANN: A shell tool for web2py

[limodou]

I've developed a shell tool for web2py today, and you can use it to
text web2py code in shell. It'll load all modules for you
automatically, just like web2py done. So you can use it to test
controller code and model code, I don't test so much, and if you found
some bugs, just tell me. Hope you have fun.

You should place this file in web2py folder. And run command:

python shell.py appname

Here appname is very import, and it'll set request.folder to that app.
If you want to switch other apps, you shuld quit from the shell first.

Here is my simple test output:

D:\project\svn\web2py>shell.py address
warning: importing mysqlite3.dbapi2 as sqlite3
warning: no MySQLdb driver
warning: no psycopg2 driver
warning: no cx_Oracle driver
Python 2.4.4 (#71, Oct 18 2006, 08:34:43) [MSC v.1310 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
>>> db=SQLDB("sqlite://d:/project/svn/web2py/db.db")
>>> db.define_table('address',
... SQLField('name', required=True,
requires=[IS_NOT_IN_DB(db, 'address.name')]),
... SQLField('telphone'),
... SQLField('mobile'),
... SQLField('homephone'),
... SQLField('email'),
... SQLField('work_address'),
... SQLField('home_address'),
... SQLField('birthday', 'date',
requires=IS_NULL_OR(IS_DATETIME())),
... SQLField('room'),
... SQLField('department'),
... )
<SQLStorage {'ALL': <gluon.sql.SQLALL instance at 0x00F410A8>,
'home_address': <gluon.sql.SQLField instance at 0x00F3FF08>, 'name':
<gluon.sql.SQLField instance
at 0x00F3F8F0>, 'room': <gluon.sql.SQLField instance at 0x00F41058>,
'_referenced_by': [], 'fields': ['id', 'name', 'telphone', 'mobile',
'homephone', 'email',
'work_address', 'home_address', 'birthday', 'room', 'department'],
'_db': <SQLStorage {'_connection': <pysqlite2.dbapi2.Connection object
at 0x00F0B320>, '_las
tsql': '', '_dbname': 'sqlite', '_execute': <function <lambda> at
0x00F3BE70>, '_folder': 'applications\\address\\databases', '_uri':
'sqlite://d:/project/svn/w
eb2py/db.db', 'tables': ['address'], '_translator': {'is null': 'IS
NULL', 'upper': 'UPPER(%(field)s)', 'lower': 'LOWER(%(field)s)',
'string': 'CHAR(%(length)s)
', 'reference': 'REFERENCES %(foreign_key)s ON DELETE
%(on_delete_action)s', 'text': 'TEXT', 'time': 'TIME', 'upload':
'CHAR(64)', 'datetime': 'TIMESTAMP', 'is
not null': 'IS NOT NULL', 'boolean': 'CHAR(1)', 'blob': 'BLOB',
'double': 'DOUBLE', 'date': 'DATE', 'integer': 'INTEGER', 'password':
'CHAR(%(length)s)', 'extra
ct': "web2py_extract('%(name)s',%(field)s)", 'id': 'INTEGER PRIMARY
KEY AUTOINCREMENT'}, 'address': <SQLStorage {...}>, '_cursor':
<pysqlite2.dbapi2.Cursor obje
ct at 0x00F3C290>}>, '_dbt':
'applications\\address\\databases\\0ce06eabddf1494b3165e3e1e9be64bf_address.table',
'telphone': <gluon.sql.SQLField instance at 0x0
0F3FD78>, 'email': <gluon.sql.SQLField instance at 0x00F3FE68>,
'mobile': <gluon.sql.SQLField instance at 0x00F3FDC8>, 'birthday':
<gluon.sql.SQLField instance
at 0x00F3FFA8>, '_tablename': 'address', 'department':
<gluon.sql.SQLField instance at 0x00F41080>, 'id': <gluon.sql.SQLField
instance at 0x00F41148>, 'homephon
e': <gluon.sql.SQLField instance at 0x00F3FE40>, 'work_address':
<gluon.sql.SQLField instance at 0x00F3FEB8>}>
>>> db().select(db.address.ALL)
<gluon.sql.SQLRows instance at 0x00F414B8>
>>> a = db().select(db.address.ALL)
>>> str(a)
'address.id,address.name,address.telphone,address.mobile,address.homephone,address.email,address.work_address,address.home_address,address.birthday,address.room
,address.department\r\n1,limodou,,,,,,,1973-12-26,,\r\n2,zoom,,,,,,,1973-12-26,,\r\n3,xlp223,,,,,,,1973-12-26,,\r\n5,ppppp,,,,,,,,,\r\n'
>>> a.response
[(1, u'limodou', u'', u'', u'', None, None, None, u'1973-12-26', u'',
u''), (2, u'zoom', u'', u'', u'', u'', None, None, u'1973-12-26', u'',
u''), (3, u'xlp223'
, u'', u'', u'', u'', None, None, u'1973-12-26', u'', u''), (5,
u'ppppp', u'', u'', u'', u'', None, None, None, u'', u'')]
>>> print FORM(TEXTAREA(_id='text'),INPUT(_type='button',_value='markdown',
... _onclick="ajax('ajaxwiki_onclick',['text'],'html')"))
<form enctype="multipart/form-data" method="post"><textarea id="text"
rows="10" cols="40"></textarea><input
onclick="ajax('ajaxwiki_onclick',['text'],'html')" v
alue="markdown" type="button"/></form>

--
I like python!
UliPad <<The Python Editor>>: http://code.google.com/p/ulipad/
meide <<wxPython UI module>>: http://code.google.com/p/meide/
My Blog: http://www.donews.net/limodou

[Massimo Di Pierro]

Great! I will play with this in the net couple of days. Very useful
and needed!

Massimo

> <shell.py>

[Massimo Di Pierro]

Could you please add a header saying that you made it and you release
it under BSD. ;-)

Massimo

On Mar 12, 2008, at 12:23 AM, limodou wrote:

> 'address.id,address.name,address.telphone,address.mobile,address.homep
> hone,address.email,address.work_address,address.home_address,address.b
> irthday,address.room
> ,address.department\r\n1,limodou,,,,,,,1973-12-26,,\r\n2,zoom,,,,,,,

> 1973-12-26,,\r\n3,xlp223,,,,,,,1973-12-26,,\r\n5,ppppp,,,,,,,,,\r\n'
>>>> a.response
> [(1, u'limodou', u'', u'', u'', None, None, None, u'1973-12-26', u'',
> u''), (2, u'zoom', u'', u'', u'', u'', None, None, u'1973-12-26', u'',
> u''), (3, u'xlp223'
> , u'', u'', u'', u'', None, None, u'1973-12-26', u'', u''), (5,
> u'ppppp', u'', u'', u'', u'', None, None, None, u'', u'')]
>>>> print FORM(TEXTAREA(_id='text'),INPUT
>>>> (_type='button',_value='markdown',
> ... _onclick="ajax('ajaxwiki_onclick',
> ['text'],'html')"))
> <form enctype="multipart/form-data" method="post"><textarea id="text"
> rows="10" cols="40"></textarea><input
> onclick="ajax('ajaxwiki_onclick',['text'],'html')" v
> alue="markdown" type="button"/></form>
>
> --
> I like python!
> UliPad <<The Python Editor>>: http://code.google.com/p/ulipad/
> meide <<wxPython UI module>>: http://code.google.com/p/meide/
> My Blog: http://www.donews.net/limodou
>
> >

> <shell.py>

[limodou]

On Wed, Mar 12, 2008 at 1:37 PM, Massimo Di Pierro
<mdip...@cs.depaul.edu> wrote:
>
> Could you please add a header saying that you made it and you release
> it under BSD. ;-)
>

Ok. As your wish.

[limodou]

New file is in the attatchment.

[Massimo Di Pierro]

Thanks. It is in trunk!
Going to bed now...

Massimo

> <shell.py>

[Michael Wills]

This is a very nice script limodou. Thank you for sharing it.

[limodou]

On Wed, Mar 12, 2008 at 4:15 PM, Michael Wills <mcw...@gmail.com> wrote:
> This is a very nice script limodou. Thank you for sharing it.
>

And I found that you can use execfile to run the model file, for example:

>>> execfile('applications/Address/models/db.py')

[Massimo Di Pierro]

I reposted shell in trunk now if you do

python shell.py someapp

and someapp does not exist it will ask if you want to create it.

Massimo

[carlo]

Amazingly useful!

Carlo Bazzo

On 12 Mar, 17:28, Massimo Di Pierro <mdipie...@cs.depaul.edu> wrote:
> I reposted shell in trunk now if you do
>
> python shell.py someapp
>
> and someapp does not exist it will ask if you want to create it.
>
> Massimo
>
> On Mar 12, 2008, at 9:26 AM, limodou wrote:
>
>
>

> > On Wed, Mar 12, 2008 at 4:15 PM, Michael Wills <mcwi...@gmail.com>

[DenesL]

First, thank you limodou, this is very useful.

> And I found that you can use execfile to run the model file, for example:
>
> >>> execfile('applications/Address/models/db.py')

This is true, but beware that the same might not work in a real web2py
environment. Let's take the example from my "importing modules
questions" thread: http://groups.google.com/group/web2py/browse_frm/thread/4a9a1258e3dab062#

In the shell I can:
execfile('applications/cookbook/modules/module1.py')
and the classes and functions inside it will be available.

Not so in the real web2py. Trying to use any function or class defined
in module1 after being execfile-d in a controller function only
produces a:
NameError: global name 'xxx' is not defined
even though 'xxx' is in dir() (???).

[limodou]

On Thu, Mar 13, 2008 at 3:28 AM, DenesL <dene...@yahoo.ca> wrote:
>
> First, thank you limodou, this is very useful.
>
>
> > And I found that you can use execfile to run the model file, for example:
> >
> > >>> execfile('applications/Address/models/db.py')
>
> This is true, but beware that the same might not work in a real web2py
> environment. Let's take the example from my "importing modules
> questions" thread: http://groups.google.com/group/web2py/browse_frm/thread/4a9a1258e3dab062#
>

Shell.py will prepare the environment which will be used for
controller, so execfile is suit for controller.

> In the shell I can:
> execfile('applications/cookbook/modules/module1.py')
> and the classes and functions inside it will be available.

And for modules, I think you should import them but not execfile them.

>
> Not so in the real web2py. Trying to use any function or class defined
> in module1 after being execfile-d in a controller function only
> produces a:
> NameError: global name 'xxx' is not defined
> even though 'xxx' is in dir() (???).
>

--

[Michael Wills]

This shell is quite cool. For those new to this stuff (like me) ,if you're working on something specific you can specify that in the environment. If you want to play with your model straight away, and it's named 'db', you can add

environment['db']=__import__('applications.%s.models.db' % app, fromlist=['*'])

Just a convenience thing for testing.

As I worked with it, I wanted to load it into iPython. My very poor way is I have a small script to just load the environment.

# file:web2pyshell.py
from shell import *
app=raw_input('app to load environment for:')
for k,v in env(app).items(): locals()[k]=v

# in ipythonrc add
execute sys.path.append('C:\\path\\to\\web2py')
execute os.chdir('C:\\path\\to\\web2py')
execfile web2pyshell.py

so when I load up iPython, it's all loaded up and waiting to go. Very nice. Many thanks!

[limodou]

On Thu, Mar 13, 2008 at 3:37 PM, Michael Wills <mcw...@gmail.com> wrote:
> This shell is quite cool. For those new to this stuff (like me) ,if you're
> working on something specific you can specify that in the environment. If
> you want to play with your model straight away, and it's named 'db', you can
> add
>
> environment['db']=__import__('applications.%s.models.db' % app,
> fromlist=['*'])
>
> Just a convenience thing for testing.
>
> As I worked with it, I wanted to load it into iPython. My very poor way is I
> have a small script to just load the environment.
>
> # file:web2pyshell.py
> from shell import *
> app=raw_input('app to load environment for:')
> for k,v in env(app).items(): locals()[k]=v
>
> # in ipythonrc add
> execute sys.path.append('C:\\path\\to\\web2py')
> execute os.chdir('C:\\path\\to\\web2py')
> execfile web2pyshell.py
>
> so when I load up iPython, it's all loaded up and waiting to go. Very nice.
> Many thanks!
>

thanks, I've finished a new shell.py version, and it can automatically
find ipython and run it now. And I've added a "-p" argument on command
line, so if you want to use default python shell, you can run like:

python shell.py -p appname

or it'll auto find ipython and run.

And I also add automatically load model files process,

model_path = os.path.join(request.folder,'models', '*.py')
from glob import glob
for f in glob(model_path):
fname, ext = os.path.splitext(f)
execfile(f, environment)
print 'Imported "%s" model file' % fname

so, when you start shell.py, it'll report all imported model files,
and you can directly use:

db().select(db.address.ALL)

to test something, don't need to load model file manually any more.

Hope Massimo can merge it to svn.

[voltron]

Errm.. this is embarrassing, but hey, silly questions are the ones not
asked :-) Why would one need a shell to develop web2py apps? Most of
the results are renderde in a view and most editors(Ullipad, Eclipse)
have access to consoles too for output. What are the benefits? Could
you guys post examples?

On Mar 13, 9:35 am, limodou <limo...@gmail.com> wrote:

> shell.py
> 4KDownload

[limodou]

On Thu, Mar 13, 2008 at 5:13 PM, voltron <nhy...@googlemail.com> wrote:
>
> Errm.. this is embarrassing, but hey, silly questions are the ones not
> asked :-) Why would one need a shell to develop web2py apps? Most of
> the results are renderde in a view and most editors(Ullipad, Eclipse)
> have access to consoles too for output. What are the benefits? Could
> you guys post examples?
>

I think it's the first step, because controller file and model file
are not real python module, so you can not directly import them to the
shell, it'll complain exceptions. And you should prepare the
environment for the running. So shell.py does this thing via evn()
function. And some editors like ulipad will use it to prepare the
environment before test web2py code. But I think someone may not like
IDE or he just don't want to open an editor, from console is very
simple and handy.

[voltron]

Ahh okay, so it is a matter of taste? I use Ullipad for quick n dirty
testing and coding.

[limodou]

On Thu, Mar 13, 2008 at 5:36 PM, voltron <nhy...@googlemail.com> wrote:
>
> Ahh okay, so it is a matter of taste? I use Ullipad for quick n dirty
> testing and coding.
>

maybe, and I think shell.py also provide a useful function can be used
in other tools.

[carlo]

can you import your models, try different ORM queries, test your views
and see what code your Html helpers generate, in Ulipad? I do not
think so. Even with Eclipse you do not have an easy way. That's why
every important python framework has a console app and I greatly
appreciated limodou's work.

carlo

[voltron]

Hi Carlo,

of course you can do all that in Ullipad! Besides I was not putting
down Linodous work( I have been using Ullipad for almost a year now) I
wanted to know If I should start using the shell for development too

[carlo]

really? Can I ask you the way you set it up because I did not get my
Wing IDE to work like that.

carlo

[Massimo Di Pierro]

would you collect this info and add a post on alterego?

Massimo

[Massimo Di Pierro]

cool.

> <shell.py>

[Michael Wills]

Yes! Very cool. I'll have to try it later. Finding and importing all the models was another thought but I didn't have time. Now I get to see how it can be done. :-)

For those who like shells, you can experiment with a lot of things and get immediate feedback. And ipython is great for that.

Eclipse is good with pydev but I found the recent version of pydev uses psyco. Psyco does something with the locals(), basically eliminating them. There is a notice about that when I ran shell.py or even ipython from within Eclipse. That's part of why I set up shell.py to run outside of Eclipse. So I'll try this from within Eclipse as well and report back.

And also one day when I get into doctests, ipython has that doctest mode. So your experiments can be copied into your code and use web2py's controller testing function.

[limodou]

On Thu, Mar 13, 2008 at 10:31 PM, Massimo Di Pierro
<mdip...@cs.depaul.edu> wrote:
>
> cool.
>
http://mdp.cti.depaul.edu/AlterEgo/default/show/64

[Massimo Di Pierro]

the latest shell.py was crashing on me without IPython. I made a
change (not very elegant I admit) and posted in trunk.

Massimo

[Michael Wills]

It works now without crashing? Just taking a look at your changes now. I have some crazy code that loads the controllers as well. I don't know if it works outside of iPython though since I rely on that. Hmm... I'll take your version and add that code and try it out. If I can get that, I'll post it and anyone without ipython can try it out if they would like.

[Massimo Di Pierro]

good. make it  command line option to load or not load the models. I say by default load.

[Michael Wills]

whoo... I'll try. Still learning! :-P web2py opened up a lot of Python to me that I was pretty clueless about before. And this stuff... I am sure someone could clean it up. I'll post as soon as possible. :-)

[Michael Wills]

Still needs some more testing. As always. :-) I am getting an error on some controller functions but not others so I am trying to see why. Ah the joys.

[limodou]

On Fri, Mar 14, 2008 at 8:00 PM, Michael Wills <mcw...@gmail.com> wrote:
> Still needs some more testing. As always. :-) I am getting an error on some
> controller functions but not others so I am trying to see why. Ah the joys.
>

I don't suggest that automatically import controller files, because
request, response are empty object, so they cann't be directly used,
and it'll has some errors when you execute the controller functions.

[Michael Wills]

I had not passed the application, controller and function values into the environment so any calls to redirect or URL, etc., would error out. So, with that, here is a version with controllers.

[limodou]

On Fri, Mar 14, 2008 at 7:08 PM, Massimo Di Pierro
<mdip...@cs.depaul.edu> wrote:
>
> the latest shell.py was crashing on me without IPython. I made a
> change (not very elegant I admit) and posted in trunk.
>
> Massimo
>

It's very strange, and I tested my code in windows, either with
ipython or without ipython are all right.

And I found you add this line:

if options.plain: raise Exception

But I don't know why your code will fail? What's the error? Because
optparse is an official module. I also test shell.py in python 2.5.

[Michael Wills]

Yeah that's what I just ran into. I haven't added an option to not automatically load the controllers yet though. That'll have to come a bit later. Plus right now, it's a bit of typing but your app name is used as a namespace... a lot of exec() in there. In ipython you get autocompletion though so it's not so bad.

[Massimo Di Pierro]

Good point

[Massimo Di Pierro]

I was getting an error in import IPython when the -p option.

[Michael Wills]

with some more options now... This whole thing is a whole lot of firsts. :-) Thanks for letting me learn with you all.

Options are now:

    parser.add_option('-p', '--plain', action='store_true', help='Default python shell, not to use IPython.')
    parser.add_option('-c', '--load-controllers', action='store_true', help='Auto load controllers', dest='controllers', default=True)
    parser.add_option('-C', '--no-controllers', action='store_false', help='Auto load controllers', dest='controllers')
    parser.add_option('-m', '--load-models', action='store_true', help='Auto load models', dest='models', default=True)
    parser.add_option('-M', '--no-models', action='store_false', help='Auto load models', dest='models')

[limodou]

On Fri, Mar 14, 2008 at 8:40 PM, Massimo Di Pierro
<mdip...@cs.depaul.edu> wrote:
>
> I was getting an error in import IPython when the -p option.
>

-p will disable ipython, and I guess that you don't install ipython.
So I can changed the file.

[limodou]

On Fri, Mar 14, 2008 at 8:56 PM, Michael Wills <mcw...@gmail.com> wrote:
> with some more options now... This whole thing is a whole lot of firsts. :-)
> Thanks for letting me learn with you all.
>
> Options are now:
>
> parser.add_option('-p', '--plain', action='store_true', help='Default
> python shell, not to use IPython.')
> parser.add_option('-c', '--load-controllers', action='store_true',
> help='Auto load controllers', dest='controllers', default=True)
> parser.add_option('-C', '--no-controllers', action='store_false',
> help='Auto load controllers', dest='controllers')
> parser.add_option('-m', '--load-models', action='store_true',
> help='Auto load models', dest='models', default=True)
> parser.add_option('-M', '--no-models', action='store_false', help='Auto
> load models', dest='models')
>

I think you don't need to set so many options, because you can set
some default option values if there are no arguments.

[Michael Wills]

Heh, true :-P I can clean that up later.

The controllers are invoked with () as well. Forgot to mention that.

welcome.default.index() will return the response.

[limodou]

On Fri, Mar 14, 2008 at 9:00 PM, Michael Wills <mcw...@gmail.com> wrote:
> Heh, true :-P I can clean that up later.
>
> The controllers are invoked with () as well. Forgot to mention that.
>
> welcome.default.index() will return the response.
>

Add nomodel option, and fix no ipython bug. But not add load
controllers process.

[limodou]

On Fri, Mar 14, 2008 at 9:18 PM, limodou <lim...@gmail.com> wrote:
> On Fri, Mar 14, 2008 at 9:00 PM, Michael Wills <mcw...@gmail.com> wrote:
> > Heh, true :-P I can clean that up later.
> >
> > The controllers are invoked with () as well. Forgot to mention that.
> >
> > welcome.default.index() will return the response.
> >
> Add nomodel option, and fix no ipython bug. But not add load
> controllers process.
>

New version. Changed some code structure.

[Massimo Di Pierro]

I am a bit lost. I assume this has all the latest features and I
added to the trunk.

I just gave a web2py hat to Guido Van Rossum.

Massimo

> <shell.py>

[limodou]

On Sat, Mar 15, 2008 at 12:53 AM, Massimo Di Pierro
<mdip...@cs.depaul.edu> wrote:
>
> I am a bit lost. I assume this has all the latest features and I
> added to the trunk.
>
> I just gave a web2py hat to Guido Van Rossum.
>

That's cool. Maybe Guido will fall in love with web2py. :)

[Michael Wills]

Just a quick minor update on the shell. If possible, could the 'run' function be defined as something else (lines 53 and 112)? I have changed it to shell_run in my version so that I can use iPython's internal run command. 'run -i' is quite helpful for having programs loaded by iPython using the variables in iPython's namespace so that's good for doing tests.

Thanks for the great work!

[Massimo Di Pierro]

send me your modified file please.

[Michael Wills]

Mine is a bit different because it adds controllers which is not a complete implementation yet. It's useful for what I am doing though. I'll attach a modified version from your current svn of shell.py. I just changed 'run' to 'shell_run' in those two places.