Integrated Development Environment with web2py

[Speedbird]

Folks,

Just wanted to share with the community a real jewel, many of you knew
this but I actually started using it "heavily" during the past couple
of weeks: the IDE is wing from wingware, basically you run web2py from
inside of it, then just open your controller/module/model from the
IDE, set up a breakpoint and voila you have a very interesting
development "studio" ala visual studio.

I've added a screenshot of my desktop running the IDE with my current
pet, pyforum.org being "debugged", the screenshot can be found here:
http://www.julioflores.com/static/debug_web2py.png

Wing IDE is not free, BUT you can get a developer's license (which
will give you the latest "Pro" release bona-fide). you have no idea
how much less time I've spent debugging the code with a tool like this
one, long live web2py

PS - Here's the web2py-specific information on their page, whoever
wrote it must've had a good understanding of the web2py framework (was
it you massimo??) - http://www.wingware.com/doc/howtos/web2py

Best regards to all,

Julio

[Iceberg]

I bet that the article is not written by Massimo, because he might not
have that good understanding ... about wing. :-)

Anyway, this is an interesting post. I never imagine an IDE for python
can even set breakpoint for a daemon program.

On Apr 23, 1:51 pm, Speedbird <ju...@techfuel.net> wrote:
> ...

> PS - Here's the web2py-specific information on their page, whoever
> wrote it must've had a good understanding of the web2py framework (was

> it you massimo??) -http://www.wingware.com/doc/howtos/web2py

[Timmie]

Thanks for posting this.
I will try how to make this work in Pydev for Eclipse (free...)

One point I noticed when I developed my controllers in Pydev is that
auto-completion of functions is not available.
I think that is due to importing mechanisms of web2py.
Web2py seems to import at runtime. But pydev needs an

import gluon.mymodule

statement in order to have the namespace available.

How do you get around this?

[Joe Barnhart]

I have to admit I've always been a non-believer in Wingware because
its cost, when there are so many excellent (and free) alternatives.
But seeing what you have accomplished on your web2py "forum" software
has me taking another look at your development tools!

I'd also be interested in your brand of coffee, since you seem to have
so much energy!

-- Joe B.

[AchipA]

Another unorthodox but very interesting feature of WingIDE is remote
debugging. While not as easy to set up, it potentially allows you to
hook/step/inspect a deployed appliance that goes wonky. That's quite a
big plus in my book :)

[Yarko Tymciurak]

Hi Julio -

Glad you enjoy Wing (I do).

Stephen Deible (Wingware) wrote the howto page, and I commented on it.  Both Stephen and John from Wingware were at the web2py coding dojo at Pycon.   They worked out the auto completion and change to catch Tickets in Wing if debugging (and grab the entire execution context).  It was great to see them at the web2py dojo (always great to see them).

I'm glad to hear you find WingIDE useful - it's one of my favorite tools.

Regards,

Yarko

[Guido Kollerie]

On 23-apr-2009, at 10:22, Timmie wrote:

> I will try how to make this work in Pydev for Eclipse (free...)

> [...]

> But pydev needs an
>
> import gluon.mymodule
>
> statement in order to have the namespace available.
>
> How do you get around this?

> --~--~---------~--~----~------------~-------~--~----~

You don't. I wrote a blog post on this topic a couple of weeks ago.
Shameless plug:

http://kollerie.wordpress.com/2009/04/07/setting-up-your-ide-for-web2py-development/

--
Guido Kollerie

[Wes James]

On Wed, Apr 22, 2009 at 11:51 PM, Speedbird <ju...@techfuel.net> wrote:
>
> Folks,
>
> Just wanted to share with the community a real jewel, many of you knew
> this but I actually started using it "heavily" during the past couple
> of weeks: the IDE is wing from wingware, basically you run web2py from

<snip>

>
> Wing IDE is not free, BUT you can get a developer's license (which

There is a free version of wingide:

http://www.wingware.com/wingide-101/index

> will give you the latest "Pro" release bona-fide). you have no idea
> how much less time I've spent debugging the code with a tool like this
> one, long live web2py

<snip>

-wj

[Speedbird]

Hey Yarko, nice to hear the tool is used by other web2py-istas :)
around here.

Indeed it really works "out of the box" with an optional minor change
in the exception catching code due to the way web2py logs in a ticket
when an error happens as you mentioned, I mean this can be used as
another one of the developer's "tools", it'll obviously be an overkill
to set up the whole environment just to change a variable value, for
instance, but pretty slick nevertheless.

And I'll "hijack" this response to also reply to Wes, the free version
of wing ide I wonder if it has less features, or is a "version"
behind, I am not sure, I requested a "developer" license about a year
ago (even before I "converted" to web2py), and after a couple of very
cordial email exchanges I received it, I praise them for that.

Keep up guys,

Julio

> > it you massimo??) -http://www.wingware.com/doc/howtos/web2py
>
> > Best regards to all,
>
> > Julio- Hide quoted text -
>
> - Show quoted text -

[Wes James]

On Thu, Apr 23, 2009 at 12:29 PM, Speedbird <ju...@techfuel.net> wrote:
>

<snip>

> And I'll "hijack" this response to also reply to Wes, the free version
> of wing ide I wonder if it has less features, or is a "version"

less features....

http://www.wingware.com/wingide/features

<snip>

-wj

[Francisco Gama]

They were kind enough to offer me a professional license for open
source projects dev. I even't yet configured it for web2py but I know
Wing very well and I love it. I use to work with a very advanced
configuration on Emacs and Eclipse with pydev and pydev extensions.
For a time, I used Komodo (and it's good... but the best version is
not free) but Wing for Python is very likely the winner. The only
thing I don't like so much about it, it's its non native interface for
Mac OS. It's the problem of GTK and of the reasons why I prefer Qt.

Anyway, I'm real Emacs fan and I love Eclipse. It's not easy for me
but It's true...
Wing is the best for python.

[Chris]

Wing is indeed a great tool.  I use it for local-machine debugging all the time and it has been a godsend.

Question -- has anyone had luck setting up Wing for remote debugging of web2py processes on a different machine?  I have been able to remotely debug simple Python scripts, but when I put the Wing hook code in web2py modules, the remote debugger connects to the IDE very briefly then disconnects.  So far I have tried putting the hooks in web2py.py and in gluon / widget.py / start().  This seems strange because when running web2py locally from the IDE, web2py.py is used as the "main debug file".  I would rather not add the hooks to individual applications (controllers, models and/or modules).  Can anyone point me in the right direction?

Thanks

[Chris]

Since my original post, I've found a way to use Wing IDE reliably for remote servers.

If anyone has need of this, write back and I will post the details.

[Massimo Di Pierro]

Please post the details. :-)

[Chris]

On Saturday, November 24, 2012 9:16:09 AM UTC-5, Massimo Di Pierro wrote:

Please post the details. :-)

Always happy to share. Massimo, if you'd like to address this in a future update of your web2py book, I will offer to collaborate with you on that.  Wingware has been an awesome tool for me & my team, would like others to benefit too.

These instructions are somewhat tailored to my setup but I think you can see how to apply elsewhere.

Items marked as remote are performed on the remote server logged in as root -- or whichever user you want to own all this -- any user with enough privileges.  Don't use the same user as owns your Apache processes for security reasons.

Items marked as local are performed on your local dev/test machine, by any user ID with sufficient privileges.

0.  Get access to the remote server

You'll need a way to connect to the remote server and issue command lines.  That usually means a private key file for the server.  For Amazon Web Services, you can get the key and command line from the AWS Management Console.

1.  Open an SSH tunnel for the app

To access the remote web2py processes via your browser or other local-machine tools -- to test the UI from your browser, to run the web2py Admin app (to view error tickets) etc. – rather than opening up security at the server and firewall level, the easy and secure approach is to create an SSH tunnel that maps ports between the two machines.

Issue these unix commands on local machine:

[local / ~] $ ssh -L 55080:localhost:80 remote-IP-address
[local / ~] $ ssh -L 55443:localhost:443 remote-IP-address

Now your local port 55xxx acts like remote port xxx.  E.g. if you would access an app as http://localhost:80/app if you were sitting in front of the remote machine, using http://localhost:55080/app from the local machine is the same thing.

2.  Install Wingware on the remote server

The details of this step change based on the OS / version of the remote server.  I'm currently using Red Hat EL 6.2.  

[]  Check the remote server's operating system and version.  You need to know the brand (e.g. Red Hat), version (e.g. 6.2) and architecture (e.g. 64 bit) since there are different Wingware versions for different environments.  You may already know this without running anything.  However if you want to verify this programmatically (this is for Red Hat only, there are different but equivalent commands for each flavor of Unix and Windows).  From the remote server:

[remote / ~] $ lsb_release -si
RedHatEnterpriseServer
[remote / ~] $ uname -m | sed 's/x86_//;s/i[3-6]86/32/'
64
[remote / ~] $ cat /etc/redhat-release
Red Hat Enterprise Linux Server release 6.2 (Santiago)

[]  Find the right download for the OS / version / architecture. All are located at http://www.wingware.com/downloads. For Red Hat, use the RPM package.  Copy the URL of the package so it can be downloaded to the server in the next step.  Note Wingware has a 30-day trial license so you can procure the license later.

[]  Download the package.  From the remote server:

[remote / ~] $ mkdir tmp
[remote / ~] $ cd tmp

[remote / ~] $ wget http://wingware.com/pub/wingide/4.1.6/wingide4.1-4.1.6-1.x86_64.rpm

[]  Install the package.  From the remote server:

[remote / ~/tmp] $ rpm -i wingide4.1-4.1.6-1.x86_64.rpm

3.  Configure Wingware for web2py on the remote server

[]  Create an empty Python module to allow import of the Wingware debugger hook. From the remote server:

## change to wherever your root web2py folder is

[remote / ~/tmp] $ cd ~/web2py/site-packages/
[remote / ~/web2py/site-packages] $ mkdir wing; cd wing
[remote / ~/web2py/site-packages/wing] $ cat >__init__.py <<EOF
EOF

[]  Copy the debugger hook, and make readable to all users (especially: Apache).  From the remote server:

[remote / ~/web2py/site-packages/wing] $ cp /usr/lib/wingide4.1/wingdbstub.py .
[remote / ~/web2py/site-packages/wing] $ chmod 644 wingdbstub.py

[]  Edit the Wingware module to test connectivity. Copy the original first.  From the remote server:

[remote / ~/web2py/site-packages/wing] $ cp wingdbstub.py original-wingstub.py

Then use whatever text editor you like to set kExitOnFailure = 1 on line 110.

And for WSGI compatibility, set kEmbedded = 1 on line 96.

[ ]  Test module setup and import (the error is expected and correct at this point in time!)  From the remote server:

[remote / ~/web2py/site-packages/wing] $ cd ~/web2py
[remote / ~/web2py] $ python2.7
Python 2.7 (r27:82500, May 22 2012, 12:22:43)
[GCC 4.4.6 20110731 (Red Hat 4.4.6-3)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.path.append('./site-packages')
>>> from wing import wingdbstub
No valid wingdebugpw file found in the following directories: ['./site-packages/wing']
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "./site-packages/wing/wingdbstub.py", line 232, in <module>
    raise ValueError('Not connected')
ValueError: Not connected

The error is the expected result because our setup isn't complete.

4.  Add Wingware hook to web2py source file

Edit web2py / gluon / main.py / wsgibase() as shown here. This works for all web requests whether via mod_wsgi or not.  (i.e. also works with the embedded Rocket server). From the remote server:

# ##################################################

# run controller

# ##################################################

if global_settings.debugging and request.application != "admin":

    import gluon.debug

    # activate the debugger and wait to reach application code

    gluon.debug.dbg.do_debug(mainpyfile=request.folder)

# ##################################################

# run controller

# ##################################################

if global_settings.debugging and request.application != "admin":

    import gluon.debug

    # activate the debugger and wait to reach application code

    gluon.debug.dbg.do_debug(mainpyfile=request.folder)

##################################################
## change begins

wingware_flag_file = 'WINGWARE-DEBUG-ON'

if os.path.exists(wingware_flag_file):

    sys.stdout.write('Wingware debug mode starting\n')

    from wing import wingdbstub

    wingdbstub.Ensure()
## change ends

##################################################

serve_controller(request, response, session)

The flag to turning Wingware on and off is a file in the base web2py directory.  if WINGWARE-DEBUG-ON exists, Wingware will be active.  This is analogous to how web2py controls app availability via the DISABLED file.

5.  Copy remote server application files to local machine

Breakpointing and other operations can't work unless the machine where the debugger IDE is running and remote server have 100% identical files under the ..../web2py folder (but do not have to be at the same path). There are many ways to do this.  The mechanism doesn't matter except you need to ensure the two file structures are the same, and know when one is changed so the other one can be updated.  Zip & copy files, tree copy via scp, refresh both from git …

I'm assuming a mapping of remote location '~/remote/web2py' to local debugging machine location '~/LOCAL/web2py'.  The actual location on the local machine isn't important except that it needs to be carried through the following steps.

From the remote server:

## cd to folder just above your web2py base
[remote / ~] $ cd ~
[remote / ~] $ zip -r web2py web2py --include \*.py
updating: web2py/ (stored 0%)
## ... umpty hundred lines

Note the --include option on the zip command includes only .py files since these are the only files required to debug.  This isn't a complete copy of a web2py instance and can't be run on its own.  It's just so you can see code in the IDE.

From the local machine:

[local / ~] $ mkdir LOCAL
[local / ~] $ cd LOCAL
## copy the zip file from the remote server
## paste that file here on the local machine
[local / ~/LOCAL] $ unzip –r web2py.zip

Now the .py files are in synch between the two machines, although at different locations.  It can actually save you some confusion later if the files are installed at the same path on both machines; but that won't always be possible.

6.  Copy local debug machine config files to remote server

As a security measure, Wingware requires that a password file it installs on the local machine (part of the Wingware user profile) to have the same content on the remote machine.  This prevents anyone from starting a debug session from a machine you haven't enabled by copying the password file there

From the local machine: find the 'wingdebugpw' file (no extension) – typically in ~/.wingide4 directory

From the remote server: copy the file to ~/remote/web2py/site-packages/wing

Note if more than one person is doing remote debugging with the same server, they will need to change this file to match the local Wingware config of the person doing the debugging at that time.  This is perhaps a good thing since there will be no question who "has the ball" at a point in time – there can only be one person debugging this way …

From the remote server, change permissions to allow all users (mainly: Apache) to read it:

[remote / ~] $ cd ~/remote/web2py/site-packages/wing
[remote / ~/remote/web2py/site-packages/wing] $ chmod 644 wingdebugpw

7.  Configure the Wingware IDE

From the local machine:

Open the Edit :: Preferences :: Debugger :: External/Remote panel

Turn on the Enable Passive Listen option

In Allowed Hosts, add both localhost and 127.0.0.1 -- if you're using IPv6, may also need ::1 or others.  I was not able to determine which of these were used.  There is no harm in listing all the localhost equivalents in this option.

Set the Server Port option to 50005

For the location map, repeat for each localhost equivalent IP address you entered previously.  (1) enter the localhost address as Remote IP Address.  (2) In the popup now (or from the Location Map later), choose "Specify mapping".  (3) Set the file mapping with ~/remote/web2py as the Remote entry and ~/LOCAL/web2py as the Local entry.  (4) In the Common Attach Hosts area, enter 50015.  Repeat these steps for each localhost equivalent IP address.

8-A.  Configure network security

Do 8-A or 8-B, not both ... B is probably easier unless you don't have the permissions required

Your local machine needs to be able to accept TCP connections on port 50005, from whatever IP address the remote server is on.  Firewall, iptables etc.  It is not a security concern if you just open this port to the world, since (a) inbound requests are harmless and (b) in any case will all be validated using the 'wingdebugpw' file that is particular to your IDE.

The remote server needs to be able to accept TCP connection on port 50015, from your local machine(s) IP address.

Note that in both cases you need to use the EXTERNAL IP address for your machine.  The IP address that is visible in the control panel or via ipconfig or ifconfig command line tools is usually an address on your local network that has no meaning outside of your network.  To check the external IP address of any machine, visit http://www.icanhazip.com

8-B.  Open an SSH tunnel for the debugger (IDE and remote agent)

Do 8-A or 8-B, not both ... B is probably easier unless you don't have the permissions required

From the local machine:

$ ssh -i ~/path/to/key.pem -L 50015:localhost:50015 -R 50005:localhost:50005 -N root@remote_name_or_ip_address -o StrictHostKeyChecking=false

Permissions required on the remote machine are SSH (port 443 and admin rights) which you have if you are connecting to the remote and issuing commands. ssh is for Unix.  A similar capability for Windows is plink (part of PuTTY).

9.  Test via a manual Python script

As a simple way to test the rest of the setup before dealing with web2py, WSGI and Apache, we can import the debug stub file into a Python interpreter session and see if we can connect.  Before the debugger is active, the lower left corner of the Wingware window looks like this:

     

Then from the remote server:

[remote / ~] $ cd ~/remote/web2py/site-packages/wing
[remote / ~/remote/web2py/site-packages/wing] $ cat >test.py <<EOF

x = 3
import wingdbstub
x = 5
import time
time.sleep(20)
x = 7
EOF

Then copy the test.py file you just created to the equivalent folder on your local machine, open the local file in the Wingware IDE, and set breakpoints on the x=5 and x=7 lines.

Then run the script on the remote server to see if the import happens and connection can be made:

[remote / ~/remote/web2py/site-packages/wing] $ python2.7 test.py

If everything is set up right, the status display turns green for 20 seconds and shows other info:

If the little bug icon turns green, this proves the server copy of Wingware, IDE preferences and network / security are all correct.  You can use watch variables or the debug probe to check the value of x, single-step and so on.

If the green display doesn't appear immediately, you will need to troubleshoot the previous steps.  You can also edit the wingdbstub.py file to have the local debug process create a log file.  I found this helpful.  Edit the existing entry to be: kLogFile = 'logs/wing.log' – NOTE file needs to be writable by Apache – the example above works because the path for any web2py process is the base web2py folder, and the web2py/logs folder is writeable by Apache.

The most common error is not having an ssh tunnel running (they die off and you have to restart them), and/or security permissions (on either machine) to allow the remote server process to call out to the debug machine or vice versa.

10.  Test using embedded Rocket server

Now see if we can debug a web2py process via its embedded (Rocket) web server.  This is needed because the simple script we used above is very simple  – web2py has a couple layers of process dispatching that interact differently with the Wingware hook.  And we want to be sure the config with web2py is right before we move on to Apache & WSGI.

From the remote server: start up web2py (from command line or GUI).   Make sure there are no error messages.  Test that web2py is properly configured / we haven't broken anything by browsing to localhost:8000, or via a command line:

curl -L 'http://localhost:8000/'

If you don't get a normal web page with no error messages, there is a problem to be fixed in an earlier step.

Now to see if everything has been set up right for Wingware, we need to create the flag file.  From the remote server:

[remote / ~] $ cd ~/remote/web2py
[remote / ~/remote/web2py] $ touch WINGWARE-DEBUG-ON

Repeat what you did above via browser or command line.  This time you should see a message on the web2py debug IO feed (command line, or Rocket GUI, or web2py log file) saying "Wingware debug mode starting".  If you don't see this, then you will probably get an error message to help troubleshoot.  If it does work, the lttle bug icon turns green.

11.  Test via Apache and mod_wsgi

From the remote server:  shut down the instance of web2py running the Rocket server.  Start or restart the Apache daemon.  You should see the same results as in section (10) -- good web page, green bug, message in debug IO.

12.  Debugging is happening

You are now able to use the Wingware IDE from your local machine to set breakpoints, do debug probes etc. on the remote machine.  Remember if you have multiple users on that machine (like consumers visiting your web site) you can't use an indiscriminate breakpoint -- because it could trap any user, and even switch between users from one trap to the next.  Perhaps not a big deal for a testing environment, but for stage/pilot or production servers, you will always want to set a conditional breakpoint using the user's session ID, IP address or something you can ID in the request.

13.  Troubleshooting

What to look at if the little bug icon won't turn green --

1. ssh tunneling – the debug machine and remote server must be able to communicate 
   

• Make sure ssh -L / -R command is still running. It can time out or have a broken pipe – restart it.
  
  
• Even if running, ssh can have a problem – usually a conflict with another instance – diagnose by running ps axu | grep ssh to see if there is another instance running or stalled; and kill it via kill -9 processid.
  
  
• Sometimes none of this seems to work, but rebooting the desktop solves it.

1. state of the Wing console
   

• The console can only manage one debug session at a time.  If you start a second request on the remote server, it won't be able to connect to back to the desktop unless the desktop finished the first request.
  
  
• One option is to hit "stop" button in the Wing console.  This makes it ready to accept the next request.  OK when you are testing on your own server but not with multiple testers (or users!) on the machine.
  
  
• It is better to use a setting in web2py / site-packages / wing / wingdbstub.py.  If "kExitOnFailure = 1" is set, if the WINGWARE-DEBUG-ON file exists and no app can connect to the debugger, the app fails as if it couldn't import a library.  That is great when you are the only user of the server (helps you see when debugging is broken so you can fix it) – but it is terrible when you aren't (since every request but one at a time will fail)!  On a shared server or production server, change this to config setting "= 0".  

Fellow Wingware users -- let me know if you have any luck with the above.

Regards -- 

[dlypka]

One GOTCHA I found with WING IDE is that you must delete all .pyc files before running, in order to have it correctly

stop on your breakpoints in non-module code, in the case where such .py source has been subsequently modified after the .pyc was created.

For code that is in a module, the breakpoints always stop correctly regardless of changes to the .py of the module.

[Chris]

On Sunday, December 2, 2012 10:20:28 AM UTC-5, dlypka wrote:

One GOTCHA I found with WING IDE is that you must delete all .pyc files before running, in order to have it correctly

stop on your breakpoints in non-module code, in the case where such .py source has been subsequently modified after the .pyc was created.

For code that is in a module, the breakpoints always stop correctly regardless of changes to the .py of the module.

That hasn't been my experience.  I'm running Wing version 4.1.4-1 (rev 26413) on OSX.