VB to Python migration
Josh
screens) that we want to get out of VB and into a better language. The
program is over 10 years old and has already been ported from VB3 to
VB6, a job which took over two years. We would like to port it to
Python, but we need to continue to offer upgrades and fixes to the
current VB6 version. Does anybody know of ways we could go about
rewriting this, one screen at a time, in Python, and calling the screens
from the existing program?
We are also looking for a graphics toolkit to use. IronPython with it's
.NET bindings and ability to integrate with Visual Studio looks good,
but leaves that bad MS taste in the mouth.
We currently use a MS Access back end and need to migrate to a proper
SQL server. We need to leave options open for SQL Server (for customers
who want to use existing infrastructure) and something like MySQL or
PostgreSQL. But in the mean time, we need to be able to access an
MSAccess (Jet) database from Python.
Any answers/suggestions/pointers are greatly appreciated.
Thanks,
Josh Isted
Nicolas Kassis
Ravi Teja
commonly do in Python (although I would be happy to see people show me
wrong). You are building a GUI app that likely need not run anywhere
except on Windows and perhaps most what of what is does is edit and
report on data from the database. This is the very task 4GL tools like
Visual Basic were created for.
VB6 has certainly been outdated for quite a while ago now and you
should probably make the switch but don't discount the tools that
exactly specialize in this domain such as Visual Studio.NET and Delphi.
Python is a great general purpose language but the above tools are
leaps and bounds ahead of the tools available to Python in this very
specific domain. Additionally your team already is familiar with the
RAD paradigm. As easy as Python is, it takes a while to develop
expertise in a new language and it's paradigms (especially when it is
such a leap - VB and Python are almost at the opposite ends of the
spectrum).
IronPython has a lot of promise but is not mature enough yet to bet a
project large enough to take 2 years (or more) to develop. At that
project size, it is not sane to attempt anything adventurous unless you
have have enough funds to buffer when things go wrong. I am not
recommending against Python, just to plan VERY VERY carefully on what
you embark on.
Start using Python gradually in your development. Refactor some
functionality into COM components for starters to simplify your VB
code. When Python is doing much of the heavy lifting, you are ready to
make the change. By the time you are ready to make the full switch, you
will know.
Personally, for a project like this, I would go with Borland Studio (it
has ECO which is a tremenduous advantage for data driven GUIs) and
supplement it with Boo (a Python like language for .NET).
Steven D'Aprano
> We have a program written in VB6 (over 100,000 lines of code and 230 UI
> screens) that we want to get out of VB and into a better language. The
> program is over 10 years old and has already been ported from VB3 to
> VB6, a job which took over two years. We would like to port it to
> Python, but we need to continue to offer upgrades and fixes to the
> current VB6 version. Does anybody know of ways we could go about
> rewriting this, one screen at a time, in Python, and calling the screens
> from the existing program?
I know little about VB, but if it can call external programs, then it
should be able to call a Python program from the command line.
[snip]
> Any answers/suggestions/pointers are greatly appreciated.
I suggest you hire a couple of experienced Python developers, pick half a
dozen screens, say three simple screens and three complex ones, and build
them in Python. That will give you an idea of how much work is involved.
Don't get your VB developers to try to port it themselves, that's a recipe
for disaster: they will be trying to learn Python, and will end up writing
VB code in Python, which is sure to be slow, inefficient, bulky and buggy.
You may also like to think about using a web front end. If your GUI app
only uses simple controls like text fields and push buttons, a web front
end might simplify things a lot.
Good luck.
--
Steven.
Jon-Pierre Gentil
Hash: SHA512
Josh wrote this on Friday 27 January 2006 05:48 pm in
<DjyCf.110918$AP5.12355@edtnps84>:
> We have a program written in VB6 (over 100,000 lines of code and 230 UI
> screens) that we want to get out of VB and into a better language. The
> program is over 10 years old and has already been ported from VB3 to
> VB6, a job which took over two years. We would like to port it to
> Python, but we need to continue to offer upgrades and fixes to the
> current VB6 version. Does anybody know of ways we could go about
> rewriting this, one screen at a time, in Python, and calling the screens
> from the existing program?
I would look into Qt, PyQt, and the Qt Designer. It'll provide one of the
easiest ways of porting your existing GUI dialogs into Python. Start
building and componentizing your code. In VB6, all classes are flat under
a single branch of a project tree, whereas in Python you'll likely
implement modules of similar functionality. Overall porting your
application will be a combination of rewriting, refactoring, and
copy-and-translate code. Good luck. :-) It actually sounds like a fun
project to be on. I had a lot of VB6 experience back in "the day" and a
project like that would be a lot of fun to me.
- --
_________________________________________________________
Jon-Pierre Gentil KC9CAF PGP: 0x7E1CBA17
jabber: jge...@sebistar.net web: www.sebistar.net
"If you think education is expensive, try ignorance."
_________________________________________________________
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.2 (GNU/Linux)
iQIcBAEBCgAGBQJD2yoeAAoJEPtxPmpAsx/rdkkP/A8oGRQF2n7UqpbTHkuMIPT5
u0LFh75ry/6RiXpOa72iLkPb9giXp+pSGGlGbR7Sgdh5NmMnhd882SJknVP8vFta
wcbdwORilBjR4I4WsnN5o+fcUz8PPERj0yfYndEEC7SBQOOahcpX6FSd9gNjbe2s
MMDy5TqDE0kTP2PtGknncLB+v4XoFkUcZtFhVtKh29y0kORCJ1jIHq+O7vN9Wgwv
BrzNse44n/TSW8Z2iCX8h+tsCK0vSzqe6GJKA5IRHmUjoRWyiQH6LKboZcA8qNrA
JQlCa9deRSLWFWOIUsHIKBeGDmiO+E7E6g1jfADwQuS+TDW1yxV+1HwGR5LCQFgV
kXVEMDzyk1SIvAVEuXIefXuZQq9nafPhDx4qpnnxqK1HK1F97meYYm7/tAoHZGKM
6kJ9tq9P5I3TkFB9dRVuNftKkKZz8f6RCJFcsHvqOu2H94Fui5upUs3/royp8TVQ
CeY74vtlFXKkPGjt07RfJFMqxacGL3ONjgYdRluV5pkDeXoer65Vhqd7kHFlpVW0
Gilw4ITDEIlIuX+iOvSP5CBixBqwrvhARGYQxCMoiwtZjfbbGo0iKYHhx4/qAGAv
qoxY0Ae6D3mTTIRXiKrVBLlXgj3kQIqCzOuCPTEhN/ydY2WeKYlArBQAL42H7Rkv
atlNHGjEcMMTcHesfK0e
=b2Qh
-----END PGP SIGNATURE-----
Luis M. González
Why use Delphi or VB when you have a more "pythonic" first class .NET
language?
You already have a very good IDE for creating your project
(SharpDevelop), which is free, open source and already has the Boo
bindings included (download here the latest version:
http://build.sharpdevelop.net/BuildArtefacts/).
Boo is a perfect language for someone wanting to develop Windows GUI
apps because it has all the nice features and syntax of Python, while
being specifically created to run on the .NET framework.
However, if you still want to use pure Python, it won't take too long
to get what you need. Ironpython is in version 1.0 Beta 2, and moving
full steam towards a stable release.
I'm sure that soon it will be integrated to Visual Studio.
DH
http://vb2py.sourceforge.net/
or if you want to convert vb6 to vb.net instead, there are tools from
microsoft and others to help with that, such as:
http://www.microsoft.com/downloads/details.aspx?FamilyId=10C491A2-FC67-4509-BC10-60C5C039A272&displaylang=en
or if you want to start over from scratch, the other recommendations are
good, like pyqt and qt designer, or else do it as a web app instead of
desktop app if it just involves basic form controls.
Thomas Ganss
You haven't specified where your main pains are.
Do you have at least rudimentary architecture ?
How often do you have code reviews / refactored your code ?
Have you been striving for good code ? Is it a total mess ?
Guessing only from the number of screens, you probably
have more than trivial amounts of data.
*Based on that assumption*, I'ld move the data FIRST.
Get rid of the JET engine! If you are certain you will
have to support SQL server, make it SQL server and
MSDE/new express version first. Only then (perhaps)
think about adding a totally free database engine.
If not, take your pick but *move the data*.
While you are on it, refactor the old code into something
at least resembling a 3 to 5 layered approach with COM.
Doesn't need to be perfect 100%, but more than 50%.
Then exchange the parts that benefit the most by
implementation inheritance, since VB's interface inheritance
is arguably the "best" technical reason for redundant code.
I'ld guess the business layer[s] would be the next things to convert -
but if your VB-forms are mostly slightly augmented copies differing
only slightly, the GUI actually might benefit most from a rewrite,
if the business rules are written redundance poor.
One of the typical scenarios asking for GUI inheritance
is insurance - customer data is only specific for few fields
and even policy field groupings resemble each other across
similar policies. A "minmally redundant" business layer could
even in VB be implemented without too much sweat- for instance
by having methods overwritten in a inheritance based OOP -
design as name mangled methods on objects responsible across
similar problem domains, keeping the "common" methods clean.
Check your code with a critical eye -
even code forced to be totally OOP for
technical reason=langauge
can be written across a wide spectrum of quality.
If there is nothing which is good enogh to be converted last
or no segregation/layering at all in the current program,
(even if it came from the DOS dinosaurs, there were
"good coding practices" known - some of the old "libraries"
were better decoupled than modern day OOP class libraries.)
get rid of most of the people responsible and start only then.
my 0.02 EUR
thomas
John M. Gabriele
> We have a program written in VB6 (over 100,000 lines of code and 230 UI
> screens) that we want to get out of VB and into a better language. The
> program is over 10 years old and has already been ported from VB3 to
> VB6, a job which took over two years. We would like to port it to
> Python, but we need to continue to offer upgrades and fixes to the
> current VB6 version. Does anybody know of ways we could go about
> rewriting this, one screen at a time, in Python, and calling the screens
> from the existing program?
Dunno what you mean by "calling the screens" from the existing
program. (?)
Is your VB6 progam object-based? If so, it sounds like you might
have your work cut out for you in duplicating the classes in Python
(I've never used VB6 and don't even know if it's an OO language.).
I'd look into how you can start with unit tests + your classes
right from the start. And, it goes without saying to use version
control from the start (svn or, if you're adventurous, maybe bzr
(i.e. Bazaar-NG)).
> We are also looking for a graphics toolkit to use.
You might take a peek at Python + PyGTK + Glade:
http://pygtk.org/ . Dunno the state of GTK on Windows,
but I've heard it's workable. No idea about Glade on
Windows.
Here's an article from 2001:
http://www.linuxjournal.com/article/4702
but if you look around there should be some newer tutorials
around.
> [snip]
>
> Any answers/suggestions/pointers are greatly appreciated.
As Steven suggested, I'd examine whether or not your app
might work as a web app. That completely frees you from Windows
altogether. Just make sure it runs alright with Firefox/Opera/
Safari. :)
---J
> Thanks,
>
> Josh Isted
--
(remove zeez if demunging email address)
Magnus Lycka
> 230 UI screens is a lot. An app of that nature is not something people
> commonly do in Python (although I would be happy to see people show me
> wrong).
Maybe not, but I don't doubt that it's reasonable to replace a
VB app with 230 UI screens with Python code.
A code of that size is something that's grown over time, and if
it's rewritten, it seems likely that it will end up with fewer
screens without fewer features.
Besides, while VB makes it very easy to duplicate a lot of code,
and end up in a situation where you have to manually change things
230 times through the GUI if something changes in all screens, it's
very convenient to use inheritence for GUI development with e.g.
wxPython. I supect the screens have a lot in common, and that it
would be fairly clear how to build a class hierachy of that, and
avoid code duplication. The end result will be a more homogenous
application and a lower maintenance cost.
The really silly thing to do is to apply VB idioms on Python.
Magnus Lycka
> We currently use a MS Access back end and need to migrate to a proper
> SQL server. We need to leave options open for SQL Server (for customers
> who want to use existing infrastructure) and something like MySQL or
> PostgreSQL. But in the mean time, we need to be able to access an
> MSAccess (Jet) database from Python.
mxODBC is probably the best way to talk to Jet in a way (DB API)
that makes it reasonably easy to port the data to another RDBMS.
Writing multi RDBMS applications is a whole chapter in itself,
if not a subject for a whole book.
Whether to use an ORM such as SQLObject on top of that is another
issue...
Ravi Teja
> Ravi Teja wrote:
> > 230 UI screens is a lot. An app of that nature is not something people
> > commonly do in Python (although I would be happy to see people show me
> > wrong).
>
> Maybe not, but I don't doubt that it's reasonable to replace a
> VB app with 230 UI screens with Python code.
>
> A code of that size is something that's grown over time, and if
> it's rewritten, it seems likely that it will end up with fewer
> screens without fewer features.
Rewrites are always good and result in smaller code base if features
are not added. However, I doubt that will make the screens fewer. Lines
of code? Certainly.
> Besides, while VB makes it very easy to duplicate a lot of code,
> and end up in a situation where you have to manually change things
> 230 times through the GUI if something changes in all screens, it's
> very convenient to use inheritence for GUI development with e.g.
> wxPython. I supect the screens have a lot in common, and that it
> would be fairly clear how to build a class hierachy of that, and
> avoid code duplication. The end result will be a more homogenous
> application and a lower maintenance cost.
You are comparing Python with VB 6. That's not what the OP is moving to
as an alternative. It's .NET. VB 6 has been out of development since 8
years? Its not something to be compared to anymore. .NET on the other
hand has all the advantages of Python accessible toolkits for this task
and then some. For example, it's not just inheritance but visual
inheritance. It's not just a GUI but GUI with data bindings, with
sophisticated caching, paging etc. GUI Builders are far more
sophisticated and integrated than, say wxGlade.
I am not saying Python is unfit for 230 screens. Just that I have not
seen anyone build such a thing and better write about it. People on the
other hand have build web sites with such volume. Result? We have a ton
of web application frameworks to address such needs. We would have the
same for Data GUIs if that was the case. We have people drooling over
TurboGears, not Dabo (apologies Dabo team).
> The really silly thing to do is to apply VB idioms on Python.
It is. But that's what people do on their first few projects and it's
perfectly natural. Didn't you :-) ?
Magnus Lycka
> Rewrites are always good and result in smaller code base if features
> are not added. However, I doubt that will make the screens fewer. Lines
> of code? Certainly.
That depends on whether you just refactor the implementation
of if you look at the problem domain with a different perspective
than you had the first time around. Obviously, I don't know anything
about this particular system, but with 230 screen it seems like
some things could be generalized. I suspect a much smaller number
of slightly context sensitive screens would suffice... My gut
tells me that if I looked at the system, I'd find lots of screens
that were basically the same, just slightly different... Whether
to use the same actual screen and perhaps let it be modified in runtime
due to some data driven approach or share most code in base classes
and have small modules with the specialized behaviour is something
to argue about in some design meetings. I'll leave that to the guys
who have to do it.
> You are comparing Python with VB 6. That's not what the OP is moving to
> as an alternative.
No, but it's what they've used and know. It's the reference for
their thinking.
> It's .NET. VB 6 has been out of development since 8
> years? Its not something to be compared to anymore. .NET on the other
> hand has all the advantages of Python accessible toolkits for this task
> and then some.
But as indicated by the OP, there might be reasons not to
sit down in that warm and soft lap from Redmond once more...
I assume they don't consider Python as a chance to avoid having
to really understand OO, or to be able to stay with the current
code base. Avoiding vendor lock-in is a very reasonable aspect,
and making it easier to port to non-windows platforms is also a
reasonable aspect.
> I am not saying Python is unfit for 230 screens. Just that I have not
> seen anyone build such a thing and better write about it.
Right, and this is interesting. After about 25 years of
programming I'm still not sure whether I should be impressed
or worried when I hear people describing systems with
hundreds of screens and hundreds of database tables.
I've worked with systems like that, and while I'm sure there
are situations where such big systems are really warranted,
the systems of this size that I saw could probably have been
shrunk to a quarter of their size if proper software development
practices had been used.
> People on the
> other hand have build web sites with such volume. Result? We have a ton
> of web application frameworks to address such needs.
I'm not 100% sure that the plethora of Python web kits is entirely
a benefit.
>>The really silly thing to do is to apply VB idioms on Python.
>
> It is. But that's what people do on their first few projects and it's
> perfectly natural. Didn't you :-) ?
No, I had used Python for years the first time I had to code VB.
I'm sure I wrote stupid code, but most of my awkward programming
experiences have been when I had to code Perl or VB6 after long
periods when I had only written Python and C++.
Josh
Thanks for the info, I took a look at SQLObject, but I'm not sure if
that's the way we want to go quite yet. We have a lot of embedded SQL in
the program and it looks like we'd have to convert it all to SQLObject
object code, which would just add to the initial complexity of the
project. I'll keep this bookmarked for future reference but I don't
think it'll work at the moment.
Josh
Josh
> I would look into Qt, PyQt, and the Qt Designer. It'll provide one of the
> easiest ways of porting your existing GUI dialogs into Python. Start
> building and componentizing your code. In VB6, all classes are flat under
> a single branch of a project tree, whereas in Python you'll likely
> implement modules of similar functionality. Overall porting your
> application will be a combination of rewriting, refactoring, and
> copy-and-translate code. Good luck. :-) It actually sounds like a fun
> project to be on. I had a lot of VB6 experience back in "the day" and a
> project like that would be a lot of fun to me.
We had looked at QT a while ago, but not in relation to Python. We were
just looking at it in relation to rewriting in C++. We'll have to
revisit it, paying attention to PyQt.
It is a rather fun project to be on. It's such a gigantic program
though, it gets difficult to comprehend how long it's going to take to
make this transision. Whether it's to Python, or some other direction.
Josh
Josh
I have taken a quick look at vb2py, but I'm not sure if that's the way
we want to go. If we do move to Python, we want to take advantage of
features of the Python language, rather than just rewriting a VB app in
Python. It may be a quicker solution in the short term, but in the long
term, I think completely embracing Python from the beginning is the best
way to go.
I tried running the project through the conversion wizard in VB.NET
Express Edition recently just for the fun of it. I come up with 16498
messages at the end of the hour and a half conversion. That's Compile
and Design errors as well as Warnings. It'd take a while to understand
those errors, possibly correcting them in VB6 before trying to do
another conversion, and again, we wouldn't be taking advantage of the
strengths of the new language. Again, a possibility, but not "good"
solution.
The web app functionality is something we also want to be very
accomodating of. We are planning on rewriting in such a way that the
functionality of the screens (input processing, error trapping etc.) is
done is an extensibly manner. That would mean using the same code for
both hardcoded windows as well as web forms.
Thanks,
Josh
Josh
> You haven't specified where your main pains are.
> Do you have at least rudimentary architecture ?
> How often do you have code reviews / refactored your code ?
> Have you been striving for good code ? Is it a total mess ?
Umm... We have been striving for good code, but being a small place,
it's rather difficult. The software in question is basically an ERP
program for the mattress industry. The program started out as a project
for one company, with a single programmer, with plans to grow.
The program, I am sure, still has much code left from those first days
of coding. With customer demands for new features, we do our best to
keep everything extensible, and open, but once a piece of code is
written, it's very rare that it will be changed unless it is being debugged.
> Guessing only from the number of screens, you probably
> have more than trivial amounts of data.
You have guessed right. One of our largest customers is using an Access
database that is around 800Mb and they have licenses for about 20
concurrent users (of our program). I know this may not sound huge for
someone who works at Walmart head office, but it's been interesting to
get Jet to work well.
> *Based on that assumption*, I'ld move the data FIRST.
> Get rid of the JET engine! If you are certain you will
> have to support SQL server, make it SQL server and
> MSDE/new express version first. Only then (perhaps)
> think about adding a totally free database engine.
> If not, take your pick but *move the data*.
This is a very interesting thought. That would mean a whole bunch of
rewriting of the current codebase.
The thought had been to write the port in an extensible manner, allowing
Jet, but being written well enough to easily use other backends. We'll
have to see.
> While you are on it, refactor the old code into something
> at least resembling a 3 to 5 layered approach with COM.
> Doesn't need to be perfect 100%, but more than 50%.
This makes sense, but OTOH, it might be more worth our time to properly
redesign in a more OOP friendly language.
> Then exchange the parts that benefit the most by
> implementation inheritance, since VB's interface inheritance
> is arguably the "best" technical reason for redundant code.
>
> I'ld guess the business layer[s] would be the next things to convert -
> but if your VB-forms are mostly slightly augmented copies differing
> only slightly, the GUI actually might benefit most from a rewrite,
> if the business rules are written redundance poor.
>
> One of the typical scenarios asking for GUI inheritance
> is insurance - customer data is only specific for few fields
> and even policy field groupings resemble each other across
> similar policies. A "minmally redundant" business layer could
> even in VB be implemented without too much sweat- for instance
> by having methods overwritten in a inheritance based OOP -
> design as name mangled methods on objects responsible across
> similar problem domains, keeping the "common" methods clean.
Unfortunately, there is little duplication across the different screens.
There definitely is some, like screens for picking a date range of a
report, with the reports being hardcoded (in our internal report writing
format) behind the print button on the screen. Here, the report should
be moved to a different module, or imported from a special format report
file, but the number of these types of screens are few and far between.
> Check your code with a critical eye -
> even code forced to be totally OOP for
>
> technical reason=langauge
>
> can be written across a wide spectrum of quality.
>
> If there is nothing which is good enogh to be converted last
> or no segregation/layering at all in the current program,
> (even if it came from the DOS dinosaurs, there were
> "good coding practices" known - some of the old "libraries"
> were better decoupled than modern day OOP class libraries.)
> get rid of most of the people responsible and start only then.
I think what needs to happen, is good coding guidelines put into place,
with a language that supports proper OOP. Inheritance doesn't even work
in VB, let alone polymorphism, unless you want to count passing Variants
and figuring out the types and then returning a variant.
> my 0.02 EUR
Thanks for your thoughts. It gives me some things to think over.
Josh
> mxODBC is probably the best way to talk to Jet in a way (DB API)
> that makes it reasonably easy to port the data to another RDBMS.
> Writing multi RDBMS applications is a whole chapter in itself,
> if not a subject for a whole book.
>
> Whether to use an ORM such as SQLObject on top of that is another
> issue...
mxODBC looks interesting. I'll need to research it some more.
Thanks,
Josh
Josh
>> Rewrites are always good and result in smaller code base if features
>> are not added. However, I doubt that will make the screens fewer. Lines
>> of code? Certainly.
>
>
> That depends on whether you just refactor the implementation
> of if you look at the problem domain with a different perspective
> than you had the first time around. Obviously, I don't know anything
> about this particular system, but with 230 screen it seems like
> some things could be generalized. I suspect a much smaller number
> of slightly context sensitive screens would suffice... My gut
> tells me that if I looked at the system, I'd find lots of screens
> that were basically the same, just slightly different... Whether
> to use the same actual screen and perhaps let it be modified in runtime
> due to some data driven approach or share most code in base classes
> and have small modules with the specialized behaviour is something
> to argue about in some design meetings. I'll leave that to the guys
> who have to do it.
For a little background, this is an ERP program. It looks after
accounting, customer orders, purchase orders, inventory, production
planning/scheduling, and product development. It is an industry specific
program, designed for the mattress manufacturing industry.
I'm not sure how many screens could be cut out. On the accounting side
of the program, there are a number that could be cut down I think. Right
now, Accounts Recievable and Accounts Payable are each separate sets of
screens. On a rewrite, we could make classes that are generalized for
accounts and then inherit these and modify them based on whether it's a
debit or credit account. This is mostly an educated guess since I don't
have a full understanding of the accounting side.
Other parts of the program I'm not so sure about. With proper OOP, I'm
sure some more screens could be cut out, or standardized using
inheritance and such.
>> You are comparing Python with VB 6. That's not what the OP is moving to
>> as an alternative.
>
>
> No, but it's what they've used and know. It's the reference for
> their thinking.
>
>> It's .NET. VB 6 has been out of development since 8
>> years? Its not something to be compared to anymore. .NET on the other
>> hand has all the advantages of Python accessible toolkits for this task
>> and then some.
>
>
> But as indicated by the OP, there might be reasons not to
> sit down in that warm and soft lap from Redmond once more...
>
> I assume they don't consider Python as a chance to avoid having
> to really understand OO, or to be able to stay with the current
> code base. Avoiding vendor lock-in is a very reasonable aspect,
> and making it easier to port to non-windows platforms is also a
> reasonable aspect.
I think in some ways, going with Python rather than just moving to
VB.NET comes from a desire to change the architecture of the program.
Going to VB.NET, we would be tempted to run the code through the upgrade
wizard and just get things working. We wouldn't be forced to change the
underlying architecture, we could just stay with the status quo.
Moving to Python would make us sit back and take a closer look at how
things are done and make taking shortcuts harder.
>> I am not saying Python is unfit for 230 screens. Just that I have not
>> seen anyone build such a thing and better write about it.
>
>
> Right, and this is interesting. After about 25 years of
> programming I'm still not sure whether I should be impressed
> or worried when I hear people describing systems with
> hundreds of screens and hundreds of database tables.
>
> I've worked with systems like that, and while I'm sure there
> are situations where such big systems are really warranted,
> the systems of this size that I saw could probably have been
> shrunk to a quarter of their size if proper software development
> practices had been used.
I understand what you are saying, and I'm sure the tasks our program
does could be made much cleaner. But, implementing an ERP which is
basically what we have, is a large project and the users need (or maybe
just want) access to lots of information.
>> People on the
>> other hand have build web sites with such volume. Result? We have a ton
>> of web application frameworks to address such needs.
>
>
> I'm not 100% sure that the plethora of Python web kits is entirely
> a benefit.
>>> The really silly thing to do is to apply VB idioms on Python.
I completely agree with this point. We want to get away from the way
that VB makes us think and therefore program. It's hard to implement OOP
in a language that won't support inheritance or polymorphism except
through the outrageous contortions, and then not even do it properly.
Josh
> I suggest you hire a couple of experienced Python developers, pick half a
> dozen screens, say three simple screens and three complex ones, and build
> them in Python. That will give you an idea of how much work is involved.
> Don't get your VB developers to try to port it themselves, that's a recipe
> for disaster: they will be trying to learn Python, and will end up writing
> VB code in Python, which is sure to be slow, inefficient, bulky and buggy.
>
> You may also like to think about using a web front end. If your GUI app
> only uses simple controls like text fields and push buttons, a web front
> end might simplify things a lot.
>
> Good luck.
Point taken on VB coders writing Python. I think part of this though, is
having programmers who are willing to learn new paradigms. I agree that
there could be many teething problems, but knowing the team, I believe
that after doing a few modules/classes (2 or 3 months, maybe more) we
could work out the bugs, and go on to new problems.
Having worked out problems with the first classes, as long as there are
no major architectural problems, we could continue on with the rest of
the program. Once good headway on the project, come back to those first
converts and refactor them, using knowledge accumulated from the last 6
months in Python.
Maybe I'm just being overly optimistic, but I think the VB programmers
we have can make the jump to Python. Their first lines of Python will
just be VB in Python, but, once the power of Python becomes apparent,
there'll be no going back.
Josh
Magnus Lycka
> I understand what you are saying, and I'm sure the tasks our program
> does could be made much cleaner. But, implementing an ERP which is
> basically what we have, is a large project and the users need (or maybe
> just want) access to lots of information.
I'm not pretending to know what this system does in detail, but the
fact that you need to access a lot of information doesn't imply that
you need lots of different user interfaces. I think e.g. wordprocessors,
spreadsheet programs and SQL monitors etc are proof of that.
I suppose that you need to present a lot of differnt kinds of data,
and that you need to provide various search parameters etc for
different data sets, but this sounds like something that might be
very few screens that adapt to some kind of meta-data, perhaps XML
descriptions or small "configuration files" containing som simple
Python data structures and possibly input validation routines etc.
Maybe this information should even be in the database?
The problem arises if there is behavior in the application that
varies with data, but with Python, it's certainly possible to handle
that to some extent even if these screens are driven by database
content.
BTW, I guess DABO http://dabodev.com/ is intended just for this kind
of application, but I doubt it was ever used for a project of this
size, it's fairly new... Never used it myself...
Whatever you use for your user interface, I suggest that you use
a multi-tier approach, and a very thin user interface layer. In my
opinion that makes testing much easier. I'd recommend that you use
a proper test tool and employ a test-driven approach.
Josh
> I suppose that you need to present a lot of differnt kinds of data,
> and that you need to provide various search parameters etc for
> different data sets, but this sounds like something that might be
> very few screens that adapt to some kind of meta-data, perhaps XML
> descriptions or small "configuration files" containing som simple
> Python data structures and possibly input validation routines etc.
> Maybe this information should even be in the database?
This all makes sense. Being hamstrung by the VB model has hampered our
thinking in areas like this I think. Having a smaller number of dynamic,
data driven screens is good for maintainability anyways.
> The problem arises if there is behavior in the application that
> varies with data, but with Python, it's certainly possible to handle
> that to some extent even if these screens are driven by database
> content.
>
> BTW, I guess DABO http://dabodev.com/ is intended just for this kind
> of application, but I doubt it was ever used for a project of this
> size, it's fairly new... Never used it myself...
I have looked through the DABO website and it looks very promising. By
the looks of it, the authors come from the same type of background
(Windows RAD application) as we are coming from. I'll have to download
it and give it a run for it's money.
> Whatever you use for your user interface, I suggest that you use
> a multi-tier approach, and a very thin user interface layer. In my
> opinion that makes testing much easier. I'd recommend that you use
> a proper test tool and employ a test-driven approach.
This is what we're planning on. If the user interface is as thin as
possible, that allows easily porting it to a web app etc.
As for the testing, that's something we'll need to learn about. I've
read some articles about test driven programming in relation to extreme
programming. Can you give me links to any good sites explaining how this
all works? (Off list may be better for these off-topic links)
Thanks,
Josh
Magnus Lycka
> As for the testing, that's something we'll need to learn about. I've
> read some articles about test driven programming in relation to extreme
> programming. Can you give me links to any good sites explaining how this
> all works? (Off list may be better for these off-topic links)
I think this might be interesting to more people...
There is some stuff from my Europython presentation in 2004 here:
http://www.thinkware.se/epc2004test/
The pdf and the "log" from a test-first-session might be of interest.
The pdf contains a number of relevant links.
You might also want to look at the TextTest testing framework, which
has been developed here at Carmen Systems. http://texttest.carmen.se/
With tools like the unittest module, you have to write and maintain
a lot of low level tests. With TextTest you have to write your
application so that it can produce textual output in a repeatable
way. TextTest will basically store all the output from a test run,
and compare that to subsequent test runs.
While unittest is more intended for low level tests written by the
programmer so that he can verify that his program does what he
intended, TextTest is more useful as a regression testing tool or
for acceptance tests, to verify that a software system behaves as
it is required to do. Their applicability overlaps though, and
depending on your type of application, your own preferences etc,
you might use either approach or a combination. At Carmen, we
mainly use TextTest.
Other interesting testing frameworks are Fit, http://fit.c2.com/
and FitNesse http://www.fitnesse.org/ but I have no hands-on
experience with them. For Python code, use PyFit, see e.g.
http://agiletesting.blogspot.com/2004/11/writing-fitnesse-tests-in-python.html
Grig Gheorghiu
I've been writing TextTest tests lately for an application that will be
presented at a PyCon tutorial on "Agile development and testing". I
have to say that if your application does a lot of logging, then the
TextTest tests become very fragile in the presence of changes. So I had
to come up with this process in order for the tests to be of any use at
all:
1) add new code in, with no logging calls
2) run texttest and see if anything got broken
3) if nothing got broken, add logging calls for new code and
re-generate texttest golden images
I've been doing 3) pretty much for a while and I find myself
regenerating the golden images over and over again. So I figured that I
won't go very far with this tool without the discipline of going
through 1) and 2) first.
>From what I see though, there's no way I can replace my unit tests with
TextTest. It's just too coarse-grained to catch subtle errors. I'm
curious to know how exactly you use it at Carmen and how you can get
rid of your unit tests by using it.
Grig
Magnus Lycka
by TextTest experts such as Goeff and Johan here at Carmen, so I'm
not a TextTest expert by any measure. I probably use it in an non-
optimal way. For really good answers, I suggest using the mailing
list at sourceforge:
http://lists.sourceforge.net/lists/listinfo/texttest-users
For those who don't know, TextTest is a testing framework which
is based on executing a program under test, and storing all
output from such runs (stdout, sterr, various files written
etc). On each test run, the output is compared to output which is
known to be correct. When there are differences, you can inspect
these with a normal diff tool such as tkdiff. In other words,
there are no test scripts that perform actual and expected values,
but there might be scripts for setting up, extracting results and
cleaning up.
Grig Gheorghiu wrote:
> I've been writing TextTest tests lately for an application that will be
> presented at a PyCon tutorial on "Agile development and testing". I
> have to say that if your application does a lot of logging, then the
> TextTest tests become very fragile in the presence of changes. So I had
> to come up with this process in order for the tests to be of any use at
> all:
The TextTest approach is different from typical unittest work,
and it takes some time to get used to. Having worked with it
for almost a year, I'm fairly happy with it, and feel that it
would mean much more work do achieve the same kind of confidence
in the software we write if would use e.g. the unittest module.
YMMV.
> 1) add new code in, with no logging calls
> 2) run texttest and see if anything got broken
> 3) if nothing got broken, add logging calls for new code and
> re-generate texttest golden images
I suppose its usefulness depends partly on what your application
looks like. We've traditionally built applications that produce
a lot of text files which have to look in a certain way. There are
several things to think about though:
We use logging a lot and we have a clear strategy for what log
levels to use in different cases. While debugging we can set a
high log level, but during ordinary tests, we have a fairly low
log level. What log messages gets emitted is controlled by an
environment variable here, but you could also filter out e.g.
all INFO and DEBUG messages from your test comparisions.
We're not really interested in exactly what the program does when
we test it. We want to make sure that it produces the right results.
As you suggested, we'll get lots of false negatives if we log too
much details during test runs.
We filter out stuff like timstamps, machine names etc. There are
also features that can be used to verify that texts occur even if
they might appear in different orders etc (I haven't meddled with
that myself).
We don't just compare program progress logs, but make sure that
the programs produce appropriate results files. I personally don't
work with any GUI applications, being confined in the database
dungeons here at Carmen, but I guess that if you did, you'd need
to instrument your GUI so that it can dump the contents of e.g.
list views to files, if you want to verify the display contents.
Different output files are assigned different severities. If
there are no changes, you get a green/green result in the test
list. If a high severity file (say a file showing produced
results) differ, you get a red/red result in the test list.
If a low severity file (e.g. program progress log or stderr)
changes, you get a green/red result in the test list.
You can obviously use TextTest for regression testing legacy
software, where you can't influence the amount of logs spewed
out, but in those cases, it's probable that the appearence of
the logs will be stable over time. Otherwise you have to fiddle
with filters etc.
In a case like you describe above, I could imagine the following
situation. You're changing some code, and that change will add
some new feature, and as a side effect change the process (but
not the results) for some existing test. You prepare a new test,
but want to make sure that existing tests don't break.
For your new test, you have no previous results, so all is red,
and you have to carefully inspect the files that describe the
generated results. If you're a strict test-first guy, you've
written a results file already and with some luck it's green,
or just have some cosmetic differences that you can accept. You
look at the other produced files as well and see that there
are no unexpected error messages etc, but you might accept the
program progress log without caring too much over every line.
For your existing tests, you might have some green/red cases,
where low severity files changed. If there are red/red test
cases among the existing tests, you are in trouble. Your new
code has messed up the results of other runs. Well, this
might bee expected. It's all text files, and if it's hundreds
of tests, you can write a little script to verify that the
only thing that changed in these 213 result files was that a
date is formatted differently or whatever. In a more normal
case, a handful of tests have a few more lines (maybe repeated
a number of times) in some progress logs, and it's quick to
verify in the GUI that this is the only change. As you suggested,
you might also raise the log level for this particular log
message for the moment, rerun the tests, see that the green/red
tunred to green/green, revert to log levels, rerun again, and
save all the new results as correct.
> I've been doing 3) pretty much for a while and I find myself
> regenerating the golden images over and over again. So I figured that I
> won't go very far with this tool without the discipline of going
> through 1) and 2) first.
If you want this it migh be convenient to write something like
LOGLEVEL_NEW=1
...
log(LOGLEVEL_NEW, 'a new log message')
...
log(LOGLEVEL_NEW, 'another new log message')
...
and then remove the first assignment and replace all LOGLEVEL_NEW
with DEBUG or whatever after the first green test.
It's also worth thinking about how much logging you want during
tests. Do you want to see implementation changes that don't cause
changed results? You are verifying the achieved results properly
in some kind of snapshot or output files, right?
>>From what I see though, there's no way I can replace my unit tests with
> TextTest. It's just too coarse-grained to catch subtle errors. I'm
> curious to know how exactly you use it at Carmen and how you can get
> rid of your unit tests by using it.
I work in the database group at Carmen R&D, and a big part of our job
is to provide APIs for the applications to use when working with
databases. Thus, we typically test our APIs rather than full programs.
This means that we need to write test programs, typically in Python,
that use the API, and present appropriate results.
The way we work in those cases isn't very different from typical unit
testing. We have to write test scripts (since the *real* program
doesn't exist) and we have to make the test scripts produce a
predictable and reproducable output. This isn't different from what
you do in unittest. If you can use .assertEqual() etc, you can also
get output to a file which is reproducable. For us, who mainly work
with content in databases, the typical thing for use to verify is that
one or several tables hold a particular content after a particular
set of operations. The easiest way to verify that is to dump the results
of one or several DB queries to text files, and to compare the files.
Typically, the files are identical, and if there is a difference, we
can directly see the difference between the files with tkdiff. This
is far superior that just writing typical unittest modules where we'd
perform a certain query and get a message that this particular query
gave the wrong result, but I don't get a chance to see any context.
Tkdiff on dumps of small tables is very informative.
It's also easier when the expected result changes. I can usually see
fairly easy in tkdiff that the new result differs from the old just
in the way I expected, and save the new result as the correct result
to compare with. One has to be very careful here of course. If you're
in a hurry and don't inspect everything properly, you could save an
incorrect result as correct, and think that your flawed program works
as intended. It might be less likely that you would write an incorrect
assertion in unittest and make the program produce this incorrect
result.
This is a fundamental difference between unittest and the way *I* use
texttest. With unittest, you state your expected results before the
test, and mechanically verify that this is correct. With TextTest, you
*could* do the same, and change the text file with expected results to
what you now think it should be. I'm usually too lazy for that though.
I just change my code, rerun the test, expect red, look at the
difference in tkdiff, determine whether this is what it should look
like, and save the new results as correct if I determine that it is.
So far, this hasn't led to any problems for me, and it's far less work
than to manually write all the assertions. TextTest (particularly in
the mature setup here) also provides support for distribution of tests
among hosts, on different operating systems, with different database
systems etc, automatic test execution as part of nightly builds etc.
TextTest originated among our optimization experts. As far as I know,
they use it more as it is intended than we do, i.e. to test complete
programs, rather than small scripts written just to check a feature in
an API. Honestly, I don't know a lot of details there though. They
seem to practice XP fairly strictly, but instead of writing unit tests,
they test the entire programs in TextTest, and they obviously manage
to produce world class software in an efficient way. As I said, I'm
not the expert...
Michael Tobis
be insurmountably difficult to acheeve both in a move to another
platform.
mt
Grig Gheorghiu
digest from your message :-) Maybe I'll continue the discussion on the
mailing list you mentioned.
Grig
dre...@yahoo.com
wrote:
>We have a program written in VB6 (over 100,000 lines of code and 230 UI
>screens) that we want to get out of VB and into a better language. The
>program is over 10 years old and has already been ported from VB3 to
>VB6, a job which took over two years. We would like to port it to
>Python, but we need to continue to offer upgrades and fixes to the
>current VB6 version. Does anybody know of ways we could go about
>rewriting this, one screen at a time, in Python, and calling the screens
>from the existing program?
>
>We are also looking for a graphics toolkit to use. IronPython with it's
>.NET bindings and ability to integrate with Visual Studio looks good,
>but leaves that bad MS taste in the mouth.
>
Because MS is too expensive ?
What you're trying to do sounds ideal for an N-tier application in
.NET, where the DAL is isolated, and you can you use both VB and
ASP.NET to access business objects.