Plans for BPBible 0.5 and beyond

[Jonathan Morgan]

Executive summary: I recommend we switch to using wxWebConnect as a renderer with the existing wxPython application, backport applicable changes from the XUL prototype if possible, and aim for a BPBible 0.5 release with the new renderer by the end of the year.  For those interested, details and rationale are below.

It's nearly 1.5 years since we branched the "next generation" XUL frontend.  In the last six months, very little work has been done on this.  There have been a number of reasons for that, but a key one is that I have increasingly been feeling that there is a very real possibility that the port will never be finished.  Another is that I really think that the XULRunner platform, while powerful, is not really very developer friendly [the last time I did significant work with it (at the end of last year) nsITreeView almost drove me insane, and I didn't complete what I wanted to.  I have not since had either the time or courage to go back to that particular work].   Having BPBible die because the XULRunner port couldn't be made to work would be a waste of a significant amount of effort, so I have spent the last few months looking around for alternatives.  When we were looking into XUL, we came to the conclusion that wxMozilla as a project was dead, and that wxWebKit was really not ready, so XUL might be a better option.  We always intended to look around for other alternatives if the path to XUL was not as good as we hoped.  I evaluated two main alternatives: wxWebKit (WebKit) and wxWebConnect (XULRunner).

wxWebKit

As WebKit is apparently the way of the future, this appeared the logical choice.  However, I tried it and decided it still didn't really feel ready.  Simple things like having a hand cursor appear if you hovered over a link just didn't work.  If I had to I might have tried adding support for these things, but I couldn't even get my hands properly on the source code (git kept on dying cloning the repository), but the codebase was huge and I couldn't easily figure out what was going on.  I was also concerned how many other things might not be properly implemented, since what we were essentially getting was a highly capable browser engine, but all of the rendering had to be provided by wxWebKit, and it wasn't all there.

wxWebConnect

This is developed by Kirix (creators of AUI), and released July last year.  It is a wrapper of XULRunner to make it easier to use as a renderer in wxWidgets applications.  I think it is fundamentally a better approach, to take an existing proven renderer and work with that rather than taking the engine and trying to port the functionality to a different platform.  I have confidence that it does work, because I do use a Firefox that is built on it.  The existing version on the Kirix website did have a few problems, the principal one being a reliance on XULRunner 1.8.1, but I have upgraded it to work with XULRunner 1.9.2, added functionality and hacked together a rough Python/SWIG binding for it.

At present I have changed quite a bit of core BPBible functionality to use WebConnect, and I think it is the best way forward for the next generation of BPBible.  It is still possible that we will find showstopper issues with it and have to stop and reconsider, but I have found with several things I have tried changing from wxHTML to wxWebConnect that it worked without a hitch (though sometimes I have had to add missing functionality to wxWebConnect first).  Compared with wxWebKit I find the wxWebConnect code comparatively easy both to use and to change (well, as easy as any code using XPCOM in C++ is to change).  I have been putting my changes in the webconnect branch in the main BPBible repository (http://code.google.com/p/bpbible/source/browse#svn/branches/webconnect), which anyone is welcome to look at, though I would be highly surprised if anyone were able to use it (it relies on changes to wxWebConnect I have published at http://github.com/jonmmorgan/wxwebconnect, Python bindings which I have not yet published anywhere, and probably the exact details of my build environment).  WebConnect is supposed to work on Linux, but I have not yet tested BPBible there.  Mac support is not really present (this may be able to change with wx 2.9.x).

At present, I'm looking at a release 0.4.7 sometime in the next 3 months or so (since there are already changes made to 0.4.6, and I think there will be more, and there are other annoyances I want to squash), and at a release 0.5 (or at least an early beta) before the end of the year with the new renderer and significant other enhancements.  I'm hoping that a lot of the changes in the BPBible XUL branch that I care about will be able to be backported relatively easily (continuous scrolling comes high on the list).  As we are still dealing with significant changes to the system (though nowhere near the madness of a complete rewrite) I do not have any idea if this timetable is realistic or not.

After 0.5, we would intend to return to what we were always able to do before the XUL mirage came up: make significant improvements to BPBible, make a few releases a year (major or minor depending on our mood and time available), and try to enhance the usefulness of our favourite Bible tool.  All such plans are of course subject to God's will.  Fairly high on my agenda is exploring the ways user editable content can be used to improve Bible software and Bible study (though that is a long term plan, and is unlikely to all fit into a single release).

As part of the uncertainty around the switch to XUL I think we have discouraged potential developers from joining us.  At times the project has felt almost dead.  The aim of this switch is to revive the project.  If anyone is interested in helping then please contact us (there will still be some uncertainty and flux while I try and get the project into a fit state where it and its dependencies can be built and used in a generic manner, but hopefully that can be resolved).

In the more distant future, I think we will have more choices to make.  The way people use computers is changing (a bit of a cliche, since it has always been changing, but anyway).  Increasingly, people expect to be able to access things on the web, store things "in the cloud", or access them on their mobile phones.  I still prefer desktop applications for a lot of things, but being a desktop application could limit shelf life.  Something we will need to consider further down the track is if we want to make BPBible available on the web, or to interact with mobiles in some way (I'm not really sure how at the moment).  For the moment, I think a good start will be to use a decent HTML renderer and put logic in HTML/CSS/Javascript where it makes sense, though it would still be likely to be a significant effort to switch platforms to the web [apart from anything else, getting copyright permissions for the job could be substantial, while we get all this "for free" through Crosswire].

We realise we are losing things by not going the XUL route.  We have had problems with wxWidgets in the past, and will probably continue to have them.  Some of the fundamental questions I have are:

* Will AUI ever be good enough as a layout system?

* Is wxWidgets powerful enough?

* Does it support Mac well enough? (not that BPBible is ever likely to be "Mac-like").

* How easy will things be to package for Linux / Windows?

* Will the renderer we pick ultimately limit us?  Will there be things that we can do in the renderer but not with wxWidgets?

* etc.

I think we will lose some of the following things we hoped to get for free from XUL, such as:

* A relatively well-designed extension system, including overlays and similar (you may recall I have some doubts about extensions, but it still could be something worth supporting).

* Automatic update? (though we might be able to get support for it elsewhere.

* Ubiquity

* Being at the "cutting edge" of technology [though sometimes I think it might cut us a bit too hard]

* Full RtoL in the UI (though we should get it in the renderer).

We will lose the ability to make fundamental changes to our existing system that were too hard in wxWidgets, or that we just thought we could do while porting to XUL.  However, I do think it's a choice between that and giving up completely, so I'm happy that it is the best choice.

Jon

[cric...@gmail.com]

Just curious - have you all looked into Qt and its python bindings at all?

-Ben

On Sunday 13 June 2010 11:06:12 Jonathan Morgan wrote:
> Executive summary: I recommend we switch to using wxWebConnect as a
> renderer with the existing wxPython application, backport applicable
> changes from the XUL prototype if possible, and aim for a BPBible 0.5
> release with the new renderer by the end of the year. For those
> interested, details and rationale are below.
>
>
>
> It's nearly 1.5 years since we branched the "next generation" XUL frontend.
> In the last six months, very little work has been done on this. There
> have been a number of reasons for that, but a key one is that I have
> increasingly been feeling that there is a very real possibility that the
> port will never be finished. Another is that I really think that the
> XULRunner platform, while powerful, is not really very developer friendly
> [the last time I did significant work with it (at the end of last year)
> nsITreeView almost drove me insane, and I didn't complete what I wanted
> to. I have not since had either the time or courage to go back to that
> particular work]. Having BPBible die because the XULRunner port couldn't
> be made to work would be a waste of a significant amount of effort, so I
> have spent the last few months looking around for alternatives. When we
> were looking into XUL, we came to the conclusion that wxMozilla as a
> project was dead, and that wxWebKit was really not ready, so XUL might be
> a better option. We always intended to look around for other alternatives
> if the path to XUL was not as good as we hoped. I evaluated two main
> alternatives: wxWebKit (WebKit) and
> wxWebConnect (XULRunner).
>

> *wxWebKit*

> As WebKit is apparently the way of the future, this appeared the logical
> choice. However, I tried it and decided it still didn't really feel ready.
> Simple things like having a hand cursor appear if you hovered over a link
> just didn't work. If I had to I might have tried adding support for these
> things, but I couldn't even get my hands properly on the source code (git
> kept on dying cloning the repository), but the codebase was huge and I
> couldn't easily figure out what was going on. I was also concerned how
> many other things might not be properly implemented, since what we were
> essentially getting was a highly capable browser engine, but all of the
> rendering had to be provided by wxWebKit, and it wasn't all there.
>

> *wxWebConnect*

> similar (you may recall I have some doubts about extensions, but it still *
> could* be something worth supporting).

[Ben Morgan]

Yes, I did look at PyQT quite a while ago, but I didn't really like it at all (the look and feel of it, programming with it).

God Bless,
Ben
-------------------------------------------------------------------------------------------
Multitudes, multitudes,
   in the valley of decision!
For the day of the LORD is near
   in the valley of decision.

Giôên 3:14 (ESV)

[Jonathan Morgan]

Having worked professionally with PyQt I agree with Ben, but to me the reasons go deeper than that.  The main thing that went wrong with the XULRunner port was not crippling problems in XULRunner (though some things about it certainly annoyed me and caused a lot of grief).  What went wrong was that converting the entire application to a different windowing system is a lot of work and takes a long time.  We always knew that could be a problem, but tried to ignore it because we didn't see a better alternative.  I reconsidered the plans because I thought there was a reasonable chance that that porting work would never be done and because new alternatives had come on the wx scene.  Continuing with wx and only changing the rendering engine means we don't need to rewrite the rest, which substantially increases the chance of having 0.5 ready by the end of the year, which I think is a critical target to keep BPBible a relevant application.

If I were looking for an alternative to wxWidgets other than XULRunner (which would equally make the project unlikely to finish), I would be more likely to look at Pyjamas Desktop first rather than Qt (as you would be more likely to be able to switch from desktop to web then).

Jon