Where are we going? (PLEASE READ AND COMMENT)

[Matt Berg]

Dear RapidSMS Community,

I wanted to pick up on an earlier thread from earlier this month on the stage of development of RapidSMS and where we see this project going as a group.

Based on conversations with many of you, I think it's fair to say that most of us our now happy with the current state of the RapidSMS core and how it's been so tightly bound with the apps.  While the current system has served the community admirably well, we've had time to kick the tires and learn how we think we could make it better.

Two things have been holding us back though from moving forward:

• There have been a lot of complaints but never succinct or unified proposals by the community on how to make RapidSMS better
• I think the community at large is unclear on whether or not there are active ongoing efforts to improve the core.  If so, it's a bit unclear what exactly is being done and what the development timeline is. This uncertainty is holding others in the group from taking this task on.

Moving forward, we need to quickly agree upon a consensus on how to move this forward. To do this it would be extremely helpful to get the following from everyone involved in this project and who has an opinion.  You can do this by trying to answer (in detail) to the following:

1. What are your current issues with RapidSMS in general?
2. What are the specific things we should do to improve the core (from this we will try and identify themes)
3. How should we handle the apps? Should we have core apps before and contribs like before?  
4. What are people's feelings on the current commit rights in the project?  Is the process still hindering user contributions or is it just apathy/busyness?

Finally, if you are actively working on improving the core, can you please explain to the group in detail what you are working on.  Please explain what your additions will mean to us in the future and what your expected timeline is?

We cannot commit resources to improving RapidSMS until we can agree on a common way forward.

We REALLY value your thoughts on this.  Please vote with your ideas.

Thanks,

Matt 

--
You received this message because you are subscribed to the Google Groups "rapidsms" group.
To post to this group, send email to rapi...@googlegroups.com.
To unsubscribe from this group, send email to rapidsms+u...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/rapidsms?hl=en.

[Andy McKay]

Would you like this in private - so you can sum them all up and start discussion? Worried that if we all reply to this, it could become one very big thread.
--
Andy McKay, @andymckay
Django Consulting, Training and Support

[Matt Berg]

Public i think the responses could help shape the discussion. Plus
want to keep this very open.

We can synthesize as we go and break up into new threads as needed.

[Jonathan Jackson]

Thanks for pushing on this Matt,

I don't actually have the pleasure of coding in rapidSMS, so I can't speak very intelligently (i.e. take this all with a huge grain of salt) on the code.  However, given the many discussions Dimagi and UNICEF programmers have had that I've watched, I would advocate for letting Adam, Evan, and others get their refactoring / repo reorganization to a point where they are happy for everyone to take a good look at it, and let that be our starting point for the new way forward.  From what I can tell, I think many of the complaints folks have had have been addressed by this work.  If this did make sense to do, perhaps Adam, Evan, and anyone else who has worked on that could suggest where they could use community support to get it over the hump.  

Cheers,

J

[Matt Berg]

Jonathan,

I think there may be some deeper underlying concerns with the overall approach we're taking things that refactoring/repo organization won't solve.  I think it would be worth hearing from the group.

It would also really help to get a detailed account of what this refactoring effort entails and when we can hope for it to be done.  

Thanks,

Matt

--
ICT Director
Millennium Villages Project
Columbia University, NYC
M: +1.646.463.1273
Skype: mlberg

[renaud gaudin]

Jonathan,

I think this is the opposite of what the community want.
The reason why Malthe started his proof-of-concept is exactly this:
always the same people driving the train and that's the ones with the
less time for it.
Others thus get stuck in this unclear state where you can't improve
things because you are waiting for someone else to.

Developers in the community have different needs and constraints ; it
is very difficult to make everybody happy. The solution is to discuss
most choices so that we have the less people unhappy.

I really like Matt's proposition of a face-to-face meeting ; I would
like it to be like the Ubuntu dev meetings.
We could then discuss all issues and identify which part needs
flexibility and which doesn't.
Doing this over email is difficult (especially if there is nobody
leading it technically) and takes months.

A one or two day meeting could settle the design principle for a
better RapidSMS.
Add some more days and you can probably get some basic prototypes.

renaud

[Matt Berg]

Renaud,

Thanks.  I agree a face-to-face would be great.  However, I don't think we should wait until that happens to have this discussion.  Also, it's likely that only a small amount of the community would be able to make it.  So for now, I think it would be great if we could use this forum to start to trying to address specifically some of the questions I'm trying to raise on behalf of the group.

Thanks,

Matt

--
ICT Director
Millennium Villages Project
Columbia University, NYC
M: +1.646.463.1273
Skype: mlberg

[Sean Blaschke]

Thanks Matt for getting this discussion started,

I am concerned that we will have another great discussion on this
board with important points raised, but at the end of the day everyone
goes back to their own projects and nothing happens. I strongly agree
with Renaud in that we need somebody technical to lead/drive/champion
this discussion (suggestions?), and that anyone working on the core
keeps the rest of the community regularly informed on progress.

There seems to be a few things everyone agrees on, yet the community
always bitches that these aren't being done:

1. Ensuring the software is properly tested
2. Ensuring the software is properly documented

To me, this seems one of the most important issues hindering
replicability and scalability of our projects. If junior programmers
can't look at code and understand what it does, we lose. If senior
programmers rewrite apps that have already been written because they
don't know what they do, we lose. If apps fail because they haven't
been properly tested, we lose - and with many of our deployments,
people might die.

Even though there is community agreement on this, there have been a
total of 7 test commits within RapidSMS since July 2009. I pose this
question back to the community... Why is this the case?

For the non technical project managers on this board, I think it is
critical we check in more often and ensure that all software being
developed for our projects is tested and documented properly. I
personally will not deem a project (or deliverable) is complete until
this is done. For the software programmers, please help and support
each other to do this.

Sean

[Jonathan Jackson]

Hi All,

All great points.  My goal is not to limit discussion, but to say that I think the new refactoring is a place where the code would need to head no matter what we do next, namely:

1.  Make it look like a normal Django App

2.  Clean up the APIs.

3.  Make is so that RapidSMS doesn't have to run the world and can easily be added to existing apps.

4.  Split out core vs. contrib apps.  (note that Evan already started this discussion a while back)

I definitely think there are some choices we need to discuss, but my sense is that where the re-factored code is headed is the right direction on those 4 points (I don't how far along) no matter where we are headed next.  For example, I'm personally certain no one is going to make a convincing argument to stray further from Django standards.  And, I think the community very much wants the above 4 points.  

By all means, lets get the discussion going now.  The reason to have a discussion with the aim of improving the code base, is if we believe that as a result of those discussions the community would spend effort implementing those changes and getting them in to core.  My argument is that some or all of that effort is better spent now getting the new code base to a place where its ready to go, and then start discussions from there.  But obviously, every community member can make that choice on their own.  If those who have worked on the new code base don't agree with me or I misrepresented it, please ignore everything I just said.

Note that with our community rules if you propose a vote to change something, someone can't veto your proposal unless they are willing to put the work in to fix it a different way.  So there's a pretty easy way to drive in your viewpoint (or new viewpoints) simply by being willing to do the work.

thanks,

Jonathan

[Matt Berg]

Jonathan,

Thanks completely agree.

We absolutely want 1-4 (that's why I'm trying to get us to agree upon).  I think that's a great way of explaining it.  If we are well on our way there that's great!  I just don't think the community has enough info right now to understand that.

Thanks,

Matt

--
ICT Director
Millennium Villages Project
Columbia University, NYC
M: +1.646.463.1273
Skype: mlberg

[Malthe Borch]

On 6 May 2010 16:21, Jonathan Jackson <jjac...@dimagi.com> wrote:
> I definitely think there are some choices we need to discuss, but my sense
> is that where the re-factored code is headed is the right direction on those
> 4 points (I don't how far along) no matter where we are headed next.  For
> example, I'm personally certain no one is going to make a convincing
> argument to stray further from Django standards.  And, I think the community
> very much wants the above 4 points.

I agree. However, parts of the community also very much want:

5. Tests
6. Documentation

If we can't have these as well, then I don't see much value in any of it.

It's absolute essential that a collaborative open-source project has
regression tests. It is absolute essential for any "rapid application
framework" to have documentation about what's there and how it works.

The very first step towards this is to put a README.txt inside every
application that briefly describes what it does, who did it and where
it was used.

\malthe

[Ian Lawrence]

Hi
I think the goal of making RapidSMS look like a normal Django
application is key.
I was part of a test automation project written in Django called
autotest (http://autotest.kernel.org/) which for me is a very well
written and documented large scale Django app. The documentation was
like:

User documentation
Test developer documentation
Core developer documentation
System administrator documentation

with the Core dev documentation particularly good
http://autotest.kernel.org/wiki/Developer ..maybe something like this
might work for RapidSMS?

Regards


Ian



--
http://ianlawrence.info

[Erica Kochi]

Hi All,

I wanted to provide an update on what is happening from our end.  As Evan said in an email from Apr 20th to the list:

Many frustrations and suggestions from the last year are currently being addressed in the development repositories (rapidsms-core-dev and rapidsms-contrib-apps-dev). 

The improvements are based on discussions from this list last fall, and the reorganization summary here: http://wiki.github.com/rapidsms/rapidsms/repository-reorganization.

The specific changes and new features currently underway are listed here: http://etherpad.unicefinnovation.org/rapidsms-refactoring-tasks

These specific tasks include: 

• out-of-the box demo app 
• separation of essential, reusable, and project-specific code
• API simplification and documentation
• easier installation and configuration
• align rapidsms more closely with django conventions

By separating code, we hope to discourage the divergent development that has plagued the project so more concerted efforts can be made on the essential (routing, parsing, etc) and reusable portions (extensible user registration, locations, messaging, etc)

Adam is leading the work on these changes. He's striking-through task as he implements them on the etherpad linked below. Hopefully these items should be complete very soon. 

I think its a great idea - as a community - to start discussions on future directions with these changes in mind.  My question to you as a group is how we are going to capture these discussions in to actual directions and who will take this on.  Any volunteers (Matt? since you started this thread?)

Best,

Erica

---------------------------------------------------------
Erica Kochi
UNICEF
Director's Office, Innovation Unit
UNICEF Supply Division, NYHQ
3 UN Plaza, 44 St, New York NY 10017

office: +1 212 326 7785
cell: +1 917 251 2954

unicefinnovation.org, www.unicef.org

unite for children

[Erica Kochi]

oh and i almost forgot.  weren't the core commit rules changed to:

send note about commit to list

wait a day or so to see if anybody objects

if not, push

[bayo opadeyi]

Sorry, I am rather new. Where can I get information on things like
commit rules, core devs, etc...

On 5/6/10, Erica Kochi <eko...@unicef.org> wrote:
> oh and i almost forgot. weren't the core commit rules changed to:
>
> send note about commit to list
> wait a day or so to see if anybody objects
> if not, push
>
>
> On 6 May 2010 14:54, Erica Kochi <erica...@gmail.com> wrote:
>
>> Hi All,
>>
>> I wanted to provide an update on what is happening from our end. As Evan
>> said in an email from Apr 20th to the list:
>>
>> Many frustrations and suggestions from the last year are currently being
>> addressed in the development repositories
>> (rapidsms-core-dev and rapidsms-contrib-apps-dev).
>>
>> The improvements are based on discussions from this list last fall, and
>> the
>> reorganization summary here:
>> http://wiki.github.com/rapidsms/rapidsms/repository-reorganization.
>>
>> The specific changes and new features currently underway are listed here:
>> http://etherpad.unicefinnovation.org/rapidsms-refactoring-tasks
>>
>> These specific tasks include:
>>

>> - out-of-the box demo app
>> - separation of essential, reusable, and project-specific code
>> - API simplification and documentation
>> - easier installation and configuration
>> - align rapidsms more closely with django conventions

>> rapidsms+u...@googlegroups.com<rapidsms%2Bunsu...@googlegroups.com>
>> .

>> > For more options, visit this group at
>> http://groups.google.com/group/rapidsms?hl=en.
>> >
>>
>>
>
> --
> You received this message because you are subscribed to the Google Groups
> "rapidsms" group.
> To post to this group, send email to rapi...@googlegroups.com.
> To unsubscribe from this group, send email to
> rapidsms+u...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/rapidsms?hl=en.
>
>

--
Bayo
Evince Systems (http://www.evincesystems.com)
http://boyombo.blogspot.com
tweet: @boyombo

"Perfection, then, is finally achieved, not when there is nothing left
to add, but when there is nothing left to take away"
- Antoine de Saint Exupery

[Jonathan Jackson]

On Thu, May 6, 2010 at 4:08 PM, bayo opadeyi <bayok...@googlemail.com> wrote:

Sorry, I am rather new. Where can I get information on things like
commit rules, core devs, etc...

http://wiki.github.com/rapidsms/rapidsms/steering-committee 

Is a good starting point.  If you have any further questions that you can't find on the wiki.  please add them and we'll make sure to answer. 

[Nic Pottier]

Thanks for starting this discussion Matt, I do feel it is important.

Before getting into things I feel like maybe a little background on me
would be helpful. I come to RapidSMS from the viewpoint of somebody
starting a software company in Kigali to build SMS based systems, I
hope to be doing a lot with RapidSMS in the next few years, and I hope
to be doing it with local Rwandan developers. I am not a Python wiz,
but there are plenty who would call me a software wiz. I say this
only so I don't get my bozo bit flipped when you see the naive Python
I write, I'm just new to it. Most of my time over the next few years
will likely be spent working on SMS based systems, and I hope that
that work can be on RapidSMS.

So with that said, I'll address Matt's question in two parts,
technical aspects of RapidSMS I would like to see changed, and
community aspects which I think could use improvement.

First the technical stuff, I am going to keep these short, as I think
either I or others have voiced this stuff in the past in more detail.

1) Core is far far too big, it should be tiny, and it should be 100%
django based. Backends should plug in via a simple HTTP API. Core
should provide the framework for taking an incoming message and
routing it to handlers reliably. That's it, period. Everything else
is add ons that can evolve and grow on their own timeline.
2) We should use Kannel as a backend by default, maintaining our own
GSM layer is insanity. I think that fact has been driven home in the
past few weeks as people struggle to get things working with an SMSC
or various GSM modem configurations.
3) We need far more decoupling. Message routing, parsing, and
handling should all be configurable and interchangeable, at each
individual level. This encourages reuse, and encourages new
approaches without affecting core. Django is a great model here.
4) We should do the hard stuff. We fail at providing the hardest
part, good and consistent parsing of the garbled messages which
invariably come in.
5) Treat things as life or death, because they might be. We fail far
too easily and don't audit the state of incoming and outgoing
messages. The core of RapidSMS should take every measure possible to
make sure messages aren't dropped on the ground from errors, whether
that be the backend failing or a server restart or whatnot. Let's
take this seriously, we aren't building casual systems here.
Persistence of incoming and outgoing messages (and their state) should
be a core competency.
6) Of course it is easy to pile on, but far more testing, far more documentation

There's more, but those are the main points.

On the community side:

1) There is no community. There are a lot of islands of RapidSMS, but
there seems to be no clear pull from the community in a particular
direction. NY has some idea as to where they want to head, but I
don't feel that the rest of the population is feeling included there.
This type of project cannot be built in an ivory tower, nor should it.
2) Release cycles. We should set a release cycle of every six (maybe
even three?) months, consistently.. aka, the Ubuntu model. At the
start of every cycle, a set of features or improvements should be
picked (new plugins to core), and a set of owners for those features
should be lined up. Those people do not need to be the same every
time, but they need to be dedicated to nailing that release. One
person should be appointed as the PM for that cycle and track and
drive the progress. Shit doesn't get done without deadlines, we know
this, let's make some deadlines and get on with it.
3) Commit rights are far too restrictive. This is probably partly due
to not having a release cycle and stable versions we can point people
to use. Let's fix that, and let's open things up. A smaller core
should help a ton there as well. But Malthe was right two months ago,
let the coders drive this and let's fight this out, the best ideas
will prevail. Let's default to granting commit rights and duke it
out, I'd rather see that than the current stagnation.

There is more of course, but that is a start. I have been busy lately
getting things off the ground, but I am looking forward to
contributing to RapidSMS in a more concrete manner in the coming
months, and to do so as a community.

-Nic

[Jonathan Jackson]

Hi Nic, 

Thanks for the thoughts.  I want to push on one area:

3) Commit rights are far too restrictive.  This is probably partly due
to not having a release cycle and stable versions we can point people
to use.  Let's fix that, and let's open things up.  A smaller core
should help a ton there as well.  But Malthe was right two months ago,
let the coders drive this and let's fight this out, the best ideas
will prevail.  Let's default to granting commit rights and duke it
out, I'd rather see that than the current stagnation.

Are you saying that you think you or others would be willing commit directly if that were the rule, but not willing to push a patch and have it reviewed?  My hypothesis is that the stagnation has been due to the lack of people who have gotten patches into a place where they could be committed, whether through a reviewer or directly.  

-J

[Nic Pottier]

>> 3) Commit rights are far too restrictive.  This is probably partly due
>> to not having a release cycle and stable versions we can point people
>> to use.  Let's fix that, and let's open things up.  A smaller core
>> should help a ton there as well.  But Malthe was right two months ago,
>> let the coders drive this and let's fight this out, the best ideas
>> will prevail.  Let's default to granting commit rights and duke it
>> out, I'd rather see that than the current stagnation.
>
> Are you saying that you think you or others would be willing commit directly
> if that were the rule, but not willing to push a patch and have it reviewed?
>  My hypothesis is that the stagnation has been due to the lack of people who
> have gotten patches into a place where they could be committed, whether
> through a reviewer or directly.

With limited snark:
<http://groups.google.com/group/rapidsms/browse_frm/thread/3b4c3e256beaf483#>

Still isn't in trunk despite everybody agreeing that it was wrong and
it is zero risk.

The problem with a gated commit process is that it makes people who
aren't actively developing gate keepers. The code should be owned by
the coders actively involved. If someone expresses a real interest to
commit, then we give it to them and watch them closely at first,
discuss tweaks etc.. But gating them slows things down a lot and
probably wouldn't be required at all if we had a much better unit
testing suite. Of course it requires people to work together and
communicate, but I think only positives come from that.

-Nic

[Jonathan Jackson]

Touche.  I think most people on the core committer list are very actively developing.  On this commit though, it does seem like this should have been pushed in, and almost immediately.  So that does speak to your point.  And if its the case that the core committers just don't have time to review, I would certainly back your position.   

-J

[Matt Berg]

Nic,

Thanks for your comments on this.  I think we've developed a good community in terms of implementation standpoint but I certainly don't feel like we've really struggled to work in concert to move towards an improved product with any kind of certainty.

I think if we don't react to these concerns (or even take the time to respond to talk about it) we are going to lose new members (for good reason) to the community like Nic who are committed to make our product better.  As a brand, RapidSMS has gained a lot of prominence and is being looked at now by governments and large organizations as a way of delivering critical public services.  While you could argue our current state gets the job done, it would be shame if we don't take ensuring it is as reliable as possible seriously.  I'm not saying individuals on this list aren't.  I just think with a little concentrated effort could really be galvanizing to our community.  Before we can do that though, we need to finish this conversation and agree upon a way forward.

Thanks,

Matt

--
ICT Director
Millennium Villages Project
Columbia University, NYC
M: +1.646.463.1273
Skype: mlberg

[Matt Berg]

I'm posting this on behalf of Kevin Samuel who is traveling.

Please read. I think he does a masterful job explaining what's wrong with RapidSMS right now and provides some suggestions (very similar to Nic's and other's thinking) on what we could to improve things.

[Jonathan Jackson]

Hi All,

Thanks for the thoughts from Kevin.  My vote for the way forward would be to find out who is willing to spend time helping to document and test Adam's new fork based on what Erica sent out.  As I said before, Adam's fork takes care of a lot of the structural complaints people have.  I don't think it addresses the documentation or testing complaints as much, but the only solution to that problem anyways is to have paid or volunteer community members add it.

Then, we can find out what people don't like about the new fork, agree on changes, create a roadmap, and find out who is willing to take on the development tasks.

Also, Erica's link seems like as good a place as any to track line items until we want to move them over to github:

http://etherpad.unicefinnovation.org/rapidsms-refactoring-tasks

Cheers,
Jonathan

[evan wheeler]

Big thanks to Matt for getting the ball rolling on this.

Nic, I think your comments are spot-on for the most part so i'll just add some commentary in-line :)

On Thu, May 6, 2010 at 5:34 PM, Nic Pottier <nicpo...@gmail.com> wrote:

1) Core is far far too big, it should be tiny, and it should be 100%
django based.  Backends should plug in via a simple HTTP API.  Core
should provide the framework for taking an incoming message and
routing it to handlers reliably.  That's it, period.  Everything else
is add ons that can evolve and grow on their own timeline.

Yes! This is one of the major issues we're tackling in the development repositories. The original architecture envisioned a tiny core with a variety of supporting functionality. Unfortunately the core was too tiny, and evolved into its present state. So, in the development repositories the core is going to be much much smaller than it is currently, but more featureful than the originally envisioned core.

Our current goal is a django-based core (its a legit django project) with a rich api for interacting with apps and backends as well as a unified web interface. The core will accept incoming messages and route them appropriately. The core will include models for keeping track of the various backends so that app authors need not concern themselves with which backends are running, for example. The core will also include basic representation of the concept of people -- designed to be extensible via an api. We've examined all of the ways that users have had to hack and customize rapidsms for various deployments and tried to create proper, formalized ways of doing so that dont result in a proliferation of forks and bloat. Everything else, including the extensions to the core models, will be add ons or libraries that live elsewhere and grow at their own pace.

 

2) We should use Kannel as a backend by default, maintaining our own
GSM layer is insanity.  I think that fact has been driven home in the
past few weeks as people struggle to get things working with an SMSC
or various GSM modem configurations.

I disagree with you on this one. Yes it is insanity, but it is necessary insanity. In my experience, every mobile network is different and many possess odd quirks (their own interpretations of the gsm standard, use of odd character encodings, etc). 

Kannel is first and foremost a WAP gateway (that can also do SMS), and can be very difficult to configure properly -- especially if you only want to send and receive SMS. Kannel is fantastic if the network implements standards well and there are no surprises -- otherwise you are faced with debugging 200000 lines of opaque C code in order to get Kannel to work with a particular character encoding or unexpected network behavior. 

Sending and receiving SMS only requires issuing a few AT commands and a good queuing system. Kannel is massive overkill for doing so, and there arent really any alternatives other than developing our own lightweight, flexible, and transparent solution. Yes, pygsm is rough around the edges, but it has been used often enough in large-scale production that i see a lot of value in devoting more time and resources to make it more robust and stable.

 

3) We need far more decoupling.  Message routing, parsing, and
handling should all be configurable and interchangeable, at each
individual level.  This encourages reuse, and encourages new
approaches without affecting core.  Django is a great model here.

Absolutely. Adam has made a lot of progress on the decoupling front in the dev repositories. Contacts (core representation of people) and Locations have an extension API so that app authors may easily add fields to these models from the comfort of their own apps without having to get their hands dirty. This way many apps can extend core models without affecting the core itself -- and the extensions within an app are discrete so they too can be reused and shared.

Adam has also developed a new mechanism called handlers which demystify and simplify both routing and parsing. Each app may define handlers based on keywords or patterns.

 

4) We should do the hard stuff.  We fail at providing the hardest
part, good and consistent parsing of the garbled messages which
invariably come in.

I agree. The hardest part is very hard and varies widely. Good and consistent parsing often depends on what message content is expected and its order, etc. Even the stringcleaning app has to be customized depending on what data will be collected. 

I think continued work on stringcleaning and improving the handlers will get us a lot farther on this front. I also think it would be valuable to create a body of real world garbled messages and their intended formats that we can test our parsing against.

 

5) Treat things as life or death, because they might be.  We fail far
too easily and don't audit the state of incoming and outgoing
messages.  The core of RapidSMS should take every measure possible to
make sure messages aren't dropped on the ground from errors, whether
that be the backend failing or a server restart or whatnot.  Let's
take this seriously, we aren't building casual systems here.
Persistence of incoming and outgoing messages (and their state) should
be a core competency.

We need a lot more QA and testing. 

 

6) Of course it is easy to pile on, but far more testing, far more documentation

and a lot more documentation!

 

On the community side:

1) There is no community.  There are a lot of islands of RapidSMS, but
there seems to be no clear pull from the community in a particular
direction.  NY has some idea as to where they want to head, but I
don't feel that the rest of the population is feeling included there.
This type of project cannot be built in an ivory tower, nor should it.

We need to build a stronger community. The idea that there is some ivory tower in new york is a wild misconception. Adam's current work is the first time there have been development resources dedicated to improving rapidsms outside of a specific in-country deployment. All of the significant work on rapidsms to date has been the result of real world deployments on extremely compressed timelines -- so the code and the community cohesion reflects the fact that development has indeed been a series of islands. We (myself in NY) has an idea of where i want to head solely based on the trials i personally endured throughout the half dozen deployments i worked on in the past year.

 

2) Release cycles.  We should set a release cycle of every six (maybe
even three?) months, consistently.. aka, the Ubuntu model.  At the
start of every cycle, a set of features or improvements should be
picked (new plugins to core), and a set of owners for those features
should be lined up.  Those people do not need to be the same every
time, but they need to be dedicated to nailing that release.  One
person should be appointed as the PM for that cycle and track and
drive the progress.  Shit doesn't get done without deadlines, we know
this, let's make some deadlines and get on with it.

This is a noble idea. We tried to just this last summer and have a three month release cycle. I was the first release manager and we had a set of features and corresponding owners. Unfortunately, our self-imposed deadlines were passed repeatedly and many features remain outstanding. Many feature owners were diverted to working on project deployments and our initial momentum evaporated. 

However, we are a much larger community today, so perhaps it could work if more than a few people can take part.

 

3) Commit rights are far too restrictive.  This is probably partly due
to not having a release cycle and stable versions we can point people
to use.  Let's fix that, and let's open things up.  A smaller core
should help a ton there as well.  But Malthe was right two months ago,
let the coders drive this and let's fight this out, the best ideas
will prevail.  Let's default to granting commit rights and duke it
out, I'd rather see that than the current stagnation.

Yes, i am a firm believer that a separate, smaller core will help a lot because 'appland' and 'libraryland' repositories will be separate and can be radically inclusive. also, with core components designed for extensibility and with richer APIs, there will be much less need to hack on internals. 

i think the current stagnation is due to the flawed architecture rather than restrictive governance. the current, flawed architecture necessitates forking the project and customizing everything to suit the needs of a deployment. having programmers duke it out would just leave the core customized for one programmers purposes and then customized for another's. rapidsms code has been stagnant only in the main repository -- there has been a lot of effort in the myriad of forks that are floating around. the refactored architecture in the dev repositories will encourage duking out to be productive for everyone

[Nic Pottier]

Thanks Evan, replies inline:

>> 2) We should use Kannel as a backend by default...

>
> I disagree with you on this one. Yes it is insanity, but it is necessary
> insanity. In my experience, every mobile network is different and many
> possess odd quirks (their own interpretations of the gsm standard, use of
> odd character encodings, etc).

I'm willing to concede that I haven't seen enough installs to
understand where Kannel fails. But the two instances where I did use
PyGSM it either didn't have the capabilities needed or didn't work in
production. I'm fine moving forward with PyGSM but I'm hesitant to
call it 'complete' at this point, except for some very particular
installs. We should have the Kannel backend be officially supported
regardless, since using a real SMSC should always be the recommended
path if at all possible.

>> 3) We need far more decoupling.
>

> Adam has also developed a new mechanism called handlers which demystify and
> simplify both routing and parsing. Each app may define handlers based on
> keywords or patterns.

Is this different than what is living in Adam's
'rapidsms-contrib-apps-dev' branch? Because if not, that isn't
decoupled.

Decoupling implies that routing is done in one place, parsing in
another, and handling in yet another.

I should be able to look at one file and see all the messages a system
supports. From there I should be able to see which parsers are used
for those messages, and where those parsed messages are sent.

As a use case I brought up long ago, suppose I want to use the
reporters app, but want a completely different set of keywords and
formats. This should be doable without writing any new Python code
and will encourage more reuse (instead of forking) going forward.

In that branch at least, that still seems to be tightly coupled:
<http://github.com/rapidsms/rapidsms-contrib-apps-dev/blob/master/reporters/handlers/register.py>

>> 4) We should do the hard stuff.  We fail at providing the hardest
>> part, good and consistent parsing of the garbled messages which
>> invariably come in.
>
> I agree. The hardest part is very hard and varies widely. Good and
> consistent parsing often depends on what message content is expected and its
> order, etc. Even the stringcleaning app has to be customized depending on
> what data will be collected.
> I think continued work on stringcleaning and improving the handlers will get
> us a lot farther on this front. I also think it would be valuable to create
> a body of real world garbled messages and their intended formats that we can
> test our parsing against.

I don't think stringcleaning goes far enough, especially because it
doesn't really help in the job of defining the format and parsing it
out, rather only taking care of common typos. We should be defining
arguments as logical units, and take care of the magic that is
deriving the user's intent within those.

>> Set release cycles:

> This is a noble idea. We tried to just this last summer and have a three
> month release cycle. I was the first release manager and we had a set of
> features and corresponding owners. Unfortunately, our self-imposed deadlines
> were passed repeatedly and many features remain outstanding. Many feature
> owners were diverted to working on project deployments and our initial
> momentum evaporated.
> However, we are a much larger community today, so perhaps it could work if
> more than a few people can take part.

I think this is certainly a big challenge, especially for the release
manager, but it is a challenge we have to take on, otherwise we won't
move forward. People either need to be honest about how much time
they can devote, or be gently nudged to bow out when they aren't
delivering. But yes, that is a tough problem, I understand that.
What is the release schedule for Adam's refactor?

I think my big concern is that my biggest pet peeves, decoupling and
parsing don't seem to be on the roadmap. Decoupling is a pretty big
architectural change, and one I think is a fundamental necessity, so
that's why I'm a bit concerned that there isn't more discussion going
on along those lines.

-Nic

[joshua masiko]

my 2 cents. As one of those 'local devs'  whose life the rewrite is supposed to make easier I don't think the situation
is as dire as it is made to appear. I had to go the 'google is your best teacher' way and pick up python, django and rapidsms
from scratch and it turned out to be a lot easier than some of the php frameworks i'd tried in the past. I'd say the fragmentation
was the biggest problem for me. It took me a while to find some of the things I wanted e.g the kannel backend. Like most opensource projects though the code is your documentation. 
All in all the proposed changes should make things a lot easier, but I think it's still easier to pick up than some of the other stuff that's out there.

[renaud gaudin]

On Mon, May 10, 2010 at 5:06 PM, Nic Pottier <nicpo...@gmail.com> wrote:
> I'm willing to concede that I haven't seen enough installs to
> understand where Kannel fails.  But the two instances where I did use
> PyGSM it either didn't have the capabilities needed or didn't work in
> production.  I'm fine moving forward with PyGSM but I'm hesitant to
> call it 'complete' at this point, except for some very particular
> installs.

I would like to comment on this because we are slightly abandoning a
major piece of RapidSMS.

Yes, pygsm is not perfect and limited to some modems currently, etc
BUT we have been using it intensively for almost a year now without
any trouble.

Having a direct access to the modem allows other actions like testing
the signal, ussd commands (for topping-up airtime), etc. All those
small yet handy features disappears with Kannel.

On the other hand, kannel is hassle-free for 99% of the modems I think
as long as we properly document its configuration.

That's why I think we should keep an abstraction layer here. Sticking
to one backend will for sure result in dirty things in the future.
It's not difficult to add that layer ; even if interface matches
kannel one.

> We should have the Kannel backend be officially supported
> regardless, since using a real SMSC should always be the recommended
> path if at all possible.

One of the RapidSMS strength used to be its ability to be used in the
field (the real field, not an hotel in kampala).
Recently, several large NGO have deployed large-scale projects which
justified an SMSC hosting thus creating that kannel fame.

But rapidsms is not only for health nor is only for ngo. This
statement is really important to me as it will ensure the
sustainability of rsms as a platform and not as a health tool.
Allowing low-end systems (someone with a usb dongle) to create his own
sms services used to be a core specificity and I want to make sure it
remains.

[joshua masiko]

some of the things I had to work through. don't know if they should be part of the core.

1. generating pdf reports. I looked at the findug branch and they were using the plain reportlab api and having to precisely declare the layout coordinates. I found it easier to use pisa as the reports i was generating werent that complex. You build the report in HTML/XHTML/CSS and it converts that to pdf.

2. blasting/spamming/scheduling of messages to groups of people. Also, having an outgoing message marked as a reply to a particular incoming message.

3. Simpler data entry akin to xforms for phones which support java me, android etc. I hacked my own solution which is a simple form designer which encodes the form as JSON and sens that  to a java me app which decodes the json and displays/stores the form and can send it back in the form expected by the keyword handlers.

4. maps. I used the google maps api but maybe openlayers would be the more generic solution.

[Malthe Borch]

On 10 May 2010 20:53, renaud gaudin <rga...@gmail.com> wrote:
> Having a direct access to the modem allows other actions like testing
> the signal, ussd commands (for topping-up airtime), etc. All those
> small yet handy features disappears with Kannel.

You can talk to your modem outside of RapidSMS. Neither the airtime or
signal strength of your GSM modem is directly important to RapidSMS,
so why do you need it to happen inside of it?

> On the other hand, kannel is hassle-free for 99% of the modems I think
> as long as we properly document its configuration.

It's a complicated piece of software, but fortunately you can
cargo-cult a standard configuration and be reasonably happy.

> That's why I think we should keep an abstraction layer here. Sticking
> to one backend will for sure result in dirty things in the future.
> It's not difficult to add that layer ; even if interface matches
> kannel one.

I don't agree. Having an abstraction layer inside RapidSMS is just an
added complication. I'd like to hear about some scenarios which aren't
better solved in an external gateway.

> One of the RapidSMS strength used to be its ability to be used in the
> field (the real field, not an hotel in kampala).
> Recently, several large NGO have deployed large-scale projects which
> justified an SMSC hosting thus creating that kannel fame.

Kannel can be used in the field; it's not restricted to hotel usage.
On Debian/Ubuntu:

$ apt-get install kannel
$ /etc/init.d/kannel start

> But rapidsms is not only for health nor is only for ngo. This
> statement is really important to me as it will ensure the
> sustainability of rsms as a platform and not as a health tool.
> Allowing low-end systems (someone with a usb dongle) to create his own
> sms services used to be a core specificity and I want to make sure it
> remains.

Anyone with a USB dongle and linux can install Kannel; this is an
insane argument against using an arguably more solid gateway than the
current RapidSMS/PyGSM combo.

It's also an argument that comes up again and again, that it's a lot
easier to use the system if it's all-inclusive. I think it goes both
ways: personally, I find it much easier if there's a clear division.
At least I can go bug the Kannel people when there are issues on its
turf.

That said, an external gateway written in Python would be excellent. I
think that's the way forward for those who want to use own GSM modems.
Just split it out, then reintegrate as a plug-in Django app.

\malthe

[renaud gaudin]

On Mon, May 10, 2010 at 7:14 PM, Malthe Borch <mbo...@gmail.com> wrote:
> You can talk to your modem outside of RapidSMS. Neither the airtime or
> signal strength of your GSM modem is directly important to RapidSMS,
> so why do you need it to happen inside of it?

I agree but it's related so maybe we can also provide (separately)
some tools for such. Being able to topup your account is _very_
important to a lot of setups and while it has nothing to do with the
core of rsms ; rsms as a platform or community should provide an easy
way to do it to ease deployments.

> I don't agree. Having an abstraction layer inside RapidSMS is just an
> added complication. I'd like to hear about some scenarios which aren't
> better solved in an external gateway.

I don't know, I just don't feel comfortable with depending only on one
3rd party (especially one this big) but I must admit you're 100% right
and should such limitation arise we'll adapt then.

btw, dumb question: is kannel available on Windows ? I'm no supporter
of it but windows compatibility seems to be a strong requirement on
current code...

> Kannel can be used in the field; it's not restricted to hotel usage.
> On Debian/Ubuntu:

Quote matters ; this part of the message respond to the
SMSC-everywhere comment ; I never said Kannel can't handle a modem.

[snipping rest of message as it's same topic]

renaud

[Adam Mckaig]

hello #rapidsms,

sorry about not jumping into this discussion sooner. i'm catching up.
here seems as good of a place to start as any.

On Mon, May 10, 2010 at 3:14 PM, Malthe Borch <mbo...@gmail.com> wrote:
> On 10 May 2010 20:53, renaud gaudin <rga...@gmail.com> wrote:
>> Having a direct access to the modem allows other actions like testing
>> the signal, ussd commands (for topping-up airtime), etc. All those
>> small yet handy features disappears with Kannel.
>
> You can talk to your modem outside of RapidSMS. Neither the airtime or
> signal strength of your GSM modem is directly important to RapidSMS,
> so why do you need it to happen inside of it?

i disagree. in situations where a toll-free number isn't available,
we've automated the sending of credit to users via ussd. (i think that
was tim's handiwork.) when the signal has been shaky, we've added a
meter to the UI. it's useful. it's also nice to be able to specify
network settings within the rapidsms config -- the band, network,
smsc, etc.

also, it's actually quite difficult to "talk to your modem" while
kannel (or any other gateway software, including pygsm) is running.
they're just serial devices, so two processes trying to read and write
to/from them at once isn't going to work reliably.

i'm not arguing against kannel. it's great. it's important that we
support it. i'd recommend running it in production over pygsm. but
it's been very useful, on occasion, to be running our own stack all
the way from HTTP to AT. so we should support that, too.

>> On the other hand, kannel is hassle-free for 99% of the modems I think
>> as long as we properly document its configuration.
>
> It's a complicated piece of software, but fortunately you can
> cargo-cult a standard configuration and be reasonably happy.

agreed. there are many example kannel configurations available. this
is a non-issue.

>> That's why I think we should keep an abstraction layer here. Sticking
>> to one backend will for sure result in dirty things in the future.
>> It's not difficult to add that layer ; even if interface matches
>> kannel one.
>
> I don't agree. Having an abstraction layer inside RapidSMS is just an
> added complication. I'd like to hear about some scenarios which aren't
> better solved in an external gateway.

every abstraction adds complexity, but in this case, it didn't even
occur to me that it wasn't worth it. why bake kannel's HTTP interface
into the heart of rapidsms, when it could be abstracted into a
backend? even if kannel was the _only_ type of backend we supported,
i'd still rather wrap it in a simple class. in fact, we did; pyKannel
influenced the original backend interface.

>> One of the RapidSMS strength used to be its ability to be used in the
>> field (the real field, not an hotel in kampala).
>> Recently, several large NGO have deployed large-scale projects which
>> justified an SMSC hosting thus creating that kannel fame.
>
> Kannel can be used in the field; it's not restricted to hotel usage.
> On Debian/Ubuntu:
>
>  $ apt-get install kannel
>  $ /etc/init.d/kannel start

no arguments there. kannel is very easy to install. however, being
written in C, it's not very easy to patch, distribute, deploy, and
maintain a fork. this is why we wrote rubygsm in the first place: a
bug (in kannel) was preventing us from using it in ethiopia, due to
some obscure amharic character set issue which we never fully
diagnosed. it wasn't supposed to be a permanent solution, but it has
been useful enough that we made it one.

>> But rapidsms is not only for health nor is only for ngo. This
>> statement is really important to me as it will ensure the
>> sustainability of rsms as a platform and not as a health tool.
>> Allowing low-end systems (someone with a usb dongle) to create his own
>> sms services used to be a core specificity and I want to make sure it
>> remains.
>
> Anyone with a USB dongle and linux can install Kannel; this is an
> insane argument against using an arguably more solid gateway than the
> current RapidSMS/PyGSM combo.
>
> It's also an argument that comes up again and again, that it's a lot
> easier to use the system if it's all-inclusive. I think it goes both
> ways: personally, I find it much easier if there's a clear division.
> At least I can go bug the Kannel people when there are issues on its
> turf.

agreed. there are benefits to both approaches, so i think we should
support both fully.

> That said, an external gateway written in Python would be excellent. I
> think that's the way forward for those who want to use own GSM modems.
> Just split it out, then reintegrate as a plug-in Django app.

it wouldn't be very difficult to add kannel's HTTP interface to pygsm.
in fact, the slingshotsms guys have already done something very
similar. (with their own HTTP protocol.)

> \malthe

thanks for your input. i'm glad this discussion is heading in a
technical direction.

</adam>

[Adam Mckaig]

On Mon, May 10, 2010 at 4:15 PM, renaud gaudin <rga...@gmail.com> wrote:
> btw, dumb question: is kannel available on Windows ? I'm no supporter
> of it but windows compatibility seems to be a strong requirement on
> current code...

not a dumb question at all. windows compatibility is important, since
(for better or worse) many developers run on it. the googles say that
kannel will run on windows via cygwin, although you have to build it
yourself. i don't know how much of a pain in the ass that is (or
isn't) -- have any of our windows developers done this?

</adam>

[evan wheeler]

Good points, Nic. A little more inline

On Mon, May 10, 2010 at 1:06 PM, Nic Pottier <nicpo...@gmail.com> wrote:

>> 3) We need far more decoupling.
>

> Adam has also developed a new mechanism called handlers which demystify and
> simplify both routing and parsing. Each app may define handlers based on
> keywords or patterns.

Is this different than what is living in Adam's
'rapidsms-contrib-apps-dev' branch?  Because if not, that isn't
decoupled.

Decoupling implies that routing is done in one place, parsing in
another, and handling in yet another.

I should be able to look at one file and see all the messages a system
supports.  From there I should be able to see which parsers are used
for those messages, and where those parsed messages are sent.

As a use case I brought up long ago, suppose I want to use the
reporters app, but want a completely different set of keywords and
formats.  This should be doable without writing any new Python code
and will encourage more reuse (instead of forking) going forward.

In that branch at least, that still seems to be tightly coupled:
<http://github.com/rapidsms/rapidsms-contrib-apps-dev/blob/master/reporters/handlers/register.py>

Ah. I was thinking of decoupling in terms of separating parsing/routing logic from app/business logic -- and handlers are a nice way to keep all the parsing and app routing outside of app.py

Your points are well taken, but i think that if all the routing/parsing for all the messages a system supports are in the same file then its going to be more difficult to share a single portion of functionality with another deployment (that is both the application logic and the accompanying parsing/routing logic)

 

>> 4) We should do the hard stuff.  We fail at providing the hardest
>> part, good and consistent parsing of the garbled messages which
>> invariably come in.
>
> I agree. The hardest part is very hard and varies widely. Good and
> consistent parsing often depends on what message content is expected and its
> order, etc. Even the stringcleaning app has to be customized depending on
> what data will be collected.
> I think continued work on stringcleaning and improving the handlers will get
> us a lot farther on this front. I also think it would be valuable to create
> a body of real world garbled messages and their intended formats that we can
> test our parsing against.

I don't think stringcleaning goes far enough, especially because it
doesn't really help in the job of defining the format and parsing it
out, rather only taking care of common typos.  We should be defining
arguments as logical units, and take care of the magic that is
deriving the user's intent within those.

Yes absolutely. I think handlers provides a structure that could accommodate that kind of specificity in the future. Not saying they're close to adequate today, but that its a decent foundation/starting point

 

>> Set release cycles:

> This is a noble idea. We tried to just this last summer and have a three
> month release cycle. I was the first release manager and we had a set of
> features and corresponding owners. Unfortunately, our self-imposed deadlines
> were passed repeatedly and many features remain outstanding. Many feature
> owners were diverted to working on project deployments and our initial
> momentum evaporated.
> However, we are a much larger community today, so perhaps it could work if
> more than a few people can take part.

I think this is certainly a big challenge, especially for the release
manager, but it is a challenge we have to take on, otherwise we won't
move forward.  People either need to be honest about how much time
they can devote, or be gently nudged to bow out when they aren't
delivering.  But yes, that is a tough problem, I understand that.
What is the release schedule for Adam's refactor?

Hmm. Adam?

 

I think my big concern is that my biggest pet peeves, decoupling and
parsing don't seem to be on the roadmap.  Decoupling is a pretty big
architectural change, and one I think is a fundamental necessity, so
that's why I'm a bit concerned that there isn't more discussion going
on along those lines.

Yeah we don't really have a single roadmap per se, and I don't mean to imply that these issues are not known or not important. Adam's current work will not be the last burst of effort on the rapidsms code, its just to tackle a few of the big outstanding issues.

I believe that many of the other issues raised in this thread will be a lot easier to tackle once we have a strong, small core that we can all work with. I think the current core's bloat and rigidity are blocking the sensible, concerted development of virtually all these other issues.

Regards,

Evan

[Tobias McNulty]

On Mon, May 10, 2010 at 5:37 PM, evan wheeler <evanmw...@gmail.com> wrote:

Our current goal is a django-based core (its a legit django project) with a rich api for interacting with apps and backends as well as a unified web interface. The core will accept incoming messages and route them appropriately. The core will include models for keeping track of the various backends so that app authors need not concern themselves with which backends are running, for example. The core will also include basic representation of the concept of people -- designed to be extensible via an api. We've examined all of the ways that users have had to hack and customize rapidsms for various deployments and tried to create proper, formalized ways of doing so that dont result in a proliferation of forks and bloat. Everything else, including the extensions to the core models, will be add ons or libraries that live elsewhere and grow at their own pace.

Does this mean the method of adding fields to Django models will live externally to RapidSMS, and the core will not depend on it?  If so, that's music to my ears.  While the model extension stuff in the dev projects (rapidsms-core-dev, rapidsms-contrib-apps-dev) is cool and useful, it's certainly not impossible to write Django (or RapidSMS) apps without them.

One of my worries while working with this code has been that RapidSMS will become a repository of cool hacks that making writing Django apps easier.  These currently include the model extensions, the magic that manifests your app at a URL (writing a urls.py is not that hard), the static media aggregator, and the tabs decorator code.  The last of these we had to remove get RapidSMS to run under Apache in a production environment.  IMHO, since these are not specific to RapidSMS, they don't really belong in RapidSMS core--and they make the learning curve steeper for existing Django developers.

I admit this may be a bit purist of me - but I do think that some of the magic is unnecessary and just leads to obfuscation of the way things work.

2) We should use Kannel as a backend by default, maintaining our own
GSM layer is insanity.  I think that fact has been driven home in the
past few weeks as people struggle to get things working with an SMSC
or various GSM modem configurations.

I disagree with you on this one. Yes it is insanity, but it is necessary insanity. In my experience, every mobile network is different and many possess odd quirks (their own interpretations of the gsm standard, use of odd character encodings, etc). 

Kannel is first and foremost a WAP gateway (that can also do SMS), and can be very difficult to configure properly -- especially if you only want to send and receive SMS. Kannel is fantastic if the network implements standards well and there are no surprises -- otherwise you are faced with debugging 200000 lines of opaque C code in order to get Kannel to work with a particular character encoding or unexpected network behavior. 

Sending and receiving SMS only requires issuing a few AT commands and a good queuing system. Kannel is massive overkill for doing so, and there arent really any alternatives other than developing our own lightweight, flexible, and transparent solution. Yes, pygsm is rough around the edges, but it has been used often enough in large-scale production that i see a lot of value in devoting more time and resources to make it more robust and stable.

 
PyGSM may be necessary, but I dislike that it's the de facto installation.  I've spent too much time in the past few weeks trying to figure out why the route process was randomly dying or not recovering if the modem went down.  In reality I don't understand how it's survived multiple large-scale installations given some of the bugs I discovered.

For the truly lean solution, I'd go one step further and say strike the "route" process altogether.  What's wrong with parsing messages as as part of the standard HTTP request/response cycle?  The fact that it runs as a separate process makes it difficult to debug and makes automated testing hard.  How many times have you had to scroll up through your test output to see the exception raised in the route process, when you really wanted it nicely printed next to your test failure?  This way, backends just become simple Django views that do some preliminary processing and pass the messages off to RapidSMS to be handled.

If one needs the flexibility of PyGSM, create a separate process/service (in an independent project) that can read messages and post them over HTTP.  I don't really see a need for it to be part of the core at all.
 

3) We need far more decoupling.  Message routing, parsing, and
handling should all be configurable and interchangeable, at each
individual level.  This encourages reuse, and encourages new
approaches without affecting core.  Django is a great model here.

Absolutely. Adam has made a lot of progress on the decoupling front in the dev repositories. Contacts (core representation of people) and Locations have an extension API so that app authors may easily add fields to these models from the comfort of their own apps without having to get their hands dirty. This way many apps can extend core models without affecting the core itself -- and the extensions within an app are discrete so they too can be reused and shared.

See my comments above about model extensions, etc.

Adam has also developed a new mechanism called handlers which demystify and simplify both routing and parsing. Each app may define handlers based on keywords or patterns.

I like the handlers and I think they offer an excellent pattern for segmenting business logic.  One comment is that they don't currently tell you what keyword was chosen when parsing the messages, which you need to know in some cases.  E.g., I tried to write a QuerySetHandler that extended KeywordHandler and based its keywords on a given model field, but hit a wall when I realized the keyword wasn't being passed to the handle method itself.
 

5) Treat things as life or death, because they might be.  We fail far
too easily and don't audit the state of incoming and outgoing
messages.  The core of RapidSMS should take every measure possible to
make sure messages aren't dropped on the ground from errors, whether
that be the backend failing or a server restart or whatnot.  Let's
take this seriously, we aren't building casual systems here.
Persistence of incoming and outgoing messages (and their state) should
be a core competency.

We need a lot more QA and testing. 

The unit tests are thoroughly broken in the current rapidsms-core-dev.  This makes developing against the core somewhat painful.  I think it goes without saying that writing good tests, and making sure they pass, should be a first priority.

6) Dependencies.  Another concern I have about the development branch of RapidSMS is its growing list of submodules and other dependencies.  E.g., I don't really think django-tables is crucial enough that it merits manifesting itself as rapidsms.tables.

In a nutshell, I think RapidSMS should do what it can using pure Django (and pure HTTP), and leave the rest up to implementors on a case-by-case basis.

Cheers,
Tobias

--
Tobias McNulty
Caktus Consulting Group, LLC
P.O. Box 1454
Carrboro, NC 27510
USA: +1 (919) 951-0052
Zambia: +26 097 400 3763
http://www.caktusgroup.com

[Adam Mckaig]

On Mon, May 10, 2010 at 5:24 PM, evan wheeler <evanmw...@gmail.com> wrote:
> Good points, Nic. A little more inline
>
> On Mon, May 10, 2010 at 1:06 PM, Nic Pottier <nicpo...@gmail.com> wrote:
>>
>> >> 3) We need far more decoupling.
>> >
>> > Adam has also developed a new mechanism called handlers which demystify
>> > and
>> > simplify both routing and parsing. Each app may define handlers based on
>> > keywords or patterns.
>>
>> Is this different than what is living in Adam's
>> 'rapidsms-contrib-apps-dev' branch?  Because if not, that isn't
>> decoupled.
>>
>> Decoupling implies that routing is done in one place, parsing in
>> another, and handling in yet another.
>>
>> I should be able to look at one file and see all the messages a system
>> supports.  From there I should be able to see which parsers are used
>> for those messages, and where those parsed messages are sent.
>>
>> As a use case I brought up long ago, suppose I want to use the
>> reporters app, but want a completely different set of keywords and
>> formats.  This should be doable without writing any new Python code
>> and will encourage more reuse (instead of forking) going forward.
>>
>> In that branch at least, that still seems to be tightly coupled:
>>
>> <http://github.com/rapidsms/rapidsms-contrib-apps-dev/blob/master/reporters/handlers/register.py>

sorry nic, i missed this point earlier. the file that you've quite
rightly called out as garbage is from the "reporters" app, which is
now entirely defunct. (i should have deleted it a long time ago.) it
has been replaced by the "registration" app, which decouples the
Contact model from the webui and sms interface. a more sane, albeit
incomplete, example of a handler:

http://github.com/rapidsms/rapidsms-contrib-apps-dev/blob/master/registration/handlers/language.py

> Ah. I was thinking of decoupling in terms of separating parsing/routing
> logic from app/business logic -- and handlers are a nice way to keep all the
> parsing and app routing outside of app.py
>
> Your points are well taken, but i think that if all the routing/parsing for
> all the messages a system supports are in the same file then its going to be
> more difficult to share a single portion of functionality with another
> deployment (that is both the application logic and the accompanying
> parsing/routing logic)

agreed. this is one of the areas which we (currently) deviate from
django conventions. if we were following them closely, then all sms
patterns should probably be listed in one file. however, this makes it
rather difficult to mix and match apps, which is something that i'm
rather fond of.

>> >> 4) We should do the hard stuff.  We fail at providing the hardest
>> >> part, good and consistent parsing of the garbled messages which
>> >> invariably come in.
>> >
>> > I agree. The hardest part is very hard and varies widely. Good and
>> > consistent parsing often depends on what message content is expected and
>> > its
>> > order, etc. Even the stringcleaning app has to be customized depending
>> > on
>> > what data will be collected.
>> > I think continued work on stringcleaning and improving the handlers will
>> > get
>> > us a lot farther on this front. I also think it would be valuable to
>> > create
>> > a body of real world garbled messages and their intended formats that we
>> > can
>> > test our parsing against.
>>
>> I don't think stringcleaning goes far enough, especially because it
>> doesn't really help in the job of defining the format and parsing it
>> out, rather only taking care of common typos.  We should be defining
>> arguments as logical units, and take care of the magic that is
>> deriving the user's intent within those.
>
> Yes absolutely. I think handlers provides a structure that could accommodate
> that kind of specificity in the future. Not saying they're close to adequate
> today, but that its a decent foundation/starting point

also agreed. we haven't produced a good generic way of doing this yet,
but abstracting the actual parsing is an important step. there's
already a simple example of a handler base class (KeywordHandler):

http://github.com/rapidsms/rapidsms-contrib-apps-dev/blob/master/handlers/handlers/keyword.py#L14

in future, more base classes can be provided, to provide more
automation. i'd like to see a FormHandler, to split and parse
something like matt's ChildCount syntax into a dict, so handlers built
on top don't have to concern themselves with the parsing.

>> >> Set release cycles:
>> > This is a noble idea. We tried to just this last summer and have a three
>> > month release cycle. I was the first release manager and we had a set of
>> > features and corresponding owners. Unfortunately, our self-imposed
>> > deadlines
>> > were passed repeatedly and many features remain outstanding. Many
>> > feature
>> > owners were diverted to working on project deployments and our initial
>> > momentum evaporated.
>> > However, we are a much larger community today, so perhaps it could work
>> > if
>> > more than a few people can take part.
>>
>> I think this is certainly a big challenge, especially for the release
>> manager, but it is a challenge we have to take on, otherwise we won't
>> move forward.  People either need to be honest about how much time
>> they can devote, or be gently nudged to bow out when they aren't
>> delivering.  But yes, that is a tough problem, I understand that.
>> What is the release schedule for Adam's refactor?
>
> Hmm. Adam?

erm, i was supposed to finish it a week or two ago. sorry about that.
i've been sidetracked by some unexpected visa issues.

i'm currently expecting to "finish" my list of tasks before the end of
the month, although it should be noted that there's nothing stopping
anyone from using the -core-dev and -contrib-dev forks right now. we
have a team in zambia right now who are building on it, and evan is
building a new project on it at HQ.

>> I think my big concern is that my biggest pet peeves, decoupling and
>> parsing don't seem to be on the roadmap.  Decoupling is a pretty big
>> architectural change, and one I think is a fundamental necessity, so
>> that's why I'm a bit concerned that there isn't more discussion going
>> on along those lines.
>
> Yeah we don't really have a single roadmap per se, and I don't mean to imply
> that these issues are not known or not important. Adam's current work will
> not be the last burst of effort on the rapidsms code, its just to tackle a
> few of the big outstanding issues.

yeah. i'm not fixing every single developer's pet peeves. just a few
big things which a few of us at HQ agreed were a pain in the ass.

[Nic Pottier]

On Mon, May 10, 2010 at 2:24 PM, evan wheeler <evanmw...@gmail.com> wrote:
> Good points, Nic. A little more inline

> Ah. I was thinking of decoupling in terms of separating parsing/routing
> logic from app/business logic -- and handlers are a nice way to keep all the
> parsing and app routing outside of app.py
> Your points are well taken, but i think that if all the routing/parsing for
> all the messages a system supports are in the same file then its going to be
> more difficult to share a single portion of functionality with another
> deployment (that is both the application logic and the accompanying
> parsing/routing logic)

Both you and Adam pointed to that amount of decoupling as somehow
impeding reuse, when it should be just the opposite.

If you want to reuse a app wholesale, you do the same thing Django
does with apps, you just include that apps urls.py. I would envision
the same thing here, you either include the default set of routers and
parsers or you build your own, mapping them to the handler of your
choice. Mixing and matching is far, far, more flexible when handlers
are dealing with simply a 'duck typed message', routing is done via
config, and creation of those 'duck typed messages' is via an
indepenent mechanism as well. (hopefully a simple definition using our
own provided parsing framework)

> I believe that many of the other issues raised in this thread will be a lot
> easier to tackle once we have a strong, small core that we can all work
> with. I think the current core's bloat and rigidity are blocking the
> sensible, concerted development of virtually all these other issues.

Totally agreed, looking forward to that!

-Nic

[Nic Pottier]

On Mon, May 10, 2010 at 2:29 PM, Tobias McNulty <tob...@caktusgroup.com> wrote:
> For the truly lean solution, I'd go one step further and say strike the
> "route" process altogether.  What's wrong with parsing messages as as part
> of the standard HTTP request/response cycle?  The fact that it runs as a
> separate process makes it difficult to debug and makes automated testing
> hard.  How many times have you had to scroll up through your test output to
> see the exception raised in the route process, when you really wanted it
> nicely printed next to your test failure?  This way, backends just become
> simple Django views that do some preliminary processing and pass the
> messages off to RapidSMS to be handled.

+1

Totally agree, all message parsing, handling and routing should take
place in the Django thread. That makes testing and development via an
'http console' trivial, and let's people debug things the way they are
used to. (at least if they are used to Django) It also provides a
defacto HTTP interface which we can use as the way of integrating new
back ends.

-Nic

[Matt Berg]

Thanks everyone for this dialogue so far.

Not to belabor this too much but Adam or Evan can you address how the new refactoring addresses some of the key issues that have been brought up namely:

• better support for testing
• how is it getting closer to pure Django (and pure HTTP)

The discussion kind of shifted (in a good way) to more about decoupling, handlers and the values of PyGSM.

I realize our thinking on some of these things may have shifted since we prioritized things in November but it would be great to get your thoughts on how the new code will address these issues.

Thanks,

Matt

[Adam Mckaig]

On Tue, May 11, 2010 at 6:13 PM, Matt Berg <mlb...@gmail.com> wrote:
> Thanks everyone for this dialogue so far.
>
> Not to belabor this too much but Adam or Evan can you address how the new
> refactoring addresses some of the key issues that have been brought up
> namely:
>
> better support for testing

testing is already "supported". you're free to write as many unit
tests as you like, for your own app(s). but the rapidsms core (and
contrib) is woefully under-tested, and that needs to be fixed. it's on
the todo list. patches are welcome, too.

> how is it getting closer to pure Django

we've already thrown out a lot of the code which was contrary to the
expectations of a django programmer. we're continuing to do so. we've
prepared a few documents on the few differences. rapidsms is now a
django *app*, and uses the django management commands and testing
framework. it's also available under the same license. so we're
getting there.

one ongoing point of disagreement, i think, is how much junk we bundle
with rapidsms to make developing on django easier. personally, i find
developing on django kind of ghetto, so i like to bundle all sorts of
toys to make it more fun (for me). this is widely regarded as a bad
idea. let's argue some more about that!

> (and pure HTTP)

if you're referring to the notion of throwing out the backends
abstraction, we're not doing that. it's a horrible idea. we've been
there, and tried that, and it wasn't flexible enough.

> The discussion kind of shifted (in a good way) to more about decoupling,
> handlers and the values of PyGSM.

yeah. thanks for starting the discussion.

> I realize our thinking on some of these things may have shifted since we
> prioritized things in November but it would be great to get your thoughts on
> how the new code will address these issues.
>
> Thanks,
>
> Matt

</adam>

[Nic Pottier]

On Thu, May 13, 2010 at 8:26 AM, Adam Mckaig <adam....@gmail.com> wrote:
> we've already thrown out a lot of the code which was contrary to the
> expectations of a django programmer. we're continuing to do so. we've
> prepared a few documents on the few differences. rapidsms is now a
> django *app*, and uses the django management commands and testing
> framework. it's also available under the same license. so we're
> getting there.

Ya it seems like we are heading in the right direction there.

I think there is also a larger philosophical debate as to whether we
do things in a way that would be expected from a Django dev, or in a
similar pattern. This goes farther than whether we just happen to run
as a Django app, but whether our architecture is modeled in a similar
way. I harp on this a lot, but I think this is the single thing that
can make RapidSMS easier to pick up, leveraging PREVIOUS knowledge is
what makes frameworks obvious/non-obvious to new developers, if we are
a Django app we should always try to mimic Django unless there is a
very strong reason not to.

I'll take that as my opportunity to harp once more that I think we
should be doing routing via an equivalent to Django's urls.py, for
various reasons, but one big one being that that would be the 'Django
way' and the most natural to a Django dev.

> one ongoing point of disagreement, i think, is how much junk we bundle
> with rapidsms to make developing on django easier. personally, i find
> developing on django kind of ghetto, so i like to bundle all sorts of
> toys to make it more fun (for me). this is widely regarded as a bad
> idea. let's argue some more about that!

I think optional tags, etc.. are all fine and good, we just shouldn't
have the core depend on that kind of stuff. That said I do kind of
like that the tabbed UI interface is fairly easy and obvious, but it
isn't like that would be a lot of code to right in a pure Django way.

> if you're referring to the notion of throwing out the backends
> abstraction, we're not doing that. it's a horrible idea. we've been
> there, and tried that, and it wasn't flexible enough.

Can you go into this more? I know both you and Evan have a strong
feeling on this, but I'm not understanding why.

To be clear, to my understanding, the idea is not to throw out
pluggable backends, or pygsm, but rather have backends pass off
incoming messages to Django via HTTP and have the routing and message
handling be done in the Django HTTP thread instead of the separate
router app. The benefits being that by living in the Django process
you get to leverage all sorts of Django goodness. (i18n plugins in
the previous thread being an example, Django standard 'debugging'
being another, lots of others too)

-Nic

[evan wheeler]

On Mon, May 10, 2010 at 5:29 PM, Tobias McNulty <tob...@caktusgroup.com> wrote:

On Mon, May 10, 2010 at 5:37 PM, evan wheeler <evanmw...@gmail.com> wrote:

Our current goal is a django-based core (its a legit django project) with a rich api for interacting with apps and backends as well as a unified web interface. The core will accept incoming messages and route them appropriately. The core will include models for keeping track of the various backends so that app authors need not concern themselves with which backends are running, for example. The core will also include basic representation of the concept of people -- designed to be extensible via an api. We've examined all of the ways that users have had to hack and customize rapidsms for various deployments and tried to create proper, formalized ways of doing so that dont result in a proliferation of forks and bloat. Everything else, including the extensions to the core models, will be add ons or libraries that live elsewhere and grow at their own pace.

Does this mean the method of adding fields to Django models will live externally to RapidSMS, and the core will not depend on it?  If so, that's music to my ears.  While the model extension stuff in the dev projects (rapidsms-core-dev, rapidsms-contrib-apps-dev) is cool and useful, it's certainly not impossible to write Django (or RapidSMS) apps without them.

One of my worries while working with this code has been that RapidSMS will become a repository of cool hacks that making writing Django apps easier.  These currently include the model extensions, the magic that manifests your app at a URL (writing a urls.py is not that hard), the static media aggregator, and the tabs decorator code.  The last of these we had to remove get RapidSMS to run under Apache in a production environment.  IMHO, since these are not specific to RapidSMS, they don't really belong in RapidSMS core--and they make the learning curve steeper for existing Django developers.

I admit this may be a bit purist of me - but I do think that some of the magic is unnecessary and just leads to obfuscation of the way things work.

Most of these hacks came from the desire to have a unified web interface for any combination of apps used in a rapidsms deployment -- add a new app and it appears as a tab among your usual apps -- and you dont have to edit several global files to do so (urls, etc). It was also an attempt to keep all of each app's sms, db, and web code in a discrete directory to ease sharing and encourage apps that are not necessarily deployment specific.

None of us were Django experts, so many things were contrary to the django way. As adam said, we're trying to move closer to django conventions so please please please share your ideas for doing so since you have much more experience with django

 

2) We should use Kannel as a backend by default, maintaining our own
GSM layer is insanity.  I think that fact has been driven home in the
past few weeks as people struggle to get things working with an SMSC
or various GSM modem configurations.

I disagree with you on this one. Yes it is insanity, but it is necessary insanity. In my experience, every mobile network is different and many possess odd quirks (their own interpretations of the gsm standard, use of odd character encodings, etc). 

Kannel is first and foremost a WAP gateway (that can also do SMS), and can be very difficult to configure properly -- especially if you only want to send and receive SMS. Kannel is fantastic if the network implements standards well and there are no surprises -- otherwise you are faced with debugging 200000 lines of opaque C code in order to get Kannel to work with a particular character encoding or unexpected network behavior. 

Sending and receiving SMS only requires issuing a few AT commands and a good queuing system. Kannel is massive overkill for doing so, and there arent really any alternatives other than developing our own lightweight, flexible, and transparent solution. Yes, pygsm is rough around the edges, but it has been used often enough in large-scale production that i see a lot of value in devoting more time and resources to make it more robust and stable.

 
PyGSM may be necessary, but I dislike that it's the de facto installation.  I've spent too much time in the past few weeks trying to figure out why the route process was randomly dying or not recovering if the modem went down.  In reality I don't understand how it's survived multiple large-scale installations given some of the bugs I discovered.

For the truly lean solution, I'd go one step further and say strike the "route" process altogether.  What's wrong with parsing messages as as part of the standard HTTP request/response cycle?  The fact that it runs as a separate process makes it difficult to debug and makes automated testing hard.  How many times have you had to scroll up through your test output to see the exception raised in the route process, when you really wanted it nicely printed next to your test failure?  This way, backends just become simple Django views that do some preliminary processing and pass the messages off to RapidSMS to be handled.

If one needs the flexibility of PyGSM, create a separate process/service (in an independent project) that can read messages and post them over HTTP.  I don't really see a need for it to be part of the core at all.

PyGSM isnt part of the core, its a separate project in its own repository. Only the gsm backend is in the core, alongside the http backend, irc backend, email backend, etc.

The second architecture of rapidsms that we wrote in ethiopia had a single process for both the message routing and web stuff. Whenever the message routing code blew up (if the sim ran out of credit or someone placed a phone call to the modem or an exception wasnt handled, etc), the website was inaccessible. A problem with the message routing shouldnt get in the way of viewing data or accessing the online dashboard. The same thing could happen in reverse: a DDOS attack on the website could prevent messages from being handled or sent. Though its probably possible to have both message routing and web stuff safely in the same process, we decided that separate processes would be safer and more flexible -- you can, for example, run your message routing process on an in-country server, and run the web stuff on amazon.

  

Adam has also developed a new mechanism called handlers which demystify and simplify both routing and parsing. Each app may define handlers based on keywords or patterns.

I like the handlers and I think they offer an excellent pattern for segmenting business logic.  One comment is that they don't currently tell you what keyword was chosen when parsing the messages, which you need to know in some cases.  E.g., I tried to write a QuerySetHandler that extended KeywordHandler and based its keywords on a given model field, but hit a wall when I realized the keyword wasn't being passed to the handle method itself.

Great catch. Lets do it

 

6) Dependencies.  Another concern I have about the development branch of RapidSMS is its growing list of submodules and other dependencies.  E.g., I don't really think django-tables is crucial enough that it merits manifesting itself as rapidsms.tables.

In a nutshell, I think RapidSMS should do what it can using pure Django (and pure HTTP), and leave the rest up to implementors on a case-by-case basis.

This is definitely a valid concern. However, I prefer submodules (because they are optional and live in separate repositories) to the way django just includes a lot of random stuff in the official repo/distribution (geodjango, lorem ipsum generator, etc).

[Nic Pottier]

On Thu, May 13, 2010 at 8:57 AM, evan wheeler <evanmw...@gmail.com> wrote:
> The second architecture of rapidsms that we wrote in ethiopia had a single
> process for both the message routing and web stuff. Whenever the message
> routing code blew up (if the sim ran out of credit or someone placed a phone
> call to the modem or an exception wasnt handled, etc), the website was
> inaccessible. A problem with the message routing shouldnt get in the way of
> viewing data or accessing the online dashboard. The same thing could happen
> in reverse: a DDOS attack on the website could prevent messages from being
> handled or sent. Though its probably possible to have both message routing
> and web stuff safely in the same process, we decided that separate processes
> would be safer and more flexible -- you can, for example, run your message
> routing process on an in-country server, and run the web stuff on amazon.

Thanks for getting into this Evan, good to hear the thinking behind
that and good points. Some comments on those concerns.

1) how was the main serving thread somehow killed by an error in the
sms handler endpoint? Hard to imagine how that would happen, this is
basically saying that you could break an entire Django site by having
one URL that has an error, which is clearly not the case. I obviously
agree that the website should never go down because of a handler
error, I'm just at a loss as to how that took place, can you
elaborate?

2) The DDOS argument is an interesting one, but appealing to that is a
cheap trick. :) We are so totally screwed in so many ways if someone
actually wanted to be malicious that it really shouldn't be guiding us
at this point.

3) I actually really disagree that decoupling handling and web views
into separate processes is a good thing. Yes, being able to have the
actual GSM modem in one place and the server in another is a good
thing, backends themselves should be separate. But if the handling
and web views are separate processes then you need to keep them in
sync. Having them be different places allows for crazy cases like
your models being different in the two instances, requires you to have
the DB accessible from both places, can cause really nuts problems
when code is out of sync etc..

-Nic

[Adam Mckaig]

On Thu, May 13, 2010 at 12:12 PM, Nic Pottier <nicpo...@gmail.com> wrote:
> On Thu, May 13, 2010 at 8:57 AM, evan wheeler <evanmw...@gmail.com> wrote:
>> The second architecture of rapidsms that we wrote in ethiopia had a single
>> process for both the message routing and web stuff. Whenever the message
>> routing code blew up (if the sim ran out of credit or someone placed a phone
>> call to the modem or an exception wasnt handled, etc), the website was
>> inaccessible. A problem with the message routing shouldnt get in the way of
>> viewing data or accessing the online dashboard. The same thing could happen
>> in reverse: a DDOS attack on the website could prevent messages from being
>> handled or sent. Though its probably possible to have both message routing
>> and web stuff safely in the same process, we decided that separate processes
>> would be safer and more flexible -- you can, for example, run your message
>> routing process on an in-country server, and run the web stuff on amazon.
>
> Thanks for getting into this Evan, good to hear the thinking behind
> that and good points.  Some comments on those concerns.
>
> 1) how was the main serving thread somehow killed by an error in the
> sms handler endpoint?  Hard to imagine how that would happen, this is
> basically saying that you could break an entire Django site by having
> one URL that has an error, which is clearly not the case.  I obviously
> agree that the website should never go down because of a handler
> error, I'm just at a loss as to how that took place, can you
> elaborate?

if i'm thinking of the correct incident (and i may not be, because
there were quite a few which lead to the decision to split the
processes), it was actually a bug with rubygsm. the power was shaky,
and the modem wasn't on a ups, so occasionally it would reset,
forgetting the init string which was set when ye olde rapidsms started
up. this lead to unexpected +WIND notifications being read from the
modem, which weren't handled at all gracefully, which ended up somehow
blowing the recursion limit.

the obvious response would be "duh, just use kannel", but we couldn't
at the time, due to a known bug receiving messages containing
characters in the amharic character set. this actually caused kannel
to peg the cpu indefinitely, which was even worse than our hack of a
solution.

i'm not fundamentally _opposed_ to merging the two processes, but i
don't see a compelling reason to do so.

and i shudder at the thought of a big list of regexen (like url.py)
containing every sms route that we've ever defined. we deliberately
abstract that stuff, because it's so gross.

> 2) The DDOS argument is an interesting one, but appealing to that is a
> cheap trick. :)  We are so totally screwed in so many ways if someone
> actually wanted to be malicious that it really shouldn't be guiding us
> at this point.

cheap indeed. the jury will disregard mr wheeler's testimony.

> 3) I actually really disagree that decoupling handling and web views
> into separate processes is a good thing.  Yes, being able to have the
> actual GSM modem in one place and the server in another is a good
> thing, backends themselves should be separate.  But if the handling
> and web views are separate processes then you need to keep them in
> sync.  Having them be different places allows for crazy cases like
> your models being different in the two instances, requires you to have
> the DB accessible from both places, can cause really nuts problems
> when code is out of sync etc..

we've split the hosting (putting the web interface on ec2 and sms
interface in-country) a couple of times now. sure, you have to sync
your databases, but when you need it (like, when you can't access your
server from the public internet due to the immutable firewall policies
of my employer), you really do need it.

on the backends, actually, i agree. backends should probably be in
separate processes, and hand their messages to the router via HTTP or
DBUS or something. maybe the apps, too. i did some work on that idea a
while back, and it was pretty sweet. i'll see if i can dig it up.

> -Nic

</adam>

[Tobias McNulty]

On Thu, May 13, 2010 at 7:13 PM, Adam Mckaig <adam....@gmail.com> wrote:

i'm not fundamentally _opposed_ to merging the two processes, but i
don't see a compelling reason to do so.

I don't think anyone's suggesting that we merge the two processes, but rather, that we decouple them even further.  In my ideal world, the code that reads and writes to the modem should live in a process, by itself.  One thread to read and write to the modem, and another to handle HTTP requests to send messages (BaseHTTPServer?).  When the modem thread reads a message, instead of processing the message directly, in the same thread, it POSTs to a Django URL specific to that backend.  Each backend is effectively a URL/view that does its own processing and then passes the message off to RapidSMS core (still in the same thread, of course) to be handled.  For example, I'd even go as far to say that the PyGSM-based SMS gateway should be its own independent project, effectively taking the current gsm backend and splitting it in two.  Think of it as a mini, PyGSM-based Kannel.

This means that each backend's gateway would live in its own independent process, ensuring that one backend can't bring down another.  And, for those who want to use it, it would fit very nicely with Kannel.  It alleviates the need for a lot of the threading logic in the route process, and lets Apache (or whatever) do the hard work.  Letting apache do all the message processing is also good because Django does not claim to be thread safe.

> 3) I actually really disagree that decoupling handling and web views
> into separate processes is a good thing.  Yes, being able to have the
> actual GSM modem in one place and the server in another is a good
> thing, backends themselves should be separate.  But if the handling
> and web views are separate processes then you need to keep them in
> sync.  Having them be different places allows for crazy cases like
> your models being different in the two instances, requires you to have
> the DB accessible from both places, can cause really nuts problems
> when code is out of sync etc..

Right now things can get out of sync during development because the route process doesn't auto-reload, while runserver does.  If everything went through runserver (or Apache in production), this would actually help!  If /all/ requests, HTTP and SMS, are handled in the same way, the whole system becomes a lot simpler.

on the backends, actually, i agree. backends should probably be in
separate processes, and hand their messages to the router via HTTP or
DBUS or something. maybe the apps, too. i did some work on that idea a
while back, and it was pretty sweet. i'll see if i can dig it up.

Backends - yes!  (Maybe all that explanation above was entirely unnecessary).  Apps - naw.  Apache can spawn additional processes as needed.

Thanks Adam & Evan for discussing, and I look forward to your hearing thoughts!

Tobias

--
Tobias McNulty
Caktus Consulting Group, LLC
P.O. Box 1454
Carrboro, NC 27510
USA: +1 (919) 951-0052
Zambia: +26 097 400 3763
http://www.caktusgroup.com

[Tobias McNulty]

On Thu, May 13, 2010 at 7:13 PM, Adam Mckaig <adam....@gmail.com> wrote:

and i shudder at the thought of a big list of regexen (like url.py)
containing every sms route that we've ever defined. we deliberately
abstract that stuff, because it's so gross.

One last point, just to be clear; I'm not suggesting anything like parsing messages in urls.py.  In fact, I see no need to change the way messages are handled at all, rather, just to change the manner (and the thread) in which they get there.

Cheers,
Tobias

--
Tobias McNulty
Caktus Consulting Group, LLC
P.O. Box 1454
Carrboro, NC 27510
USA: +1 (919) 951-0052
Zambia: +26 097 400 3763
http://www.caktusgroup.com

[Nic Pottier]

On Thu, May 13, 2010 at 1:40 PM, Tobias McNulty <tob...@caktusgroup.com> wrote:
> I don't think anyone's suggesting that we merge the two processes, but
> rather, that we decouple them even further.  In my ideal world, the code
> that reads and writes to the modem should live in a process, by itself.  One
> thread to read and write to the modem, and another to handle HTTP requests
> to send messages (BaseHTTPServer?).  When the modem thread reads a message,
> instead of processing the message directly, in the same thread, it POSTs to
> a Django URL specific to that backend.  Each backend is effectively a
> URL/view that does its own processing and then passes the message off to
> RapidSMS core (still in the same thread, of course) to be handled.  For
> example, I'd even go as far to say that the PyGSM-based SMS gateway should
> be its own independent project, effectively taking the current gsm backend
> and splitting it in two.  Think of it as a mini, PyGSM-based Kannel.

Yes, exactly, thanks for explaining that in a better way Tobias.

To your point that each backend has its own URL/view which does its
own processing, on that I would hope that almost all end points would
do almost no processing of their own there, really just parsing the
HTTP parameters to get the txt attributes (sender, time, body) and
then passing it off to the real RapidSMS handler.

-Nic

[Adam Mckaig]

On Thu, May 13, 2010 at 8:38 PM, Nic Pottier <nicpo...@gmail.com> wrote:
> On Thu, May 13, 2010 at 1:40 PM, Tobias McNulty <tob...@caktusgroup.com> wrote:
>> I don't think anyone's suggesting that we merge the two processes, but
>> rather, that we decouple them even further.  In my ideal world, the code
>> that reads and writes to the modem should live in a process, by itself.  One
>> thread to read and write to the modem, and another to handle HTTP requests
>> to send messages (BaseHTTPServer?).  When the modem thread reads a message,
>> instead of processing the message directly, in the same thread, it POSTs to
>> a Django URL specific to that backend.  Each backend is effectively a
>> URL/view that does its own processing and then passes the message off to
>> RapidSMS core (still in the same thread, of course) to be handled.  For
>> example, I'd even go as far to say that the PyGSM-based SMS gateway should
>> be its own independent project, effectively taking the current gsm backend
>> and splitting it in two.  Think of it as a mini, PyGSM-based Kannel.
>
> Yes, exactly, thanks for explaining that in a better way Tobias.
>
> To your point that each backend has its own URL/view which does its
> own processing, on that I  would hope that almost all end points would
> do almost no processing of their own there, really just parsing the
> HTTP parameters to get the txt attributes (sender, time, body) and
> then passing it off to the real RapidSMS handler.

ah. thanks. good explanation, both of you. this makes a lot more sense
to me, now. that sounds quite reasonable. i'm not going to leap out of
my chair to implement it, since i'm still skeptical that it would be
flexible enough, but i'd be very interested to see an example.

all being well, it wouldn't be terribly difficult to port the existing
'handlers' app to this, so all of the current work going into those
would be easy to port.

i don't see the router+apps model ever totally going away, because
some complex interactive apps (like training, injector, etc) are
tightly wrapped around it -- but it might be worth making these 2% of
apps more difficult to implement, to make the other 98% simpler.
perhaps we could introduce some sort of sms middleware system, for
those complex cases.

</adam>

[Malthe Borch]

On 13 May 2010 18:26, Adam Mckaig <adam....@gmail.com> wrote:
> testing is already "supported". you're free to write as many unit
> tests as you like, for your own app(s). but the rapidsms core (and
> contrib) is woefully under-tested, and that needs to be fixed. it's on
> the todo list. patches are welcome, too.

It doesn't need to be fixed, it needs to be entirely integrated into
the development process and this view has indeed been voiced many
times on this list already.

Why is this so? Let's take an example.

This is how RapidSMS handles outgoing messages (the description is
taken carbon-copy from comments in the code):

# call outgoing phases in the opposite order of the incoming
# phases, so the first app called with an incoming message
# is the last app called with an outgoing message

This behavior isn't tested anywhere. If you're in doubt, double-check here:

- http://github.com/rapidsms/rapidsms-core-dev/blob/master/lib/rapidsms/tests/test_router.py

However, it's also not obvious that it's a good thing. Django signals
for instance, are not called in any particular order. This actually
makes it easier to write, test and understand signal handlers.

But maybe for RapidSMS it's a good thing. If we had tests and
documentation, it would be clear whether this was the case. I would
imagine that there'd be a good example then of why you *need* this
behavior. And if you don't need it, throw it out.

Without knowing of these examples, I would imagine that it would work
well enough if there was just a single backend for an outgoing
message, but that a signal was fired before and after it was sent.

Tests, documentation and development good together, hand in hand. If
you can't test or document something, chances are that it shouldn't be
in there in the first place.

I took a day or two out this week to work on documentation for the
system I've been putting together:

- http://uganda.rapidsms.org/docs/

The system really has no name at this point, but tentatively I'm
calling it "UgandaSMS".

Let's get the ball rolling on whether RapidSMS is realistically going
to move in this direction in a reasonable timeframe or if it's time to
change gears. I don't really care much as the long as the system of
choice meets the basic requirements of being tested and documented.
But if you have time, please take a look and see if it's not something
we could use as a foundation.

There's a lot of things missing obviously, but it's a firm first step
in my opinion, for what that's worth. In the end it's all nuts and
bolts and how to put them together.

\malthe

[Nic Pottier]

> The system really has no name at this point, but tentatively I'm
> calling it "UgandaSMS".

I like a lot of the things you've done in this implementation,
specifically I like that routing, parsing and handling are separate.

One thing that I'm not sure about is the requirement of the parsing
stage to return an actual model of the message parameters. If I
understand this correctly, the idea is that then we have a DB log of
the parsed parameters, which I do agree could be a good thing.

But I have some reservations that it is might be too burdensome.

For messages that contain 0..n fields or codes the model format it
turns pretty complicated to serve just as a 'logging layer'. IE, a
common case for health systems is to allow the reporter to send in a
series of action codes. In the final implementation, at the business
layer, that obviously turns into a one to many relationship between
the report and the action codes. Having to have two of those one to
many tables just to support the parsed message model seems like a lot
of duplicated work. I'd rather see the parsing layer pass back a
simple dict of values, or perhaps a base class, but not a DB Model.

That said, I think not separating the parsing layer is the one
fundamental flaw of RapidSMS as it is, so although I recognize
Malthe's work is still new and lacking in features that will be
needed, I support it because it addresses that shortcoming.

I will go into that more in my next message in a new thread.

-Nic

[Malthe Borch]

On 15 May 2010 20:16, Nic Pottier <nicpo...@gmail.com> wrote:
> For messages that contain 0..n fields or codes the model format it
> turns pretty complicated to serve just as a 'logging layer'.  IE, a
> common case for health systems is to allow the reporter to send in a
> series of action codes.  In the final implementation, at the business
> layer, that obviously turns into a one to many relationship between
> the report and the action codes.  Having to have two of those one to
> many tables just to support the parsed message model seems like a lot
> of duplicated work.  I'd rather see the parsing layer pass back a
> simple dict of values, or perhaps a base class, but not a DB Model.

True. That's a good point.

Perhaps the parser should indeed return kwargs, but they wouldn't be
set on the model; instead, they would be passed into the handler,
e.g.:

>>> model, kwargs = message.parse(text)
>>> message = model(text=text)
>>> message.handle(**kwargs)

Then, if you wanted to persist anything, you could do it yourself
either in the handler, or using a signal listener. It wouldn't be a
default behavior.

\malthe

[Matt Berg]

Just wanted to follow up on Nic's last points.

Adam et al - what's the release schedule for what you are working
on?

Does our current and soon to be updated architecture handle decoupling
and parsing in a way we are fundamentally ok with? I know we have a
great discussion on parsing going on now.

I think we need to arrive at a group consensus on where we are going
with everything soon...