Redirection for Mobile Sites

[Ahmed Al-Saadi]

Hello good folks of Zotonistan:

I have started developing a couple of sites for my clients using
Zotonic. One of these sites is to have a mobile version that needs to be
activated when a mobile user agent is detected. Since my home page
points to resource_template.erl, I copied that file to my site's
resource directory and edited the relevant resource functions and
successfully redirected when I used my iPhone (by detecting the
User-Agent header).

However, this is not a good solution and was only meant to explore
Webmachine's facilities. The drawback is that if another site uses a
different resource for the home page, or if I wanted the redirect to
occur on another resource (e.g., on all site pages), I'd have to hack
other resources, too. In other words, this solution is not modular.
Other suboptimal solutions:

1) Point the home page to a special "routing" resource: this also cannot
redirect on all urls
2) Use an external url rewriting (nginx or apache): this will involve
putting part of the app logic in the web server
3) Go below Webmachine and use Mochiweb to handle this: this is yet
another ugly hack

Ideally, I would be able to install a zotonic module and configure it
with the ability to choose which urls are redirected with the option to
overwrite the redirect (e.g., "view full site" option on a mobile
device). The problem is that Webmachine has a very "horizontal" view of
the execution stack (i.e., using resources; as opposed to the
vertical/onion view of underlying web server pipeline).

Anyone has a better (possibly obvious) solution?

Cheers,
-signed(ahmed).

[Marc Worrell]

Good questions.

I am also pondering what the best solution is for mobile.

I was thinking along the line:

• Sniff browser for mobile
• When in doubt send the mobile site with a reload when the screen was found to be bigger
• Have an extra template set (maybe in templates.mobile or templates/mobile/...)

Of course for a really mobile website you need to use jquery mobile, not the full jquery with ui and the kitchen sink.

Any ideas/preferences?

Problems I still see are:
• file upload (we use an iframe where the javascript posts to)
• rich text editing (markdown for the win)

- Marc

[Alain O'Dea]

Hi Marc:

I think this should be solved the same way language is.

If I understand it correctly language is sniffed, but you can override it with cookies.

During dispatch detect mobile user agents and set the mobile property in the Zotonic request Context unless it has been overridden by cookie. When resource_template goes to lookup a template it would check if the mobile property is set and choose templates from templates.mobile/ in lieu of templates/.  If templates.mobile/ is not present it would fall back to templates/.

Does that sound reasonable?

Cheers,

Alain

[Marc Worrell]

It sounds good to me.

And the 'mobile' setting can be stored, just like the language, in a persistent cookie.

Does the extra directory for templates make sense?

- Marc

[Alain O'Dea]

I think the directory structure needs more discussion.  We need a sense of the pros and cons.

With templates/mobile/ do we run the risk of a template binding in dispatch/dispatch like "mobile/home.tpl" being misinterpreted?  Does anyone currently use subdirectories of templates for organization now?  In either case I'd advocate for templates.mobile/ since it is sufficiently obvious and doesn't cause conflicts with existing complex template management strategies.

What are the cons of templates.mobile/?  They definitely exist, but I am failing to come up with any right now.

[Ahmed Al-Saadi]

This process of mobile template selection sounds too "magical" and I advise against it in favor of something more general, for example:
1) parameter/setting-aware dispatch rules (through extending Webmachine dispatch... again!). This is less general than the next one which is a...
2) non-declarative dispatch through a special (pluggable) module

The latter can possibly be called mod_dispatch and can either redirect to another "regular" url or is parameterized to accommodate *any* possible template name "rewrite". It's really like the nginx or apache mod_rewrite, but with the ability to rewrite not necessarily the url, but the template name instead.

I'll look into doing a proof of concept next week.

Ahmed

[Alain O'Dea]

Hi Ahmed:

There is definitely a risk in magical behavior.  Your solution of providing a rewrite module sounds sensible and I look forward to seeing you proof of concept :)

One possible application of your mod_rewrite would be to allow arbitrary property rewriting through Zotonic's observe_* callbacks.  This would be the best of all worlds since a mod_mobile could implement the template.mobile/ solution, but yet still be possible to disable where unnecessary.  Beyond that it is easy to add observe_* callbacks to a site module so you can cleanly manage site-specific rewrites.

I like it :)

Cheers,

Alain

[Arjan Scherpenisse]

What most websites do is implementing a second domain, e.g. m.mysite.com
and direct mobile visitors there if they visit mysite.com.

This m.mysite.com could just be a second Zotonic site but with the same
content (database), right?

So the only part where you actually make a decision based on the user
agent is when you do the redirecting. This could be a one line
javascript on mysite.com.

Anyway, I'm also in favor of Ahmed's approach opposed to hardcoding a
concept of "mobile" in zotonic. (what is "mobile" anyway? it ranges from
everything between a tablet to a watch running Android..)

So I'm curious to any proposals you come up with, Ahmed :-)

Arjan

On 09/14/2011 03:21 AM, Alain O'Dea wrote:
> Hi Ahmed:
>
> There is definitely a risk in magical behavior. Your solution of
> providing a rewrite module sounds sensible and I look forward to seeing
> you proof of concept :)
>

> One possible application of your *mod_rewrite* would be to allow
> arbitrary property rewriting through Zotonic's *observe_** callbacks.
> This would be the best of all worlds since a *mod_mobile* could
> implement the *template.mobile/* solution, but yet still be possible to

> disable where unnecessary. Beyond that it is easy to add

> *observe_** callbacks to a site module so you can cleanly manage

[Alain O'Dea]

I like your suggestion too Arjan.

Having a separate site makes it much cleaner.  I didn't realize that would work.  I thought there would be conflicts in things like pivot maintenance.

[ludovic Demblans]

Hello,

I had the same 'fear' about the pivot and queries conflicts about two
sites querying the same database.

So, what about this ?

I may result in race conditions with two modifications of the same rsc at
the same time, or there is only one queue in zotonic for each connection ?

Thanks !

Le Wed, 14 Sep 2011 15:29:54 +0200, Alain O'Dea <alain...@gmail.com> a
écrit:

--
Utilisant le logiciel de courrier révolutionnaire d'Opera :
http://www.opera.com/mail/

[Marc Worrell]

Hi,

Actually multiple sites on the same db won't work.

Every Zotonic thinks it has a consistent view on the database.

And it caches quite aggressively data in memory.

As sites don't communicate these caches will get out of sync.

As for the pivot queue, that one is maintained in the database.

So no problems there :)

I don't think you want to have two sites on the same db.

It also will mean duplicating models, template snippets, business logic, hell.

And which site receives e-mail? Does pubsub? Polls other services?

And that is separate from the cache coherency issues.

I have been thinking about this mobile/desktop/pad/whatever presentation.

And I think it is good to go back to the basics, and then reason from there.

Let's start with what a html page is.

It can be:

1. A presentation of a resource (information resource uri)

2. A search result (or combination of)

3. A special page template

4. A web app

Ad 1) 

This is typically shown at the url you get from the page_url property of a resource.

It is the html representation of a resource.

The real url of the resource is its non-informational resource-uri:  http://example.com/id/1234

Whenever you request that uri, you het a 303 See Other redirect depending on the requested content-type.

Think of a person resource. Then you can have a html page, a vcard, a photo etc.

All representing that particular resource.

Let's now think about the html representation.

This representation is the html representation for a desktop sized/capable user agent (aka browser).

But it could also be the html representation for another kind of html capable user agent.

Ad 2/3)

You will need to tailor the amount of data, it's size and presentation to the screen size and the size of the mouse/finger using it.

Ad 4)

Mobile will need different apps than desktop. Though you might still be able to use a mobile app on a desktop.

I see the smaller screen sizes primarily as information consumption devices.

The bigger ones (iPad and up) are information production as well.

Note that there is a huge difference in designing for touch and mouse pointers.

Then think from  the other direction: a search engine.

When I am on my iPhone, then Google shows me the same results as when I am on my MacBook.

That same search result url must show differently, though with the same primary content, on those different user agents.

And that is just because of the capabilities of the user agent and it's input methods.

You can have quite a bit of variation.

Though I do think that with responsive design you can quite well close the gap between (say) phones and pads.

(That is separate from web apps, that is a whole different story).

Conclusion:

Keep all models/logic/etc the same.

Add different html presentations.

Keep informative urls the same between different user agents (think Google)

Now how to do that.

I am proponent of browser sniffing.

Let's assume a new site and follow the advice of front end people.

Start with mobile (phone/pad).

Then add extra stuff for desktop (mouse, keyboard, bigger screen).

Now we have to classify the user agent.

This we can do when accepting the request, before dispatch.

We can also inspect the persistent cookie at this point.

The classification itself can be a notification.

So that we can add other classifications later.

I would classify something along the following dimensions:

- small/medium/big screen

- touch/mouse

- keyboard

Then we can create dispatch rules and templates for these classifications.

Think of them as extra Accept headers with the request.

The system will then have a preference for the classification and can check fallbacks by removing one or more of them.

In Zotonic we could have different template sets / dispatch rules:

- A shared set

- A set per kind of user agent

More thoughts?

- Marc

[Dmitrii Dimandt]

>
> In Zotonic we could have different template sets / dispatch rules:
> - A shared set
> - A set per kind of user agent
>
> More thoughts?
>
> - Marc

I'm all for browser sniffing.

However, this means we need to add more processing power to dispatch rules. I've had this idea for a long time, because it doesn't concern just mobile sites. Below is just ranting without any real thoughts gone into the rant :)

Let's start with mobile first.

Basically, what we usually have is different presentations of the same data. So I, the developer, would want my templates folder to be structured like this:

templates
- mobile
- phones
home.tpl
base.tpl
etc..
- tablets
home.tpl
base.tpl
etc..
- desktop
home.tpl
base.tpl
etc..

So, probably, my dispatch rules should look like this (somewhat pseudo-code):

---------------code-------------------
{on_before_dispatch, {
if mobile and phone -> {template_base_dir, 'mobile/phones'};
if mobile and tablets -> {template_base_dir, 'mobile/tablets'};
else -> {template_base_dir, 'desktop'};
}}
------------end code----------------

Or, better still:

---------------code-------------------
{on_before_dispatch, fun browser_test:test/1}
------------end code----------------

Where browser_test is:

---------------code-------------------
-module(browser_test).
-export([is_cacheable/0, test/1]).

is_cacheable() -> true.
test(Context) ->
case test_browser(Context) of
mobile_phone -> {template_base_dir, 'mobile/phones'};
mobile_tablet -> {template_base_dir, 'mobile/tablets'};
_ -> {template_base_dir, 'desktop'}
end.
------------end code----------------

Obviously, Zotonic would have to recognize the template_base_dir option/variable and behave accordingly. It could also cache responses from the module if it's responses can be cached (see the is_cacheable/0 above).

This way I can have nice templates with the same names to reduce the clutter.

The other reason this kind of preprocessing would be nice is multilingual sites.

I strongly believe that Russian version of http://site/id/23 should be viewed as different from the English version of that same resource. Both from the user point of view and the search engine point of view.

So, I would love to have something like this: http://site/lang-id/regular/zotonic/urls And have a dispatch rule along the lines of:

---------------code-------------------
{on_before_dispatch, '/:lang-id/*', [{dispatch_base, '/*'}, {lang, 'lang-id'}]}
------------end code----------------

In this case Zotonic would dispatch a http://site/en/page/123/page-slug correctly (parsing dispatch at /page, not at /en) all the while displaying the user the correct language version of the resource.

This, of course, means that Zotonic should know how to process options dispatch_base and lang.

These conditions could also be handled by a module:

---------------code-------------------
{on_before_dispatch, fun lang_test:test/1}
------------end code----------------

---------------code-------------------
-module(lang_test).
-compile(export_all).

-is_cacheable(true, [lang, dispatch_base]). % could we cache just certain variables/options?

test(Context) ->
[Lang, DispatchBase] = parse_url(Context),
[{lang, Lang}, {dispatch_base, DispatchBase}].
------------end code----------------

One more thing closely related to mobile sites hat could be handled with such rules (pseudo code below), the idea should be clear:

---------------code-------------------
{on_before_dispatch, fun host_test:test/1}
------------end code----------------

---------------code-------------------
-module(host_test).
-compile(export_all).

-is_cacheable(true, [{context, host}]). % could we cache based on certain external conditions?

test(Context) ->
case Context.host of
'm.facebook.com' ->
{template_base_dir, 'mobile'};
'api.facebook.com' ->
{template_base_dir, 'dev'};
_ ->
{template_base_dir, 'default'}
end.

------------end code----------------

Of course, we would have to discuss LOTS of things in this case:
- do we want on_before_dispatch conditions?
- if yes, how much preprocessing on Context do we do before we invoke these conditions?
- how do we handle multiple conditions?
- which options can these conditions can modify or set?
- how much power will these conditions have? Can we redirect the user from these conditions? Log him in or log him out? Manipulate session state? Manipulate passed data? etc.

One more thing I'd love to see besides all the above is template folder structure.

If the site is quite large, you will end up with lots of templates in a flat list each looking like this: module_action_specificcondition.tpl. And there are lots of such modules.

Let's give them structure.

If we have templates/pages/user/login.tpl, make it compile (internally) to pages_user_login.tpl

For instance, wight no we may have:
article.tpl
_article_chapeau.tpl
_article_keywords.tp
_article_meta.tpl
_article_prevnext.tpl
_article_sidebar.tpl
_article_summary.tpl

This is unnecessary visual clutter that gets quite hard to navigate in time. Why not structure them like this:
article.tpl
article/
chapeau.tpl
keywords.tp
meta.tpl
prevnext.tpl
sidebar.tpl
?

This way we could have some additional ways to include and extend templates:

Regular:
{% include "article.tpl" %}

Full name.
{% include "article_chapeau.tpl" %} — article/chapeau.tpl is compiled internally into article_chapeau.tpl and then included

Paths
{% include "article/chapeau.tpl" %} — include article_chapeau.tpl. Same, as above

In template article/sidebar.tpl:
{% include "./keywords.tpl" %} — include keywords.tpl on the same level as article/sidebar.tpl, i.e. include article/keywords.tpl

We can also have something along the lines of
{% include "./../../profiles/user/info.tpl" %}

Well, that's all I have to rant about :)

[Andreas Stenius]

Wow. A lot of ideas.

And quite a few good ones too :)

Although I'm a little bit too pressed for time right now to dwell
further on them :/

//Andreas

2011/9/15 Dmitrii Dimandt <dmi...@dmitriid.com>:

[Ahmed Al-Saadi]

Hi Marc et. al.

I have implemented mod_mobile which allows the rendering of alternative templates for mobile (and other alternative devices). This module does browser sniffing to find the type of the user-agent and then observes notifications from resources that attempts to render a template, offering an alternative template depending on variables specified in the dispatch rules (thanks for the observe_* callbacks hint, Alain!)

For this module to work, I have modified resource_template.erl and resource_page.erl in mod_base to notify interested modules that it is about to render a template (using z_notifier:first/2). If mod_mobile is active, it will observe that notification, detect the type of user agent and will look for a dispatch parameter ('template_mobile', for now) and will return the template referenced by that parameter to the calling resource. The latter will then use that alternative template for rendering.

The benefit of such a solution:
1) The template path is not hard-coded: it can be specified in the dispatch rule as such:
{gallery,      ["gallery"], resource_template,      [ {template, "gallery.tpl"}, {template_mobile, "mobile/gallery.tpl"} ]},

2) The template choice can have a cascade by trying the most specific template first: e.g., iphone then mobile then regular:
{gallery,      ["gallery"], resource_template,      [ {template, "gallery.tpl"}, {template_mobile, "mobile/gallery.tpl"}, {template_iphone, "mobile/iphone/gallery.tpl"} ]},

3) The template chosen can be based on the parameters Marc suggested: e.g., small screen, touch device, no keyboard:
{gallery,      ["gallery"], resource_template,      [ {template, "gallery.tpl"}, {template_small_touch_nokey, "mobile/gallery.tpl"},

I'm new to Zotonic (and Erlang for that matter), so you may have better ideas for the architecture of this solution about which I look forward to hear. Also, mod_mobile still needs to implement #2 and #3 above as well as do other optimizations such as saving the user-agent detection in the session and possibly caching the regular expression (though the former probably makes the latter less of an issue).

You will find the code here:
https://github.com/thatpythonguy/zotonic (fork of Zotonic with changes to the page and template resources)
https://github.com/thatpythonguy/mod_mobile

Cheers,
-signed(ahmed).

[Alain O'Dea]

And the power of Zotonic events wins again. Nicely done Ahmed!  A very simple notification of render_template with a fallback of current behavior and a simple module that detects the user agent and subs in an explicitly specified template_mobile from dispatch.  I am impressed.  Very nice indeed :)

[Ahmed Al-Saadi]

Thanks, Alain! Hope to make this production-ready soon.

[Manuel Durán Aguete]

Hello Ahmed,
I've modified your mod_mobile code to add support for tera wurfl
https://github.com/ncgmoviles/zotonic/commit/19a70daf253151e9038b2d9f77915289f019083fThe
patch needs some work, configuration related issues,  but this first
try seems to work fine.

Thank you for the initial code.

Regards.

[Ahmed Al-Saadi]

That's interesting, Manuel! Certainly device detection is one of the
high-priority tasks among the many others that will make mod_mobile
production-ready. However, I personally wouldn't make mod_mobile
dependent on an external service, local or remote, unless it is an OTP
app. I'm especially wary of having to install a PHP (or Java) server
just to be able to serve an alternative template! Other than an
aesthetic preference, there are scenarios where such an architecture
could cause a bottleneck.

I suppose that if you wanted a mature solution to sniff a large
selection of mobile devices and serve a different template for each,
you'd have no easier path than to use a (ready-made) wurfl solution.
Maybe an ideal solution would involve implementing a wurfl server in
Erlang as a gen_server? Not sure about your schedule, but mine is
currently full ;)

Thanks for sharing the info on your change and on wurfl; I hadn't heard
of the latter before!

Ahmed

[Marc Worrell]

Wurfl is a nice service, though I am a bit wary about their license.
They use the AGPLv3 license, and they use a private API.
When my memory serves me correct then using the AGPLv3 license over a private API extends that license to your code.
Which would make your whole project AGPLv3.
Which I most definitely would not like to happen.

Does anyone else have an (educated) opinion on this license issue?

- Marc

[Marc Worrell]

On 3 okt 2011, at 11:36, Marc Worrell wrote:

Does anyone else have an (educated) opinion on this license issue?

Actually it is already answered on their own website:

Q: If I use the AGPL API, will I need to open-source my code even if I don't modify the API?

A:  Yes, you will. Exactly like regular GPL, linking your code to GPL code creates derivative work (in the copyright sense of the term) and this is enough to trigger the 'copyleft' provisions. FSF is adamant on this interpretation and so is ScientiaMobile. 

http://wurfl.sourceforge.net/faqlicensing.php#135

So I am afraid we won't be able to use Wurfl in Zotonic.

You can use it in your own AGPLv3 licensed project if you want, the Zotonic license (APLv2) allows this.

- Marc

[Manuel Durán Aguete]

Hello Marc,

You're rigth,  but a question , is the apache 2.0 license compatible with AGPL ?

Q: Why is AGPL not good enough for commercial software?

[....]In other words, all of the software that links to the AGPL library must also be released with a compatible Free or Open-Source license [.....]

Regards.

2011/10/3 Marc Worrell <mwor...@me.com>

[Manuel Durán Aguete]

Hello again,

I've copied Luca Passani, creator of wurfl.

I've asked scientiamobile about the licensing issue and Luca answered

"

Hi Manuel,

I suspect that you will need to ask users to download Tera-WURFL (now called DB API) separately and deal with licensing issue themselves

Cheers

Luca

"

my question was:

"

Hello,

I'm a old user of wurfl.

I'm trying to develop a mobile adaptor for zotonic. Zotonic (http://www.zotonic.com) is a open source cms with apache license 2.0. The module that I'm develpping, licensed as apache 2.0 license too , calls tera-wurfl webservices to resolve device properties.

I'm not a license expert, the question is can I use tera-wurfl ? is your license compatible with apache one? 

Thank you.

"

regards.

2011/10/3 Manuel Durán Aguete <man...@aguete.org>

[Marc Worrell]

Hi Luca,

I think what he means is that you can distribute your code as long it does not contain their API or code.

Which is one way of shifting the problem one step further, but not far enough.

The AGPLv3 is a _very_ contagious license. It make everything it touches AGPLv3. 

And touching might be using a protocol or service licensed under the AGPLv3.

Or referring to the library routines of AGPLv3 code, which is enough to contaminate your code...

My advice: don't touch anything that is AGPLv3, when you see that license: look away and leave.

On an aside, when you follow the letter of the GPLv2 then all WordPress extensions and templates are GPLv2 as well.

Which is a rather interesting issue with the commercially sold extensions and templates.

All these issues are the reason why we didn't choose the GPL.

The APLv2 allows any use of the code, commercial or non-commercial.

You don't need to share your modifications and can combine it with any other code without affecting that code.

The other code can affect the license of your APLv2 code though (as does the AGPLv3).

- Marc