New ESME hooks/actions

[vdichev]

Hello all,

I wanted to have all the powerful Twitter tools in ESME, too, without
bloating the core. Fortunately, there's already a concept unique to
ESME, which makes a lot possible- actions.

The actions engine is simple, but very powerful. There's a *test
condition* and an *action* to be run. I recently realized that many of
the ad-hoc Twitter tools could be implemented in ESME if the limits of
tests and actions are pushed a bit. Introducing hooks to different
ESME events could bring many benefits to this messaging platform.

*Hooks* that could be added to the test conditions:

-someone followed me
-someone unfollowed me
-I logged in

In addition, *regular/scheduled events* could be test conditions as
well. Maybe not a full-blown cron alternative, but at least:

-every N minutes
-at a specified time

There are many *actions* that could be included, as well, but so far
let's start with one- fetching all messages from a *feed*, checking
out which ones are unique from the messages in one's inbox and posting
them in the user's current timeline.

Let me give some *combinations* of these as examples to whet your
appetite:

-every 30 minutes, fetch new entries from my blog and post them in my
timeline
-every 5 minutes, fetch new entries from my Twitter feed (that's what
I started from, look where I got)
-when someone follows me, send me an email
-when I log in, post a message "Hello everyone!"
-at 12 AM tomorrow, post a delayed message

This is something which several different *tools* are trying to
accomplish for Twitter- Twitterfeed (for posting blog headings),
TweetLater(for delayed messages & follower autoreply), identi.ca/
Twitter/ping.fm/whatever cross-post bridges.

My current proposal for hooks/timed tests *syntax* is:

-"hook=login"
-"hook=follow"
-"hook=unfollow"
-"every=5 min"
-"at=12 AM" or "at 200810011200" (something flexible along the lines
of GNU "at")

Since we already have an HTTP POST action with a similar syntax, the
feed action syntax could be:
-"atom:http://..."
-"rss:http://..."

So, what does everyone think? Is it trying to stretch ESME to areas it
wasn't meant to fulfill? Is it too complicated? Or do you think that
these simple building blocks are flexible enough to create complex
behavior? Do you believe that this is a feature, which could make a
difference for ESME and distinguish it from the rest of the social
messaging applications?

I currently have a proof-of-concept version, which includes login,
follow and unfollow hooks and an atom feed fetch action. I'm sure many
things could be improved in my current design, so you could make a big
difference with your comments.

I'm eagerly awaiting your feedback.

Best Regards,
Vassil Dichev

P.S. If someone is interested in *implementation* details, I decided
to include the MailboxReason as an argument to the test function and
created a LoginReason, FollowReason, UnfollowReason, etc. Not sure if
it was designed to work like that, but seems the best way to integrate
my solution in the current design.

P.P.S. Eventually, *expansions* in different actions/hooks could be
created, like "%t: %100c... at %p" in the atom action could generate a
message with the title, up to 100 characters from the content and the
published date at the end. There might be also *parameters* for
actions and hooks, for instance something like "hook=follow following
> 5000 & updates < 5" would generate a hook for dealing with spammers,
who have followed more than 5000 people including you, but have less
than 5 updates to contribute.

With this off my chest, I can go to a short vacation till the end of
the week. I'll gladly reply to any suggestions and opinions when I'm
back and the concept has matured a bit

[Hirsch, Richard]

Cool stuff. What I like is that Vassil's hooks give the end user more power to determine how the social network that is the foundation in ESME is really used.

I think there are two hooks missing:
-I follow someone
-I unfollow someone

I was reading Vassil 's mail and I thought about business rules that might also be relevant as well and then I remembered that there already is a business rule engine in the NetWeaver platform: https://www.sdn.sap.com/irj/sdn/go/portal/prtroot/docs/library/uuid/702315d9-7901-2b10-eaa1-d9e4f5470cac. These are also available via web-service : https://www.sdn.sap.com/irj/sdn/weblogs?blog=/pub/wlg/11081. Now that would be an interesting way to bring "business logic" into ESME.

D.


-----Ursprüngliche Nachricht-----
Von: esme...@googlegroups.com [mailto:esme...@googlegroups.com] Im Auftrag von vdichev
Gesendet: Dienstag, 30. September 2008 16:05
An: esme-dev
Betreff: [ESME-dev] New ESME hooks/actions

[Mrinal Wadhwa]

Vassil,

Absolutely brilliant !!
This definitely has a lot of very interesting possibilities.


Cheers,
Mrinal

[David Pollak]

Vissil,

This is wicked good stuff.  I'm going to start a spreadsheet with "inputs" "actions" and "plugins"  Let's see where we can take these excellent ideas.

Thanks,

David

--
Lift, the simply functional web framework http://liftweb.net
Collaborative Task Management http://much4.us
Follow me: http://twitter.com/dpp
Git some: http://github.com/dpp

[David Pollak]

Oh... I forgot to mention, it's great to link the idea to a goal that an end user wants to achieve.  This helps everyone understand why a particular feature exists.

[David Pollak]

On Wed, Oct 1, 2008 at 6:05 AM, David Pollak <feeder.of...@gmail.com> wrote:

Vissil,

This is wicked good stuff.  I'm going to start a spreadsheet with "inputs" "actions" and "plugins"  Let's see where we can take these excellent ideas.

I've invited you to collaborate on the spreadsheet.

Everyone can view the spreadsheet at http://spreadsheets.google.com/ccc?key=p3rFpv-lucPWq3szApfnsFw&hl=en
 

[vdichev]

Thanks to all for the great feedback!

I tried to summarize 3 of the most important user stories so far in
the spreadsheet:

-notify by mail when followed (unfollowed is identical)
-post message on login
-regularly get in timeline unique messages from a feed

The third case is the most difficult so far because there must be an
actor which could be stopped when the action is deleted/disabled and
started on application start.

I'm trying not to restrict what the user can do but create building
blocks which would satisfy needs we cannot predict yet. If we get it
right, we can add many more actions/filters (e.g. Dick's suggestions)
and let the user be creative and combine them.

I've reconsidered the "hook=" syntax, as it makes things more
complicated without serving much purpose.

Perhaps I could add a branch or share my code somehow so you can check
it out and see how it feels. David could then say if using a
MailboxReason for the events is ok or abuse of the design.

Thanks,
Vassil


On Oct 1, 4:18 pm, "David Pollak" <feeder.of.the.be...@gmail.com>
wrote:

> On Wed, Oct 1, 2008 at 6:05 AM, David Pollak

> <feeder.of.the.be...@gmail.com>wrote:

>
> > Vissil,
>
> > This is wicked good stuff. I'm going to start a spreadsheet with "inputs"
> > "actions" and "plugins" Let's see where we can take these excellent ideas.
>
> I've invited you to collaborate on the spreadsheet.
>

> Everyone can view the spreadsheet athttp://spreadsheets.google.com/ccc?key=p3rFpv-lucPWq3szApfnsFw&hl=en
>
>
>
>
>
> > Thanks,
>
> > David

> >> Twitter/ping.fm/whatever <http://identi.ca/Twitter/ping.fm/whatever>cross-post bridges.

> > Collaborative Task Managementhttp://much4.us

> > Follow me:http://twitter.com/dpp
> > Git some:http://github.com/dpp
>
> --
> Lift, the simply functional web frameworkhttp://liftweb.net

> Collaborative Task Managementhttp://much4.us

[vdichev]

Not a new action, but a feature request for an existing one: define
custom HTTP POST message, where the actual esme message is substituted
via some sort of expansion like %m. Then one could e.g. cross-post to
Twitter as Darren has already noticed:

http://user:pass...@twitter.com/statuses/update.xml
status=%m

[David Pollak]

On Mon, Oct 6, 2008 at 12:41 PM, vdichev <vdi...@gmail.com> wrote:

Thanks to all for the great feedback!

I tried to summarize 3 of the most important user stories so far in
the spreadsheet:

-notify by mail when followed (unfollowed is identical)
-post message on login
-regularly get in timeline unique messages from a feed

The third case is the most difficult so far because there must be an
actor which could be stopped when the action is deleted/disabled and
started on application start.

I'm trying not to restrict what the user can do but create building
blocks which would satisfy needs we cannot predict yet. If we get it
right, we can add many more actions/filters (e.g. Dick's suggestions)
and let the user be creative and combine them.

I've reconsidered the "hook=" syntax, as it makes things more
complicated without serving much purpose.

Perhaps I could add a branch or share my code somehow so you can check
it out and see how it feels. David could then say if using a
MailboxReason for the events is ok or abuse of the design.

Right now, the Actions are applied *after* the mailbox reason is determined.

I was actually thinking that having Actions apply "highlights" to a message might make more sense.  If Actions were cascading and recursive, then one action could "highlight" a message as "important" and another action could forward all "important" messages to SMS.  Each message could have an number of highlights attached to it. 

Making a branch and playing with things on the branch makes a lot of sense.  I'm noodling on a unification of Actions, Following, and Tracking.  I'll probably be playing with a lot of different designs... and the syntax for expressing these things can't be too obtuse.  So... let's create some branches... do some playing... and see what we come up with.  I wish we were hosting this on Git which has much cleaner and easier branching... sigh.

Thanks,

David
 

--

Lift, the simply functional web framework http://liftweb.net

Collaborative Task Management http://much4.us

[Hirsch, Richard]

Take a look at http://code.google.com/p/esmeproject/issues/detail?id=16

-----Ursprüngliche Nachricht-----
Von: esme...@googlegroups.com [mailto:esme...@googlegroups.com] Im Auftrag von vdichev

Gesendet: Montag, 06. Oktober 2008 21:47
An: esme-dev
Betreff: [ESME-dev] Re: New ESME hooks/actions

[vdichev]

That issue is similar, but I think we cannot create arbitrary POST
data with the message inserted at any place the user chooses. From
what I can see the liftweb GetPoster is used and as a body msg.toXml
is sent. Not sure about twitter, but for many services http parameters
won't do, so probably the example "http://rhirsch:******@twitter.com/
statuses/update.xml?status=" won't work even if basic authentication
worked.


On Oct 7, 11:36 am, "Hirsch, Richard" <richard.hir...@siemens.com>
wrote:
> Take a look athttp://code.google.com/p/esmeproject/issues/detail?id=16

>
> -----Ursprüngliche Nachricht-----
> Von: esme...@googlegroups.com [mailto:esme...@googlegroups.com] Im Auftrag von vdichev
> Gesendet: Montag, 06. Oktober 2008 21:47
> An: esme-dev
> Betreff: [ESME-dev] Re: New ESME hooks/actions
>
> Not a new action, but a feature request for an existing one: define custom HTTP POST message, where the actual esme message is substituted via some sort of expansion like %m. Then one could e.g. cross-post to Twitter as Darren has already noticed:
>

> http://user:passw...@twitter.com/statuses/update.xml

> > > > me:http://twitter.com/dppGit some:http://github.com/dpp

[vdichev]

Hello all,

There is now a basic prototype of the hooks and actions ideas in the
"hooks-actions" branch. It's usable enough to get the idea (read on
for what doesn't work), so you are free to check it in and give it a
try. Until I update the quick syntax help on the actions page, these
are the changes:

Test:
-"login"
-"followed", "unfollowed": seem clearer than "follow" & "unfollow";
besides Dick suggested that there are hooks for when the user follows/
unfollows another user.
-"every N mins" where N is an integer. We could add different time
units later on or accept abbreviated/long forms, but this seems to
suffice for now.

Action:
-"atom:http://...."

These are enough to create quite a bit more combinations already. Here
are the bugs/feature enhancements in my list:

* Most apparent of all, each hook generates a message in the user's
timeline regardless of the action assigned for this hook. Other users
don't see this message though unless the "resend" action is used. I
kinda like this for easy debugging, but it could get annoying quickly
I guess.
* When the server instance is restarted, the actions associated with
actors should restart their actors. Some of that already happens via
the ActorWatcher, but the corresponding dispatching actor loses the
hashtable which keeps track of already running actors, so they can't
be stopped when action is deactivated/removed.
* I'd like that when an atom feed is activated, it detects previous
messages from the same feed and doesn't generate duplicate messages.
The "source" message attribute seems appropriate for this.
* URL parsing might be improved- e.g. a hash sign in an URL breaks the
URL. We could safely make a tag start only when the previous character
is a non-word character without risking to break hash-tag syntax.
* basic user authentication in URL syntax- very useful for e.g. the
Twitter atom feed.
* more advanced Internet date format parsing according to all the
variants of RFC 822/RFC 3339 (the time formats allowed by Atom)
* refactor: for instance message sending has some unnecessary
duplication
* new xml-based fetch actions- RSS, Twitter. These are similar to
Atom, so very easy to implement
* the Atom-generated message should be a link
* Loads of parameters to customize the actions
- the message generated by a hook
- tags to assign to a message generated by Atom
- truncate Atom-fetched message to a number of characters

Phew, seems there are many more to-do items than completed ones, but
I'm not sure the google code issue tracker is appropriate for
experimental branches yet.

I hope that this already shows that ESME actions can make it a micro-
messaging platform for power users. Props to David for this immensely
useful idea!

Happy hacking,
Vassil

P.S. Just got an idea for another hook- when the profile image is
added/changed. This would make @chinposin-lovin' folks really happy-
instant win for ESME ;-)

[vdichev]

Hello again,

The hooks-actions branch currently has the feature requests I've
planned. I've submitted a request for review.

To summarize, I wanted tests for the actions which would hook into
ESME system events, like login, being followed, profile changed, etc.
It's a feature which can't be implemented with bots. I also wanted to
implement a general format to pull messages in, like Atom or RSS. From
the corporate perspective, it could be more secure and reliable to not
let an external service push new messages to the server instance. All
of this would target the power user, who would be able to assemble
services, which are otherwise created in an ad-hoc manner (numerous
Twitter tools).

The hook is actually a message generated when the respective event
occurs. The message generated by such an event can be used in the
actions test expression, since it carries attached with it the event
cause. Messages generated by events appear by default in the user's
timeline, but not in follower's timeline. The message can be resent to
followers using the existing "resend" action, or suppressed from
user's own timeline by means of the "filter" action. Any and all saved
messages appear in the public timeline, which can't be avoided; I
expect that this would be addressed by these messages being private by
default in the future (when private messages are implemented).

The only exception is the "every N mins" regular action- these
messages are not saved in the timeline (expect in the case of explicit
resend) because they're awfully noisy. Besides, this is the only test
where two identical "every N mins" tests never refer to the same
generated event (so can't be turned off by the "filter" action).

I've added a "profile" test, which is triggered when the user changes
their profile (nickname, first or last name or image url among
others). Apart from that the other actions are unchanged: "login",
"followed", "unfollowed", "every N mins".

In addition to the "atom" action I've implemented an "rss" action.
Currently these actions are restarted properly on server restart and
stopped when the action is removed or disabled. Whenever one of these
feed actions is re-activated, messages already in the ESME database
from the same feed are not duplicated- only new messages are posted.
Messages generated by feeds are saved in both the user's timeline and
the one of their followers.

In addition, several bug fixes allow more flexible message forwarding.
HTTP POST and mail actions now have a customizable body with
expansions for the user nickname, full name, image url and follower
nickname (in the relevant followed/unfollowed action). It is even
possible to use basic http authentication by specifying the header
"header:Authentication=Basic%20" followed by base64-encoded
"user:password" string (necessary for e.g. twitter). These might not
be the epitome of usability, but it's a good start.

I am going to slow down work on the hooks-actions branch now and give
you time to try it out. I'm open for comments on implementation, style
and especially syntax- I'm not sure whether "profile" or %u, %i, %w
and %f look particularly intuitive, for instance.

The experience I built with Scala and Lift was quite gratifying, and I
would be even more pleased if you find some of the features useful.

Have fun,
Vassil

[Hirsch, Richard]

Can't wait to test this stuff out. We will definitely have to document this and post new examples on blog.esme.us.

D.

-----Ursprüngliche Nachricht-----
Von: esme...@googlegroups.com [mailto:esme...@googlegroups.com] Im Auftrag von vdichev

Gesendet: Montag, 17. November 2008 23:49

An: esme-dev
Betreff: [ESME-dev] Re: New ESME hooks/actions

[vdichev]

Documentation is a must. I feel the mini-help on the actions page is
quite insufficient for this (even as a cheat sheat it starts to grow)

On Nov 18, 9:33 am, "Hirsch, Richard" <richard.hir...@siemens.com>
wrote:

[Hirsch, Richard]

Do you know this document: http://esmeproject.googlecode.com/files/actions.pdf

We will have to add your new functionality. Probably be better if it was in the wiki instead of a PDF.

D.

-----Ursprüngliche Nachricht-----
Von: esme...@googlegroups.com [mailto:esme...@googlegroups.com] Im Auftrag von vdichev

Gesendet: Dienstag, 18. November 2008 10:01