About Our GSoC Projects
Jon
SilCC (Swift Language Computation Core)
SiLCC should parse incoming text from XML feeds and extract relevant keywords. There should also be robust classes for dealing with Twitter picoformats and SMS txtspk. Developers interested in this project should be experienced in Python, Natural Language Processing, or the Twitter API. After evaluating what we had developed originally, we're open to starting over from scratch with the SiLCC project for a number of reasons; partly to make it work with our new Swift core, partly to leverage work already completed by others in this field. GSoC Participants will help determine the new direction.
Example Use Case - Feeds coming into a Swift River instance also posted to the RESTful SiLCC interface, tags are extracted and reapplied to the incoming content. The tags are posted back to the interface, allowing users to sift through related content. Users can then vote on the accuracy of these tags in relation to the content they've been applied to, which in turn helps to train the SiLCC algorithm for the future.Reverberations
Examples of similar projects - TagThe.net (tagging) and OpenCalais.com (tagging, determines taxonomic relationships between datasets)
Reverberations is a project that uses 'Plurk' style timeline to visualize the long tail of retweets on Twitter and trackbacks to Blog content. We consider the initial Tweet to be the stimuli, while all other retweets are echoes of that stimuli. Although Swift's UI actively filters out duplicates, we are very much interested in how many echoes an item of content is as it indicates influence.
Example Use Case - Users of Swift click a button that opens a panel, that displays these reverberations on a grid where the 'x' axis represents time (from left to right)SULSa (Swift User Location Services)
Mockup - Here's a mockup of what this might look like http://appfrica.net/blog/2009/09/15/visualizing-crisis-related-crosstalk/ also see Plurk.com
RESTful location detection service. One purpose of SULSa is to extract location data (Lat and Lon coordinates, as well as City and Country names) from items that have none. The other is to retun that location data preformated in specific XML formats (PFIF, EDXL, JSON, and GeoRSS).
Example Use Case - Incoming content has no location data, how do we determine where it might be coming from? One is to use the IP address of the originating server. The other is to try to infer other options from a users social graph (blog, Facebook page, Google profile etc.) or by using contextual information in the actual content itself.
Jon Gosier
+256.773806071
Director of Swift River
url - http://swift.ushahidi.com/
skype - j.gosier
twitter - jongos
M. Edward (Ed) Borasky
> SilCC (Swift Language Computation Core)*_
>
> SiLCC should parse incoming text from XML feeds and extract relevant
> keywords. There should also be robust classes for dealing with Twitter
> picoformats and SMS txtspk. Developers interested in this project
> should be experienced in Python, Natural Language Processing, or the
> Twitter API. After evaluating what we had developed originally, we're
> open to starting over from scratch with the SiLCC project for a number
> of reasons; partly to make it work with our new Swift core, partly to
> leverage work already completed by others in this field. GSoC
> Participants will help determine the new direction.
>
> *Example Use Case - * Feeds coming into a Swift River instance also
> posted to the RESTful SiLCC interface, tags are extracted and
> reapplied to the incoming content. The tags are posted back to the
> interface, allowing users to sift through related content. Users
> can then vote on the accuracy of these tags in relation to the
> content they've been applied to, which in turn helps to train the
> SiLCC algorithm for the future.
Do you have any idea what the incoming data rates are likely to be?
I've got some pretty good data on current raw Twitter data rates, but
I don't know where else you're expecting to get data.
> _*SULSa (Swift User Location Services)*_
>
> RESTful location detection service. One purpose of SULSa is to extract
> location data (Lat and Lon coordinates, as well as City and Country
> names) from items that have none. The other is to retun that location
> data preformated in specific XML formats (PFIF, EDXL, JSON, and GeoRSS).
>
> *Example Use Case - *Incoming content has no location data, how do
> we determine where it might be coming from? One is to use the IP
> address of the originating server. The other is to try to infer
> other options from a users social graph (blog, Facebook page, Google
> profile etc.) or by using contextual information in the actual
> content itself.
This is tricky. Twitter just fired up their new "place-based" location
API a week or so ago. I did a blog post on it
http://borasky-research.net/2010/03/11/a-challenge-to-the-location-based-services-community/. Right now, their database is USA-only. I've asked them on the developer list what it would take for someone, say, a crisis management team or organization, to supply them with geo data and have them insert it into their database. But I haven't heard anything back. I'm guessing someone high up in the disaster relief community should reach out to Twitter executives if that's something you think is worth
pursuing.
As far as "traversing social graphs" and "connecting the dots about
people" is concerned, there are a lot of very frightened people in the
world where privacy is concerned. The academics are coming up with
ways of figuring out stuff like who's gay, who's in debt up to their
ears, and even *millions* of social security numbers!
In any event, the algorithms are there and probably accurate enough to
deal with incoming documents that have been sent by people who aren't
trying to conceal their location. But I think you're going to have
significant difficulties getting allowed to deploy them, even in a
worthwhile context like disaster relief.
--
M. Edward (Ed) Borasky
borasky-research.net/m-edward-ed-borasky/
"A mathematician is a device for turning coffee into theorems." ~ Paul Erdos
>
> --
> Jon Gosier
> +256.773806071
> Director of Swift River
>
> url - http://swift.ushahidi.com/
> skype - j.gosier
> twitter - jongos
>
> --
> You received this message because you are subscribed to the Google
> Groups "Swift River" group.
> To post to this group, send email to swift...@googlegroups.com.
> To unsubscribe from this group, send email to
> swiftriver+...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/swiftriver?hl=en.
Jon
1) It's impossible to predict exactly but I can let you know the logic we're using to prepare. Let's say the app isn't popular and no one but Ushahidi uses Swiftriver. During an event like Haiti (worst-case scenario) we had 100,000 incoming messages in the first four days (let's call that 25,000 per day). 80% from twitter, 15% from blogs, news, rss feeds and 5% from SMS. Now let's look at the other side, let's say Swift is mildly used in a number of places. Now we're exponentially increasing our transactions (posting content and receiving tags back). For every instance someone deploys that uses our RESTful service (rather than deploying their own) we're adding an unknown variable of transactions per day.
So our data sources, all dealt with simultaneously are SMS, Twitter, Email, Web, RSS/Blogs/News Media, and API Posts. We're already filtering out exact duplicates from all sources, next is retweets with no unique content from Twitter. Our only concern with SiLCC is that we can write in something that can 'teach it' so that it gets better over time. We do need something that aggregates and parses tweets, but we're aiming for much more than just handling Twitter data. And even Twitter has an 'evolving' µformat vocabulary.
2) This is exactly why we roled our own service. If anything better ever comes along, we'll just use that instead. SULSa won't be perfect but it allows us to get as detailed as we want on the fly (as far as the location datasets in the DB).
3) As far as mining the social graph, you're right. But most of this stuff is already public information. If someone posts that they live in New York on Twitter, is it really that much of a stretch to pair that data with info coming from Twitter? I suppose the way I wrote that made it sound like we'd be looking on the users friend's profiles to try to find out where they live, but all we're doing is looking at the next best public place beyond the originating RSS.
Jon Gosier
+256.773806071
Director of Swift River
url - http://swift.ushahidi.com/
skype - j.gosier
twitter - jongos