Re: [CrisisMappers] Re: Per Jen's suggestion in comments, we'd love your thoughts here for Global Pulse
Sara Farmer
I've created a testers list at
https://github.com/global-pulse/oliver/wiki/System-test - your eye in
particular will be especially useful in working out how well the "crisis
toolkit" (see the toolkit notes at
https://github.com/global-pulse/shunt/wiki for this) covers what's
needed in the flip-over between "normal" and "crisis" modes of operation.
We'll also need some analysis eyes on the toolset to support
vulnerability monitoring, if anyone wants to help with building out and
testing those lists of tools and needs too.
As an aside, I agree totally about Alabama/Georgia - I was on the
HumanityRoad monitoring team the night the tornadoes swirled through,
and I can remember thinking at one point that that level of coordinated
technology response wouldn't have been imaginable just over a year ago.
Sj.
On 04/30/2011 11:27 PM, te...@citizenactionteam.org wrote:
>
> please put me on the list for people to test the product to see if i
> can interpret the info from the perspective of someone that operates
> "on the ground". although we work "virtually", we support groups on
> the ground and the better we can be prepared to react the better we
> can serve them.
>
> already, because of the heads up from our news sources that
> Alabama/Georgia had reached the level that they'd need us, we were
> better able to react quickly, and get our ducks lined up...
>
> terra
>
> Quoting Sara Farmer <sara....@btinternet.com>:
>
>> After lots of conversations about frameworks and their relative merits
>> for building the Global Pulse platforms and tools (and absolutely no
>> floods of Python people coming out of the woodwork to help), the tech
>> side of Global Pulse is back on track with the 'how the heck we're
>> going to do this thing' plan again.
>>
>> There are actually two systems that are fighting to be built as the
>> "Global Pulse Platform". One is a secure, lightweight repository,
>> giving a place where analysts and other users can find and compare
>> datasets, tools and 'hunches'. The other is a monitoring framework,
>> composed of several of those tools and datasets/datastreams linked
>> together. The first lends itself well to a Ruby/Rails/nosql
>> implementation; the second is better built using Python. This is not
>> the problem that it first appears, because there are very few points
>> at which these two systems actually need to talk to each other. So
>> the plan is this:
>>
>> * Build the repository (aka "Global Pulse Platform") and hunch sharing
>> tool in Ruby/Rails/Mongodb.
>>
>> * Build the toolset (where sensible) in Python.
>>
>> * Connect the two with a "recipe manager" that allows users to pipe
>> data, tools and tool outputs into each other (noting that the user or
>> system manager will generally access most of the available tools, e.g.
>> things like Ushahidi, directly).
>>
>> How you can help? Come on over to the GP Platform Architecture and
>> join in the conversation there before we get thrown out of here for
>> being too geeky. Pull the code and designs down from the global-pulse
>> github area, and tell us what you want and think. Or if you're in a
>> hurry, tag a tweet with #gpbuild so we can find and reply to it.
>>
>> * http://groups.google.com/group/unglobalpulse-platform
>> * https://github.com/global-pulse
>>
>> Thank you,
>>
>>
>>
>> Sara.
>>
>> On Apr 25, 7:29 pm, Sara Farmer <sara.far...@btinternet.com> wrote:
>>> Hi Tim,
>>>
>>> It was a horrendously difficult decision to use Rails/Ruby over
>>> Python/Django for Aorta: we need something fast to build, a mature
>>> set of libraries for datastore management etc etc. Of all the
>>> frameworks available, and the 15 or so criteria that we compared
>>> across, Python had the edge in algorithm libraries - Rails had the
>>> edge for build speed (Java was sound but harder to get up to speed
>>> in, Scala also looked good but had less mature libraries; Drupal's
>>> open data management tools are maturing well at the moment, etc).
>>>
>>> In the end, the decision between Rails and Django was made on the
>>> relative availabilty to us of volunteers who were skilled in these
>>> languages/frameworks. If you can tell me that there is a huge
>>> untapped pool of Django/Python people who are willing to help build
>>> the global pulse prototypes quickly, then we are so early in the
>>> tech that the cost to switch is minimal (although several Ruby/Rails
>>> people might cry).
>>>
>>> Like I said, it's early in the tech part of the program, and this
>>> was a very tough decision to make. If you can add our knowledge on
>>> it, we are definitely open to help.
>>>
>>> Sj.
>>>
>>> --- On Mon, 25/4/11, Tim McNamara <mcnamara....@gmail.com> wrote:
>>>
>>> From: Tim McNamara <mcnamara....@gmail.com>
>>> Subject: Re: [CrisisMappers] Per Jen's suggestion in comments, we'd
>>> love your thoughts here for Global Pulse
>>> To: crisis...@googlegroups.com
>>> Cc: puls...@googlegroups.com
>>> Date: Monday, 25 April, 2011, 22:20
>>>
>>> Hi Sara,
>>>
>>> Sorry in advance for taking this thread in a fairly technical
>>> nature. Feel free to contact me off the list to follow up :)
>>>
>>> I'll be tracking the Global Pulse repositories and contributing
>>> where I can. (FWIW, I would release the code under AGPL, rather than
>>> GPL)
>>> May I ask, why is Aorta being developed in Rails? Ruby has very
>>> immature machine learning/NLP libraries and has a small adoption
>>> within academic circles. You are likely to get far further with
>>> something written in Java or Python.
>>>
>>> Yes, stopwords are highly context dependent. For example, embodiment
>>> is a stopword when dealing with patents. You can use a classifier
>>> that may not be put off my noise as much, such as a max entropy
>>> classifier or a neural net. Another tactic is to generate a complete
>>> feature set and prune the least helpful features out
>>> for classification.
>>>
>>> Ontology development is a fairly vexed issue. I tend to think that
>>> the time taken to develop the world's perfect ontology is time that
>>> could have been spent making a tool that helps people. Perhaps
>>> utilise the work of Freebase.com if you would like the benefits of
>>> structure and the the looseness of a general graph. This would also
>>> make sure that the spirit of openness that Global Pulse seems to
>>> fostering.
>>>
>>> Tim McNamara | @timClicks | timmcnamara.co.nz
>>>
>>> On Mon, Apr 25, 2011 at 1:00 PM, Sara Farmer
>>> <sara.far...@btinternet.com> wrote:
>>>
>>> Absolutely on the lexicons front - although we need to start somewhere,
>>> and structured ontologies seems like a sane place to start (please
>>> guys,
>>> if you have experience that this isn't sane, then please tell us, and
>>> why).
>>>
>>> This weekend, I realised to my horror that stopwords are also
>>> heavily context-dependent. I know this shouldn't be a surprise, given
>>> that stopwords are closely related to the signals that they're letting
>>> through, but it was enough to make me think hard about their concept
>>> and
>>> use.
>>>
>>> Still, baby steps... and it's wonderful to see you sharing
>>> experience and approach on this.
>>>
>>> Sj.
>>>
>>> --- On Mon, 25/4/11, Giles Crouch <gi...@mediabadger.com> wrote:
>>>
>>> From: Giles Crouch <gi...@mediabadger.com>
>>>
>>> Subject: Re: [CrisisMappers] Per Jen's suggestion in comments, we'd
>>> love your thoughts here for Global Pulse
>>> To: crisis...@googlegroups.com
>>>
>>> Cc: "sara farmer" <s...@unglobalpulse.org>
>>> Date: Monday, 25 April, 2011, 1:35
>>>
>>> We've been working on an algorithm that detects a number of "weak
>>> signals" to "understand" them, connect them and understand when they
>>> become "strong" to indicate a significant issue, such as civil
>>> unrest that puts people in danger. This is a complex set of issues.
>>> Although our approach is commercialization, some form will be made
>>> free for
>>> use by crisis mapping applications.
>>>
>>> Beyond Ontologies: We've spent 4 years working on text analysis,
>>> mostly with sentiment and such. Beyond an "ontology" you also need a
>>> "Lexicon", which is the slang, short codes, emoticons and
>>> colloquialisms that may be used and need to be layered into the
>>> ontology so that you can better understand the "context" in text
>>> analysis (i.e. blogs, tweets etc.), then you get into geographical
>>> identifiers (such as tribe or political afiliation) that provide
>>> additional contextual insight.
>>>
>>> Just some elements for consideration
>>> Giles
>>>
>>> On Sun, Apr 24, 2011 at 8:40 PM, Sara Farmer
>>> <sara.far...@btinternet.com> wrote:
>>>
>>> Absolutely. A plan would be to hypothesise about these behaviours
>>> and the structures that could represent them, given the commentaries
>>> and knowledge that we already have, and then go see what the data
>>> tells us around those hypotheses (i.e. what are the interesting data
>>> co-occurances, where are the spikes etc). Sometimes we'll be right,
>>> sometimes wrong, but most likely we'll be close enough with the
>>> hypotheses to be learning more about human behaviour and its online
>>> footprints every time we do this (BTW, we also need help and
>>> suggestions on our existing hypotheses, ontologies etc).
>>>
>>> And yes, different crises will affect different areas and
>>> demographics differently - that's kinda the point, i.e. it's not
>>> enough to make sweeping generalisations (like "stop exports to
>>> reduce food insecurity") but we need to understand more about who is
>>> being
>>> affected, when and where, and to do it quickly enough to
>>> potentially make a difference. We're not talking granularity (and
>>> potential privacy issues) down to the "mrs smith on acacia avenue"
>>> level, but knowing where to look more closely with the conventional
>>> methods (e.g. using negative information like which parts of a
>>> community are vulnerable but *aren't* being sent money from outside).
>>>
>>> You'll note a lot of hedging terms like "might" above. The truth is
>>> that we don't yet know what all the data traces were, what they
>>> currently look like, or what they're likely to look like in the
>>> future. But the more we investigate, the more we ask the questions,
>>> the more we test out different ontologies, frameworks and ways of
>>> looking, the closer we get to something useful.
>>>
>>> Sj.
>>>
>>> --- On Sun, 24/4/11, Nicholas Doiron <ndoi...@andrew.cmu.edu> wrote:
>>>
>>> From: Nicholas Doiron <ndoi...@andrew.cmu.edu>
>>> Subject: Re: [CrisisMappers] Per Jen's suggestion in comments, we'd
>>> love your thoughts here for Global Pulse
>>>
>>> To: crisis...@googlegroups.com
>>> Date: Sunday, 24 April, 2011, 23:54
>>>
>>> Hi Robert, Terra, and others:
>>>
>>> Since much of the Global Pulse post was detecting when people cut
>>> back or
>>> change their phone use, I'd ask whether those are leading or trailing
>>> indicators. Take a serious read of this New York Times article...
>>>
>>> families that "can't afford" bed nets or education for their kids, can
>>> afford their more expensive mobile
>>> bill:http://www.nytimes.com/2010/05/23/opinion/23kristof.html?_r=1
>>>
>>> Also, how much do we know about how mobile money works in a famine or
>>> drought? Perhaps
>>> you'd see a flood of money sent to relatives in the
>>> first areas to be hit, then a cautious hault as the crisis appears to
>>> spread. But a different drought could affect one group more than
>>> others
>>> (for example, most Rutooro people are concentrated in one part of
>>> Uganda),
>>>
>>> so few families would have relatives to send money from outside the
>>> crisis.
>>>
>>> Regards,
>>> Nick Doiron
>>> Montevideo, Uruguay
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> On Sun, April 24, 2011 4:02 pm, te...@citizenactionteam.org wrote:
>>> > Hi Robert/All,
>>>
>>> > This is a very important issue that you have articulated brilliantly.
>>> > We have been worried about it for a long time, but have not had the
>>> > bandwidth to even start thinking about it. I'd be thrilled to
>>> help with
>>> > the logic for the coding of solutions. Because what you pose is a
>>> tool
>>> to
>>> > combat poverty and disease, which I see as perfectly doable.
>>>
>>> > But I want to cage things in a certain way.
>>>
>>> > Our heads have often been swimming in data, as we've tried to
>>> > determine if it's an opportunity or just a waste of precious
>>> bandwidth
>>> > and/or carbon-resources that distract us from "doing the job".
>>> And often,
>>> > while we can "see" a longer term, coded solution, we have to focus on
>>> > digging out the guy under the rubble and so put the longer term
>>> solution
>>> > on the back burner.
>>>
>>> > So, of course, THANK YOU for the planet for taking this on.
>>>
>>> > But I see two big problems, not one. The first is to identify that a
>>> > problem is occurring, or is likely to occur. And the second is
>>> what do
>>> > actually DO about it.
>>>
>>> > To illustrate this, I'll pose how we've looked at things,
>>>
>>> > I'll use SMS
>>>
>>> alerts as an easy target here, but it's just an example,
>>>
>>> > so anyone doing SMS alerts, please don't take it personally. If
>>> 10,000
>>> > alerts come in and no one sorts through them and verifies or even
>>> actively
>>> > addresses them, are they a "waste"? or an opportunity. I like to
>>> see it
>>> > was both, because we think we're better off WITH them than
>>> WITHOUT. So we
>>> > sort through what we can, pick out the few that seem worth
>>> following up on
>>> > and then kick some butt on those.
>>>
>>> > What about the rest? The rest ARE just wasted, in our minds. But as
>>> > some wise sage once told me, "99% of your efforts in life will be
>>> wasted in
>>> > your work to have the 1% matter." So we then think about those 99%
>>> > wasted. In that process, lots of trails have been left, which are
>>> often
>>> > picked up by someone else that either has the bandwidth to do the
>>> work
>>> > that needed to
>>>
>>> complete those tasks, or had a better idea that was lost in
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> > the initial rush.
>>>
>>> > Bottom line... YES YES YES...there's an automated solution waiting to
>>> > solve the data exhaust problem in disaster relief. And YES YES
>>> YES I'd be
>>> > willing to work on coded
>>>
>>> ...
>>>
>>> read more �
>>
>> --
>> You received this message because you are subscribed to the Google
>> Groups "CrisisMappers" group.
>> To post to this group, send email to crisis...@googlegroups.com.
>> To unsubscribe from this group, send email to
>> crisismapper...@googlegroups.com.
>> For more options, visit this group at
>> http://groups.google.com/group/crisismappers?hl=en.
>>
>>
>
>
>