Protocol Idea: Federated Search Like Status

[Mark Essel (@victusfate)]

One of the challenges for distributed/federated social protocols is
search. Is there a protocol that is dedicated to broadcasting search
queries to other servers that subscribe to perform local search
queries (could be a combination of Salmon/Push)?

hypothetical search:
User/Agent on federated network A) submits a search query "needle in a
haystack". The protocol immediately PusHes the search query to all
subscriber search nodes. Each server does a local search and sends a
message back up stream (Salmon) to the original query server which
alerts the User/Agent and shows a selection of matches. Search quality
is dependent on local server implementations.

some thoughts from last year:
http://www.victusspiritus.com/2009/08/21/real-time-search-over-federated-networks/

[Brian Behlendorf]

Gnutella? If you built something new, could you build it on XMPP?

Brian

[Mark Essel (@victusfate)]

I'll read up on Gnutella.
I think the communication could be broadcast with XMPP, but do you
think the query/search format should be tied to a specific format?

> >http://www.victusspiritus.com/2009/08/21/real-time-search-over-federa...

[Mark Essel (@victusfate)]

Wow this sounds fantastic:
http://en.wikipedia.org/wiki/Gnutella

they even mention an aspect of search:
"Instead of treating every user as client and server, some users were
now treated as "ultrapeers", routing search requests and responses for
users connected to them."

But I think the format I'm thinking of could work with any
architecture. It's just a type of message and answer interface, that
will hopefully yield responsive and useful results.

<searchlikestatus>
<query>needle in a haystack</query>
<origin>URL of asking server</origin>
</searchlikestatus>

Various opt in subscriber servers/clients could probe their local
networks feeding the question down with an identical propagating
search

of course results could be cached to avoid heavy repeat searches.

On Mar 9, 11:02 am, "Mark Essel (@victusfate)" <mes...@gmail.com>
wrote:

[Mark Essel (@victusfate)]

we'd need to set it up to avoid traffic/querying the same server
twice.
Maybe a query ID?

On Mar 9, 11:08 am, "Mark Essel (@victusfate)" <mes...@gmail.com>

[Mark Essel (@victusfate)]

Couple of issues to prevent adoption for this type of search (thanks
to my coworker Jim for playing the devil's advocate):

Routing: Would every server get every question? Information overload,
network clogging, cats chasing dogs. Routing could results in misses
if not done with forethought (my local server may not respond with
some great info if I never get the question). We'd want to design it
so that the statistical chance of missing a high quality answer is low
(replicate answers on several servers?) Search answers can find their
way up by a quality score from people that ask? (users upvote high
quality search matches, user influence may matter so experts have more
weight and can fend off the law of averages, perhaps individual users
will affect the query by allowing their personal information to affect
the answer (2way search/separate issue)

Why bother: Servers would have to do more work than they do now, why
would they willingly choose to do so? Why connect to the Internet, to
consume and provide information. I had hopes that this could be setup
in a way that would be a standard built into web servers.

Servers will need knowledge of their local data, local index (so
they'd need quality local info). We'd need something like great local
crawlers to keep our index fresh and correct.

On Mar 9, 11:09 am, "Mark Essel (@victusfate)" <mes...@gmail.com>

[Mark Essel (@victusfate)]

Final issue, with local search vision the broad view of an
interconnected (linked) web is lost. Somehow the value of inbound
links needs to be incorporated into this search system.

Basically don't ignore the work of all the web crawling bots from
Google, Bing, etc.

On Mar 9, 12:14 pm, "Mark Essel (@victusfate)" <mes...@gmail.com>

[tyler gillies]

here is the model in my head.

Search request server. Sends out search requests to search enabled servers via PuSH.

Search servers receive PuSH request and query their local database for the result using any method they deem necessary.

something like result = Content.search_function(search_term)

they then POST the result values (as an array) to the search request server's callback URL example.com/search_results

On Tue, Mar 9, 2010 at 9:24 AM, Mark Essel (@victusfate) <mes...@gmail.com> wrote:

Final issue, with local search vision the broad view of an
interconnected (linked) web is lost. Somehow the value of inbound
links needs to be incorporated into this search system.

Basically don't ignore the work of all the web crawling bots from
Google, Bing, etc

--
You received this message because you are subscribed to the Google Groups "Open Web Foundation Discussion" group.
To post to this group, send email to open-web...@googlegroups.com.
To unsubscribe from this group, send email to open-web-discu...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/open-web-discuss?hl=en.

--
Everyone Loves Tea
http://www.everyonelovestea.com

[Mark Essel (@victusfate)]

that would do it.

still gotta get over the overloaded traffic hump. would every query
hit every server, nope.

how can we mix broad web knowledge like links, upvotes from crowd
sourced sites etc with the beauty of a locally response search engine.

scoring and quality of answers is an open issue, maybe lots of
companies could compete in this space?

> > open-web-discu...@googlegroups.com<open-web-discuss%2Bunsubscrib e...@googlegroups.com>

[Peter Saint-Andre]

On 3/9/10 12:08 PM, Mark Essel (@victusfate) wrote:

> how can we mix broad web knowledge like links, upvotes from crowd
> sourced sites etc with the beauty of a locally response search engine.
>
> scoring and quality of answers is an open issue, maybe lots of
> companies could compete in this space?

You might look into the work that Jeremie Miller (inventor of Jabber)
put into Wikia Search a while back.

http://en.wikipedia.org/wiki/Wikia_Search

And yes, it's a hard problem. :)

Peter

--
Peter Saint-Andre
https://stpeter.im/

[Mark Essel (@victusfate)]

Thanks Peter, that's very relevant. Appreciate you pointing it out.
The entire protocol really gets to the heart of the web, do we want it
to be a question/query driven model? Does a distributed search
architecture benefit the web in the long term?

>  smime.p7s
> 9KViewDownload

[tyler gillies]

According to google it does ;)

> --
> You received this message because you are subscribed to the Google Groups "Open Web Foundation Discussion" group.
> To post to this group, send email to open-web...@googlegroups.com.

> To unsubscribe from this group, send email to open-web-discu...@googlegroups.com.

[Will Norris]

might also want to look at Open Search, if you haven't already.

http://www.opensearch.org/

[Mark Essel (@victusfate)]

whoa this is a perfect pattern for what I'm after.
So what it needs is adoption, and a mechanism for many distributed
social platforms to opt in as opensearch providers.

On Mar 11, 2:51 pm, Will Norris <w...@willnorris.com> wrote:
> might also want to look at Open Search, if you haven't already.
>
> http://www.opensearch.org/
>

[Steven Livingstone-Perez]

It's pretty well adopted - i think all the major browsers now have it built
in. The UX is the tricky bit.

Still you can go to http://search.twitter.com and add them as a provider
(for example).

Also, check this list http://www.opensearchlist.com

Hit one of the sites, add one of the providers and then fire a search in
your browser.

/steven
http://livz.org

--

You received this message because you are subscribed to the Google Groups
"Open Web Foundation Discussion" group.
To post to this group, send email to open-web...@googlegroups.com.
To unsubscribe from this group, send email to

open-web-discu...@googlegroups.com.