Do you plan to add JSON support..

[Pix]

Following this previous post http://groups.google.com/group/solrnet/browse_thread/thread/d4a8ebc554def043
and some others of yours from 2010/2009...

When do you plan to add a JSON a parser to solrnet? ... or are you
just going to leave it parsing xml responses only?

[Mauricio Scheffer]

My position is still the one I explained here: http://code.google.com/p/solrnet/issues/detail?id=106

I still don't see a valid usecase for a JSON parser, so I'd love to hear your reasons for needing one.

I do plan to implement binary communication (see http://code.google.com/p/solrnet/issues/detail?id=71 ) because it should be more efficient than XML, although it's low-priority to me. The current XML parser/serializer has good enough performance for my apps...

--

Mauricio

--
You received this message because you are subscribed to the Google Groups "SolrNet" group.
To post to this group, send email to sol...@googlegroups.com.
To unsubscribe from this group, send email to solrnet+u...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/solrnet?hl=en.

[Pix]

Well... simply put... parsing json is far quicker and is less
"bloated" than xml. Take a scenario where you have a very large index
and with a query pulling back a few thousand records, a json response
from Solr will be substantially smaller over the network. It is
increasingly being used to replace xml as the method of choice for
data transport over networks.... not just http calls from javascript.

Have a read through this article I came across yesterday:
http://www.codeproject.com/KB/IP/fastJSON.aspx

I appreciate the fact that implementing a new parser is a low priority
for you, but just because the performance is good enough for your
apps, doesn't mean it's good for other people out there who is using
your library in .net. Looking at some other posts, it seems to me
that there is quite a bit of interest in the json parser.....

if it's purely a case that you don't have enough time to implement a
json parser, then I'd be happy to help on that side and contribute the
code back....



On Feb 24, 4:18 pm, Mauricio Scheffer <mauricioschef...@gmail.com>
wrote:

> My position is still the one I explained here:http://code.google.com/p/solrnet/issues/detail?id=106
> <http://code.google.com/p/solrnet/issues/detail?id=106>I still don't see a
> valid usecase for a JSON parser, so I'd love to hear your reasons for
> needing one.

> I do plan to implement binary communication (seehttp://code.google.com/p/solrnet/issues/detail?id=71) because it should be

> more efficient than XML, although it's low-priority to me. The current XML
> parser/serializer has good enough performance for my apps...
>
> --
> Mauricio
>
>
>
>
>
>
>
> On Thu, Feb 24, 2011 at 12:30 PM, Pix <ddecamp...@gmail.com> wrote:
> > Following this previous post

> >http://groups.google.com/group/solrnet/browse_thread/thread/d4a8ebc55...

[Howard van Rooijen]

Mauricio has produced an excellent framework that has been of benefit to a huge number of projects - but one thing to realise is that this type of OSS project is primarily about solving your own problems and sharing the results with the rest of the world (a very generous thing to do). If users of SolrNet want JSON support they should contribute that functionality rather than relying on Mauricio to provide free development effort, this is after all an Open Source Project...

We used SolrNet on a public ecommerce site that had 100k+ pageviews per day and network saturation of Solr XML Responses was not an issue, in facetpage response times were still well below 1s.

[[mRg]]

While the arguments are certainly there for JSON being smaller 'across
the wire' etc I would debate how it would effect a larger site.

I run a very large, highly available site which is very dependant on
SolrNet and it has never failed to perform or be the fastest part of
the site. Also why would you ~want~ to pull back a few thousand
records in one request ? This is of no use to the user and Solr's
paging and filtering/faceting should eliminate the need for having to
pull a lot of fields and then apply logic to them. A lot of the time
if you are talking about a larger scaled site then other factors such
as load balancing and aggressive caching of the solr requests come
into play that effect core performance.

Don't get me wrong though, there is no reason why you shouldn't code
up the JSON parsers for SolrNet and it can only make the project
stronger and fully featured if stuff like this is committed although
the java binary format would be smaller still and if you were to
prioritise work then it should probably be that :D (I think there is a
branch for this already)

[Mauricio Scheffer]

inline:

On Thu, Mar 3, 2011 at 8:24 AM, Pix <ddeca...@gmail.com> wrote:

Well... simply put...    parsing json is far quicker and is less
"bloated" than xml.  Take a scenario where you have a very large index
and with a query pulling back a few thousand records,

mRg already answered this. Pulling thousands of documents is a very rare usage of Solr. It may even indicate that you're using Solr for something it wasn't designed to do.

 

a json response
from Solr will be substantially smaller over the network.

Have you tried response compression? mRg added compression support a few months ago: http://code.google.com/p/solrnet/issues/detail?id=115

 

 It is
increasingly being used to replace xml as the method of choice for
data transport over networks.... not just http calls from javascript.

Have a read through this article I came across yesterday:
http://www.codeproject.com/KB/IP/fastJSON.aspx

Benchmarks should be taken with a grain of salt. In the specific case of SolrNet, I think overall JSON would only be marginally faster than XML. I've seen benchmarks of System.Xml.Linq.XDocument being much faster than System.Xml.XmlDocument, but when I made the switch in SolrNet I re-benchmarked and saw no difference in performance at all. Still, I'm keeping XDocument because I was actually looking for lazy parsing, not pure performance.

Another thing to consider with JSON is dependencies. Taking a dependency to JSON.net or whatever shouldn't be taken lightly in a library. DataContractJsonSerializer is another option, but is it flexible enough for our needs? Can we implement it without requiring the user to add any additional attributes? Can I hook up my own serialization for a specific type? Does it support anonymous objects? dynamic (.net 4.0) objects?

 

I appreciate the fact that implementing a new parser is a low priority
for you, but just because the performance is good enough for your
apps, doesn't mean it's good for other people out there who is using
your library in .net.  Looking at some other posts, it seems to me
that there is quite a bit of interest in the json parser.....

Howard already answered this.

 

if it's purely a case that you don't have enough time to implement a
json parser, then I'd be happy to help on that side and contribute the
code back....

I really think our efforts should concentrate on the binary protocol rather than JSON, I think it will be faster and way more efficient. It's what the Java client uses by default.

I'm also looking to make response parsing more lazy, so in the case of a large response you would be able to start processing results immediately.

--

Mauricio

[Steven Livingstone Pérez]

Just to back up what Mauricio says - parsing an Xml document (using DOM or a reader) is much faster than you think. Having spoken with the guy team who wrote it i know that their parser is very very fast - and that was a few years back.

I suspect parsing Xml is much faster than JSON mainly because internal tree optimizations etc are done for that. In addition you have the reader option for huge document sets.

Also, i much prefer consuming Xml (as i recently found out while trying to pull in Google spreadsheet feeds which emit JSON and Xml) - it has namespace & schema support and more that are useful in the more complex scenarios.

Doing Xml -> Json should be fine for smaller document sets i'd think.

/steven

---

Date: Thu, 3 Mar 2011 11:05:51 -0300
Subject: Re: [SolrNet] Re: Do you plan to add JSON support..
From: mauricio...@gmail.com
To: sol...@googlegroups.com