Solr-Ajax Installation

[Praful Bagai]

Hi all,

I am not able to set Ajax-Solr on my system.I did a bit of R&D on Ajax-Solr for the same .

I understood what it tries to do with the tutorial provided at https://github.com/evolvingweb/ajax-solr/ , but unfortunately, I am not able to able to set it up on my system.

I dont know how to integrate it with Solr. Into which directory of Solr instance does Ajax-Solr goes?

Please suggest something.

Thanks 

Regards

Praful Bagai

[Sidd]

The only place where Ajax-Solr references Solr is in the reuters.js file (in the tutorial) where you specify the solrUrl. It doesn't matter where Ajax-Solr goes. For example, my solrUrl is http://localhost:8983/solr/.

[Praful Bagai]

Thanks for quick response.

I have kept my Ajax-Solr folder inside the Tomcat Directory.

I changed the solrURL as http://localhost:8080/solr/ in reuters .js file.

Now, when i browse to http://localhost:8080/solr/, it still shows the same backend Solr UI. (This may sound a kinda noobish)

Can you please suggest me how to get the Ajax-Solr front end UI ??

Thanks

Regards

Praful Bagai

[Sidd]

It shows the backend Solr UI? What I did to set up Ajax-Solr is just download the whole API for this, and i went into reuters.js. First I had to change that solrUrl. Secondly, the fields that the demo program assumes certain fields which I am guessing your program won't have. You will have to look through that file and get rid of the instances of the fields that you don't have, and add instances that you need. For example, when you add the ResultsWidget, you want to change the 'target' to the field you want to search up. Also, I know there were some arrays in that file with topics, organizations, and something else, you should get rid of that too.

[James McKinney]

If you want to replace the default Solr UI, you may have to ask the Solr mailing lists: http://lucene.apache.org/solr/discussion.html

ajax-solr is very easy to deploy. In fact, if you open the index.html file of the Reuters demo in your browser, it should work. If you've ever run a web server like Apache or Nginx, all you need to do is upload (via FTP or SSH) the JS, CSS and HTML files to an appropriate directory for that web server. I've never used Tomcat as a web server.

[Praful Bagai]

Ajax-Solr isnt working yet. I am using Solr 3.4, edited the schema.xml file according to the tutorial provided at https://github.com/evolvingweb/ajax-solr/wiki/Reuters-tutorial. Changed my solrURL to http://http://localhost:8080/solr/. but still it does not run.

I am opening index.html via tomcat. It just shows the index1 template with loading circle in the result section and thats it. No calender,no map nothing at all. Where does the problem lies??

Moreover when i edited schema.xml and browse to http://localhost:8080/solr/, it then pops up an error saying "Server errors in solr configuration"??

Plus what does these statements mean?

"If you are using your own Solr instance, you may want to change the values assigned to the fields keys."

AND

" If you are not using your own Solr instance, you may want to change (or remove) the lines adding facet links for topics, organisations and exchanges."

Kindly help. I am stuck with this for past 2 days now.

Thanks

Regards

Praful Bagai

[James McKinney]

If http://localhost:8080/solr/ says "server errors in Solr configuration" then you have a problem with your Solr configuration that is not related to ajax-solr. I would ask for help configuring Solr on the Solr mailing lists, giving a little more detail about the error message shown: http://lucene.apache.org/solr/discussion.html

I am opening index.html via tomcat. It just shows the index1 template with loading circle in the result section and thats it. No calender,no map nothing at all. Where does the problem lies??

You can just open index.html in your browser when you are first setting ajax-solr up. If it's not working, then if you're using Firefox, install Firebug and open the JavaScript console to see the errors. If you're using Chrome or Safari, you may need to enable developer tools and open the JavaScript console.

Plus what does these statements mean?

"If you are using your own Solr instance, you may want to change the values assigned to the fields keys."

AND

" If you are not using your own Solr instance, you may want to change (or remove) the lines adding facet links for topics, organisations and exchanges."

The second statement should not have "not" in it. I fixed it now. All it means is that if you are not using the same solrUrl as in the tutorial, then you may not have values for topics, organisations, exchanges, etc. and that you should tailor the tutorial to your own needs.

[Praful Bagai]

Hi James,

Sorry for the trouble caused. I just made a noobish mistake. I integrated the Solr with nutch. So, as a result of it,I had to replace schema.xml of Solr directory with schema.xml of nutch directory. So, I was editing the wrong schema.xml file.SOORY!!! I rectified the mistake caused by me and now solr is all up and running.Thanks for your concern!!

But I cannot find these 2 fields in solr 3.5.0 schema.xml So, I manually added these 2 fields under <field> tag.

1) <field name="date" type="pdate" indexed="true" stored="true" multiValued="true" omitNorms="true" termVectors="true" />

2) <field name="dateline" type="string" indexed="true" stored="true" multiValued="true" omitNorms="true" termVectors="true" />

But still experiencing the same problem,ie regarding index.html.

the error that is flashed up in the firebug is-

 >undefined field date</u></p><p><b>description</b> <u>The request sent by the client was syntactically incorrect (undefined field date)

I dont know know why it is now flashing an error although i manually included the date field in schema.xml.

Kindly help!!!

Thanks

Praful Bagai

[Praful Bagai]

Hey James,

I got Ajax-solr running. But it aint running on Chrome. It runs on Firefox though. Any specific issue under Chrome. My chrome is the latest one. 

Now, i want to ask that although ajax-solr is running but it's searching under only reuters data. If I want to crawl the web using nutch and integrate it with solr,then i have to replace solr's schema.xml file with nutch's schema.xml file which will again cause the earlier problems.

How would I now integrate Solr with Nutch so as Ajax-Solr can search other data on the web as well??

and yes THANKS A LOT for helping me. That was soo nice of you. Thank You

Regards

Praful Bagai

[James McKinney]

I would recommend going through the tutorial from the beginning, writing the code as you go, to see which step causes the error. It will be easier to figure things out that way than by trying to load the whole demo at once and working backwards.

[Praful Bagai]

Can I still integrate Solr with Nutch while continuing with Ajax-Solr knowing the fact that it will replace the schema.xml file??

[James McKinney]

As long as you can query a Solr instance, you can use ajax-solr. If localhost:8080/solr (or wherever your Solr server is) can return Solr responses, then ajax-solr will work.

[Praful Bagai]

Hi James,

(Correct me if I am wrong)

I earlier had a query that whether we can we integrate Solr with Nutch while continuing with Ajax-Solr knowing the fact that it will replace the schema.xml file which will then hinder us from using Ajax-Solr??

I guess, Yes we can. If we can set multiple cores. One for keeping crawled data using Nutch ,ie we can query the Solr instance in one core and search in another via Solr-Ajax.

Just to be sure, Can we do this??

PS: I tried setting up multiple cores at first place but couldnt set them up, thats why was not able to try the method specified above. Can you kindly tell me whether is it possible or not???

Thanks

Regards

Praful Bagai

[James McKinney]

I have no experience with Nutch, but like I said, if there's some URL that can return a typical Solr response (e.g. an XML file with a list of matching documents, like Solr usually does) then you can use ajax-solr. The Solr mailing lists will probably be more helpful in terms of setting up your cores: http://lucene.apache.org/solr/discussion.html

[Michael]

Somehow my mail got lost... :-(  - so second try....

Hi,

Nutch indexes his documents to Solr. The data you can crawl differ of
course from the Reuters-example, so Nutch uses a different schema.xml.

Nutch has no search-user-interface any more. So you have to build one of
your own. This was when I discovered Ajax-Solr and I want to thank all
of the developers very greatly! It's not only a great tool, but together
with the examples it was quite straight forward to customize it for my
needs. Thank you!

Ok - I wrote "customize": when you look at the examples you only have to
understand the index.html, the reuters.js and the reuters.theme.js. The
rest of the code is nearly identical to the examples. You have to
understand it in so far, as you must look at the fields and how they are
used. Personally I'm no Javascript guru, but the coding was pretty clear.

I can't offer you my code, because I extended nutch with an individual
plugin and integrated several other datasources. So my schema.xml
differs totally (the website is in German: https://www.netluchs.de/).
But it's all in one index (no cores!).

To make a long post short: integrating nutch with ajax-solr is not only
a matter of configuration-files! You have to get your fingers dirty and
code a little bit.

Regards from Germany

	Michael

[Sidd]

For me, I just copied and pasted JUST the fields from the nutch schema to the solr schema..that should do the trick

[Praful Bagai]

@Michael--

I could not understand in which way you are asking me to code. I am very good in writing codes but I am not able to get in which way you are asking me to code. Can you please elaborate with little help explaining it exactly in what way you want me to code??

How and where do I need to code?

Thanks

Regards

[Praful Bagai]

@Sid

I just tried your method. It works completely fine. But I faced a problem which is as follow

When reuters data folder is replaced with already present data folder in example/solr directory, Solr still indexes crawled data via cygwin but when I browse to http://localhost:8080/solr/admin/ it only shows the reuters data not any other crawled data. Also in ajax-solr, reuters data is shown.

Next when I deleted the reuters data from example/solr directory , Solr indexes the crawl data via cygwin and I am also able to see it on my localhost, but not on ajax-solr. Why is so???

Thanx

Regards

Praful Bagai

[Michael]

Hi Praful (? is this your short name),

first of all: I meant no offense by telling you to code. I just wanted to make the point, that you have to make more than just copy some files and configure something. Please excuse me, if my answer sounded rude and I offended you. Sorry!

Let's take one step back. You successfully indexed some webpages with nutch and you can query the resulting solr-index by using the default solr-admin-frontend. Is this right? So you can do something like:

http://my.host.name:PORT/.../select/?q=*%3A*&version=2.2&start=0&rows=10&indent=on&facet=on&facet.field=title&facet.field=url

The part before the "select is the solrurl you need.

Next you have to take a look at the reuters.js file. There are three fields

 var fields = [ 'topics', 'organisations', 'exchanges' ];

and three

 'facet.field': [ 'topics', 'organisations', 'exchanges' ],

They does not fit with the nutch index. So you have to make three changes in this file: solrUrl, fields and facet.field. 

Next you have to look at reuters.theme. In the snippet-function is a reference to the full text in the field "text". Nutch calls this field "content". 

Does this description help you?

Regards

     Michael

PS.: I live in GMT+2 so please give me some time to respond. I'll stay online this evening so feel free to send a private message if you need further help.

[Praful Bagai]

Hey Michael,

First thing first, Your message did not offended me at all. Why did you thought like that? If, by chance, I was harsh in my comments, please forgive me. Thank you Michael. :-)

Next, You asked me to make changes to variables 'fields' and 'facet.fields', but in what way should I customize them so as to make them work according to my needs. Can you please provide me with an example. 

Spare me for my poor knowledge on Nutch and Solr.

And you have all the time in the world to respond me. Please don't feel sorry. :-)

Thanks

Regards

Praful Bagai 

PS: Praful is my first name!! Cheers !!!

[Sidd]

var params = {
            facet : true,
            'hl' : true,
            'hl.fl' : '*',
            'hl.fragsize' : '-12321',
            'facet.field' : [ 'text', 'title', 'id', 'description', 'keywords',
                    'host', 'subdomains' ],
            'facet.limit' : 20,
            'facet.mincount' : 1,
            'f.topics.facet.limit' : 50,
            'json.nl' : 'map'
        };
        for ( var name in params) {
            Manager.store.addByValue(name, params[name]);
        }

See above on what my facet.field is (in the reuters.js file). Those are the fields I want to have faceted, and those fields have been defined in my schema.xml. Same deal for the "var field" values. You don't need to put every single field in those arrays, only the fields you want to have faceted. For more information about faceting, look at http://lucene.apache.org/solr/api-3_6_0/doc-files/tutorial.html and go down to the heading called "Faceted search"

[Sidd]

Are you still having hte problem where you can only see reuters data?

[Praful Bagai]

@Sidd.

 

My problem got fixed. I appreciate your concern. 

Thanx

Regards

[Michael Nebel (gmail)]

Hi,

Nutch indexes his documents to Solr. The data you can crawl differ of
course from the Reuters-example, so Nutch uses a different schema.xml.

Nutch has no search-user-interface any more. So you have to build one of
your own. This was when I discovered Ajax-Solr and I want to thank all
of the developers very greatly! It's not only a great tool, but together
with the examples it was quite straight forward to customize it for my
needs. Thank you!

Ok - I wrote "customize": when you look at the examples you only have to
understand the index.html, the reuters.js and the reuters.theme.js. The
rest of the code is nearly identical to the examples. You have to
understand it in so far, as you must look at the fields and how they are
used. Personally I'm no Javascript guru, but the coding was pretty clear.

I can't offer you my code, because I extended nutch with an individual
plugin and integrated several other datasources. So my schema.xml
differs totally (the website is in German: https://www.netluchs.de/).
But it's all in one index (no cores!).

To make a long post short: integrating nutch with ajax-solr is not only
a matter of configuration-files! You have to get your fingers dirty and
code a little bit.

Regards from Germany

Michael

[Praful Bagai]

For all those who need to search data other than reuter's data, follow these steps.r

You just need to customize reuter.js and reuters.theme.js.

Step 1) replace the word 'text' with 'content' in reuters.theme.js. Nutch has no field "text". It's called "content".

except this---

The following line should read:

AjaxSolr.theme.prototype.tag = function (value, weight, handler) {
    return $('<a href="#"
class="tagcloud_item"/>').text(value).addClass('tagcloud_size_' +
weight).click(handler);
};

This is a different "text" - no doc.text....

2) When you look at one result - you see the possible

fields a solr document (doc) has. For example:

<doc>
        <float name="boost">0.0</float>
        <str name="content">nebel.de - .....</str>
        <str name="digest">b861eda2c87742c51058fcc1b19cd770</str>
        <long name="documentContentLength">2106</long>
        <str name="host">www.nebel.de</str>
        <str name="id">http://www.nebel.de/</str>
        <str name="segment">20120305215919</str>
        <str name="site">www.nebel.de</str>
        <str name="title">nebel.de - Dunst und Durchblick</str>
        <date name="tstamp">2012-03-05T21:03:44.036Z</date>
        <str name="url">http://www.nebel.de/</str>
</doc>

This are the fields you can search over - the fields you set in the "var
fields".... So I would guess

        var fields = [ 'title', 'url', 'content' ];

would work.

The facets are a different thing. If you only have a small index, you
can user nearyl any field. I would suggest to start with

'facet.field': [ 'title' ],

Step 3)  Remove the "doc.dateline" from the snippet. This field misses in the nutch-index.

Step 4) Check the following value in your nutch-*xml:

<property>
  <name>fetcher.store.content</name>
  <value>true</value>
  <description>If true, fetcher will store content.</description>
</property>

<property>
  <name>parser.caching.forbidden.policy</name>
  <value>content</value>
  <description>If a site (or a page) requests through its robot metatags
  that it should not be shown as cached content, apply this policy.
Currently
  three keywords are recognized: "none" ignores any "noarchive" directives.
  "content" doesn't show the content, but shows summaries (snippets).
  "all" doesn't show either content or summaries.</description>
</property>

And that's it. You are done. !!!! Now you can search data other than reuter's.

Praful Bagai

[Michael]

There is nothing special about using ajax-solr as frontend for nutch. Anyone else with nutch-problems is also welcome to drop me a note. I use ajax-solr as frontend for nutch-indexes.

Michael

[Praful Bagai]

@Sidd-

I saw your thread on highlighting keywords in the search result, and I guess you were able to overcome the problem you had been facing but unfortunately ,while reading your post, I was not able to implement it on my system.

Can you please stage me through this difficult (or should I say simple) process of highlighting the keywords.

PS:

The <highlighting> tag is active in solrconfig.xml and I also included the following parameters in reuters.js

    var params = {

      facet: true,

      'facet.field': ['title'],

      'facet.limit': 20,

'hl':true,

      'hl.fl':'text',-----------------------------------------------------since for nutch it is content, so I replaced text with content,but to no luck

      'facet.mincount': 1,

      'f.topics.facet.limit': 50,

      'f.countryCodes.facet.limit': -1,

      'facet.date': 'date',

      'facet.date.start': '1987-02-26T00:00:00.000Z/DAY',

      'facet.date.end': '1987-10-20T00:00:00.000Z/DAY+1DAY',

      'facet.date.gap': '+1DAY',

      'json.nl': 'map'

    };

Please guide me through this.

Thanks

Regards

Praful Bagai

[Sidd]

I'm not too sure how I did it, and I didn't really do a good job of logging it, but I'll try my best to remember. Remember that if you want the highlighting field to show up, you have to I know that I didn't like the way the default files did it (by using the q parameter files) because they would 'set' the q value, whereas the other files would 'add' to the fq field. This means that if I had more than one search term, it'll only do the last search term.

Make sure to use the autocomplete widget, that helped me with some reason.

I guess the important part is that in the currentsearchwidget, I made that extend AbstractTextWidget, and I retrieved all of the fq parameters after it was updated, and I put them together into one string so that it would be separated like "TEXT:SEARCHTERMS, TEXT:SEARCHTERMS, TEXT:SEARCHTERMS, [field]:[whateveryouaresearchinginthisfield]...." Make sure you have colons between the field and the search term, and make sure all of the terms are separated by a comma AND a space (that messed me up before). Then I set the q parameter to that long string, and my results came up in manager.response.highlighting[doc.id]...

Hope that helped.

[Praful Bagai]

Can you please guide me through this as I am unable to do it on my own because of lack of knowledge in Solr/Ajax-Solr.

Kindly help me please. Been stuck up with from over 8-9 days now.

ThankYou

Regards

[Sidd]

I'll help you because I feel your pain. I think at this point, the easiest thing to do would be to find a place to post your code so I can take a look at it? Also before you do that, make sure you have a browser that supports JavaScript consoles. for example, I use FireFox with Firebug. It let's me know when there are javascript errors. If that doesnt help, then post your directory and I'll take a look at it and tell you exactly what to change and why it won't work.

[Praful Bagai]

Thanks a lot Sid for replying back and Sorry Sid, I just got busy with some very important work of mine ,that's why couldn't connect with you. Apologies.

I am attaching my reuters.js and reuters.theme.js. Kindly have a look and please help me in solving this mammoth query of mine.

Thank you

Regards

Praful Bagai

[Cristian Ispas]

   1. @James McKinney: <<...If localhost:8080/solr (or wherever your Solr server is) can return Solr responses, then ajax-solr will work>> question: if I want to use Solr at www.example.org, do I need to have a hosting which allows java servlet container like Jetty or Tomcat? it not, how would I be able to setup Solr instance within my website since Solr is a server that works in a java container?

    2. looking at solurl "http://evolvingweb.ca/solr/reuters/" from reuters.js example, does it mean you uploaded your Solr instance to evolvingweb.ca/solr/reuters directory? 

    thanks 

[James McKinney]

For setting up a Solr server, read Solr documentation or ask questions on the Solr mailing lists: http://lucene.apache.org/solr/discussion.html

For ajax-solr, you don't need Jetty, Tomcat or any Java Servlet container. Anything that can serve HTML, CSS, and JavaScript is fine. That includes Apache, Nginx, etc. - anything that a usually webhost will offer you.

[Praful Bagai]

I knew someone would need it again (or atleast I would definately need it, if i format my laptop), so I had written the steps to install Ajax-Solr in a notepad file. Attaching the notepad file. Follow the simple steps and if still you are not able to make it through, just ping me.

PS: I didnt implemented the highlight function. However, I did ask for help in this forum but no-one was kind enough to help me. But, now my project has been successfully submitted, I dont not need the highlight function anymore. So, you have to implement it on your own. :-)

Praful Bagai

[James McKinney]

Thanks Praful, I've added this info the the wiki https://github.com/evolvingweb/ajax-solr/wiki/Tutorial:-Nutch

[J. Gobel]

Hi Praful

Thank you for your short guide, this is what I was looking for ! However I do not follow all the steps.

I am using Nutch 1.6 and Solr 3.6 .. I am able to crawl the web using Nutch and look for the crawled information in Solr 3.6 .. I 'pushed' the index to Solr so there is no need to use the Schema.xml from Nutch.

Now obviously I want to search my crawled data using ajax-solr.. Your guide is a good start but please explain the steps in a bit more detail.
I can follow Step 1 without a problem.
step 2:

2) When you look at one result - you see the possible
fields a solr document (doc) has. For example:

What should I look at? When you look at one result?

Step 3)  Remove the "doc.dateline" from the snippet.

What snippet?

Step 4) Check the following value in your nutch-default.xml:
Check the following, I should change according to your guide or add them .. ?

Looking forward to your reply

J.

Op zondag 25 november 2012 09:05:04 UTC+1 schreef Praful Bagai het volgende:

[J. Gobel]

By mistake I emailed Praful directly.

Just to make it clear on exactly where I am with my Nutch 1.6 and SOlr 3.6

http://wiki.apache.org/nutch/NutchTutorial

In specific : 6. Integrate Solr with Nutch .. I have copied the schema.xml of Nutch to the Solr appropriate folder.

now...

Where do I go from here? What do I need to change ? please explain in babysteps ;-) Just a hint, if you write 'Remove the "doc.dateline"' .. please copy/paste how the lines of code currently are, and then show what it needs to be. I noticed that this tutorial is also on the ajax-solr tutorial pages .. truly, believe it from a total noob .. it is far from being clear. perhaps if you write code it is.

I think that a guide that will follow up on the NutchTutorial on Ajax-Solr will be greatly appreciated. Especially for noobs such as me :-)

Sorry if I offended anyone in here.

Many thanks in advance,

J.

Op zaterdag 29 december 2012 00:23:33 UTC+1 schreef J. Gobel het volgende:

[J. Gobel]

Already found a mistake in my document. please use the one attached to this message please.



Op maandag 26 november 2012 20:22:40 UTC+1 schreef James McKinney het volgende:

[James McKinney]

Thanks J! I've updated the tutorial. 

https://github.com/evolvingweb/ajax-solr/wiki/Tutorial:-Nutch

[getan...@gmail.com]

Hi All,

 

I am new to Solr and trying to set up the Ajax Solr reuters tutorial on my own indexed data on Windows 7 m/c. I am working with Solr 4.5.1 and using default port 8983, I am able to query Solr on my indexed data using URL - http://localhost:8983/solr/#/collection1/query. My schema.xml has all the fields required for Indexing and faceting.

 

However I am facing challenges during the set up

 

1. I downloaded the zip ball from https://github.com/evolvingweb/ajax-solr and unzipped it with name AJAX_HOME on my c drive.

 

SOLR_HOME - C:\solr-4.5.1

AJAX_HOME - C:\solr-4.5.1\AJAX_HOME

 

However I did not find reuters.theme.js script which has been mentioned in this post several times. Can I use the one posted in this group and modify the fields which I am using? something like below

 

AjaxSolr.theme.prototype.tag = function (cat, manu_exact) {

  return $('<a href="#" class="tagcloud_item"/>').text(value).addClass('tagcloud_size_' + weight).click(handler);
};

 

2. I modified the following components. PFA the same

 

C:\solr-4.5.1\AJAX_HOME\examples\reuters-requirejs\js\reuters.js

C:\solr-4.5.1\AJAX_HOME\examples\reuters-requirejs\index.html

 

No changes done to

C:\solr-4.5.1\AJAX_HOME\examples\reuters-requirejs\widgets\TagcloudWidget.js

C:\solr-4.5.1\AJAX_HOME\examples\reuters-requirejs\widgets\CurrentSearchWidget.js

C:\solr-4.5.1\AJAX_HOME\examples\reuters-requirejs\widgets\CalendarWidget.js

 

Now when i open the index.html page in my browser (IE 8 ) C:\solr-4.5.1\AJAX_HOME\examples\reuters-requirejs\index.html, I am able to see a empty results on the right pane (moving circle as if the page is loading) and no tag cloud or calender on my left pane (it more like a static html page)

 

I am not sure if I have missed anything. Also I do not understand why we have mutiple versions of the widget and index html pages. Any help is greatly appreciated.

 

Thanks

Geetanjali

[James McKinney]

reuters.theme.js no longer exists in the most recent version of the example. I don't think there are any references to it in the documentation. All theming was removed in April, to encourage the use of other, existing templating libraries.

The numbered versions of index.html, etc. match the steps in the tutorial at https://github.com/evolvingweb/ajax-solr/wiki/Reuters-tutorial You can just use the non-numbered versions.

If nothing is loading, please use Chrome or Firefox and check the JavaScript console for any errors which will help you debug.

James

--
You received this message because you are subscribed to the Google Groups "ajax-solr" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ajax-solr+...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
<index.html><reuters.js>

[gariashi]

hi Praful,

Am new to Ajax Solr. Am trying to make the reuters example stated on the ajax sold library work. needed help with this. Can you please share your email address so I can email you ?

thanks :)