[Dspace-tech] Dspace going down
23 views
Skip to first unread message
Blanco, Jose
Aug 24, 2015, 5:17:56 PM8/24/15
to dspac...@lists.sourceforge.net
Not sure what happened last night, but our Dspace instance went down.
We got bombarded by a whole bunch of these errors in the mail:
Exception:
java.lang.ArrayIndexOutOfBoundsException: -2147483648
at
java.text.SimpleDateFormat.subFormat(SimpleDateFormat.java:1049)
at java.text.SimpleDateFormat.format(SimpleDateFormat.java:882)
at java.text.SimpleDateFormat.format(SimpleDateFormat.java:852)
at java.text.DateFormat.format(DateFormat.java:316)
Unfortunately all the dspace.log files got owned by root, so I have no
information from the Dspace application. Once in a while we get a few
dsapce.log files that get owned by root, but not all for one day, that I
have noticed. What could be causing the dspace.log file to get owned by
root?
Our sys admin guy rebooted our server and our Dspace instance came back
up. But dspace.log file was owned by root again, so I deleted it and
restarted tomcat, which changes the ownership to dspace. Everything is
running fine now.
But why would this have happened? In looking at the tomcat log files.
I don't see much more than crawlers accessing our site at what appears
to be a normal rate. I do see a few access to images at our site like:
76.107.55.179 - - [13/Sep/2007:00:01:05 -0500] "GET /styles.css.jsp
HTTP/1.1" 200 13754
76.107.55.179 - - [13/Sep/2007:00:01:05 -0500] "GET /favicon.ico
HTTP/1.1" 200 3978
76.107.55.179 - - [13/Sep/2007:00:01:05 -0500] "GET /utils.js HTTP/1.1"
200 9537
76.107.55.179 - - [13/Sep/2007:00:01:07 -0500] "GET
/image/deepbluetype1.gif HTTP/1.1" 200 849
76.107.55.179 - - [13/Sep/2007:00:01:07 -0500] "GET /image/drop1.gif
HTTP/1.1" 200 7912
76.107.55.179 - - [13/Sep/2007:00:01:07 -0500] "GET
/image/searchbtbl.gif HTTP/1.1" 200 932
76.107.55.179 - - [13/Sep/2007:00:01:07 -0500] "GET /image/arrow.gif
HTTP/1.1" 200 61
76.107.55.179 - - [13/Sep/2007:00:01:07 -0500] "GET /image/spacer.gif
HTTP/1.1" 200 49
Which I guess is OK behavior for crawlers, no? Most or the access are
to items, bitstreams, or browse-title.
Over the weekend we also had an IP that was crawling us, and we got a
whole bunch of SQL connection errors where the connection pool was
exhausted. I'm pretty certain that this IP was the cause because when
we denied it access, the error went away. I looked at the log files,
both dspace and tomcat, and did not see anything that looked suspicious.
The IP was crawling us at a slow rate, and was basically doing a search
and then harvesting the results. If any one has any thoughts on why this
IP would have been causing the SQL connection errors while others
aren't, please let me know.
Many thanks!
Jose
We got bombarded by a whole bunch of these errors in the mail:
Exception:
java.lang.ArrayIndexOutOfBoundsException: -2147483648
at
java.text.SimpleDateFormat.subFormat(SimpleDateFormat.java:1049)
at java.text.SimpleDateFormat.format(SimpleDateFormat.java:882)
at java.text.SimpleDateFormat.format(SimpleDateFormat.java:852)
at java.text.DateFormat.format(DateFormat.java:316)
Unfortunately all the dspace.log files got owned by root, so I have no
information from the Dspace application. Once in a while we get a few
dsapce.log files that get owned by root, but not all for one day, that I
have noticed. What could be causing the dspace.log file to get owned by
root?
Our sys admin guy rebooted our server and our Dspace instance came back
up. But dspace.log file was owned by root again, so I deleted it and
restarted tomcat, which changes the ownership to dspace. Everything is
running fine now.
But why would this have happened? In looking at the tomcat log files.
I don't see much more than crawlers accessing our site at what appears
to be a normal rate. I do see a few access to images at our site like:
76.107.55.179 - - [13/Sep/2007:00:01:05 -0500] "GET /styles.css.jsp
HTTP/1.1" 200 13754
76.107.55.179 - - [13/Sep/2007:00:01:05 -0500] "GET /favicon.ico
HTTP/1.1" 200 3978
76.107.55.179 - - [13/Sep/2007:00:01:05 -0500] "GET /utils.js HTTP/1.1"
200 9537
76.107.55.179 - - [13/Sep/2007:00:01:07 -0500] "GET
/image/deepbluetype1.gif HTTP/1.1" 200 849
76.107.55.179 - - [13/Sep/2007:00:01:07 -0500] "GET /image/drop1.gif
HTTP/1.1" 200 7912
76.107.55.179 - - [13/Sep/2007:00:01:07 -0500] "GET
/image/searchbtbl.gif HTTP/1.1" 200 932
76.107.55.179 - - [13/Sep/2007:00:01:07 -0500] "GET /image/arrow.gif
HTTP/1.1" 200 61
76.107.55.179 - - [13/Sep/2007:00:01:07 -0500] "GET /image/spacer.gif
HTTP/1.1" 200 49
Which I guess is OK behavior for crawlers, no? Most or the access are
to items, bitstreams, or browse-title.
Over the weekend we also had an IP that was crawling us, and we got a
whole bunch of SQL connection errors where the connection pool was
exhausted. I'm pretty certain that this IP was the cause because when
we denied it access, the error went away. I looked at the log files,
both dspace and tomcat, and did not see anything that looked suspicious.
The IP was crawling us at a slow rate, and was basically doing a search
and then harvesting the results. If any one has any thoughts on why this
IP would have been causing the SQL connection errors while others
aren't, please let me know.
Many thanks!
Jose
Christian Voelker
Aug 24, 2015, 5:17:57 PM8/24/15
to Blanco, Jose, dspac...@lists.sourceforge.net
Hello Jose,
I dont believe in a specific IP to be the culprit for
exhausting your connection pool. This is a bit like
thinking in terms of enemies out there on the hostile
internet. I would follow the issue with you logfiles
being owned by root though. That looks serious to me.
There are logfiles at different levels and you might
find out more if you consult all of them e.g. the
dspace log, tomcat log (not written by default), the
apache logs, either access log or error log. postgres
will also write logs. Another way is to take a look
from the outside.
Are you monitoring your machine?
We use the most simplistic thing available, a free
service called mon.itor.us which ticks our machine
on port 80 every fifteen minutes or so and reports
availybility and average round trip time, which is
not much but gives me a baseline so that I know
whether the measures I take are for better of for
worse. You might find out that there is a time pat-
tern, e.g. it gets slow every day at 10pm and you
find similar pattern in the logs. That would be a
better hint. I dont know your platform, but there
has to be some basic facility to log cpu and memory
usage or connection attempts at the network inter-
face level. Maybe your firewall knows about that.
There is cool new tool called dstat for Linux that
I would like to use, but it is not available on
MacOS.
There Error message you posted does not tell me a thing.
If you tell us some more about your system setup, then
maybe, somebody can tell you more, how to track this
down. Beside OS and version numbers of all parts of
your software, regarding the SQL connection errors,
dont forget to include the JDBC connector version.
Is the DB on the same machine? You recently mentioned
that you had issues with large files. Now you tell,
that crawlers also load bitstreams. If your DB is
not on the same machine but communicating over the
same physical interface, the download of bitstreams
might compete for bandwidth with your DB connection.
Maybe this solution is much too complicated, but
you might understand which kind of things are inter-
esting to know about your setup to discuss the type
of problems you reported.
Bye, Christian
I dont believe in a specific IP to be the culprit for
exhausting your connection pool. This is a bit like
thinking in terms of enemies out there on the hostile
internet. I would follow the issue with you logfiles
being owned by root though. That looks serious to me.
There are logfiles at different levels and you might
find out more if you consult all of them e.g. the
dspace log, tomcat log (not written by default), the
apache logs, either access log or error log. postgres
will also write logs. Another way is to take a look
from the outside.
Are you monitoring your machine?
We use the most simplistic thing available, a free
service called mon.itor.us which ticks our machine
on port 80 every fifteen minutes or so and reports
availybility and average round trip time, which is
not much but gives me a baseline so that I know
whether the measures I take are for better of for
worse. You might find out that there is a time pat-
tern, e.g. it gets slow every day at 10pm and you
find similar pattern in the logs. That would be a
better hint. I dont know your platform, but there
has to be some basic facility to log cpu and memory
usage or connection attempts at the network inter-
face level. Maybe your firewall knows about that.
There is cool new tool called dstat for Linux that
I would like to use, but it is not available on
MacOS.
There Error message you posted does not tell me a thing.
If you tell us some more about your system setup, then
maybe, somebody can tell you more, how to track this
down. Beside OS and version numbers of all parts of
your software, regarding the SQL connection errors,
dont forget to include the JDBC connector version.
Is the DB on the same machine? You recently mentioned
that you had issues with large files. Now you tell,
that crawlers also load bitstreams. If your DB is
not on the same machine but communicating over the
same physical interface, the download of bitstreams
might compete for bandwidth with your DB connection.
Maybe this solution is much too complicated, but
you might understand which kind of things are inter-
esting to know about your setup to discuss the type
of problems you reported.
Bye, Christian
Lawrence K. Chen, P.Eng.
Aug 24, 2015, 5:17:58 PM8/24/15
to dspac...@lists.sourceforge.net
Actually, our dspace went down last night as well. I haven't looked to
see why it might have, but I have it being monitored with our nagios
system. After the last outage, I added an event-handler to
automatically restart it. It worked.
Re: dspace.log owned by root. We had this problem a while back, and it
was because the rc script was initially starting dspace as root. I
changed the rc script to su appropriately when starting the tomcat.
I have noticed previous outages occur when our dspace is being heavily
crawled. Including once by our Google search appliance....
Lawrence
> -------------------------------------------------------------------------
> This SF.net email is sponsored by: Microsoft
> Defy all challenges. Microsoft(R) Visual Studio 2005.
> http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
> _______________________________________________
> DSpace-tech mailing list
> DSpac...@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/dspace-tech
>
>
--
Who: Lawrence K. Chen, P.Eng. What: Unix Systems Administrator
For: Enterprise Server Technologies (EST) Where: 11 Hale Library
Snail: Computing and Telecommunications Services (IT-CTS)
Kansas State University, 109 East Stadium, Manhattan, KS 66506
Phone: (785)532-4916 Fax: (785)532-3515 Email: lkc...@k-state.edu
* * * * * * * * Web: http://www-personal.k-state.edu/~lkchen * * * * * * * *
see why it might have, but I have it being monitored with our nagios
system. After the last outage, I added an event-handler to
automatically restart it. It worked.
Re: dspace.log owned by root. We had this problem a while back, and it
was because the rc script was initially starting dspace as root. I
changed the rc script to su appropriately when starting the tomcat.
I have noticed previous outages occur when our dspace is being heavily
crawled. Including once by our Google search appliance....
Lawrence
> This SF.net email is sponsored by: Microsoft
> Defy all challenges. Microsoft(R) Visual Studio 2005.
> http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
> _______________________________________________
> DSpace-tech mailing list
> DSpac...@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/dspace-tech
>
>
--
Who: Lawrence K. Chen, P.Eng. What: Unix Systems Administrator
For: Enterprise Server Technologies (EST) Where: 11 Hale Library
Snail: Computing and Telecommunications Services (IT-CTS)
Kansas State University, 109 East Stadium, Manhattan, KS 66506
Phone: (785)532-4916 Fax: (785)532-3515 Email: lkc...@k-state.edu
* * * * * * * * Web: http://www-personal.k-state.edu/~lkchen * * * * * * * *
Blanco, Jose
Aug 24, 2015, 5:18:03 PM8/24/15
to dspac...@lists.sourceforge.net
For some reason the dspace.log file is not being written to by dspace.
I ran filter-media around 12:30, and I lost track of when it ended, but
at 1:05 is the last entry in dspace.log from filter-media. Is there any
reason why dspace would stop writing to the dspace.log file under these
circumstances? The permissions look fine:
-rw-r--r-- 1 dspace dspace 548742 Sep 13 13:05 dspace.log
I know if I restart tomcat it would start writing again, but why would
it stop? Did running filter-media have something to do with it?
Thanks!
Jose
I ran filter-media around 12:30, and I lost track of when it ended, but
at 1:05 is the last entry in dspace.log from filter-media. Is there any
reason why dspace would stop writing to the dspace.log file under these
circumstances? The permissions look fine:
-rw-r--r-- 1 dspace dspace 548742 Sep 13 13:05 dspace.log
I know if I restart tomcat it would start writing again, but why would
it stop? Did running filter-media have something to do with it?
Thanks!
Jose
Graham Triggs
Aug 24, 2015, 5:18:04 PM8/24/15
to Blanco, Jose, dspac...@lists.sourceforge.net
Hi,
It's possible that filter-media had something to do with it, but under
normal circumstances it shouldn't.
What is likely here is that something has renamed or replaced the
dspace.log file that Tomcat was writing to. As Tomcat retains an open
handle on the log file, it will continue writing to the 'old' file, even
though you can only see the new one.
If you have a dspace.log.1, take a look at it - there is a distinct
possibility that your 'missing' log entries may be in there.
G
For more information please visit http://www.postini.com
It's possible that filter-media had something to do with it, but under
normal circumstances it shouldn't.
What is likely here is that something has renamed or replaced the
dspace.log file that Tomcat was writing to. As Tomcat retains an open
handle on the log file, it will continue writing to the 'old' file, even
though you can only see the new one.
If you have a dspace.log.1, take a look at it - there is a distinct
possibility that your 'missing' log entries may be in there.
G
> -------------------------------------------------------------------------
> This SF.net email is sponsored by: Microsoft
> Defy all challenges. Microsoft(R) Visual Studio 2005.
> http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
> _______________________________________________
> DSpace-tech mailing list
> DSpac...@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/dspace-tech
>
This email has been scanned by Postini.
> This SF.net email is sponsored by: Microsoft
> Defy all challenges. Microsoft(R) Visual Studio 2005.
> http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
> _______________________________________________
> DSpace-tech mailing list
> DSpac...@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/dspace-tech
>
For more information please visit http://www.postini.com
Blanco, Jose
Aug 24, 2015, 5:18:05 PM8/24/15
to Graham Triggs, dspac...@lists.sourceforge.net
Yes. I see it. It's writing to dspace.log.8. What could have renamed
the handle?
the handle?
Mika Stenberg
Aug 24, 2015, 5:18:09 PM8/24/15
to dspac...@lists.sourceforge.net
Im having issues with DSpace statistics. My log file seems ok but my
statistics are 4 months old and DSpace wont update them. What might be
causing this? I have set the periodic stat-runs on cron as instructed in
the documentation.
-Mika
statistics are 4 months old and DSpace wont update them. What might be
causing this? I have set the periodic stat-runs on cron as instructed in
the documentation.
-Mika
Christian Voelker
Aug 24, 2015, 5:18:11 PM8/24/15
to Mika Stenberg, dspac...@lists.sourceforge.net
Am 14.09.2007 um 08:44 schrieb Mika Stenberg:
> Im having issues with DSpace statistics. My log file seems ok but my
> statistics are 4 months old and DSpace wont update them. What might be
> causing this?
If no, see above, especially directories to write to,
and whether there are files with the same name
blocking the creation of new reports,
maybe left over from a former crash?
If yes, take a look into your cron system.
Do other jobs run? Problems with su or sudo?
Is it in the personal or system wide crontab?
IIRC the script does not need to be executable
by itself because the actual job is run through
dsrun, but I may be wrong on this.
Bye, Christian
Nason Bimbe
Aug 26, 2015, 10:59:21 AM8/26/15
to dspace-tech
I am running Dspace 3.1 and I am interested in getting the number of downloads from SOLR.
and when I look at the results in the facets section, the numbers seems too high for a month's data. Looking into some of the docs returned I can see that some are bots yet my config says bots should be omitted.
Can some please help me explain this? I have looked at http://iplists.com/google.txt and this ip address is not on the list. What is going on here?
[config snippet]
# control solr statistics querying to filter out spider IPs
# false by default
#query.filter.spiderIp = false
# control solr statistics querying to look at "isBot" field to determine
# if record is a bot. true by default.
#query.filter.isBot = true
[result snippet]
<doc>
<float name="score">3.504466</float>
<arr name="bundleName"><str>ORIGINAL</str></arr>
<str name="city">Mountain View</str>
<str name="continent">NA</str>
<str name="countryCode">US</str>
<str name="dns">crawl-66-249-75-222.googlebot.com.</str>
<int name="id">1673</int>
<str name="ip">66.249.75.222</str>
<bool name="isBot">false</bool>
<float name="latitude">37.419205</float>
<float name="longitude">-122.0574</float>
<arr name="owningColl"><int>12</int></arr>
<arr name="owningComm"><int>9</int><int>2</int><int>2</int></arr>
<arr name="owningItem"><int>829</int></arr>
<str name="statistics_type">view</str>
<date name="time">2013-05-08T17:20:01.295Z</date>
<int name="type">0</int>
<str name="uid">7f7a6135-6c0a-4e2d-b8a3-2c022c2ab20e</str>
<str name="userAgent">Mozilla/5.0 (iPhone; U; CPU iPhone OS 4_1 like Mac OS X; en-us) AppleWebKit/532.9 (KHTML, like Gecko) Version/4.0.5 Mobile/8B117 Safari/6531.22.7 (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)</str>
</doc>
<doc>
<float name="score">3.504466</float>
<arr name="bundleName"><str>ORIGINAL</str></arr>
<str name="city">Moscow</str>
<str name="continent">EU</str>
<str name="countryCode">RU</str>
<str name="dns">spider-178-154-243-100.yandex.com.</str>
<int name="id">1056</int>
<str name="ip">178.154.243.100</str>
<bool name="isBot">false</bool>
<float name="latitude">55.752197</float>
<float name="longitude">37.6156</float>
<arr name="owningColl"><int>10</int></arr>
<arr name="owningComm"><int>1</int></arr>
<arr name="owningItem"><int>175</int></arr>
<str name="statistics_type">view</str>
<date name="time">2013-05-08T16:52:12.106Z</date>
<int name="type">0</int>
<str name="uid">1943f4fe-3ce8-4793-8c3e-3c5e34fa7975</str>
<str name="userAgent">Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)</str>
</doc>
Thanks
Nason
Nason Bimbe
Aug 26, 2015, 10:59:35 AM8/26/15
to dspac...@lists.sourceforge.net
Hello, People,
Has anyone got an answer to this problem. I am kind of stuck here.
Regards
Nason
--
View this message in context: http://dspace.2283337.n4.nabble.com/DSpace-statistics-tp4664534p4664559.html
Sent from the DSpace - Tech mailing list archive at Nabble.com.
Has anyone got an answer to this problem. I am kind of stuck here.
Regards
Nason
--
View this message in context: http://dspace.2283337.n4.nabble.com/DSpace-statistics-tp4664534p4664559.html
Sent from the DSpace - Tech mailing list archive at Nabble.com.
Andrea Schweer
Aug 26, 2015, 10:59:55 AM8/26/15
to Nason Bimbe, dspace-tech
Hi Nason,
On 01/06/13 02:01, Nason Bimbe wrote:
I am running Dspace 3.1 and I am interested in getting the number of downloads from SOLR.
and when I look at the results in the facets section, the numbers seems too high for a month's data. Looking into some of the docs returned I can see that some are bots yet my config says bots should be omitted.
The dspace config setting doesn't apply to manual queries like the
one you're running. You will need to manually append
&fq=-isBot:true to your query to exclude downloads by known
bots.
http://localhost:8080/solr/statistics/select?indent=on&version=2.2&start=0&rows=100&fl=*%2Cscore&qt=standard&wt=standard&explainOther=&hl.fl=&facet=true&facet.field=owningComm&q=type:0&fq=-isBot:true
You may wish to run
[dspace]/bin/dspace stats-util -u
[dspace]/bin/dspace stats-util -i
beforehand if you aren't running these regularly -- the first one ensures your bot IP lists are at the most recent version and the second one goes through all stats and marks hits from these IPs with "isBot:true".
Finally, you say you want a month's data yet your query shows no restriction by date. So unless your stats data only covers 1 month, your results will be for a much longer time period.
cheers,
Andrea
http://localhost:8080/solr/statistics/select?indent=on&version=2.2&start=0&rows=100&fl=*%2Cscore&qt=standard&wt=standard&explainOther=&hl.fl=&facet=true&facet.field=owningComm&q=type:0&fq=-isBot:true
You may wish to run
[dspace]/bin/dspace stats-util -u
[dspace]/bin/dspace stats-util -i
beforehand if you aren't running these regularly -- the first one ensures your bot IP lists are at the most recent version and the second one goes through all stats and marks hits from these IPs with "isBot:true".
Finally, you say you want a month's data yet your query shows no restriction by date. So unless your stats data only covers 1 month, your results will be for a much longer time period.
cheers,
Andrea
-- Dr Andrea Schweer IRR Technical Specialist, ITS Information Systems The University of Waikato, Hamilton, New Zealand
Nason Bimbe
Aug 26, 2015, 11:00:05 AM8/26/15
to Andrea Schweer, dspace-tech
Hi Andrea,
Thank you very much for your reply. Yes the stats data was for one month only (May) that's why I did not add the date filter.
I have run
[dspace]/bin/dspace stats-util -u
[dspace]/bin/dspace stats-util -i
[dspace]/bin/dspace stats-util -i
Then the SOLR query and nothing has changed.
If I may ask do you now the query used by the 'View Usage Statistics' link in XMLUI because the stats displayed look more reasonable, in other words can I replicate the query by querying SOLR directly? Would you also be in a position to explain the possible values for type in the query for me please? For example I have seen posts describing number of downloads as being of type = 0 in SOLR.
Thanking you in advance.
Nason
helix84
Aug 26, 2015, 11:00:06 AM8/26/15
to Andrea Schweer, dspace-tech
Hi Andrea,
I'll use this opportunity to ask a related commonly asked question -
how do I turn on logging of queries sent to Solr? I assume they would
appear in the Tomcat log.
Regards,
~~helix84
Compulsory reading: DSpace Mailing List Etiquette
https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette
I'll use this opportunity to ask a related commonly asked question -
how do I turn on logging of queries sent to Solr? I assume they would
appear in the Tomcat log.
Regards,
~~helix84
Compulsory reading: DSpace Mailing List Etiquette
https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette
Andrea Schweer
Aug 26, 2015, 11:00:12 AM8/26/15
to hel...@centrum.sk, dspace-tech
Hi,
On 05/06/13 02:25, helix84 wrote:
I'll use this opportunity to ask a related commonly asked question - how do I turn on logging of queries sent to Solr? I assume they would appear in the Tomcat log.
On my RHEL boxes (DSpace 1.8.2, Tomcat 6), these appear in
/var/log/tomcat/catalina.out like so:
I can't remember doing anything to enable this, so I assume this is the standard behaviour.
Jun 5, 2013 9:22:45 AM org.apache.solr.core.SolrCore execute
INFO: [search] webapp=/solr path=/select
params={facet=true&facet.offset=0&facet.mincount=1&f.language_filter.facet.limit=11&wt=javabin&rows=0&version=2&f.language_filter.facet.sort=count&f.subject_filter.facet.sort=count&facet.query=dateIssued.year:[2010+TO+2013]&facet.query=dateIssued.year:[2000+TO+2009]&facet.query=dateIssued.year:[1990+TO+1999]&facet.query=dateIssued.year:[1980+TO+1989]&facet.query=dateIssued.year:[1970+TO+1979]&facet.query=dateIssued.year:[1966+TO+1969]&start=0&f.author_filter.facet.limit=11&q=*:*&f.subject_filter.facet.limit=11&f.author_filter.facet.sort=count&facet.field=author_filter&facet.field=subject_filter&facet.field=language_filter}
hits=6792 status=0 QTime=7I can't remember doing anything to enable this, so I assume this is the standard behaviour.
Andrea Schweer
Aug 26, 2015, 11:00:13 AM8/26/15
to hel...@centrum.sk, dspace-tech
Hi again,
On 05/06/13 09:28, Andrea Schweer wrote:
> On 05/06/13 02:25, helix84 wrote:
>> I'll use this opportunity to ask a related commonly asked question -
>> how do I turn on logging of queries sent to Solr? I assume they would
>> appear in the Tomcat log.
>
> On my RHEL boxes (DSpace 1.8.2, Tomcat 6), these appear in
> /var/log/tomcat/catalina.out like so:
Nevermind -- these are the discovery queries. That's what I get for
responding before I've had a coffee.
I suppose Kevin, Mark D or someone else at @mire might know why the
logging behaviour is different for discovery vs the solr stats.
On 05/06/13 09:28, Andrea Schweer wrote:
> On 05/06/13 02:25, helix84 wrote:
>> I'll use this opportunity to ask a related commonly asked question -
>> how do I turn on logging of queries sent to Solr? I assume they would
>> appear in the Tomcat log.
>
> On my RHEL boxes (DSpace 1.8.2, Tomcat 6), these appear in
> /var/log/tomcat/catalina.out like so:
responding before I've had a coffee.
I suppose Kevin, Mark D or someone else at @mire might know why the
logging behaviour is different for discovery vs the solr stats.
Andrea Schweer
Aug 26, 2015, 11:00:14 AM8/26/15
to Nason Bimbe, dspace-tech
Hi Nason,
On 05/06/13 02:18, Nason Bimbe wrote:
Thank you very much for your reply. Yes the stats data was for one month only (May) that's why I did not add the date filter.
I have run[dspace]/bin/dspace stats-util -u
[dspace]/bin/dspace stats-util -i
Then the SOLR query and nothing has changed.
Did you add &fq=-isBot:true to your query? You need to add this,
otherwise "bot" hits will still be included in your query results
even after the two commands above.
If I may ask do you now the query used by the 'View Usage Statistics' link in XMLUI because the stats displayed look more reasonable, in other words can I replicate the query by querying SOLR directly?
I've customised the stats for "my" repository so much that I have no
idea what's normally on that page. From the demo server, it looks
like this lists the 10 items with the most downloads. To get those,
you'd use:
http://localhost:8080/solr/statistics/select?indent=true&rows=0&q=type:0&facet=true&facet.limit=10&facet.field=owningItem&fq=-isBot:true
The English version of this query is something like
http://localhost:8080/solr/statistics/select?indent=true&rows=0&q=type:0&facet=true&facet.limit=10&facet.field=owningItem&fq=-isBot:true
The English version of this query is something like
- Show me the 10 most frequent (facet=true&facet.limit=10)
- items (facet.field=owningItem)
- where "frequent" means going by bitstream downloads (q=type:0)
- but I don't want to see any individual events in additio(rows=0)
- and exclude all hits by bots (fq=-isBot:true) -- this counts
only events that don't have an isBot value at all or where the
isBot value is false
Would you also be in a position to explain the possible values for type in the query for me please? For example I have seen posts describing number of downloads as being of type = 0 in SOLR.
The type refers to the constants in Constants.java
(https://github.com/DSpace/DSpace/blob/master/dspace-api/src/main/java/org/dspace/core/Constants.java).
Probably the most important ones are 0 (bitstream downloads) and 2
(item page views). The id is always the internal id of the object
concerned; for bitstreams, owningItem will hold the id of the
bitstream's item. All other fields for view/download events look
pretty self-explanatory to me; please ask if you have specific
questions.
There is documentation about all fields here: https://wiki.duraspace.org/display/DSDOC3x/DSpace+Statistics
In particular, this section explains what fields are available for querying / faceting: https://wiki.duraspace.org/display/DSDOC3x/DSpace+Statistics#DSpaceStatistics-Whatisexactlybeinglogged?
And this section explains a bit about custom queries: https://wiki.duraspace.org/display/DSDOC3x/DSpace+Statistics#DSpaceStatistics-CustomReporting-QueryingSOLRDirectly though much of figuring this out is just getting your head around Solr. Again, ask if you have specific questions.
There is documentation about all fields here: https://wiki.duraspace.org/display/DSDOC3x/DSpace+Statistics
In particular, this section explains what fields are available for querying / faceting: https://wiki.duraspace.org/display/DSDOC3x/DSpace+Statistics#DSpaceStatistics-Whatisexactlybeinglogged?
And this section explains a bit about custom queries: https://wiki.duraspace.org/display/DSDOC3x/DSpace+Statistics#DSpaceStatistics-CustomReporting-QueryingSOLRDirectly though much of figuring this out is just getting your head around Solr. Again, ask if you have specific questions.
Nason Bimbe
Aug 26, 2015, 11:00:47 AM8/26/15
to Andrea Schweer, dspace-tech
Hi Andrea,
Thank you very much, it is what I needed and the information has been helpful.
Regards
Nason
Lewatle Phaladi
Aug 26, 2015, 12:17:26 PM8/26/15
to dspac...@lists.sourceforge.net
Hi DSpace Team,
I am configuring dspace stats module and I am encountering some problems while setting it up, after I followed online documentations on how to enable the module I found that GeoLiteCity.dat is not available in /dspace3.2/config directory and I downloaded it manually and restarted tomcat as the documents indicates and when I refresh dspace on my browser I get blank page, I have also done symbolic link for all my dspace webapps to Tomcat, I have followed the following documentations https://wiki.duraspace.org/display/DSDOC3x/DSpace+Statistics and http://wiki.lib.sun.ac.za/index.php/SUNScholar/SOLR_Statistics/3.2
Regards,
Lewatle
|
Hilton Gibson
Aug 26, 2015, 12:17:27 PM8/26/15
to Lewatle Phaladi, dspac...@lists.sourceforge.net
Hi Lewatle
Did this start to happen after an upgrade or are you talking about a totally new system?
If it's an upgrade, then these links are important;
Cheers
hg
Hilton Gibson
Ubuntu Linux Systems Administrator
JS Gericke Library
Room 1025D
Stellenbosch University
Private Bag X5036
Stellenbosch
7599
South Africa
Tel: +27 21 808 4100 | Cell: +27 84 646 4758
------------------------------------------------------------------------------
Managing the Performance of Cloud-Based Applications
Take advantage of what the Cloud has to offer - Avoid Common Pitfalls.
Read the Whitepaper.
http://pubads.g.doubleclick.net/gampad/clk?id=121051231&iu=/4140/ostg.clktrk
_______________________________________________
DSpace-tech mailing list
DSpac...@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/dspace-tech
List Etiquette: https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette
Lewatle Phaladi
Aug 26, 2015, 12:17:28 PM8/26/15
to Hilton Gibson, dspac...@lists.sourceforge.net
Good Day Hilton,
It happened after upgrade from 1.8.2 to 3.2
Regards,
Lewatle
From: Hilton Gibson [mailto:hilton...@gmail.com]
Sent: 05 February 2014 03:13 PM
To: Lewatle Phaladi
Cc: dspac...@lists.sourceforge.net
Hilton Gibson
Aug 26, 2015, 12:17:30 PM8/26/15
to Lewatle Phaladi, dspac...@lists.sourceforge.net
Ok. Did you do this before the upgrade: http://wiki.lib.sun.ac.za/index.php/SUNScholar/SOLR_Statistics/3.2#Please_note: ??
Hilton Gibson
Ubuntu Linux Systems Administrator
JS Gericke Library
Room 1025D
Stellenbosch University
Private Bag X5036
Stellenbosch
7599
South Africa
Tel: +27 21 808 4100 | Cell: +27 84 646 4758
Lewatle Phaladi
Aug 26, 2015, 12:17:31 PM8/26/15
to Hilton Gibson, dspac...@lists.sourceforge.net
Yes I did
Lewatle Phaladi
Aug 26, 2015, 12:17:33 PM8/26/15
to Hilton Gibson, dspac...@lists.sourceforge.net
Not yet
From: Hilton Gibson [mailto:hilton...@gmail.com]
Sent: 05 February 2014 04:00 PM
To: Lewatle Phaladi
Cc: dspac...@lists.sourceforge.net
Subject: Re: [Dspace-tech] DSpace statistics
Ok. Did you check SOLR is working?
Cheers
hg
Hilton Gibson
Ubuntu Linux Systems Administrator
JS Gericke Library
Room 1025D
Stellenbosch University
Private Bag X5036
Stellenbosch
7599
South Africa
Tel: +27 21 808 4100 | Cell: +27 84 646 4758
Hilton Gibson
Aug 26, 2015, 12:17:33 PM8/26/15
to Lewatle Phaladi, dspac...@lists.sourceforge.net
Ok. Did you check SOLR is working?
Cheers
hg
Hilton Gibson
Ubuntu Linux Systems Administrator
JS Gericke Library
Room 1025D
Stellenbosch University
Private Bag X5036
Stellenbosch
7599
South Africa
Tel: +27 21 808 4100 | Cell: +27 84 646 4758
On 5 February 2014 15:56, Lewatle Phaladi <Lewatle...@wits.ac.za> wrote:
Hilton Gibson
Aug 26, 2015, 12:17:34 PM8/26/15
to Lewatle Phaladi, dspac...@lists.sourceforge.net
Ok. It should be your first troubleshooting step.
Cheers
hg
Hilton Gibson
Ubuntu Linux Systems Administrator
JS Gericke Library
Room 1025D
Stellenbosch University
Private Bag X5036
Stellenbosch
7599
South Africa
Tel: +27 21 808 4100 | Cell: +27 84 646 4758
Hilton Gibson
Aug 26, 2015, 12:17:45 PM8/26/15
to Lewatle Phaladi, dspac...@lists.sourceforge.net
Ok. Great. What was the fix?
Hilton Gibson
Ubuntu Linux Systems Administrator
JS Gericke Library
Room 1025D
Stellenbosch University
Private Bag X5036
Stellenbosch
7599
South Africa
Tel: +27 21 808 4100 | Cell: +27 84 646 4758
On 6 February 2014 11:13, Lewatle Phaladi <Lewatle...@wits.ac.za> wrote:
Hi Hilton,
Thanks I have stats working at the moment.
Lewatle Phaladi
Aug 26, 2015, 12:17:45 PM8/26/15
to Hilton Gibson, dspac...@lists.sourceforge.net
Hi Hilton,
Thanks I have stats working at the moment.
Regards,
Lewatle Phaladi
Aug 26, 2015, 12:17:46 PM8/26/15
to Hilton Gibson, dspac...@lists.sourceforge.net
The problem was on ports clash while I changed port on tomcat setting file from 8080 to 80 it was still using 8080 instead of 80, and on /config/modules/solr-statistics.cfg file I put removed 8080, now is on 80 and I can see few info on test system showing stats
Reply all
Reply to author
Forward
0 new messages