Issue running Solr 7.3 via systemctl on RHEL 7.6?

19 views
Skip to first unread message

sukdith punjasthitkul

unread,
Jul 23, 2019, 1:37:01 PM7/23/19
to Dataverse Users Community
 Hi Dataverse folks,

 We are currently running a small Dataverse instance at dartmouth
 (dataverse.dartmouth.edu) and while trying to update Dataverse from 4.8.6
 to 4.9, we ran into issues running Solr 7.3 using systemctl.


 * RHEL 7.6 (Red Hat Enterprise Linux Server release 7.6 (Maipo)
 * psql (PostgreSQL) 9.2.24
 * using jdbc drive postgresql-9.2-1004.jdbc4.jar
 * openjdk version "1.8.0_212"
 * OpenJDK Runtime Environment (build 1.8.0_212-b04)
 * OpenJDK 64-Bit Server VM (build 25.212-b04, mixed mode)
 * GlassFish Server Open Source Edition 4.1 (build 13)

 Issue: we've set up solr.service but when we try starting solr by doing
 sudo systemctl start solr

 The logs show it initially starts up and then immediately afterwards shuts
 down, saying there was an error:

ERROR (coreContainerWorkExecutor-2-thread-1) [   ] o.a.s.c.CoreContainer Error waiting for SolrCore to be created
java.util.concurrent.ExecutionException: org.apache.solr.common.SolrException: Unable to create core [collection1]

 We can start solr manually (ie, from solr folder, bin/solr start -m 1g).

 When we used systemctl to stop solr, it also stopped glassfish. In
 addition, glassfish gets stuck in a loop while trying to restart (a series
 of activate/deactivate). Our current workaround is:

 * stop glassfish
 * manually stop solr
 * start glassfish (NOTE: during start, logs showed that solr via systemctl
 also tried--and failed-- to start)
 * manually start solr

 While non-ideal, using this manually-run process allowed us to finish the
 update to 4.9 and then to 4.9.1, but we would like to figure out what could
 be causing this.

 We checked and confirmed that we are using the solrconfig.xml provided from
 the 4.9 update. Attached are copies of the solr.log with the error and the
 solr.service we used. Because we are on RHEL, solr.service is put in
 /etc/systemd/system with solr user as owner:group.

 Any help troubleshooting this would be greatly appreciated!

 Regards,

sukdith "sukie" punjasthitkul

solr.service
solr-system-BROKEN-2.log

Philip Durbin

unread,
Jul 23, 2019, 2:51:20 PM7/23/19
to dataverse...@googlegroups.com
Hi Sukdith,

First, thanks for installing Dataverse! I don't believe you're on the community map so when you get a chance please follow the procedure at http://guides.dataverse.org/en/4.15.1/installation/config.html#putting-your-dataverse-installation-on-the-map-at-dataverse-org

This will help us also add your installation at https://dataverse.org/metrics since you're now running Dataverse 4.9+.

With regard to your specific init script troubles, it's super strange that stopping Solr also stops Glassfish for you.

Contributors keep improving the startup scripts so I would take a look at http://guides.dataverse.org/en/4.15.1/_downloads/solr.service from http://guides.dataverse.org/en/4.15.1/installation/prerequisites.html#solr-init-script to see if works better for you.

Also, the folks who hack on these scripts tend to hang out in #dataverse on freenode. You can join from a browser at http://chat.dataverse.org

A lot of them are in a better position that I am to answer your questions (I'm still getting used to the "new" init system used in CentOS 7) and hopefully they'll reply here.

I hope this helps,

Phil


--
You received this message because you are subscribed to the Google Groups "Dataverse Users Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-commu...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/dataverse-community/d10e44aa-cbce-4413-a3f3-5eb66cf2867c%40googlegroups.com.


--

sukdith punjasthitkul

unread,
Jul 24, 2019, 2:08:29 PM7/24/19
to Dataverse Users Community
Hi Phil, 

Thank you, this was very helpful. The link to the solr.service config from 4.15.1 fixed the issue of Solr starting up and then immediately shutting down. The change in solr.service was:

* changing Type from Simple to Type
* adding an ExecStop 


I can now start solr via systemctl. 
Stopping solr via systemctyl still also stops glassfish though. 
Restarting glassfish also restarted solr, so I'm wondering if there's something in the glassfish service config that's triggering this. 

--sukie--

To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-community+unsub...@googlegroups.com.

Philip Durbin

unread,
Jul 25, 2019, 9:23:54 AM7/25/19
to dataverse...@googlegroups.com
I'm glad you're making progress but all this "stopping service x is also stopping service y" stuff is scary. It smells like a bug.


If you are using the latest scripts and you're still seeing this very odd behavior, please open an issue at https://github.com/IQSS/dataverse/issues

Also, again, you're very welcome in http://chat.dataverse.org if you'd like to discuss this there. The people there who hack on these script are me, donsizemore, and jri. And others there are probably happy to hack on them as well. :)

Thanks,

Phil

To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-commu...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Dataverse Users Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-commu...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/dataverse-community/cf8a6bfc-7c8d-41fe-9511-82589c77fd39%40googlegroups.com.

sukdith punjasthitkul

unread,
Jul 26, 2019, 2:16:02 PM7/26/19
to Dataverse Users Community
Hi Phil, 

The glassfish.service config you listed fixed the issue, thank you :)

--sukie--
To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-community+unsub...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Dataverse Users Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-community+unsub...@googlegroups.com.
Reply all
Reply to author
Forward
0 new messages