org.apache.solr.client.solrj.SolrServerException: IOException occurred when talking to server at: {zookeeper_solr_url}/authority

[Manish Kumar]

Hi there,

I installed fresh dspace-backend-7.6.1 on dev server with google storage integration for asset store and instaed of standalone solr we have used solr cloud and through zookeeper we are connecting to solr from dspace.

UI and Backend both are running fine except solr, we are unable to push data in solr, no error on ui but data is not reflecting on solr instance. I found below error logs in dspace.log.

" 2024-04-03 12:18:38,951 INFO  unknown unknown org.ehcache.jsr107.Eh107CacheManager @ Registering Ehcache MBean javax.cache:type=CacheStatistics,CacheManager=file./dspace/config/ehcache.xml,Cache=canvasdimensions

2024-04-03 12:18:39,616 INFO  unknown unknown org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor @ Autowired annotation should only be used on methods with parameters: public java.lang.String org.dspace.storage.bitstore.S3BitStoreService.getBucketName()
2024-04-03 12:18:40,510 INFO  unknown unknown org.dspace.importer.external.service.ImportService @ Loading 6 import sources.
2024-04-03 12:18:41,230 INFO  unknown unknown org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor @ Autowired annotation is not supported on static methods: public static void org.dspace.rdf.storage.DOIURIGenerator.setFallback(org.dspace.rdf.storage.URIGenerator)
2024-04-03 12:18:41,235 INFO  unknown unknown org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor @ Autowired annotation is not supported on static methods: public static void org.dspace.rdf.storage.DOIHandleURIGenerator.setFallback(org.dspace.rdf.storage.URIGenerator)
2024-04-03 12:18:41,457 INFO  unknown unknown org.dspace.service.impl.ClientInfoServiceImpl @ Trusted proxies (configure via 'proxies.trusted.ipranges'): 34.107.229.188-34.107.229.188, 127.0.0.1-127.0.0.1
2024-04-03 12:18:41,468 INFO  unknown unknown org.dspace.statistics.SolrStatisticsCore @ solr-statistics.server:  {zookeeper_solr_url}/statistics
2024-04-03 12:18:41,468 INFO  unknown unknown org.dspace.statistics.SolrStatisticsCore @ usage-statistics.dbfile:  null
2024-04-03 12:18:41,479 ERROR unknown unknown org.dspace.statistics.SolrLoggerServiceImpl @ java.lang.IllegalStateException: The required 'dbfile' configuration is missing in usage-statistics.cfg!
2024-04-03 12:18:41,698 INFO  unknown unknown org.dspace.app.rest.Application @ No active profile set, falling back to 1 default profile: "default"
2024-04-03 12:18:44,042 INFO  unknown unknown org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext @ Root WebApplicationContext: initialization completed in 2329 ms
2024-04-03 12:18:46,542 INFO  unknown unknown org.springframework.aop.framework.CglibAopProxy @ Unable to proxy interface-implementing method [public final java.lang.Iterable org.dspace.app.rest.repository.DSpaceRestRepository.findAll(org.springframework.data.domain.Sort)] because it is marked as final: Consider using interface-based JDK proxies instead!
2024-04-03 12:18:46,543 INFO  unknown unknown org.springframework.aop.framework.CglibAopProxy @ Unable to proxy interface-implementing method [public final java.lang.Iterable org.dspace.app.rest.repository.DSpaceRestRepository.findAll()] because it is marked as final: Consider using interface-based JDK proxies instead!
2024-04-03 12:18:46,761 INFO  unknown unknown org.dspace.storage.rdbms.DatabaseUtils @ Loading Flyway DB migrations from: classpath:org/dspace/storage/rdbms/sqlmigration/postgres, classpath:org/dspace/storage/rdbms/migration, classpath:org/dspace/storage/rdbms/xmlworkflow
.....
2024-04-03 12:18:48,877 INFO  unknown unknown org.ehcache.jsr107.ConfigurationMerger @ Cache org.dspace.alerts.SystemWideAlert will use expiry configuration from template default
2024-04-03 12:18:48,882 INFO  unknown unknown org.ehcache.core.EhcacheManager @ Cache 'org.dspace.alerts.SystemWideAlert' created in Eh107InternalCacheManager.
2024-04-03 12:18:49,923 INFO  unknown unknown org.dspace.discovery.SolrSearchCore @ Zookeeper URL :-  {zookeeper_solr_url}
2024-04-03 12:18:49,924 INFO  unknown unknown org.dspace.discovery.SolrSearchCore @ Zookeeper defaultCollection :-  search
2024-04-03 12:18:49,953 INFO  unknown unknown org.dspace.discovery.SolrSearchCore @ CloudSolrClient data : org.apache.solr.client.solrj.impl.CloudSolrClient@6a7fb0f6
2024-04-03 12:18:49,979 INFO  unknown unknown org.apache.zookeeper.ZooKeeper @ Client environment:zookeeper.version=3.4.14-4c25d480e66aadd371de8bd2fd8da255ac140bcf, built on 03/06/2019 16:18 GMT
2024-04-03 12:18:49,979 INFO  unknown unknown org.apache.zookeeper.ZooKeeper @ Client environment:host.name=dspace-backend-test-6cb6f66724-h5xyz
2024-04-03 12:18:49,979 INFO  unknown unknown org.apache.zookeeper.ZooKeeper @ Client environment:java.version=11.0.22
2024-04-03 12:18:49,979 INFO  unknown unknown org.apache.zookeeper.ZooKeeper @ Client environment:java.vendor=Eclipse Adoptium
2024-04-03 12:18:49,979 INFO  unknown unknown org.apache.zookeeper.ZooKeeper @ Client environment:java.home=/opt/java/openjdk
2024-04-03 12:18:49,979 INFO  unknown unknown org.apache.zookeeper.ZooKeeper @ Client environment:java.class.path=/usr/local/tomcat/bin/bootstrap.jar:/usr/local/tomcat/bin/tomcat-juli.jar
2024-04-03 12:18:49,979 INFO  unknown unknown org.apache.zookeeper.ZooKeeper @ Client environment:java.library.path=/usr/local/tomcat/native-jni-lib:/usr/java/packages/lib:/usr/lib64:/lib64:/lib:/usr/lib
2024-04-03 12:18:49,979 INFO  unknown unknown org.apache.zookeeper.ZooKeeper @ Client environment:java.io.tmpdir=/usr/local/tomcat/temp
2024-04-03 12:18:49,979 INFO  unknown unknown org.apache.zookeeper.ZooKeeper @ Client environment:java.compiler=<NA>
2024-04-03 12:18:49,979 INFO  unknown unknown org.apache.zookeeper.ZooKeeper @ Client environment:os.name=Linux
2024-04-03 12:18:49,979 INFO  unknown unknown org.apache.zookeeper.ZooKeeper @ Client environment:os.arch=amd64
2024-04-03 12:18:49,979 INFO  unknown unknown org.apache.zookeeper.ZooKeeper @ Client environment:os.version=5.4.202+
2024-04-03 12:18:49,979 INFO  unknown unknown org.apache.zookeeper.ZooKeeper @ Client environment:user.name=root
2024-04-03 12:18:49,979 INFO  unknown unknown org.apache.zookeeper.ZooKeeper @ Client environment:user.home=/root
2024-04-03 12:18:49,979 INFO  unknown unknown org.apache.zookeeper.ZooKeeper @ Client environment:user.dir=/usr/local/tomcat
2024-04-03 12:18:49,980 INFO  unknown unknown org.apache.zookeeper.ZooKeeper @ Initiating client connection, connectString={zookeeper_solr_url} sessionTimeout=45000 watcher=org.apache.solr.common.cloud.SolrZkClient$ProcessWatchWithExecutor@54a49e48
2024-04-03 12:18:49,990 INFO  unknown unknown org.apache.solr.common.cloud.ConnectionManager @ Waiting for client to connect to ZooKeeper
2024-04-03 12:18:50,011 INFO  unknown unknown org.apache.zookeeper.ClientCnxn @ Opening socket connection to server {zookeeper_solr_url}/10.32.29.117:2181. Will not attempt to authenticate using SASL (unknown error)
2024-04-03 12:18:50,018 INFO  unknown unknown org.apache.zookeeper.ClientCnxn @ Socket connection established to {zookeeper_solr_url}/10.32.29.117:2181, initiating session
2024-04-03 12:18:50,048 INFO  unknown unknown org.apache.zookeeper.ClientCnxn @ Session establishment complete on server {zookeeper_solr_url}/10.32.29.117:2181, sessionid = 0x28e3b8ac2c4000a, negotiated timeout = 40000
2024-04-03 12:18:50,055 INFO  unknown unknown org.apache.solr.common.cloud.ConnectionManager @ zkClient has connected
2024-04-03 12:18:50,055 INFO  unknown unknown org.apache.solr.common.cloud.ConnectionManager @ Client is connected to ZooKeeper
2024-04-03 12:18:50,075 INFO  unknown unknown org.apache.solr.common.cloud.ZkStateReader @ Updated live nodes from ZooKeeper... (0) -> (5)
2024-04-03 12:18:50,099 INFO  unknown unknown org.apache.solr.client.solrj.impl.ZkClientClusterStateProvider @ Cluster at {zookeeper_solr_url} ready
2024-04-03 12:18:50,219 INFO  unknown unknown org.dspace.discovery.SolrSearchCore @ Final cloud solr data ------:> org.apache.solr.client.solrj.impl.CloudSolrClient@6a7fb0f6
2024-04-03 12:18:52,357 WARN  unknown unknown org.springframework.security.config.annotation.web.builders.WebSecurity @ You are asking Spring Security to ignore Ant [pattern='/api/authn/login', GET]. This is not recommended -- please use permitAll via HttpSecurity#authorizeHttpRequests instead.
2024-04-03 12:18:52,358 INFO  unknown unknown org.springframework.security.web.DefaultSecurityFilterChain @ Will not secure Ant [pattern='/api/authn/login', GET]
2024-04-03 12:18:52,358 WARN  unknown unknown org.springframework.security.config.annotation.web.builders.WebSecurity @ You are asking Spring Security to ignore Ant [pattern='/api/authn/login', PUT]. This is not recommended -- please use permitAll via HttpSecurity#authorizeHttpRequests instead.
2024-04-03 12:18:52,359 INFO  unknown unknown org.springframework.security.web.DefaultSecurityFilterChain @ Will not secure Ant [pattern='/api/authn/login', PUT]
2024-04-03 12:18:52,359 WARN  unknown unknown org.springframework.security.config.annotation.web.builders.WebSecurity @ You are asking Spring Security to ignore Ant [pattern='/api/authn/login', PATCH]. This is not recommended -- please use permitAll via HttpSecurity#authorizeHttpRequests instead.
2024-04-03 12:18:52,359 INFO  unknown unknown org.springframework.security.web.DefaultSecurityFilterChain @ Will not secure Ant [pattern='/api/authn/login', PATCH]
2024-04-03 12:18:52,359 WARN  unknown unknown org.springframework.security.config.annotation.web.builders.WebSecurity @ You are asking Spring Security to ignore Ant [pattern='/api/authn/login', DELETE]. This is not recommended -- please use permitAll via HttpSecurity#authorizeHttpRequests instead.
2024-04-03 12:18:52,359 INFO  unknown unknown org.springframework.security.web.DefaultSecurityFilterChain @ Will not secure Ant [pattern='/api/authn/login', DELETE]
2024-04-03 12:18:52,643 INFO  unknown unknown org.springframework.boot.autoconfigure.web.servlet.WelcomePageHandlerMapping @ Adding welcome page: ServletContext resource [/index.html]
2024-04-03 12:18:53,120 ERROR unknown unknown org.dspace.authority.AuthoritySolrServiceImpl @ An error occurs querying authority solr core
org.apache.solr.client.solrj.SolrServerException: IOException occurred when talking to server at: {zookeeper_solr_url}/authority
        at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:695) ~[solr-solrj-8.11.2.jar:8.11.2 17dee71932c683e345508113523e764c3e4c80fa - mdrob - 2022-06-13 11:27:56]
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:266) ~[solr-solrj-8.11.2.jar:8.11.2 17dee71932c683e345508113523e764c3e4c80fa - mdrob - 2022-06-13 11:27:56]
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:248) ~[solr-solrj-8.11.2.jar:8.11.2 17dee71932c683e345508113523e764c3e4c80fa - mdrob - 2022-06-13 11:27:56]
        at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:214) ~[solr-solrj-8.11.2.jar:8.11.2 17dee71932c683e345508113523e764c3e4c80fa - mdrob - 2022-06-13 11:27:56]
        at org.apache.solr.client.solrj.SolrClient.query(SolrClient.java:1003) ~[solr-solrj-8.11.2.jar:8.11.2 17dee71932c683e345508113523e764c3e4c80fa - mdrob - 2022-06-13 11:27:56]
        at org.apache.solr.client.solrj.SolrClient.query(SolrClient.java:1018) ~[solr-solrj-8.11.2.jar:8.11.2 17dee71932c683e345508113523e764c3e4c80fa - mdrob - 2022-06-13 11:27:56]
        at org.dspace.authority.AuthoritySolrServiceImpl.getSolr(AuthoritySolrServiceImpl.java:71) ~[dspace-api-7.6.1.jar:7.6.1]
        at org.dspace.app.rest.configuration.ActuatorConfiguration.solrAuthorityCoreHealthIndicator(ActuatorConfiguration.java:74) ~[classes/:7.6.1]
        at org.dspace.app.rest.configuration.ActuatorConfiguration$$EnhancerBySpringCGLIB$$b1ced5a.CGLIB$solrAuthorityCoreHealthIndicator$3(<generated>) ~[classes/:7.6.1]
        at org.dspace.app.rest.configuration.ActuatorConfiguration$$EnhancerBySpringCGLIB$$b1ced5a$$FastClassBySpringCGLIB$$a3721a40.invoke(<generated>) ~[classes/:7.6.1]
...
Caused by: org.apache.http.client.ClientProtocolException
        at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:187) ~[httpclient-4.5.13.jar:4.5.13]
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) ~[httpclient-4.5.13.jar:4.5.13]
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56) ~[httpclient-4.5.13.jar:4.5.13]
        at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:571) ~[solr-solrj-8.11.2.jar:8.11.2 17dee71932c683e345508113523e764c3e4c80fa - mdrob - 2022-06-13 11:27:56]
        ... 226 more
Caused by: org.apache.http.ProtocolException: Target host is not specified
        at org.apache.http.impl.conn.DefaultRoutePlanner.determineRoute(DefaultRoutePlanner.java:71) ~[httpclient-4.5.13.jar:4.5.13]
        at org.apache.http.impl.client.InternalHttpClient.determineRoute(InternalHttpClient.java:125) ~[httpclient-4.5.13.jar:4.5.13]
        at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184) ~[httpclient-4.5.13.jar:4.5.13]
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) ~[httpclient-4.5.13.jar:4.5.13]
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56) ~[httpclient-4.5.13.jar:4.5.13]
        at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:571) ~[solr-solrj-8.11.2.jar:8.11.2 17dee71932c683e345508113523e764c3e4c80fa - mdrob - 2022-06-13 11:27:56]
        ... 226 more
2024-04-03 12:18:53,499 INFO  unknown unknown org.springframework.boot.actuate.endpoint.web.EndpointLinksResolver @ Exposing 2 endpoint(s) beneath base path '/actuator'
2024-04-03 12:18:53,572 INFO  unknown unknown org.springframework.security.web.DefaultSecurityFilterChain @ Will secure Or [Ant [pattern='/api/**'], Ant [pattern='/iiif/**'], Ant [pattern='/actuator/**'], Ant [pattern='/signposting/**']] with [org.springframework.security.web.session.DisableEncodeUrlFilter@702052d1, org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter@512d3398,
.....
2024-04-03 12:18:55,924 INFO  unknown unknown org.springframework.aop.framework.CglibAopProxy @ Unable to proxy interface-implementing method [public final java.lang.Iterable org.dspace.app.rest.repository.DSpaceRestRepository.findAll()] because it is marked as final: Consider using interface-based JDK proxies instead! "

Need your help to fix it.

Thanks,

Manish Kumar

[Manish Kumar]

Hi team,

Please help me on my posted issue.

Thanks,

Manish Kumar

[Manish Kumar]

Hi all,

Let me add more context.

We are using dspace-backend-7.6.1 and solr-8.11.1 and solr is running on cloud mode connected with zookeeper. While performing some action on UI, I am getting below error message.

2024-04-04 12:33:50,666 ERROR unknown unknown org.dspace.authority.AuthoritySolrServiceImpl @ An error occurs querying authority solr core
org.apache.solr.client.solrj.SolrServerException: IOException occurred when talking to server at: zk-headless.solrcloud.svc.poc.cluster.local/authority.

Caused by: org.apache.http.client.ClientProtocolException
        at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:187) ~[httpclient-4.5.13.jar:4.5.13]
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) ~[httpclient-4.5.13.jar:4.5.13]
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56) ~[httpclient-4.5.13.jar:4.5.13]
        at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:571) ~[solr-solrj-8.11.2.jar:8.11.2 17dee71932c683e345508113523e764c3e4c80fa - mdrob - 2022-06-13 11:27:56]
        ... 226 more
Caused by: org.apache.http.ProtocolException: Target host is not specified

And same for  statistics also. Due to this not able to index in solr. Zookeeper is suceessfully connecting with solr. 

org.apache.zookeeper.ZooKeeper @ Initiating client connection, connectString=  zk-headless.solrcloud.svc.poc.cluster.local   sessionTimeout=45000 watcher=org.apache.solr.common.cloud.SolrZkClient$ProcessWatchWithExecutor@27b41889
2024-04-04 12:33:47,489 INFO  unknown unknown org.apache.solr.common.cloud.ConnectionManager @ Waiting for client to connect to ZooKeeper
2024-04-04 12:33:47,507 INFO  unknown unknown org.apache.zookeeper.ClientCnxn @ Opening socket connection to server   zk-headless.solrcloud.svc.poc.cluster.local/10.32.29.117:2181. Will not attempt to authenticate using SASL (unknown error)
2024-04-04 12:33:47,513 INFO  unknown unknown org.apache.zookeeper.ClientCnxn @ Socket connection established to  zk-headless.solrcloud.svc.poc.cluster.local  /10.32.29.117:2181, initiating session
2024-04-04 12:33:47,526 INFO  unknown unknown org.apache.zookeeper.ClientCnxn @ Session establishment complete on server  zk-headless.solrcloud.svc.poc.cluster.local/10.32.29.117:2181, sessionid = 0x28e3b8ac2c4000b, negotiated timeout = 40000
2024-04-04 12:33:47,531 INFO  unknown unknown org.apache.solr.common.cloud.ConnectionManager @ zkClient has connected
2024-04-04 12:33:47,531 INFO  unknown unknown org.apache.solr.common.cloud.ConnectionManager @ Client is connected to ZooKeeper
2024-04-04 12:33:47,553 INFO  unknown unknown org.apache.solr.common.cloud.ZkStateReader @ Updated live nodes from ZooKeeper... (0) -> (5)
2024-04-04 12:33:47,576 INFO  unknown unknown org.apache.solr.client.solrj.impl.ZkClientClusterStateProvider @ Cluster at   zk-headless.solrcloud.svc.poc.cluster.local     ready
2024-04-04 12:33:47,662 INFO  unknown unknown org.dspace.discovery.SolrSearchCore @ Final cloud solr data ------:> org.apache.solr.client.solrj.impl.CloudSolrClient@303549a5

I found one key inside dspace.cfg : 
# Solr core name prefix.
# If you connect multiple instances of DSpace to a single Solr instance, you
# can organize them with a common core name prefix.
solr.multicorePrefix =

What would be value in my case(I mean zookeeper and solr with 3-3 instances) ? since we have used solr.server = zk-headless.solrcloud.svc.poc.cluster.local (it's a kubernetes cluster url) and internally dspace is trying to connect through  zk-headless.solrcloud.svc.poc.cluster.local/authority or  zk-headless.solrcloud.svc.poc.cluster.local/statistics. 

Existing system is using dspace 7.2.1, same zookeeper and solr version what we have used in dspace 7.2.1 &  7.6.1 and have not faced this issue on 7.2.1.

Thanks,

Manish Kumar