Issue starting Vocbench 10.2.1 in a container (running on a Kubernetes pod)
306 views
Skip to first unread message
charly....@univ-grenoble-alpes.fr
Apr 7, 2022, 4:36:47 PM4/7/22
to vocbench-user
Dear all,
I'm having a weird behaviour with a Vocbench 10.2.1 image that I try to launch on a Kubernetes Pod. I'm using this Dockerfile (slightly modified to download the .zip using cURL instead of copying it). The application fail to start properly and I'm having the following exception:
---------------------------------------------------------------------------
0:24:55.439 ERROR [fileinstall-/opt/vocbench3/semanticturkey-10.2.1/deploy] Failed to install artifact: /opt/vocbench3/semanticturkey-10.2.1/deploy/vocbench3-10.2.1.war
java.lang.IllegalStateException: Unknown protocol: webbundle
at org.apache.felix.framework.URLHandlersStreamHandlerProxy.toExternalForm(URLHandlersStreamHandlerProxy.java:482)
at org.apache.felix.framework.URLHandlersStreamHandlerProxy.toExternalForm(URLHandlersStreamHandlerProxy.java:474)
at java.net.URL.toExternalForm(URL.java:972)
at java.net.URL.toString(URL.java:958)
at org.apache.felix.fileinstall.internal.DirectoryWatcher.install(DirectoryWatcher.java:961)
at org.apache.felix.fileinstall.internal.DirectoryWatcher.install(DirectoryWatcher.java:884)
at org.apache.felix.fileinstall.internal.DirectoryWatcher.doProcess(DirectoryWatcher.java:489)
at org.apache.felix.fileinstall.internal.DirectoryWatcher.process(DirectoryWatcher.java:365)
at org.apache.felix.fileinstall.internal.DirectoryWatcher.run(DirectoryWatcher.java:316)
Exception in thread "Karaf Lock Monitor Thread" java.lang.OutOfMemoryError: Java heap space
at sun.nio.cs.UTF_8.newEncoder(UTF_8.java:72)
at java.lang.StringCoding$StringEncoder.<init>(StringCoding.java:282)
at java.lang.StringCoding$StringEncoder.<init>(StringCoding.java:273)
at java.lang.StringCoding.encode(StringCoding.java:338)
at java.lang.String.getBytes(String.java:918)
at java.io.UnixFileSystem.getBooleanAttributes0(Native Method)
at java.io.UnixFileSystem.getBooleanAttributes(UnixFileSystem.java:242)
at java.io.File.isDirectory(File.java:860)
at org.apache.karaf.main.Main.doMonitor(Main.java:399)
at org.apache.karaf.main.Main.access$100(Main.java:75)
at org.apache.karaf.main.Main$2.run(Main.java:376)
org.apache.karaf.features.internal.util.MultiException: Error:
Error downloading wrap:file:/opt/vocbench3/semanticturkey-10.2.1/system/org/apache/servicemix/bundles/org.apache.servicemix.bundles.elasticsearch/6.5.4_1/org.apache.servicemix.bundles.elasticsearch-6.5.4_1.jar$overwrite=merge&Import-Package=!org.apache.logging.log4j.core.*,*;resolution:=optional
Error downloading wrap:file:/opt/vocbench3/semanticturkey-10.2.1/system/org/apache/servicemix/bundles/org.apache.servicemix.bundles.elasticsearch/6.5.4_1/org.apache.servicemix.bundles.elasticsearch-6.5.4_1.jar$overwrite=merge&Import-Package=!org.apache.logging.log4j.core.*,*;resolution:=optional
at org.apache.karaf.features.internal.download.impl.MavenDownloadManager$MavenDownloader.<init>(MavenDownloadManager.java:91)
at org.apache.karaf.features.internal.download.impl.MavenDownloadManager.createDownloader(MavenDownloadManager.java:72)
at org.apache.karaf.features.internal.region.Subsystem.downloadBundles(Subsystem.java:457)
at org.apache.karaf.features.internal.region.Subsystem.downloadBundles(Subsystem.java:452)
at org.apache.karaf.features.internal.region.SubsystemResolver.resolve(SubsystemResolver.java:224)
at org.apache.karaf.features.internal.service.Deployer.deploy(Deployer.java:401)
at org.apache.karaf.features.internal.service.FeaturesServiceImpl.doProvision(FeaturesServiceImpl.java:1063)
at org.apache.karaf.features.internal.service.FeaturesServiceImpl.lambda$doProvisionInThread$13(FeaturesServiceImpl.java:998)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
Suppressed: java.io.IOException: Error downloading wrap:file:/opt/vocbench3/semanticturkey-10.2.1/system/org/apache/servicemix/bundles/org.apache.servicemix.bundles.elasticsearch/6.5.4_1/org.apache.servicemix.bundles.elasticsearch-6.5.4_1.jar$overwrite=merge&Import-Package=!org.apache.logging.log4j.core.*,*;resolution:=optional
at org.apache.karaf.features.internal.download.impl.AbstractRetryableDownloadTask.run(AbstractRetryableDownloadTask.java:81)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
... 3 more
Caused by: java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOfRange(Arrays.java:3664)
at java.lang.String.<init>(String.java:207)
at aQute.lib.io.IO.readUTF(IO.java:1634)
at aQute.lib.io.ByteBufferDataInput.readUTF(ByteBufferDataInput.java:117)
at aQute.bnd.classfile.ConstantPool.readUtf8Info(ConstantPool.java:189)
at aQute.bnd.classfile.ConstantPool.read(ConstantPool.java:103)
at aQute.bnd.classfile.ClassFile.parseClassFile(ClassFile.java:75)
at aQute.bnd.osgi.Clazz.parseClassFileData(Clazz.java:800)
at aQute.bnd.osgi.Clazz.parseClassFileData(Clazz.java:788)
at aQute.bnd.osgi.Clazz.parseClassFileWithCollector(Clazz.java:776)
at aQute.bnd.osgi.Clazz.parseClassFile(Clazz.java:766)
at aQute.bnd.osgi.Analyzer.analyzeJar(Analyzer.java:2666)
at aQute.bnd.osgi.Analyzer.analyzeBundleClasspath(Analyzer.java:2582)
at aQute.bnd.osgi.Analyzer.analyze(Analyzer.java:212)
at aQute.bnd.osgi.Analyzer.calcManifest(Analyzer.java:999)
at org.ops4j.pax.swissbox.bnd.BndUtils.createBundle(BndUtils.java:161)
at org.ops4j.pax.url.wrap.internal.Connection.getInputStream(Connection.java:83)
at java.net.URL.openStream(URL.java:1093)
at org.apache.karaf.features.internal.download.impl.SimpleDownloadTask.download(SimpleDownloadTask.java:78)
at org.apache.karaf.features.internal.download.impl.AbstractRetryableDownloadTask.run(AbstractRetryableDownloadTask.java:60)
... 7 more
Suppressed: [CIRCULAR REFERENCE: java.io.IOException: Error downloading wrap:file:/opt/vocbench3/semanticturkey-10.2.1/system/org/apache/servicemix/bundles/org.apache.servicemix.bundles.elasticsearch/6.5.4_1/org.apache.servicemix.bundles.elasticsearch-6.5.4_1.jar$overwrite=merge&Import-Package=!org.apache.logging.log4j.core.*,*;resolution:=optional]
20:25:37.446 ERROR [activator-1-thread-2] Error installing boot features
org.apache.karaf.features.internal.util.MultiException: Error:
Error downloading wrap:file:/opt/vocbench3/semanticturkey-10.2.1/system/org/apache/servicemix/bundles/org.apache.servicemix.bundles.elasticsearch/6.5.4_1/org.apache.servicemix.bundles.elasticsearch-6.5.4_1.jar$overwrite=merge&Import-Package=!org.apache.logging.log4j.core.*,*;resolution:=optional
Error downloading wrap:file:/opt/vocbench3/semanticturkey-10.2.1/system/org/apache/servicemix/bundles/org.apache.servicemix.bundles.elasticsearch/6.5.4_1/org.apache.servicemix.bundles.elasticsearch-6.5.4_1.jar$overwrite=merge&Import-Package=!org.apache.logging.log4j.core.*,*;resolution:=optional
at org.apache.karaf.features.internal.download.impl.MavenDownloadManager$MavenDownloader.<init>(MavenDownloadManager.java:91)
at org.apache.karaf.features.internal.download.impl.MavenDownloadManager.createDownloader(MavenDownloadManager.java:72)
at org.apache.karaf.features.internal.region.Subsystem.downloadBundles(Subsystem.java:457)
at org.apache.karaf.features.internal.region.Subsystem.downloadBundles(Subsystem.java:452)
at org.apache.karaf.features.internal.region.SubsystemResolver.resolve(SubsystemResolver.java:224)
at org.apache.karaf.features.internal.service.Deployer.deploy(Deployer.java:401)
at org.apache.karaf.features.internal.service.FeaturesServiceImpl.doProvision(FeaturesServiceImpl.java:1063)
at org.apache.karaf.features.internal.service.FeaturesServiceImpl.lambda$doProvisionInThread$13(FeaturesServiceImpl.java:998)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
Suppressed: java.io.IOException: Error downloading wrap:file:/opt/vocbench3/semanticturkey-10.2.1/system/org/apache/servicemix/bundles/org.apache.servicemix.bundles.elasticsearch/6.5.4_1/org.apache.servicemix.bundles.elasticsearch-6.5.4_1.jar$overwrite=merge&Import-Package=!org.apache.logging.log4j.core.*,*;resolution:=optional
at org.apache.karaf.features.internal.download.impl.AbstractRetryableDownloadTask.run(AbstractRetryableDownloadTask.java:81)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
... 3 more
Caused by: java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOfRange(Arrays.java:3664)
at java.lang.String.<init>(String.java:207)
at aQute.lib.io.IO.readUTF(IO.java:1634)
at aQute.lib.io.ByteBufferDataInput.readUTF(ByteBufferDataInput.java:117)
at aQute.bnd.classfile.ConstantPool.readUtf8Info(ConstantPool.java:189)
at aQute.bnd.classfile.ConstantPool.read(ConstantPool.java:103)
at aQute.bnd.classfile.ClassFile.parseClassFile(ClassFile.java:75)
at aQute.bnd.osgi.Clazz.parseClassFileData(Clazz.java:800)
at aQute.bnd.osgi.Clazz.parseClassFileData(Clazz.java:788)
at aQute.bnd.osgi.Clazz.parseClassFileWithCollector(Clazz.java:776)
at aQute.bnd.osgi.Clazz.parseClassFile(Clazz.java:766)
at aQute.bnd.osgi.Analyzer.analyzeJar(Analyzer.java:2666)
at aQute.bnd.osgi.Analyzer.analyzeBundleClasspath(Analyzer.java:2582)
at aQute.bnd.osgi.Analyzer.analyze(Analyzer.java:212)
at aQute.bnd.osgi.Analyzer.calcManifest(Analyzer.java:999)
at org.ops4j.pax.swissbox.bnd.BndUtils.createBundle(BndUtils.java:161)
at org.ops4j.pax.url.wrap.internal.Connection.getInputStream(Connection.java:83)
at java.net.URL.openStream(URL.java:1093)
at org.apache.karaf.features.internal.download.impl.SimpleDownloadTask.download(SimpleDownloadTask.java:78)
at org.apache.karaf.features.internal.download.impl.AbstractRetryableDownloadTask.run(AbstractRetryableDownloadTask.java:60)
... 7 more
Suppressed: [CIRCULAR REFERENCE: java.io.IOException: Error downloading wrap:file:/opt/vocbench3/semanticturkey-10.2.1/system/org/apache/servicemix/bundles/org.apache.servicemix.bundles.elasticsearch/6.5.4_1/org.apache.servicemix.bundles.elasticsearch-6.5.4_1.jar$overwrite=merge&Import-Package=!org.apache.logging.log4j.core.*,*;resolution:=optional]----------------------------------------------------------------------------------------
Although this exceptions are raised only running in a Kubernetes pod and I can't reproduce it locally using docker (appart from the first exception dealing with "webbundle" but if I understood well this one is harmless). Also everything seems to work fine using 9.1.0 release.
I'm aware that this one will be hard to reproduce but If anyone has already encountered these exception that could be a clue!
I'm aware that this one will be hard to reproduce but If anyone has already encountered these exception that could be a clue!
Charly
Tiziano Lorenzetti
Apr 11, 2022, 8:57:13 AM4/11/22
to charly....@univ-grenoble-alpes.fr, vocbench-user
Dear Charly,
the webbundle issue should not be a big deal. It should depend on non-synchronic factors related to installation of bundles at the very first start of SemanticTurkey and usually it solves simply by restarting ST.
I see that the stacktrace reports also a
java.lang.OutOfMemoryError: Java heap space
so, I suppose you could try to increase the maximum Java heap size.
Best regards,
Tiziano
--
You received this message because you are subscribed to the Google Groups "vocbench-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vocbench-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/vocbench-user/19aa9e03-1473-4b43-b093-40ffbd52a334n%40googlegroups.com.
Charly Coussot
Apr 11, 2022, 12:27:01 PM4/11/22
to Tiziano Lorenzetti, vocbench-user
Dear Tiziano,
Thank you for your answer. As I was mentioning this does not happen in my local machine, only on my Kubernetes cluster. For now on, I suspect the older version of Docker installed on the kubernetes. This lead to this strange circular reference. I've already try to increase the heap size up to 1gb and I still have the same issue.
Best regards,
Charly
Juan Amador
Apr 26, 2022, 9:06:52 AM4/26/22
to vocbench-user
its a dns / firewall / network issue - it cannot connect to a server
Rob Kaesehagen
May 16, 2022, 2:15:22 PM5/16/22
to vocbench-user
Hi, I a having the same issue starting Vocbench. If it is a network/dns./firewall issue: has anyone here had this as well and has a solution to solve it? As far as I can see, there are no limitations on the firewall for outgoing traffic. We have network connectivity. Any clues?
Op dinsdag 26 april 2022 om 15:06:52 UTC+2 schreef juane....@gmail.com:
Armando Stellato
May 17, 2022, 3:48:58 AM5/17/22
to Rob Kaesehagen, vocbench-user
Dear Rob,
I’m sorry for your experience with the issue. This is something annoying due to asynchronous installation of the various bundles in OSGi and, even by setting dependencies, we have not always got rid of it.
However, as reported in here:
https://groups.google.com/g/vocbench-user/c/tT1K8y1z_b0/m/7OAcJ2W5CAAJ
it shouldn’t be a big deal. The first time you run VB you might encounter in this error. Let it rest a little and let it install all other bundles. Then, close, restart, and the webbundle will be installed eventually.
From this second start on, all bundles should be in the unpacked directory of bundles and you shouldn’t experience this anymore,
Pls let us know if this solves the issue,
Kind Regards,
Armando
--
You received this message because you are subscribed to the Google Groups "vocbench-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to
vocbench-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/vocbench-user/bbd8af89-6d02-4549-9545-c261847e71bfn%40googlegroups.com.
Rob Kaesehagen
May 24, 2022, 7:14:08 AM5/24/22
to vocbench-user
Hi Armando,
This does not solve the problem. Because of the error, karaf won't start ant therefore vocbench keeps asking for username/password in the frontend. I have tried to install VB3 on a different server, but get the same error messages. Server 1: ubuntu, Server 2: Alma linux.
Rob
Op dinsdag 17 mei 2022 om 09:48:58 UTC+2 schreef stel...@uniroma2.it:
Rob Kaesehagen
May 24, 2022, 9:00:48 AM5/24/22
to vocbench-user
Hi all,
i did not solve the problem. I installed a previous build (10.1.1) and it worked out of the box like a charm. So it looks like som glitch in the latest version.
Rob
Op dinsdag 24 mei 2022 om 13:14:08 UTC+2 schreef Rob Kaesehagen:
Robert Chavez
Dec 21, 2022, 11:14:14 AM12/21/22
to vocbench-user
Rob,
Did you ever get to the bottom of this issue?
I am experiencing the same problem and have gone through the same steps that you have. Only 10.1.1 and earlier is working for me.
All attempts with versions later than 10.1.1 result in the error:
Suppressed: [CIRCULAR REFERENCE:
java.io.IOException: Error downloading
wrap:file:/opt/vocbench3/semanticturkey-10.2.1/system/org/apache/servicemix/bundles/org.apache.servicemix.bundles.elasticsearch/6.5.4_1/org.apache.servicemix.bundles.elasticsearch-6.5.4_1.jar$overwrite=merge&Import-Package=!org.apache.logging.log4j.core.*,*;resolution:=optional]
It's a head scratcher.
Thanks,
Rob C.
Rob Kaesehagen
Dec 22, 2022, 2:38:52 AM12/22/22
to Robert Chavez, vocbench-user
Ho Robert,
No, we use the omder version now
Van: vocben...@googlegroups.com <vocben...@googlegroups.com> namens Robert Chavez <nae...@gmail.com>
Verzonden: Wednesday, December 21, 2022 5:14:13 PM
Aan: vocbench-user <vocben...@googlegroups.com>
Onderwerp: Re: [vocbench-user] Re: Issue starting Vocbench 10.2.1 in a container (running on a Kubernetes pod)
Verzonden: Wednesday, December 21, 2022 5:14:13 PM
Aan: vocbench-user <vocben...@googlegroups.com>
Onderwerp: Re: [vocbench-user] Re: Issue starting Vocbench 10.2.1 in a container (running on a Kubernetes pod)
To view this discussion on the web visit
https://groups.google.com/d/msgid/vocbench-user/2ff7bf07-acd2-49ea-9f9e-24d3fc1e5fc8n%40googlegroups.com.
Armando Stellato
Jan 29, 2023, 7:22:47 AM1/29/23
to Rob Kaesehagen, Robert Chavez, vocbench-user
Dear all,
By first, our sincere apologies for the issue and for the headaches ingenerated :-).
One thing we were consider for future releases is if it is not the case to bundle everything, so you wouldn’t have this asynchronous issue. On the other hand, we will probably, within end of the year, bring big changes to the architecture, including most probably abandoning Karaf.
That said, a few more words that could help:
- I have run into this issue, and managed to solve it by applying what I said
- I’m assuming Java is version 8; pls verify formally, which could be done by injecting a print to console on the karaf batch and not simply checking which versions you have installed, that it is one of the latest versions of Java 8, this could be an issue even though I can’t say for sure it is that.
- In reference to Rob’s reply, when I said: “run it, wait quite some time (even after it gives the error), then close and restart” it’s something independent of whether semantic turkey will start or not. Differently from your sentence, it is semantic turkey not starting, while Karaf is started and working. So, exactly because the first time you run a new version all the bundles are downloaded (those not already in the system), unpacked and installed, even though there is an error with one of them, the others will be installed. So you can wait patiently for them to be installed (if you want to follow the process, you can open the “data/cache” directory within semantic turkey installation and check that new bundle-directories are being created. When it’s finished (wait a little without new bundles being deployed), close and restart. The bundle that was missing before will be installed and you will be able to access VB
To view this discussion on the web visit https://groups.google.com/d/msgid/vocbench-user/AM0PR03MB5076FBEB675A011FEFE2BE70B5E89%40AM0PR03MB5076.eurprd03.prod.outlook.com.
Hugo van Versendaal
Feb 8, 2023, 5:18:38 PM2/8/23
to vocbench-user
Hi all,
That is some interesting news about changing the architecture!
Regarding the Kubernetes issue: thanks for the tips, but I haven't managed to fix it by restarting the server. As soon as I stop it from the console, the pod with the container is killed and the new pod starts the server with the same issue. Maybe this could be solved by mounting a volume with the "data/cache" directory. I haven't tried it yet though.
However, I got the latest version of the software (11.3.0) to run properly on our Kubernetes cluster, by separating SemanticTurkey and the VocBench webapp as described in the chapter "Separate HTTP Server" of the VB documentation. Hooray! Below are the steps I took to achieve this, in case anyone else would like take this road.
1. Create a Docker image from Tomcat (e.g. jre-8) and add or unzip the vocbench3 WAR file to the webapps dir. This WAR file can be found in the folder "semanticturkey-${st_version}/deploy/" of the vocbench3-${vb_version}-full.zip download.
2. Create a Docker image of SemanticTurkey. I used this Dockerfile as a template: https://bitbucket.org/art-uniroma2/vocbench3-docker/src/master/11.3.0/Dockerfile. Instead of the COPY command on line 6, I used RUN wget to download semanticturkey-${st_version}.zip.
3. Create pods (or deployments) to deploy both containers. I tried running both containers in one pod, but I got stuck, so I decided to go with one pod per container in the end.
Important note: be sure to configure the vbconfig.js file in the VB container, so it can properly contact the ST container. See: http://vocbench.uniroma2.it/doc/sys/#further_configuration_for_reaching_semantic_turkey_on_a_different_host_port. In my case, I had to set st_protocol to https, st_port to 443 and st_host to the ingress hostname. This is probably due to our company policies, so you have to figure out the right configuration for your cluster.
Please let me know if you have thoughts or questions about this method.
Kind regards,
Hugo
Op zondag 29 januari 2023 om 13:22:47 UTC+1 schreef stel...@uniroma2.it:
Armando Stellato
Feb 9, 2023, 11:43:55 AM2/9/23
to Hugo van Versendaal, vocbench-user
Dear Hugo,
I’m glad you sorted this out and thanks for the report!.
In effect I did not think about suggesting the next-by-ease solution, which is to keep client and separated (since the dependency issue emerges for providing the client app via karaf) and which, for the docker distribution, to remodel the distribution like you did and which applies what we already suggested in the page you cited to the docker distribution.
Kind Regards,
Armando
To view this discussion on the web visit https://groups.google.com/d/msgid/vocbench-user/b9e260d9-5df8-416b-85c7-3989dde11e65n%40googlegroups.com.
Reply all
Reply to author
Forward
0 new messages