[PATCH] fix kafka mirror url (error 404)

[geraldo netto]

Signed-off-by: geraldo netto <gerald...@gmail.com>
---
apache-kafka/Makefile | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/apache-kafka/Makefile b/apache-kafka/Makefile
index 63ed524..6fbc98d 100644
--- a/apache-kafka/Makefile
+++ b/apache-kafka/Makefile
@@ -3,7 +3,7 @@ VERSION:=0.8.2.1
NAME:=kafka
DIR:=kafka_$(SCALA)-$(VERSION)
TARBALL:=$(DIR).tgz
-URL:=http://mirror.ventraip.net.au/apache/$(NAME)/$(VERSION)/$(TARBALL)
+URL:=http://archive.apache.org/dist/$(NAME)/$(VERSION)/$(TARBALL)


module: ROOTFS usr.manifest
--
2.7.4

[Commit Bot]

From: geraldo netto <gerald...@gmail.com>
Committer: Nadav Har'El <n...@scylladb.com>
Branch: master

fix kafka mirror url (error 404)

Signed-off-by: geraldo netto <gerald...@gmail.com>
Message-Id: <1515724275-3769-1-git-...@gmail.com>

---
diff --git a/apache-kafka/Makefile b/apache-kafka/Makefile

[Nadav Har'El]

Thanks. I committed this (and going through your other patches as well).

However, if I do
scripts/build image=apache-kafka

scripts/run.py

I get all sorts of connection refused errors, and messages like

org.I0Itec.zkclient.exception.ZkTimeoutException: Unable to connect to zookeeper server within timeout: 6000

until a few seconds later, the application shuts down.

Were you able to use this package? How?

--
Nadav Har'El

n...@scylladb.com

--
2.7.4

--
You received this message because you are subscribed to the Google Groups "OSv Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to osv-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Geraldo Netto]

Hello Nadav/All,

Indeed, I'm also facing the same issue and I think the problem is this:

[2018-01-15 14:27:27,398] INFO Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn)

[2018-01-15 14:27:27,409] ERROR Unable to open socket to localhost/127.0.0.1:2181 (org.apache.zookeeper.ClientCnxnSocketNIO)

I have the feeling that Kafka is concatenating localhost and 127.0.0.1 as a single string

Maybe some config. is wrong?

I try to check that later

In any case, not only Kafka but there are some other issues

dpdk-example looks broken to me

erjang is pointing to a dropbox url to download the jar and the file is not available anymore

most of the apps are using wget without -c

Full log:

OSv v0.24-477-g29fa668

eth0: 192.168.122.89

java.so: Starting JVM app using: io/osv/isolated/RunIsolatedJvmApp

java.so: Setting Java system classloader to IsolatingOsvSystemClassLoader and lo  gging manager to IsolatingLogManager

[2018-01-15 14:27:26,928] INFO Verifying properties (kafka.utils.VerifiableProperties)

[2018-01-15 14:27:27,010] INFO Property broker.id is overridden to 0 (kafka.utils.VerifiableProperties)

[2018-01-15 14:27:27,013] INFO Property log.cleaner.enable is overridden to false (kafka.utils.VerifiableProperties)

[2018-01-15 14:27:27,016] INFO Property log.dirs is overridden to /tmp/kafka-logs (kafka.utils.VerifiableProperties)

[2018-01-15 14:27:27,018] INFO Property log.retention.check.interval.ms is overridden to 300000 (kafka.utils.VerifiableProperties)

[2018-01-15 14:27:27,021] INFO Property log.retention.hours is overridden to 168 (kafka.utils.VerifiableProperties)

[2018-01-15 14:27:27,023] INFO Property log.segment.bytes is overridden to 1073741824 (kafka.utils.VerifiableProperties)

[2018-01-15 14:27:27,026] INFO Property num.io.threads is overridden to 8 (kafka.utils.VerifiableProperties)

[2018-01-15 14:27:27,028] INFO Property num.network.threads is overridden to 3 (kafka.utils.VerifiableProperties)

[2018-01-15 14:27:27,030] INFO Property num.partitions is overridden to 1 (kafka.utils.VerifiableProperties)

[2018-01-15 14:27:27,032] INFO Property num.recovery.threads.per.data.dir is overridden to 1 (kafka.utils.VerifiableProperties)

[2018-01-15 14:27:27,034] INFO Property port is overridden to 9092 (kafka.utils.VerifiableProperties)

[2018-01-15 14:27:27,036] INFO Property socket.receive.buffer.bytes is overridden to 102400 (kafka.utils.VerifiableProperties)

[2018-01-15 14:27:27,039] INFO Property socket.request.max.bytes is overridden to 104857600 (kafka.utils.VerifiableProperties)

[2018-01-15 14:27:27,041] INFO Property socket.send.buffer.bytes is overridden to 102400 (kafka.utils.VerifiableProperties)

[2018-01-15 14:27:27,044] INFO Property zookeeper.connect is overridden to localhost:2181 (kafka.utils.VerifiableProperties)

[2018-01-15 14:27:27,046] INFO Property zookeeper.connection.timeout.ms is overridden to 6000 (kafka.utils.VerifiableProperties)

[2018-01-15 14:27:27,179] INFO [Kafka Server 0], starting (kafka.server.KafkaServer)

[2018-01-15 14:27:27,188] INFO [Kafka Server 0], Connecting to zookeeper on localhost:2181 (kafka.server.KafkaServer)

[2018-01-15 14:27:27,239] INFO Starting ZkClient event thread. (org.I0Itec.zkclient.ZkEventThread)

[2018-01-15 14:27:27,257] INFO Client environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 0 9:09 GMT (org.apache.zookeeper.ZooKeeper)

[2018-01-15 14:27:27,261] INFO Client environment:host.name=osv (org.apache.zookeeper.ZooKeeper)

[2018-01-15 14:27:27,265] INFO Client environment:java.version=1.7.0_55 (org.apache.zookeeper.ZooKeeper)

[2018-01-15 14:27:27,267] INFO Client environment:java.vendor=Oracle Corporation (org.apache.zookeeper.ZooKeeper)

[2018-01-15 14:27:27,269] INFO Client environment:java.home=/usr/lib/jvm/java/jre (org.apache.zookeeper.ZooKeeper)

[2018-01-15 14:27:27,271] INFO Client environment:java.class.path=:/kafka/core/build/dependant-libs-2.10.4*/*.jar:/kafka/contrib/hadoop-consumer/build/libs/kafka-hadoop-consumer*.jar:/kafka/contrib/hadoop-pro  ducer/build/libs/kafka-hadoop-producer*.jar:/kafka/clients/build/libs/kafka-clients*.jar:/kafka/libs/jop  t-simple-3.2.jar:/kafka/libs/kafka_2.10-0.8.2.1.jar:/kafka/libs/kafka-clients-0.8.2.1.jar:/kafka/libs/lo  g4j-1.2.16.jar:/kafka/libs/lz4-1.2.0.jar:/kafka/libs/metrics-core-2.2.0.jar:/kafka/libs/scala-library-2.  10.4.jar:/kafka/libs/slf4j-api-1.7.6.jar:/kafka/libs/slf4j-log4j12-1.6.1.jar:/kafka/libs/snappy-java-1.1  .1.6.jar:/kafka/libs/zkclient-0.3.jar:/kafka/libs/zookeeper-3.4.6.jar:/kafka/core/build/libs/kafka_2.10*  .jar (org.apache.zookeeper.ZooKeeper)

[2018-01-15 14:27:27,283] INFO Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib (org.apache.zookeeper.ZooKeeper)

[2018-01-15 14:27:27,285] INFO Client environment:java.io.tmpdir=/tmp (org.apache.zookeeper.ZooKeeper)

[2018-01-15 14:27:27,287] INFO Client environment:java.compiler=<NA> (org.apache.zookeeper.ZooKeeper)

[2018-01-15 14:27:27,289] INFO Client environment:os.name=Linux (org.apache.zookeeper.ZooKeeper)

[2018-01-15 14:27:27,291] INFO Client environment:os.arch=amd64 (org.apache.zookeeper.ZooKeeper)

[2018-01-15 14:27:27,294] INFO Client environment:os.version=3.7.0 (org.apache.zookeeper.ZooKeeper)

[2018-01-15 14:27:27,296] INFO Client environment:user.name=osv (org.apache.zookeeper.ZooKeeper)

[2018-01-15 14:27:27,298] INFO Client environment:user.home=/ (org.apache.zookeeper.ZooKeeper)

[2018-01-15 14:27:27,300] INFO Client environment:user.dir=/kafka (org.apache.zookeeper.ZooKeeper)

[2018-01-15 14:27:27,304] INFO Initiating client connection, connectString=localhost:2181 sessionTimeout=6000 watcher=org.I0Itec.zkclient.ZkClient@6cbc2704 (org.apache.zookeeper.ZooKeeper)

[2018-01-15 14:27:27,398] INFO Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn)

[2018-01-15 14:27:27,409] ERROR Unable to open socket to localhost/127.0.0.1:2181 (org.apache.zookeeper.ClientCnxnSocketNIO)

[2018-01-15 14:27:27,412] WARN Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect (org.apache.zookeeper.ClientCnxn)

java.net.ConnectException: Connection refused

  at sun.nio.ch.Net.connect0(Native Method)

  at sun.nio.ch.Net.connect(Net.java:465)

  at sun.nio.ch.Net.connect(Net.java:457)

  at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670)

  at org.apache.zookeeper.ClientCnxnSocketNIO.registerAndConnect(ClientCnxnSocketNIO.java:277)

  at org.apache.zookeeper.ClientCnxnSocketNIO.connect(ClientCnxnSocketNIO.java:287)

  at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:967)

  at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003)

[2018-01-15 14:27:28,530] INFO Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn)

[2018-01-15 14:27:28,536] ERROR Unable to open socket to localhost/127.0.0.1:2181 (org.apache.zookeeper.ClientCnxnSocketNIO)

[2018-01-15 14:27:28,540] WARN Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect (org.apache.zookeeper.ClientCnxn)

java.net.ConnectException: Connection refused

  at sun.nio.ch.Net.connect0(Native Method)

  at sun.nio.ch.Net.connect(Net.java:465)

  at sun.nio.ch.Net.connect(Net.java:457)

  at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670)

  at org.apache.zookeeper.ClientCnxnSocketNIO.registerAndConnect(ClientCnxnSocketNIO.java:277)

  at org.apache.zookeeper.ClientCnxnSocketNIO.connect(ClientCnxnSocketNIO.java:287)

  at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:967)

  at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003)

[2018-01-15 14:27:29,653] INFO Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn)

[2018-01-15 14:27:29,659] ERROR Unable to open socket to localhost/127.0.0.1:2181 (org.apache.zookeeper.ClientCnxnSocketNIO)

[2018-01-15 14:27:29,661] WARN Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect (org.apache.zookeeper.ClientCnxn)

java.net.ConnectException: Connection refused

  at sun.nio.ch.Net.connect0(Native Method)

  at sun.nio.ch.Net.connect(Net.java:465)

  at sun.nio.ch.Net.connect(Net.java:457)

  at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670)

  at org.apache.zookeeper.ClientCnxnSocketNIO.registerAndConnect(ClientCnxnSocketNIO.java:277)

  at org.apache.zookeeper.ClientCnxnSocketNIO.connect(ClientCnxnSocketNIO.java:287)

  at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:967)

  at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003)

[2018-01-15 14:27:30,790] INFO Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn)

[2018-01-15 14:27:30,798] WARN Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect (org.apache.zookeeper.ClientCnxn)

java.net.ConnectException: Connection refused

  at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)

  at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)

  at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)

  at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)

[2018-01-15 14:27:31,912] INFO Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn)

[2018-01-15 14:27:31,918] WARN Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect (org.apache.zookeeper.ClientCnxn)

java.net.ConnectException: Connection refused

  at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)

  at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)

  at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)

  at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)

[2018-01-15 14:27:33,027] INFO Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn)

[2018-01-15 14:27:33,034] WARN Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect (org.apache.zookeeper.ClientCnxn)

java.net.ConnectException: Connection refused

  at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)

  at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)

  at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)

  at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)

[2018-01-15 14:27:33,427] INFO Terminate ZkClient event thread. (org.I0Itec.zkclient.ZkEventThread)

[2018-01-15 14:27:34,145] INFO Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn)

[2018-01-15 14:27:34,258] INFO Session: 0x0 closed (org.apache.zookeeper.ZooKeeper)

[2018-01-15 14:27:34,263] INFO EventThread shut down (org.apache.zookeeper.ClientCnxn)

[2018-01-15 14:27:34,269] FATAL [Kafka Server 0], Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)

org.I0Itec.zkclient.exception.ZkTimeoutException: Unable to connect to zookeeper server within timeout: 6000

  at org.I0Itec.zkclient.ZkClient.connect(ZkClient.java:880)

  at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:98)

  at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:84)

  at kafka.server.KafkaServer.initZk(KafkaServer.scala:157)

  at kafka.server.KafkaServer.startup(KafkaServer.scala:82)

  at kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:29)

  at kafka.Kafka$.main(Kafka.scala:46)

  at kafka.Kafka.main(Kafka.scala)

  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)

  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

  at java.lang.reflect.Method.invoke(Method.java:606)

  at io.osv.Jvm.runMain(Jvm.java:119)

  at io.osv.isolated.IsolatedJvm.access$400(IsolatedJvm.java:30)

  at io.osv.isolated.IsolatedJvm$3.run(IsolatedJvm.java:116)

[2018-01-15 14:27:34,303] INFO [Kafka Server 0], shutting down (kafka.server.KafkaServer)

[2018-01-15 14:27:34,319] INFO [Kafka Server 0], shut down completed (kafka.server.KafkaServer)

[2018-01-15 14:27:34,323] FATAL Fatal error during KafkaServerStartable startup. Prepare to shutdown (kafka.server.KafkaServerStartable)

org.I0Itec.zkclient.exception.ZkTimeoutException: Unable to connect to zookeeper server within timeout: 6000

  at org.I0Itec.zkclient.ZkClient.connect(ZkClient.java:880)

  at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:98)

  at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:84)

  at kafka.server.KafkaServer.initZk(KafkaServer.scala:157)

  at kafka.server.KafkaServer.startup(KafkaServer.scala:82)

  at kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:29)

  at kafka.Kafka$.main(Kafka.scala:46)

  at kafka.Kafka.main(Kafka.scala)

  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)

  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

  at java.lang.reflect.Method.invoke(Method.java:606)

  at io.osv.Jvm.runMain(Jvm.java:119)

  at io.osv.isolated.IsolatedJvm.access$400(IsolatedJvm.java:30)

  at io.osv.isolated.IsolatedJvm$3.run(IsolatedJvm.java:116)

[2018-01-15 14:27:34,347] INFO [Kafka Server 0], shutting down (kafka.server.KafkaServer)

Kind Regards,

Geraldo Netto
Sapere Aude => Non dvcor, dvco
http://exdev.sf.net/

[Waldek Kozaczuk]

In order to operate Kafka requires zookeeper to store and retrieve some configuration data. So the reason it fails is because there is no zookeeper running on port 2181.

I am not sure how to run 2 OSv instances on same host (one kafka and one zookeeper) due to some networking port conflicts. But you can run zookeeper natively on the host just to test kafka by going to apps/apache-zookeeper/ROOTFS/zookeeper and call ./bin/zkServer.sh start.

Then you have to modify apps//apache-kafka/assets/config/server.properties to point to zookeeper. It cannot be localhost:2181 but instead 192.168.122.1:2181. The 192.168.122.1 is what you need to use to make OSv instance running in QEMU/KVM to communicate with apps running on the same host.

So change this line:

zookeeper.connect=localhost:2181

to 

zookeeper.connect=192.168.122.1:2181

Waldek

PS. It would be nice to upgrade kafka and zookeeper to newer versions as well. But I think it would be event better to upgrade the apps build process to use new mikelamgelo capstan. I will describe my thoughts in a separate email. 

To unsubscribe from this group and stop receiving emails from it, send an email to osv-dev+u...@googlegroups.com.