Lagom Framework for scala

Mangesh Deshmukh

unread,

May 19, 2017, 5:43:10 PM5/19/17

to Lagom Framework Users

Hi

I am trying out Lagom for Scala using simple instructions provided here.

https://www.lagomframework.com/get-started-scala.html

Love the simplicity but have few questions. The end step in the documentation is to go to http://localhost:9000/api/hello/World. When I invoke this, I get following message:

Bad Gateway

The gateway could not establish a connection to the service

I see this in log:

[info] Loading project definition from /Users/mdeshmukh/work/evaluate/lagom/helloWorld/hello/project

[info] Set current project to hello (in build file:/Users/mdeshmukh/work/evaluate/lagom/helloWorld/hello/)

[info] Starting Kafka

[info] Starting Cassandra

.SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".

SLF4J: Defaulting to no-operation (NOP) logger implementation

SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.

.......

[info] Cassandra server running at 127.0.0.1:4000

Kafka Server closed unexpectedly.

[info] Service locator is running at http://localhost:8000

[info] Service gateway is running at http://localhost:9000

[info] Service hello-impl listening for HTTP on 0:0:0:0:0:0:0:0:57797

[info] Service hello-stream-impl listening for HTTP on 0:0:0:0:0:0:0:0:58322

[info] (Services started, press enter to stop and go back to the console...)

When I go to Gateway page at http://localhost:9000, I see this:

GET\Q/stream\EService: hello-stream (http://0.0.0.0:58322)

GET\Q/api/hello/\E([^/]+)Service: hello (http://0.0.0.0:57797)

POST\Q/api/hello/\E([^/]+)Service: hello (http://0.0.0.0:57797)

I have some questions:

1. How does Lagom decide the IP address to "0.0.0.0" while when starting Cassandra, it correctly set it as "127.0.0.1"?

2. Kafka does not come up. See following message:

"Kafka Server closed unexpectedly."

Checked Kafka logs under "hello/target/lagom-dynamic-projects/lagom-internal-meta-project-kafka/target/server.log" (for the benefit of people who do not know where to look for Kafka logs) and see this message:

[2017-05-19 14:10:53,838] INFO starting (kafka.server.KafkaServer)
[2017-05-19 14:10:53,844] INFO Connecting to zookeeper on localhost:2181 (kafka.server.KafkaServer)
[2017-05-19 14:10:53,850] INFO Starting ZkClient event thread. (org.I0Itec.zkclient.ZkEventThread)
[2017-05-19 14:10:53,863] INFO Waiting for keeper state SyncConnected (org.I0Itec.zkclient.ZkClient)
[2017-05-19 14:10:53,895] INFO zookeeper state changed (SyncConnected) (org.I0Itec.zkclient.ZkClient)
[2017-05-19 14:10:53,934] INFO Loading logs. (kafka.log.LogManager)
[2017-05-19 14:10:53,939] INFO Logs loading complete. (kafka.log.LogManager)
[2017-05-19 14:10:54,004] INFO Starting log cleanup with a period of 300000 ms. (kafka.log.LogManager)
[2017-05-19 14:10:54,005] INFO Starting log flusher with a default period of 9223372036854775807 ms. (kafka.log.LogManager)
[2017-05-19 14:10:54,031] INFO Awaiting socket connections on 0.0.0.0:9092. (kafka.network.Acceptor)
[2017-05-19 14:10:54,033] INFO [Socket Server on Broker 0], Started 1 acceptor threads (kafka.network.SocketServer)
[2017-05-19 14:10:54,044] INFO [ExpirationReaper-0], Starting  (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2017-05-19 14:10:54,044] INFO [ExpirationReaper-0], Starting  (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2017-05-19 14:10:54,142] INFO [ExpirationReaper-0], Starting  (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2017-05-19 14:10:54,143] INFO [ExpirationReaper-0], Starting  (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2017-05-19 14:10:54,153] INFO [GroupCoordinator 0]: Starting up. (kafka.coordinator.GroupCoordinator)
[2017-05-19 14:10:54,154] INFO [GroupCoordinator 0]: Startup complete. (kafka.coordinator.GroupCoordinator)
[2017-05-19 14:10:54,156] INFO [Group Metadata Manager on Broker 0]: Removed 0 expired offsets in 4 milliseconds. (kafka.coordinator.GroupMetadataManager)
[2017-05-19 14:10:54,165] INFO [ThrottledRequestReaper-Produce], Starting  (kafka.server.ClientQuotaManager$ThrottledRequestReaper)
[2017-05-19 14:10:54,166] INFO [ThrottledRequestReaper-Fetch], Starting  (kafka.server.ClientQuotaManager$ThrottledRequestReaper)
[2017-05-19 14:10:54,170] INFO Will not load MX4J, mx4j-tools.jar is not in the classpath (kafka.utils.Mx4jLoader$)
[2017-05-19 14:10:54,185] INFO Creating /brokers/ids/0 (is it secure? false) (kafka.utils.ZKCheckedEphemeral)
[2017-05-19 14:10:54,190] INFO Result of znode creation is: NODEEXISTS (kafka.utils.ZKCheckedEphemeral)
[2017-05-19 14:10:54,190] FATAL [Kafka Server 0], Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)
java.lang.RuntimeException: A broker is already registered on the path /brokers/ids/0. This probably indicates that you either have configured a brokerid that is already in use, or else you have shutdown this broker and restarted it faster than the zookeeper timeout so it appears to be re-registering.
   at kafka.utils.ZkUtils.registerBrokerInZk(ZkUtils.scala:305)
   at kafka.utils.ZkUtils.registerBrokerInZk(ZkUtils.scala:291)
   at kafka.server.KafkaHealthcheck.register(KafkaHealthcheck.scala:70)
   at kafka.server.KafkaHealthcheck.startup(KafkaHealthcheck.scala:51)
   at kafka.server.KafkaServer.startup(KafkaServer.scala:244)
   at kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:37)
   at com.lightbend.lagom.internal.kafka.KafkaLocalServer.start(KafkaLocalServer.scala:38)
   at com.lightbend.lagom.internal.kafka.KafkaLauncher$.main(KafkaLauncher.scala:45)
   at com.lightbend.lagom.internal.kafka.KafkaLauncher.main(KafkaLauncher.scala)
[2017-05-19 14:10:54,192] INFO [Kafka Server 0], shutting down (kafka.server.KafkaServer)

3. Where do I find Gateway/ServiceLocator logs? How/where do I change the log level for Gateway/ServiceLocator?

4. Why do we see SLF4J error? What can we do about it. I see some documentation in the given link. I see that I can apply it to the project but what about changing log levels for Gateway and ServiceLocator?

.SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".

SLF4J: Defaulting to no-operation (NOP) logger implementation

SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.

Thanks,

Mangesh

Ignasi Marimon-Clos i Sunyol

unread,

May 22, 2017, 6:53:12 AM5/22/17

to Mangesh Deshmukh, Lagom Framework Users

Lagom services are bound to a pseudo-random port in all the network interfaces (0.0.0.0).

Cassandra, on the other hand in only bound to 127.0.0.1 (port 4000).

This probably means you either have another kafka server already running of the file storage for kafka or zookeeper got corrupted.

Try using the jps command in the Oracle JDK to list all the java processes to locate a KafkaLauncher and kill -9 that.

$ jps

34115 NailgunRunner

35801 KafkaLauncher

35802 CassandraLauncher

35771 sbt-launch.jar

31342

35807 Jps

Also try removing your "hello/target/" folder completely to make sure the issue was not related to a corrupted ZK storage.

Thanks,
Mangesh

--
You received this message because you are subscribed to the Google Groups "Lagom Framework Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to lagom-framework+unsubscribe@googlegroups.com.
To post to this group, send email to lagom-framework@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/lagom-framework/99323f90-33da-4822-b2d3-741555885319%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Mangesh Deshmukh

unread,

May 22, 2017, 12:20:28 PM5/22/17

to Ignasi Marimon-Clos i Sunyol, Lagom Framework Users

Hi Ignasi,

Tried running "jps" but there was no stray Kafka process.

Then I deleted hello/target folder. That seems to have removed "Kafka stopped unexpectedly" message but I don't see any "Kafka Started successfully" kind of message. But I do see "KafkaLauncher" when I run "jps".

My original problem remains, which is that the Gateway is unable to reach underlying service and I see "Bad Gateway" message.

Also, it would be nice to know, as to how can I control the log settings for Gateway and ServcieLocator. The documentation talks about Logback.xml and I tried placing it at several places but don't see it talking effect.

Thanks,

Mangesh

Ignasi Marimon-Clos i Sunyol

unread,

May 23, 2017, 9:43:23 AM5/23/17

to Mangesh Deshmukh, Lagom Framework Users

Hi Mangesh,

> Then I deleted hello/target folder. That seems to have removed "Kafka stopped unexpectedly" message but I don't see any "Kafka Started successfully" kind of message. But I do see "KafkaLauncher" when I run "jps".

We do a best effort to trace a message when a service or Cassandra are available. Such a message doesn't exist for Kafka unfortunately (we only alert when Kafka crashed) so if you don't see "Kafka server closed unexpectedly" you can assume it is running.

I forgot to answer your second half of the e-mail the other day.

It is possible you are doing the right setup but are being impacted by a known issue in Lagom scala (https://github.com/lagom/lagom/issues/534) where logback.xml is not properly loaded.

Finally, you mentioned you are getting a Bad Gateway error.

This could be caused by several network-related issues. Are you using an OS with a PROXY set up?

When loading the application you should a message like:

[info] Service hello-impl listening for HTTP on 0:0:0:0:0:0:0:0:57797

[info] Service hello-stream-impl listening for HTTP on 0:0:0:0:0:0:0:0:58322

Are you able to run the following command successfully?

$ curl http://localhost:57797/api/hello/Alice

Hello, Alice!

That curl command is a direct access to the service where

$ curl http://localhost:9000/api/hello/Alice

Hello, Alice!

is an access via the Service Gateway.

I want to understand better which of the two fails since the cause for error differs.

I read your original mail again and everything looks good to me. Keep in mind that a Service Gateway is not a UI but a simple proxy that can redirect requests into each service. You original mail mentioned you see a 404 and a list of alternative endpoints to try. Are you seeing "Bad Gateway" when trying to access one of the valid endpoints (like in my Alice samples above)?

HTH,

--

Ignasi Marimon-Clos

Software Developer @ Lagom

Mangesh Deshmukh

unread,

May 23, 2017, 12:01:20 PM5/23/17

to Ignasi Marimon-Clos i Sunyol, Lagom Framework Users

Hi Ignasi,

Thanks for taking time to respond to my queries.

I am able to reach the endpoints by themselves. Only have problem when routing through Gateway.

Also, I don't believe I am behind proxy (don't see any settings in browser and I am successfully able to connect to internet). This is why I wanted to see if there are any logs on Gateway that I can check to see what is going on behind the scene.

Thanks,

Mangesh

Ignasi Marimon-Clos i Sunyol

unread,

May 24, 2017, 10:56:12 AM5/24/17

to Mangesh Deshmukh, Lagom Framework Users

Hi Mangesh,

would it be possible for you to copy paste the console output of the following commands:

curl -v http://localhost:57797/api/hello/Alice

curl -v http://localhost:9000/api/hello/Alice

Thanks,

To view this discussion on the web visit https://groups.google.com/d/msgid/lagom-framework/CA%2B-wor3r%3D946T4JXWQBMzhGukh4Mp_ozLpTdQRTzcwi8L6ezzg%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

Mangesh Deshmukh

unread,

May 26, 2017, 12:38:35 PM5/26/17

to Ignasi Marimon-Clos i Sunyol, Lagom Framework Users

Hi Ignasi,

Here is the output for those 2 requests.

In case of Gateway request, it sits there for few seconds before getting 502.

mdeshmukh-ltm1:~ mdeshmukh$ curl -v http://localhost:57797/api/hello/Alice

* Trying 127.0.0.1...

* TCP_NODELAY set

* Connected to localhost (127.0.0.1) port 57797 (#0)

> GET /api/hello/Alice HTTP/1.1

> Host: localhost:57797

> User-Agent: curl/7.51.0

> Accept: */*

>

< HTTP/1.1 200 OK

< Content-Length: 13

< Content-Type: text/plain

< Date: Fri, 26 May 2017 16:36:10 GMT

<

* Curl_http_done: called premature == 0

* Connection #0 to host localhost left intact

Hello, Alice!mdeshmukh-ltm1:~ mdeshmukh$

mdeshmukh-ltm1:~ mdeshmukh$

mdeshmukh-ltm1:~ mdeshmukh$ curl -v http://localhost:9000/api/hello/Alice

* Trying 127.0.0.1...

* TCP_NODELAY set

* Connected to localhost (127.0.0.1) port 9000 (#0)

> GET /api/hello/Alice HTTP/1.1

> Host: localhost:9000

> User-Agent: curl/7.51.0

> Accept: */*

>

< HTTP/1.1 502 Bad Gateway

< Content-Type: text/html; charset=utf8

< Content-Length: 4879

< Date: Fri, 26 May 2017 16:36:54 GMT

<

<!DOCTYPE html>

<head>

<title>Bad gateway</title>

<style>

html, body, pre {

margin: 0;

padding: 0;

font-family: Monaco, 'Lucida Console', monospace;

background: #ECECEC;

}

h1 {

margin: 0;

background: #A31012;

padding: 20px 45px;

color: #fff;

text-shadow: 1px 1px 1px rgba(0,0,0,.3);

border-bottom: 1px solid #690000;

font-size: 28px;

}

a {

color: #D36D6D;

}

p#detail {

margin: 0;

padding: 15px 45px;

background: #F5A0A0;

border-top: 4px solid #D36D6D;

color: #730000;

text-shadow: 1px 1px 1px rgba(255,255,255,.3);

font-size: 14px;

border-bottom: 1px solid #BA7A7A;

}

p#detail.pre {

white-space: pre;

font-size: 13px;

overflow: auto;

}

p#detail input {

background: #AE1113;

background: -webkit-linear-gradient(#AE1113, #A31012);

background: -o-linear-gradient(#AE1113, #A31012);

background: -moz-linear-gradient(#AE1113, #A31012);

background: linear-gradient(#AE1113, #A31012);

border: 1px solid #790000;

padding: 3px 10px;

text-shadow: 1px 1px 0 rgba(0, 0, 0, .5);

color: white;

border-radius: 3px;

cursor: pointer;

font-family: Monaco, 'Lucida Console';

font-size: 12px;

margin: 0 10px;

display: inline-block;

position: relative;

top: -1px;

}

h2 {

margin: 0;

padding: 5px 45px;

font-size: 12px;

background: #333;

color: #fff;

text-shadow: 1px 1px 1px rgba(0,0,0,.3);

border-top: 4px solid #2a2a2a;

}

pre {

margin: 0;

border-bottom: 1px solid #DDD;

text-shadow: 1px 1px 1px rgba(255,255,255,.5);

position: relative;

font-size: 12px;

}

pre span.line {

text-align: right;

display: inline-block;

padding: 5px 5px;

width: 30px;

background: #D6D6D6;

color: #8B8B8B;

text-shadow: 1px 1px 1px rgba(255,255,255,.5);

font-weight: bold;

}

pre span.code {

padding: 5px 5px;

position: absolute;

right: 0;

left: 40px;

}

pre:first-child span.code {

border-top: 4px solid #CDCDCD;

}

pre:first-child span.line {

border-top: 4px solid #B6B6B6;

}

pre.error span.line {

background: #A31012;

color: #fff;

text-shadow: 1px 1px 1px rgba(0,0,0,.3);

}

pre.error {

color: #A31012;

}

pre.error span.marker {

background: #A31012;

color: #fff;

text-shadow: 1px 1px 1px rgba(0,0,0,.3);

}

</style>

</head>

<h1>Bad gateway</h1>

<p id="detail" class="pre">The gateway could not establish a connection to the service</p>

</body>

</html>

* Curl_http_done: called premature == 0

* Connection #0 to host localhost left intact

mdeshmukh-ltm1:~ mdeshmukh$

Thanks,

Mangesh

Ignasi Marimon-Clos i Sunyol

unread,

May 30, 2017, 6:53:45 AM5/30/17

to Mangesh Deshmukh, Lagom Framework Users

Hi Mangesh,

sorry for the delay.

I see in the output you provide that the Lagom Service gateway is available but can't locate a service.

But I also see in your output that the service is actually ready and usable.

In that case, this could be a service location issue. The gateway seems unable to locate and/or connect to the backend service.

"The gateway could not establish a connection to the service"

What is the output for curl -v http://localhost:8000/services ?

That endpoint is the lagom's service locator API where you can see where each service is bound. The lagom service gateway will use that to route the request from http://localhost:9000/api/hello/Alice to http://localhost:57797/api/hello/Alice .

There are possible issues:

- the service is not published in the service locator

- the service is locatable under an invalid IP that the gateway can't connect to

- the service is locatable under 127.0.0.1 but the gateway (running on a JVM) picked up some PROXYing setup in your OS that redirects the requests to a PROXY that can enroute the request back to the service

Hope this helps!

Stephen Couchman

unread,

Sep 6, 2017, 8:09:50 PM9/6/17

to Lagom Framework Users

Sorry to bring up an old topic, but I seem to be having EXACTLY this problem.

The service gateway times out trying to find the hello service, running completely off defaults on a newly rebooted OSX with no firewall.

I can talk to the services directly without issue.

The runtime log from sbt is spamming out events involving Kafka. It's long so I put it here https://pastebin.com/TrVggp2n

The service gateway answers:

curl http://localhost:8000/services

[{"name":"cas_native","url":"tcp://127.0.0.1:4000/cas_native"},{"name":"kafka_native","url":"tcp://localhost:9092/kafka_native"},{"name":"hello","url":"http://0.0.0.0:57797"},{"name":"hello-stream","url":"http://0.0.0.0:58322"}]

Any help is appreciated.

Thanks,

Stephen

Thanks,
Mangesh

To unsubscribe from this group and stop receiving emails from it, send an email to lagom-framewo...@googlegroups.com.
To post to this group, send email to lagom-f...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/lagom-framework/99323f90-33da-4822-b2d3-741555885319%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Ignasi Marimon-Clos
Software Developer @ Lagom

--
You received this message because you are subscribed to the Google Groups "Lagom Framework Users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to lagom-framewo...@googlegroups.com.
To post to this group, send email to lagom-f...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/lagom-framework/CA%2B-wor3r%3D946T4JXWQBMzhGukh4Mp_ozLpTdQRTzcwi8L6ezzg%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--
Ignasi Marimon-Clos
Software Developer @ Lagom

Ignasi Marimon-Clos i Sunyol

unread,

Sep 7, 2017, 4:01:42 AM9/7/17

to Stephen Couchman, Lagom Framework Users

Hi Stephen,

I think you're hit by the dreaded OSX-java issue:

https://thoeni.io/post/macos-sierra-java/

https://plumbr.eu/blog/java/macos-sierra-problems-with-java-net-inetaddress-getlocalhost

Long story short: review the contents of your /etc/hosts

HTH,

To unsubscribe from this group and stop receiving emails from it, send an email to lagom-framework+unsubscribe@googlegroups.com.
To post to this group, send email to lagom-framework@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/lagom-framework/d22b7b95-bd60-41f7-a611-612d783d8dd5%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

Reply all

Reply to author

Forward