Nameko Web Server Issues

[some...@vahanalytics.com]

Hi everyone,

I have been using Nameko in production for the last 10 months now and I've suddenly run into some weird issues for my http endpoints. Out of all the calls made to any of the http endpoints in the services, about 50% of the requests end up giving a 404. 

Can someone please point me to the right direction as to what I might be missing.

I have tried increasing max_workers without any effect.

I am attaching the typical behaviour. The requesting server is set to retry as a result it does get through in 2-3 tries. But I am unable to figure out the cause for this error in the first place. 

[13/Dec/2017 10:27:26] "POST /api/v1/analytics/device_gps HTTP/1.1" 404 361 0.000279

[13/Dec/2017 10:27:26] "POST /api/v1/analytics/device_gps HTTP/1.1" 200 195 0.003991

[13/Dec/2017 10:27:29] "POST /api/v1/analytics/device_gps HTTP/1.1" 404 361 0.000294

[13/Dec/2017 10:27:29] "POST /api/v1/analytics/device_gps HTTP/1.1" 200 195 0.003918

[13/Dec/2017 10:27:29] "POST /api/v1/analytics/device_gps HTTP/1.1" 200 195 0.003909

[13/Dec/2017 10:27:39] "POST /api/v1/analytics/device_gps HTTP/1.1" 404 361 0.000287

[13/Dec/2017 10:27:39] "POST /api/v1/analytics/device_gps HTTP/1.1" 404 361 0.000244

[13/Dec/2017 10:27:39] "POST /api/v1/analytics/device_gps HTTP/1.1" 404 361 0.000234

[13/Dec/2017 10:27:39] "POST /api/v1/analytics/device_gps HTTP/1.1" 404 361 0.000227

[13/Dec/2017 10:27:39] "POST /api/v1/analytics/device_gps HTTP/1.1" 200 195 0.003943

[13/Dec/2017 10:27:39] "POST /api/v1/analytics/device_gps HTTP/1.1" 200 195 0.003900

[13/Dec/2017 10:27:47] "POST /api/v1/analytics/device_gps HTTP/1.1" 404 361 0.000288

[13/Dec/2017 10:27:47] "POST /api/v1/analytics/device_gps HTTP/1.1" 200 195 0.005321

[13/Dec/2017 10:27:49] "POST /api/v1/analytics/device_gps HTTP/1.1" 200 195 0.003912

[13/Dec/2017 10:28:07] "POST /api/v1/analytics/device_gps HTTP/1.1" 404 361 0.000278

[13/Dec/2017 10:28:07] "POST /api/v1/analytics/device_gps HTTP/1.1" 200 195 0.004134

[13/Dec/2017 10:28:07] "POST /api/v1/analytics/device_gps HTTP/1.1" 404 361 0.000276

[13/Dec/2017 10:28:07] "POST /api/v1/analytics/device_gps HTTP/1.1" 404 361 0.000232

[13/Dec/2017 10:28:07] "POST /api/v1/analytics/device_gps HTTP/1.1" 404 361 0.000243

[13/Dec/2017 10:28:07] "POST /api/v1/analytics/device_gps HTTP/1.1" 404 361 0.000292

[13/Dec/2017 10:28:07] "POST /api/v1/analytics/device_gps HTTP/1.1" 200 195 0.003832

Another instance where I am seeing this behaviour:

[13/Dec/2017 07:22:47] "POST /api/v1/analytics/write_distance HTTP/1.1" 200 198 0.007928

[13/Dec/2017 07:26:01] "POST /api/v1/analytics/write_distance HTTP/1.1" 200 198 0.007584

[13/Dec/2017 07:27:12] "POST /api/v1/analytics/write_distance HTTP/1.1" 404 361 0.000307

[13/Dec/2017 07:28:44] "POST /api/v1/analytics/write_distance HTTP/1.1" 200 198 0.007077

[13/Dec/2017 07:31:08] "POST /api/v1/analytics/write_distance HTTP/1.1" 404 361 0.000284

[13/Dec/2017 07:32:43] "POST /api/v1/analytics/write_distance HTTP/1.1" 200 198 0.007327

[13/Dec/2017 07:33:49] "POST /api/v1/analytics/write_distance HTTP/1.1" 404 361 0.000291

[13/Dec/2017 07:36:53] "POST /api/v1/analytics/write_distance HTTP/1.1" 404 361 0.000303

[13/Dec/2017 07:38:15] "POST /api/v1/analytics/write_distance HTTP/1.1" 404 361 0.000247

[13/Dec/2017 07:39:16] "POST /api/v1/analytics/write_distance HTTP/1.1" 404 361 0.000289

[13/Dec/2017 07:42:37] "POST /api/v1/analytics/write_distance HTTP/1.1" 404 361 0.000322

[13/Dec/2017 07:43:20] "POST /api/v1/analytics/write_distance HTTP/1.1" 200 198 0.007246

I am running Nameko 2.6.0, Python 2.7.12 

Please let me know if any additional info is needed.

PS: All the rpc related entrypoints are working absolutely fine for me

Thanks,

Someshwar

[Matt Yule-Bennett]

Hi Someshwar,

There's not enough info here to know what's going on. Can you post your service code and more details about how you're hosting and running them?

Are these logs from the same nameko service instance, or a cluster? (i.e. does a single instance alternate between 200 and 404?)

Matt.

[some...@vahanalytics.com]

Hi Matt,

Thanks for getting back.

This issue is happening to only one of my projects

Now, I have 4 services in this file called service.py and I'm not running it with a ServiceContainer or ServiceRunner. Just 4 classes inside a file.

This is only 1 service instance and it is alternating between 200 and 404 for the same. I am using Supervisor to run the service with the usual: 

exec ./analytics_env/bin/nameko run --config ./config.yml service

Surprisingly enough, I removed one of the http entrypoints that I could move somewhere else and the problems seems to have been mitigated. Now, I am consistently getting 200s for the requests. 

My followup comments:
1) The above seems to make no sense.

2) There is no chance that there was an error in that function

3) The function was calling another api in return internally and processing a few things (I am using this function as sort of a router to hit other services further along my stack).

4) I do not believe that having another api call inside this http entrypoint could be the issue because I have other entrypoints that are doing so without any issues.

Even though it seems to suddenly have been resolved, I want to understand if I should continue to have http api endpoints with nameko or should I use a more robust platform like nodejs/flask to cater to apis while nameko works well with rpc services. (I already do most of my api servings from a nodejs server anyways).

Thanks,

Someshwar

[Matt Yule-Bennett]

Well that is curious. There is probably something funny going on with the four services sharing the same werkzeug WebServer (SharedExtensions are shared per-process, not per-service). If you're able to share the code (privately) I will try to figure out what was going on.

Re: Nameko HTTP vs NodeJS, I think it depends on your use-case. There are some things that you will get for free by using Nameko, such as the built-in RPC proxies to talk to other services and event handlers. Having said that even at my workplace our client-facing APIs are in NodeJS.

[Someshwar Dash]

Thanks Matt for this information (SharedExtensions are shared per-process, not per-service).

I have sent you the offending entrypoint function on gitter and we can continue there. I don't think there is any point following up on this post since nothing really came out of it. You can actually go ahead and delete this post entirely from the google group to reduce unactionable clutter :P

--
You received this message because you are subscribed to a topic in the Google Groups "nameko-dev" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/nameko-dev/SWUysVY77ps/unsubscribe.
To unsubscribe from this group and all its topics, send an email to nameko-dev+unsubscribe@googlegroups.com.
To post to this group, send email to namek...@googlegroups.com.

To view this discussion on the web, visit https://groups.google.com/d/msgid/nameko-dev/7b6f946d-a5a3-4478-ae15-204d9ba9f675%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Someshwar Dash

Co-Founder and CTO,

Vahanalytics