Contributors wanted: Quantitative performance improvement study

[thom...@gmail.com]

There is a general agreement/notion in the developer community that go/golang is faster than python or node.Js for a typical i/o intensive web application.

However, there has been no comprehensive quantitative study on this so far that I am aware of.

I would like to propose a joint development of benchmark
1. go program
2. Python program
3. Node.Js program

Which can then be used for the experiments. We can report our findings and analysis in a co-authored paper for publication.

If anyone is interested, please reach me at thom...@gmail.com

Regards
Milind

[robert engels]

Some thoughts… not to throw cold water on your idea, but

Not many people write using the “http server/client” layers directly - almost everyone uses a framework of some sort - now you have hundreds of options on each platform.

Even if you wrote the programs from scratch as “equally using the built-in facilities”, most will probably dismiss the results as “they don’t matter as no one would write it that way…”

Given the above, you are probably just better off writing low-level networking, and disk IO tests, and then extrapolating that platform X is better suited to web servers.

Just MO.

> --
> You received this message because you are subscribed to the Google Groups "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Milind Thombre]

Good thx. I should have specified frameworks. Though there is value in collecting statistics for a raw benchmarking program too in my opinion. 

Django with python

Nodejs with Angular frontend

And idk what framework go user's use.

[robert engels]

Oh, and before anybody flames me, I am not implying that just because you are faster at network and disk IO makes it a better web server platform - there are other considerations (like dynamic code loading and others) that play a major part - I only offered in the context of the original question.

[Shulhan]

On Fri, 1 Feb 2019 15:13:35 -0800 (PST)
thom...@gmail.com wrote:

>
> However, there has been no comprehensive quantitative study on this
> so far that I am aware of.
>

Did you mean like Web Framework Benchmark by TechEmpower? [1]

[1] https://www.techempower.com/benchmarks/#section=intro&hw=ph&test=plaintext


--
{ "github":"github.com/shuLhan", "site":"kilabit.info" }

[robert engels]

Yes, but even those tests don’t provide a real analysis as to which is the better platform, because there is far more to consider than just the runtime performance - although for some cases it would immediately exclude certain platforms & frameworks.

[Milind Thombre]

Wow!

Django is an order of magnitude slower than nodejs. Good to know these numbers. Thanks for the link Shulhan! I don't see *go* in the list, which framework does a typical go web app use? 

But they used a *hello world* program! 

[robert engels]

No, there are a lot of different tests - you need to click on the tabs.

[Milind Thombre]

True. e.g. Development time/effort, availability of developers, support community strength and longetivity of the chosen technology platform to name a few!

[robert engels]

Exactly. In a lot of cases if the application can horizontal scale, the raw performance matters little if time to market is the greatest concern.

[robert engels]

Also, the sites testing methodology is seriously flawed, but for people that take stock in those sort of tests I guess it serves its purpose.

On Feb 1, 2019, at 6:39 PM, Milind Thombre <thom...@gmail.com> wrote:

[Milind Thombre]

Agreed, though even if it scales horizontally, your cloud service provider bill could scale vertically as well. :).

Time to market is a greater consideration for me currently , so I've already chosen python/Django for v1. 0. I am comfortable with Python and  am going to finish faster. 

But I wanted an exploration into golang running in parallel (different team) that could rewrite the cloud modules for future cost savings and faster response times. 

Does this strategy sound good to you guys? 

Regards 

Milind

[robert engels]

Depending on your scale, even 10% greater efficiency can mean a lot of dollars… But that is where cost / benefit analysis comes in… Cost to develop and maintain vs. decreased ongoing deployment costs (less servers/cpu needed, etc.)

[Milind Thombre]

Do you think we could jointly author

1. An unbiased research study with an improved methodology, better statistical analysis and appropriate and varied benchmarking program(s) ? 

2. CSP and other cost savings/increase  study (ROI) by shifting to go long term.

Also: if you think this idea is great, do you know anyone I can approach who may be willing to pay nominal research costs for such a proposal? 

Thx

Milind

[Robert Engels]

I’m sorry, but I don’t have the bandwidth. I also think it may be a fools errand, or so narrowly focused that it isn’t very valuable. Every case is somewhat unique which is why the architects get the big bucks to make the “right” call given all the factors. 

[Milind Thombre]

Thanks for the critique. 

I guess I will proceed to do a small study for my specific use case without external contributors then.....Keeping the big bucks for myself and not contributing anything to the community. 

Now off to lick my wounds ;) 

[Robert Engels]

Just because I’m not available doesn’t mean someone else isn’t interested. Like I said, just MO. 

[Jesper Louis Andersen]

On Sat, Feb 2, 2019 at 1:37 AM Milind Thombre <thom...@gmail.com> wrote:

Wow!

Django is an order of magnitude slower than nodejs. Good to know these numbers. Thanks for the link Shulhan! I don't see *go* in the list, which framework does a typical go web app use? 

This can be important, but I too want to stress that this is but one factor of a system. Earlier, some of the tests in the techempower benchmark could outperform other tests by the omission of headers. In a hello world program, the HTTP body is small, so the header size is a serious optimization target. In a real system, where you move many kilobytes per request, chances are that compression/encoding speed is more important to optimize. Also, as you add more functionality to your system, which can be baked into other frameworks, then your numbers will start going down.

But they used a *hello world* program!

The obvious limitation of Node.js is twofold:

* You have to work around cooperative concurrency. If a single event blocks the event handler, then both your throughput and latency can be severely hurt. A hello world style of program hides this. Go is preemptively scheduled, and even more so in later releases. This allows for far better latency responses, though at the expense of some throughput.

* Node.js is not a priori parallel and thus doesn't utilize more than a single CPU core. Modern CPUs are many-core machines, NUMA, and requires more work in Node.js to utilize fully. In Go, this usage comes "for free" because it runs a web server per request in a goroutine.

However, the hello program doesn't reflect any of these realities.

My experience is that systems are often slow due to unforeseen consequences[0]. Modern software is complex and the bottlenecks can occur in all parts of the system. Said bottlenecks are not always due to the speed of the programming language, the HTTP server, the framework and so. Rather, modern software requires statistics and experimental analysis through the scientific method. This is also why telemetry, observability, instrumentation, dynamic tracing, and measurement have become key factors to good system design in the later years. And some of these have the potential to slow down the system. However, the benefit they bring allows further optimization.

[0] Indeed, Test Lab C-33/a in the Black Mesa Research Facility style of problems often occur in software development.

[Milind Thombre]

We are on the same page! Excellent points.

[Amnon Baron Cohen]

https://www.techempower.com/benchmarks/#section=data-r15&hw=ph&test=json