"Cloud native", Containers, Lambda's, ... fast startup and low memory

[Rob Bygrave]

So firstly, some of you may be aware I've been on a big Kubernetes, Docker, "Right sized services" project. There is a lot of articles coming out on this but I've seen some recent ones that miss some important details from a JVM perspective.

I co-presented on this topic to our local JVM group just last week.

So IMO (and keeping it short):

1) I do believe in the Docker Kubernetes trend.  It's a big and growing trend and I personally believe it will become the dominant deployment mechanism for services. 

2) JVM developers need to be able to go small with fast startup. We need to be able to get high deployment density on shared hardware and be able to compete with golang here.

3) The JVM can go small with fast startup with some caveats:

    - CGroup awareness is vital when going small (Java 8 with +UseCGroupMemoryLimitForHeap) 

    - Java 10 also made a big jump (measured as halving startup time for one specific case)

    - We (very likely) need to use frameworks and libraries that have reduced their use of classpath scanning, reflection and "automatic configuration" 

Note: By small I measured at  -cpus=0.5 -m100m  (That is, half a CPU and 100M total memory so the JVM heap would be ~25M)

4) What works on 8 CPU's and 32G (my laptop) can be very very poor at 0.5 CPU and 100M (e.g. Spring boot)

5) Frameworks (including Ebean) need to think about improving their startup times

What does this mean?

- Lots of libraries and frameworks on the JVM have tended to use a lot of classpath scanning, reflection etc to ease developer experience at the cost of startup CPU consumption and runtime memory.

- JVM developers are going to become more aware of this trade off and I believe there will be a move to use libraries and frameworks that are good in low resource environments (e.g. a 0.5 CPU and 100M total memory type environment). 

It's not just me and my colleagues saying this but it's stated clearly by micronaut.io where all Dependency Injection, AOP and Proxy generation happens at compile time and they compare it directly to the approach of SpringBoot. https://github.com/micronaut-projects/presentations/blob/master/Greach-2018.md .

What does this mean for Ebean?

- We already do lots of stuff better than the competition so we are pretty good already thankfully

- We may encourage the use of build time enhancement (but I suspect the majority are already doing this)

- We should encourage more explicit entity registration over scanning (e.g. serverConfig.addClass() or ServerConfigProvider ) or look to add a mechanism (like APT) to use code generation to do this automatically at compile or build time. 

- At some point we could look to profile and speed up our startup using optional code generation.  Look to avoid all reflection reads of bean properties, annotations etc and instead use code generation.

What are other projects doing?

- We had an interesting discussion on DI.  Guice 4 is apparently pretty quick. micronaut.io has taken the step of doing "dagger like" code generation for DI 

- Many "Web frameworks"  forgo scanning and reflection and are explicit (which makes them fast and light) 

So as I see it, right now we are ok.  We are better than the competition and my personal experience to date with low resource environments shows no issue with Ebean startup.  However,  at some point we need to look to do some profiling and thinking about startup.  If there are people struggling with going small / fast startup etc then let us know.

If anyone has a thought or experience in this area then by all means add to the discussion.

Cheers, Rob.

[robert engels]

I would love to hear your rationale in detail as to why?

Personally, I don’t get it. The JVM/Tomcat has always been the VM/container. Why put that in another container which has less services? The Tomcat/web/j2ee already supports easy horizontal and vertical scaling.

If anything, the Java ThreadGroup needs integration with CGroups, to allow better memory and CPU resource allocations/management by containers like Tomcat.

On the other side, even if you assume Docker like containers, if services become too small, they have little value, and tons of IPC overhead. For example, do you think a database is going to be lightweight? They all have considerable start-up times - and even longer if running in a Docker image.

Infrastructure components like EBean probably have very little value in a micro-services architecture, and who cares, almost all of the AWS like services are macro, not micro.

Anyway, rather than saying ‘I believe…’ and ‘need high deployment density’, why ? Why compete with Golang?

The architecture should fit the requirements, not the other way around - otherwise it is just marketing hype. See https://www.nemil.com/mongo/1.html for a great series on how the ‘architecture of the day’ really messes things up.

Just my two cents.

--

---
You received this message because you are subscribed to the Google Groups "Ebean ORM" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ebean+un...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Rob Bygrave]

> The Tomcat/j2ee 

First step depends on the opinion regarding putting "multiple wars/ears" into a single servlet container or jee container.  To me there are bunch of issues that occur with that.  You can't just patch a library that the container provides without effecting the other apps, one set of JVM parameters for all the applications, 1 JVM version for all applications, Thread locals and hot deployments etc.

If you want or like multiple wars in tomcat (or JEE) and you have never had problems with that ... don't go further - stay happy.

For people who are not so happy with that, the next step is deciding to go to 1 JVM per application. That avoids any and all issues with the above setup which is why some people prefer it. This could be called "runnable war" or "embedded server". Basically it is 1 JVM per application. With this we can usually patch everything up to the JVM (but typically we expect a JVM to be installed as a dependency).  Some people used RPM or .deb deployment with this.

The next step is docker where we use docker layers and effectively now patch down to the OS. To me the interesting point here is that we get to effectively provide the JVM with the service (using docker layers which is nice simple and efficient for this).

> if services become too small, they have little value, and tons of IPC overhead

Totally agree.  I use the term "Right sized services" myself and try to emphasis that it is important not to go small for kicks - on the JVM we have other mechanisms for modular development that give us a better result.  That said, some services are naturally small and when that is the case we don't want to suck (well, I don't anyway).  I've built a few services that are maybe 10 or less rest endpoints and I'd call them naturally small.  I've also merged 5 services back into one because they should never have been separated and this is your point - people are not thinking critically and clearly and there is a lot of hype based architecture going on.

> need high deployment density’, why ? 

If we deploy into the cloud we pay for the resources that we need (Memory and CPU).  If we are on premise then we maybe have relatively fixed hardware costs and are not impacted by "density" and so maybe don't notice too much.  

If we are deploying to the cloud and we can't get density (say we deploy a small application but it needs 500M to run because it is terribly inefficient) then we become relatively expensive from a deployment perspective.  Management will wonder why that simple service costs so much. 

> Why compete with Golang?

I personally think JVM developers / development teams are competing with .Net, Golang and NodeJS.  Roughly speaking, If we don't do a good job then they will get the work. For me, I'd say I think this is already happening.  

Also worth looking at is GraalVM native-images - compiling Java bytecode to native amongst other things.

Also there is JLink with the option to create smaller JVM for distribution.

> think a database is going to be lightweight

No.  I'm not talking about databases but services (typically Rest, Soap, SPA services).

To put some numbers out there ... at 0.5 CPU we can start a service in 3 seconds and the exact same functionality can start in 36 seconds.  It's nothing to do with the JVM in this case (Java 10 / CGroup aware) but purely down to the framework and library implementation.  The people who are impacted by slow startup and high memory usage care about this.  I'd say most people who deploy to Kubernetes will care about this (and yes in the past we haven't really had to worry about it).

Cheers, Rob.

[robert engels]

You can use multiple wars and still update the libs - the libs just need to local to the war. For some libs, like a database driver, it makes far more sense to put that in ‘shared’. Also, depending on the container, you can even update the ‘shared libs’, restart the ‘one or more apps’ and it will see the updated jars - depends on the container though. We did this routinely in a system I developed for a previous client.

But yes, the JVM per application does make things easier - as an app that generates lots of garbage, uses lots of CPU, etc. can be isolated/constrained far easier at the process level. Maybe I am not familiar enough with Docker, but having the JVM in a Docker container seems bad - you lose the ability to easily update the JVM for security issues, etc. - now you have to go and update all of the containers, re-test, etc. Whereas is you designed the application well from the start - the JVM version should make little difference. Java has been pretty awesome when it comes to backward compatibility.

As for your services argument, where is the rest data coming from? Most likely a database… so these small services are still dwarfed/limited by the database, so that governs your architecture and cost structure.

As for the Node ecosystem and other alternatives... Have you worked with it? It’s pretty bad. Node with Typescript is serviceable for some enterprise applications. The single threaded design prevents many types of applications from being written - at least easily. There is a reason that Java has not only survived but dominated over the last 20 years, it was well designed from the start. Node with JavaScript as its basis was poorly conceived from the beginning. Node/JS on the server is clearly a product of the hype machine, and poor engineering.

Golang is pretty horrible as well for many classes of applications - there is a reason that much of the Google back-end is written in Java, not Go. Also, their reasons for moving from Java to something like Go, are marketing/business, not technical. Google can’t stand it that they rely on technology designed/developed elsewhere - their ego doesn’t allow it. Most other organizations don’t have those concerns. Furthermore, Google has been pretty bad about releasing technologies, then abandoning them for something else, because they can - this is a horrific pattern/prospect for most other organizations.

Regards,

Robert

[Rob Bygrave]

> but having the JVM in a Docker container seems bad - you lose the ability to easily update the JVM for security issues,

IMO it is the opposite. Docker makes it easier to patch the JVM. This is really about the Kubernetes and Docker combination though and what they bring together and not just Docker.

> now you have to go and update all of the containers, re-test, etc.

That is what build pipelines should do and they can do this pretty seamlessly.

>  so these small services are still dwarfed/limited by the database

In my experience in terms of money/dollars the database cost doesn't dwarf the cost of the services (RDS Postgres for example) and the cost of the services is material but obviously that will depend on many many things.  The database cost occurs regardless of the "density" of the services though so it somewhat becomes a question of if the cost of the services is material.

> As for the Node ecosystem and other alternatives

I've written enough to know that we have a great eco-system on the JVM.  I've had to replace NodeJS services with Java based ones for performance reasons (let alone the tooling and libraries etc) but for me I think it is a case of not taking things for granted and questioning the relatively expensive techniques we have been using for the past few years. There is a lot of people who are (incorrectly in my opinion) saying that the JVM can't go into this "small" space but really it is more the case that we can't take some libraries/frameworks there (due to large amounts of scanning and reflection used).  

> Golang

You might be right but for me I don't see it that way. For me, I see many people (not just Google) using golang and a lot of them are going for the fast startup and low memory consumption.

 

Time will tell I figure. I have been using the JVM for 20 years and really like many many things about it and the eco-system ... so I'd like to use it and the same techniques for as wide a problem space as I can and that includes "going small".

Cheers, Rob.

[robert engels]

I am definitely not a Docker expert, so maybe many of my concerns are incorrect or out-of-date.

Thinking about this some more though, if the containerized applications are Java based, and third-party images provided, then how? You need to rely on them to update the Java version in their build setup. I see how it could be fairly trivial for locally produced images, even with lots of them (small services). Compare that with just ensuring the Java on the server is the latest.

As for ‘cloud ready’. It’s my understanding that for most (all?) of the major cloud vendors, CPU usage is billed. My testing shows a ‘docker under OSX’ node installation performs 30-50% slower than running the native node under OSX.  This is probably because at it’s heart Docker is Linux based as it relies on CGroups, so I would guess that running a Docker containerized app under linux itself should be more efficient, but it doesn’t seem possible as it still needs a virtual machine if it is going to ensure certain library versions, as these are in many cases tied to the kernel version. I thought Docker originally just relied on CGroup manipulation, but it seems to have gone beyond that??? But I’m probably wrong.

As an aside, I think programmatic configuration vs. ‘auto load’ is always better anyway. ‘auto load’ is only maintainable for the smallest of projects. So if the side-effect of this is faster start-up, that’s a great benefit anyway.

Cheers.

[robert engels]

I was incorrect on the performance numbers, the docker container is 5x slower than the native, 235 requests per second, vs 1187 for the native.

Probably not an insignificant cost difference in the cloud.

[robert engels]

In case someone stumbles across this thread, I have a deeper understanding on the containers, and it easily explains the performance difference, and why it probably doesn’t matter to most people.

Maybe it will be helpful to others.

A ‘container’ is a Linux only feature based on CGroups. Docker requires a Linux ‘host OS’ running kernel 3.10+ in order to ensure binary compatibility at the user land kernel call level. No ‘containerized’ applications can expect / configure a specific kernel version (unless you run a kvm process in a container… ugh).

When using Docker for Mac, or Docker for Windows, you need to run a Linux OS in a VM in order to host the containers. The local Docker commands communicate with the ‘true” Docker daemon running in the VM.

So, the performance difference I saw is caused by my host OS being OSX.

I would expect a Linux OS to experience performance numbers similar to the native even when running in a Docker container, BUT, if using something like AWS, where you are running in a VM to start with, I expect similar performance degradation.

I didn’t mean to hijack the thread, but since the original topic was on making EBean more efficient in containers, which may be a bit of work, it seemed that at least a ‘why should we do this’, as a means to keep the marketing hype in check, was prudent.

There’s a part of me that honestly thinks this is a cloud vendor money grab - you’re having performance problems? just scale up (out) ! oh and by the way we charge by the CPU second…

Cheers.

[Daryl Stultz]

On Tue, Jul 3, 2018 at 7:25 AM robert engels <ren...@ix.netcom.com> wrote:

In case someone stumbles across this thread, I have a deeper understanding on the containers, and it easily explains the performance difference, and why it probably doesn’t matter to most people.

I run on Mac OSX. I have recently begun running Tomcat (and Postgresql) through Docker. Postgresql doesn't appear to give any issues with performance. Tomcat is slow to start due to file system throughput being rather slow on the Mac. I mounted my volumes with :delegate flag in docker-compose and things get better but still not great. In any case, not an Ebean problem.

/Daryl

[Rob Bygrave]

> I didn’t mean to hijack the thread

No that's all good, nice additions to the general discussion.

> I would expect a Linux OS to experience performance numbers similar to the native

On Linux I've heard from various sources that the overhead is really small.  I myself am not an OSX person. 

> There’s a part of me that honestly thinks this is a cloud vendor money grab

I am sure many others will be thinking something similar.

Note that Kubernetes is not cloud only - it can run anywhere. It has orchestration and load balancing built in supporting fully automated rollout (rolling update of releases). We can just declare that we want a service to go from version 1 to version 2 and it will perform the rolling update.  Regardless of whether we want a service to auto scale etc the automatic rolling update is nice, especially for people wanting to move towards Continuous delivery.

I've hit cases where scaling up via adding pods is more an anti-pattern for specific services. Instead it can be a lot better to give more CPU to fewer services (for these services in question).  Additionally, less pods/instances is also generally better for "near caching" which can get somewhat forgotten about.

A fair bit to think about ...

Cheers, Rob.

--

[robert engels]

Just a final follow-up on this. Internally monitoring of the VM container shows the actual request processing time to be fairly similar to the native, so it appears that much of the difference in speed is probably related to the virtual networking overhead, etc. so based on the specific performance needs of the application, using a docker container - even in a VM - may not be that big of a deal.