death by too many threads

[Dave Dyer]

Investigating the behavior of IOS threads, I wrote a simple torture program
to serially create a lot of threads, each thread creates a little garbage
and exits.  I found that this program rapidly crashes IOS, having run out
of memory.

In Nativemethods.m I discovered that the threadsToDelete array fills up, and
once full the code in ThreadRunner just drops it if there's no place in the
array, which results in permanent leakage of the thread's resources.

This ought to be rewritten to use a non-finite resource for the threads
pending deletion.   If you agree with the diagnosis, I will do that
and submit a push.

[Shai Almog]

Other than supporting a theoretical, yet impractical case what would that accomplish?
Limiting the number of threads created is essential in every OS.

[Dave Dyer]

My test creates threads one at a time, at the modest rate of 10 per second.  There
are never more than one or two of them live at any time.  That doesn't seem out of
the ordinary to me.  A server app that spawned threads at that rate would be considered
low impact.

[Dave Dyer]

I've got a working version that avoids running out of memory at 10 threads/second,
but if the rate is 100 instead of 10, thread creation still outruns the gc and we run
out of memory.

I suspect the culprit here is the over-simple, arbitrary 30 second pause in the
system process.   This ought to be modified somehow;  My thought is that
this delay ought to be interrupted if the heap reaches some target of
memory full.

For some reason, it takes 2 complete GC cycles for thread memory to actually be reclaimed,
so for example when the first GC-mark phase is triggered, 300 processes are slated for deletion, when
the second gc mark phase is triggered, they are all still there, so the pending count is 600,
then the first 600 are actually finalized and the pending deletion count stabilizes at 600.

Another oddity I noticed is that GC-mark frequently waits for 10-30 seconds for
for process 3, which seems to be the EDT process.

Finally, no matter how slowly processes are created, the IOS simulator returns an error 35 "too many threads"
at around 5000. It's unclear what the problem may be.  Perhaps its only a limit in the simulator.

[Shai Almog]

There is interruption if you are running out of RAM not if you are running out of threads.

The two cycle cleanup is true for JavaSE too, read about finalizer() in JavaSE VM's the concepts are universally true.

[Dave Dyer]

On Monday, May 1, 2017 at 9:38:51 PM UTC-7, Shai Almog wrote:

There is interruption if you are running out of RAM not if you are running out of threads.

According to the IOS simulator control panel, the ram in use is modest.   The pthread_create
fails with error 35 at around the 5500'th process, regargless how long the elapsed time.

--

The two cycle cleanup is true for JavaSE too, read about finalizer() in JavaSE VM's the concepts are universally true.

Always looking for good reads. :)

[Dave Dyer]

I verified that the behavior is very similar on a real device, in particular that even creating
short-lived threads at the sedate rate of one per second, after a few thousand seconds
pthread_create fails with error 35.   I haven't found any suggestion that this should be
the case.

This isn't an unreasonable test; if reading a web page, or sending a datagram,
spawns a couple threads, this number of threads would be created in a short
session.

[Shai Almog]

Spawns a couple isn't 100 per minute. A GC won't occur if RAM is in low use, thread garbage doesn't implicitly trigger a GC.

[Dave Dyer]

I've instrumented gcMark to show the size of the dead process queue.
Gc seems to happen every 30 seconds like clockwork.

[Dave Dyer]

Here's a little supporting log from one of my tests.  This one creates 1000 short lived processes
at the rate of 10 per second, then switches to only creating a few strings.

2017-05-03 11:28:07.160 Dtest[31914:891522] GC mark, 0 dead processes pending
2017-05-03 11:28:13.783 Dtest[31914:891522] GC mark, 0 dead processes pending
2017-05-03 11:28:14.027 Dtest[31914:891522] GC mark, 0 dead processes pending
2017-05-03 11:28:44.145 Dtest[31914:891522] GC mark, 293 dead processes pending
2017-05-03 11:29:14.209 Dtest[31914:891522] GC mark, 586 dead processes pending
2017-05-03 11:29:44.243 Dtest[31914:891522] GC mark, 879 dead processes pending
2017-05-03 11:30:14.320 Dtest[31914:891522] GC mark, 706 dead processes pending
2017-05-03 11:30:44.372 Dtest[31914:891522] GC mark, 413 dead processes pending
2017-05-03 11:31:14.424 Dtest[31914:891522] GC mark, 120 dead processes pending
2017-05-03 11:31:44.438 Dtest[31914:891522] GC mark, 0 dead processes pending

As you can see, the GCs occur at completely regular intervals.  The "706" line
corresponds to where new processes were no longer being spawned.

[Shai Almog]

We GC every 30 seconds if there is no reason to trigger GC sooner. Thread death isn't considered as a reason to trigger GC.

[Dave Dyer]

Thread creation doesn't trigger a gc either.  creating a lot of short lived threads will run
out of memory even before the mysterious ios thread limit, and even with the thread
gc patch i submitted.

Trying to limit the number of balls in the air, I haven't opened a discussion of a thread pool yet.

[Shai Almog]

Those don't make as much sense on a mobile device. A thread is pretty expensive, if you are hitting those limits the problem is conceptual. We probably shouldn't crash but degrading performance to such an extent might be worse than crashing.

[Dave Dyer]

I'm not concerned about thread factories today, but one has to stress these systems to know
Where they will and will not perform well in the future. The trajectory for mobiles can only
Be toward more intensive uses.

[Dave Dyer]

I added a commit to my currently pending push request which fixes the "error 35" problem.  With
this fix, creating new threads (at a modest rate) runs indefinitely. 

It's still a problem that creating threads will never trigger a GC, so you're likely to run out of
memory if you work too fast.

[Shai Almog]

Nope. The trajectory of mobile is for battery life and that doesn't move as fast as Moors law.