GC, safepoints & IOs

[Jean-Philippe BEMPEL]

Hello,

From the excellent newsletter Java Performance Tuning, Jack Shirazi comments about this article: http://blogs.mulesoft.org/performance-impact-io-intensive-app/

I quote:

"If one of application threads is trying to write while the kernel is busy flushing, this thread will get stuck behind the flushing job and won’t be able to respond to the summon to pause. This would cause a ripple effect: The busy disk blocks the write thread, the thread prolonged the GC pause, and the pause makes the application appear non-responsive."

From my understanding of safepoint mechanism, it seems not correct. IO operations are performed trough JNI calls and therefore are already in safepoint when the call in native world is blocked by the flushing job. So there shouldn't be issue in TTSP (Time To SafePoint) due to this flushing job.

Is my understanding is correct, or I missed something ?

Thanks.

[Vladimir Sitnikov]

The wording in article is moot. It would be great if it could be
updated/enhanced.

>IO operations are performed trough JNI calls

Not all IOs are equal.

1) What if APP is using memory mapped files to perform IO?
What if it is using Unsafe.copyMemory?

I think can affect TTSP. Can it?

2) If GC log is placed on the busy drive, then pause times might be
affected due to GC threads being blocked on gc log write.
Well, it has nothing to do with TTSP, but it is related with GC pause.

Vladimir

[Jean-Philippe BEMPEL]

On Monday, March 2, 2015 at 11:21:25 AM UTC+1, Vladimir Sitnikov wrote:

The wording in article is moot. It would be great if it could be
updated/enhanced.


>IO operations are performed trough JNI calls

Not all IOs are equal.

Agreed, but I think the article tlak about regular basic disk IO (like FileOutputStream), but it is my assumption.

 

1) What if APP is using memory mapped files to perform IO?
What if it is using Unsafe.copyMemory?

I think can affect TTSP. Can it?

Seems to me it can totally be affected.

[Nitsan Wakart]

MMapped buffer writes are not at a safepoint, so can delay it. 'Normal' file writes are at a safepoint.

Unsafe copy memory does not include a safepoint, neither does System.arrayCopy. Curiously, if you are using ByteBuffer.put() on a direct buffer you will get a chunking loop via Bits.copyFrom/ToArray(note the loop is technically counted but on a long counter, a bit fragile?) but only if you are copying in a heap buffer (and the other way around heap.put(direct)). So chunking happens for the cross type case, but not for the same type. Memory copies are not the issue discussed in the article by the sound of it, but as the array/buffer copied is of an arbitrary size they are potentially safepoint delayers.

--
You received this message because you are subscribed to the Google Groups "mechanical-sympathy" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mechanical-symp...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Vladimir Sitnikov]

> Memory copies are not the issue discussed in the article

I was just listing a couple of ways to write value to mmaped file. I
meant "if app is using Unsafe.copyMemory to perform IO, then it might
get stuck".
I'm not an expert in using mmap, so I was curious if it indeed might
affect TTSP.

> but as the array/buffer copied is of an arbitrary size they are potentially safepoint delayers.

That is a bit different issue.

Vladimir

[Kirk Pepperdine]

Seems to me it can totally be affected.

I’ve seen this happen on more than 1 occasion…. 

 

2) If GC log is placed on the busy drive, then pause times might be
affected due to GC threads being blocked on gc log write.
Well, it has nothing to do with TTSP, but it is related with GC pause.

Indeed, seen this also.

— Kirk

[Gil Tene]

This certainly happens for GC mechanisms that are silly enough to do potentially blocking writes to the GC log while in a pause... Which [unfortunately] seems to be most GC mechanisms.

In Zing, GC logging lines are accumulated in a memory buffer that is only flushed outside of safepoints, for this very reason...

-- Gil.

[Gil Tene]

Normal mapped files in Java (MappedByteBuffer) do exhibit TTSP issues as described in the top of this thread (no need to use Unsafe to get hurt by this). We've measured TTSPs as high as 800msec on systems that do high volumes of journaling or other heavy-write operations. The process that is TTSP-affected doesn't have to have anything to do with the heavy journaling, it just has to be unlucky enough to run on the same system, and to have a page fault on a single MappedByteBuffer access (either get or put) get stuck behind 120+ other disk operations in the Linux i/o scheduler, each of which can take several milliseconds to complete. It is [unfortunately] common to see such disk i/o bursts initiated by the kernel when flushing accumulated dirty page caches...

So [ironically] non-mapped i/o in Java is much less susceptible to TTSP than mapped I/O is... At least currently.