BufferedReader vs NIO Buffer
arun....@gmail.com
Could someone pls tell me why is the NIO buffering faster than the IO
BufferedReader class.
I agree NIO uses FileChannels to access the file. But it in turn uses
the FileInputStream.
Example 1: BufferedReader
BufferedReader reader = new BufferedReader(new
FileInputStream("File.txt"));
reader.read(....);
Example 2: NIO Bufferes
FileChannel fc = new FileInputStream("File.txt").getChannel();
ByteBuffer buffer = ByteBuffer.allocate(1024);
fc.read(buffer);
Keeping the fact in mind, in both cases the data is being buffered -
then how is the NIO method faster than IO Buffer.
Peter Duniho
> Hi,
>
> Could someone pls tell me why is the NIO buffering faster than the IO
Is it? If you have a concise-but-complete code example that
demonstrates that one is reliably faster than the other, you should
include that here as part of your question. That way, it be ensured
that everyone is talking about the same implementation, and the same
factors that might account for a difference (including the possibility
of implementation errors in which one i/o technique winds up favored
over the other).
If you don't have a concise-but-complete code example that demonstrates
that one is reliably faster than the other, then it seems to me you've
put the cart before the horse.
Pete
Arne Vajhøj
My expectation would be that:
- the difference is very small
- NIO may be slightly faster because it was designed later
and performance could have been a design criteria possibly
resulting in a faster implementation
Actual difference will depend on Java vendor, Java version,
OS, CPU, IO system etc..
For SUN Java 1.6, WinXP and my PC:
Unbuffered small reads : 26844
Unbuffered large reads : 687
Buffered small reads : 1406
Buffered large reads : 688
Nio map small reads : 3656
Nio map large reads : 563
Nio read small reads : 3547
Nio read large reads : 531
Random access small reads : 26359
Random access large reads : 656
Unbuffered small reads : 26875
Unbuffered large reads : 500
Buffered small reads : 938
Buffered large reads : 469
Nio map small reads : 3453
Nio map large reads : 328
Nio read small reads : 3437
Nio read large reads : 375
Random access small reads : 26125
Random access large reads : 469
Unbuffered small reads : 26313
Unbuffered large reads : 453
Buffered small reads : 937
Buffered large reads : 485
Nio map small reads : 3406
Nio map large reads : 328
Nio read small reads : 3469
Nio read large reads : 437
Random access small reads : 26719
Random access large reads : 734
(see code below)
Arne
=======================
import java.io.BufferedInputStream;
import java.io.FileInputStream;
import java.io.FileOutputStream;
import java.io.RandomAccessFile;
import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStream;
import java.nio.ByteBuffer;
import java.nio.channels.FileChannel;
public class ReadTest {
static final int TOT = 100000000;
static final int SMALL = 10;
static final int LARGE = 1000000;
static final String FILE_NAME = "C:/readtest.dat";
private static final int REP = 3;
public static void main(String[] args) throws Exception {
createFile();
for(int i = 0; i < REP; i++) {
(new UnbufferedSmallReadTest()).doTest();
(new UnbufferedLargeReadTest()).doTest();
(new BufferedSmallReadTest()).doTest();
(new BufferedLargeReadTest()).doTest();
(new NioMapSmallReadTest()).doTest();
(new NioMapLargeReadTest()).doTest();
(new NioReadSmallReadTest()).doTest();
(new NioReadLargeReadTest()).doTest();
(new RandomAccessSmallReadTest()).doTest();
(new RandomAccessLargeReadTest()).doTest();
}
}
private static void createFile() throws IOException {
OutputStream f = new FileOutputStream(FILE_NAME);
for(int i = 0; i < TOT; i++) {
f.write((i % 256));
}
}
}
abstract class GenericReadTest {
public void doTest() throws Exception {
long t1 = System.currentTimeMillis();
readAndCheck();
long t2 = System.currentTimeMillis();
System.out.println(getType() + " : " + (t2 - t1));
}
protected abstract void readAndCheck() throws Exception;
protected abstract int getReadSize();
public abstract String getType();
}
abstract class UnbufferedReadTest extends GenericReadTest {
protected void readAndCheck() throws IOException {
InputStream f = new FileInputStream(ReadTest.FILE_NAME);
byte[] b = new byte[getReadSize()];
int n;
int ix = 0;
while((n = f.read(b)) >= 0) {
for(int i = 0; i < n; i++) {
if(b[i] != (byte)(ix % 256)) {
throw new IOException("Bad data read - offset " +
ix + " expected " + (byte)(ix % 256) + " found " + b[i]);
}
ix++;
}
}
}
}
abstract class BufferedReadTest extends GenericReadTest {
protected void readAndCheck() throws IOException {
BufferedInputStream f = new BufferedInputStream(new
FileInputStream(ReadTest.FILE_NAME));
byte[] b = new byte[getReadSize()];
int n;
int ix = 0;
while((n = f.read(b)) >= 0) {
for(int i = 0; i < n; i++) {
if(b[i] != (byte)(ix % 256)) {
throw new IOException("Bad data read - offset " +
ix + " expected " + (byte)(ix % 256) + " found " + b[i]);
}
ix++;
}
}
}
}
abstract class NioMapReadTest extends GenericReadTest {
protected void readAndCheck() throws IOException {
FileChannel f2 = (new
FileInputStream(ReadTest.FILE_NAME)).getChannel();
ByteBuffer f = f2.map(FileChannel.MapMode.READ_ONLY, 0,
ReadTest.TOT);
byte[] b = new byte[getReadSize()];
int ix = 0;
while(ix < ReadTest.TOT) {
f.get(b);
for(int i = 0; i < b.length; i++) {
if(b[i] != (byte)(ix % 256)) {
throw new IOException("Bad data read - offset " +
ix + " expected " + (byte)(ix % 256) + " found " + b[i]);
}
ix++;
}
}
}
}
abstract class NioReadReadTest extends GenericReadTest {
protected void readAndCheck() throws IOException {
FileChannel f = (new
FileInputStream(ReadTest.FILE_NAME)).getChannel();
ByteBuffer b2 = ByteBuffer.allocate(getReadSize());
byte[] b = new byte[getReadSize()];
int n;
int ix = 0;
while((n = f.read(b2)) >= 0) {
b2.get(b);
for(int i = 0; i < n; i++) {
if(b[i] != (byte)(ix % 256)) {
throw new IOException("Bad data read - offset " +
ix + " expected " + (byte)(ix % 256) + " found " + b[i]);
}
ix++;
}
}
}
}
abstract class RandomAccessReadTest extends GenericReadTest {
protected void readAndCheck() throws IOException {
RandomAccessFile f = new RandomAccessFile(ReadTest.FILE_NAME, "r");
byte[] b = new byte[getReadSize()];
int n;
int ix = 0;
while((n = f.read(b)) >= 0) {
for(int i = 0; i < n; i++) {
if(b[i] != (byte)(ix % 256)) {
throw new IOException("Bad data read - offset " +
ix + " expected " + (byte)(ix % 256) + " found " + b[i]);
}
ix++;
}
}
}
}
class UnbufferedSmallReadTest extends UnbufferedReadTest {
protected int getReadSize() {
return ReadTest.SMALL;
}
public String getType() {
return "Unbuffered small reads";
}
}
class UnbufferedLargeReadTest extends UnbufferedReadTest {
protected int getReadSize() {
return ReadTest.LARGE;
}
public String getType() {
return "Unbuffered large reads";
}
}
class BufferedSmallReadTest extends BufferedReadTest {
protected int getReadSize() {
return ReadTest.SMALL;
}
public String getType() {
return "Buffered small reads";
}
}
class BufferedLargeReadTest extends BufferedReadTest {
protected int getReadSize() {
return ReadTest.LARGE;
}
public String getType() {
return "Buffered large reads";
}
}
class NioMapSmallReadTest extends NioMapReadTest {
protected int getReadSize() {
return ReadTest.SMALL;
}
public String getType() {
return "Nio map small reads";
}
}
class NioMapLargeReadTest extends NioMapReadTest {
protected int getReadSize() {
return ReadTest.LARGE;
}
public String getType() {
return "Nio map large reads";
}
}
class NioReadSmallReadTest extends NioMapReadTest {
protected int getReadSize() {
return ReadTest.SMALL;
}
public String getType() {
return "Nio read small reads";
}
}
class NioReadLargeReadTest extends NioMapReadTest {
protected int getReadSize() {
return ReadTest.LARGE;
}
public String getType() {
return "Nio read large reads";
}
}
class RandomAccessSmallReadTest extends RandomAccessReadTest {
protected int getReadSize() {
return ReadTest.SMALL;
}
public String getType() {
return "Random access small reads";
}
}
class RandomAccessLargeReadTest extends RandomAccessReadTest {
protected int getReadSize() {
return ReadTest.LARGE;
}
public String getType() {
return "Random access large reads";
}
}
Mike Amling
> Hi,
>
> Could someone pls tell me why is the NIO buffering faster than the IO
> BufferedReader class.
>
> I agree NIO uses FileChannels to access the file. But it in turn uses
> the FileInputStream.
>
> Example 1: BufferedReader
>
> BufferedReader reader = new BufferedReader(new
> FileInputStream("File.txt"));
> reader.read(....);
How do you get new BufferedReader(new FileInputStream(...)) to
compile? All the BufferedReader constructors require a Reader argument.
>
> Example 2: NIO Bufferes
>
> FileChannel fc = new FileInputStream("File.txt").getChannel();
> ByteBuffer buffer = ByteBuffer.allocate(1024);
> fc.read(buffer);
>
> Keeping the fact in mind, in both cases the data is being buffered -
> then how is the NIO method faster than IO Buffer.
Any Reader would have to attempt to parse an incoming byte sequence
into 16-bit chars according to an encoding. Your NIO example is not
doing any such parsing of the data.
--Mike Amling
EJP
>
> Could someone pls tell me why is the NIO buffering faster than the IO
> BufferedReader class.
Is it?
> I agree NIO uses FileChannels to access the file.
And ByteBuffers.
> But it in turn uses the FileInputStream.
No it doesn't.
> BufferedReader reader = new BufferedReader(new
> FileInputStream("File.txt"));
Doesn't compile. Even if you fix that, you are introducing byte-to-char
overheads here that you don't have in your NIO test. So you are
comparing apples and oranges.
> Keeping the fact in mind, in both cases the data is being buffered -
> then how is the NIO method faster than IO Buffer.
I doubt that it is any faster at all, unless you use direct buffers, of
a much larger size.
Christian
I have actually never seen direct ByteBuffers perform better than normal
ByteBuffers... no matter what situation...
Roedy Green
<arun....@gmail.com> wrote, quoted or indirectly quoted someone who
said :
>
>Could someone pls tell me why is the NIO buffering faster than the IO
>BufferedReader class.
Except for the Mapped stuff, both use the same physical i/o.
nio offers a few benefits:
1. easier multithreading.
2. you can pluck a few bytes out of the buffer. With DataInputStream,
typically you read and process every byte. (Granted you can skip with
BufferedInputStream too.)
I asked myself the same question and did not come up with a
satisfactory answer. It seemed the added complexity would not provide
that much payback except for the Mapped stuff. Some benchmarks are in
order.
--
Roedy Green Canadian Mind Products
http://mindprod.com
If you tell a computer the same fact in more than one place, unless you have an automated mechanism to ensure they stay in sync, the versions of the fact will eventually get out of sync.
Christian
> On Sun, 28 Mar 2010 09:15:35 -0700 (PDT), "arun....@gmail.com"
> <arun....@gmail.com> wrote, quoted or indirectly quoted someone who
> said :
>
>>
>> Could someone pls tell me why is the NIO buffering faster than the IO
>> BufferedReader class.
>
> Except for the Mapped stuff, both use the same physical i/o.
>
> nio offers a few benefits:
>
> 1. easier multithreading.
>
> 2. you can pluck a few bytes out of the buffer. With DataInputStream,
> typically you read and process every byte. (Granted you can skip with
> BufferedInputStream too.)
>
> I asked myself the same question and did not come up with a
> satisfactory answer. It seemed the added complexity would not provide
> that much payback except for the Mapped stuff. Some benchmarks are in
> order.
>
Isn't the main Payback of nio that you can use it non blocking mode...
You certainly don't want to have as many threads open as connections in
a setting where several hundred parallel connections are possible...
Christian
Arne Vajhøj
> Isn't the main Payback of nio that you can use it non blocking mode...
> You certainly don't want to have as many threads open as connections in
> a setting where several hundred parallel connections are possible...
On high-end HW you can support hundreds of threads.
Go to thousands and something else is necessary.
Arne
EJP
> Except for the Mapped stuff, both use the same physical i/o.
But with a ByteBuffer and FileChannel there is less implied data copying
than with e.g. a BufferedReader/InputStreamReader/FileInputStream.
Thomas Pornin
> On high-end HW you can support hundreds of threads.
Apart from the creation cost, having a thread around costs a bit in
RAM (actually address space), and a bit in scheduler efficiency.
The address space cost is for the thread stack. Thread stack size
depends on the operating system and can be changed (with Sun's JVM,
there is a -XX:ThreadStackSize parameter just for that). A typical size
is 1 MB. Most of it will remain untouched and thus not really allocated
(the address space chunk will be reserved, but the pages will not
exist). One thousand threads is 1 GB of address space; this may cause a
bit of trouble on 32-bit architectures, if the Java heap is also quite
big (thread stacks are allocated _outside_ of the Java heap). The
thousand stacks will use only a few dozen megabytes of actual RAM, no
worry here. On a 64-bit architecture, there is no shortage of address
space, and you can create thousands of threads without any problem.
The scheduler is the part of the operating system which decides which
thread should run next. Depending on the operating system, this may be
entirely in the OS kernel (Linux) or partially handled in userland
(Solaris). Either way, what happens which so many threads depends on the
algorithms which are used. Traditional, old-style schedulers are fine
with hundreds of threads but may begin to lose efficiency when reaching
the thousands. However, it is said that Solaris is very good at handling
many threads, and "recent" Linux kernels (I mean, since the 2.6.0
kernel, which is not in fact very recent) have been greatly improved:
they can handle _millions_ of runnable threads, with no worry.
Also, if you go the NIO road, then what this means is that you are
handling your own scheduling, in Java, with the constraint that
information about who may run must fit in the NIO API. It may provide a
gain only if the NIO API happens not to constrain too much, _and_ you
beat the carefully tuned kernel algorithms at their own game. I find
this to be a bit lacking in plausibility. And even if you succeed, you
end up with your own set of kind-of threads, except that Java does not
have coroutines, so things remain irksome.
To sum up:
-- You can go to thousands of threads with Java.
-- The limits are bound to address space allocation, and can be lifted
by using the -XX:ThreadStackSize flag, or, more appropriately, by using
the hardware at its full potential, i.e. using a 64-bit OS.
-- NIO will not help you with scheduling issues, unless you happen to
use a primitive operating system _and_ you are much better at
programming than the OS designers. With a non-primitive OS there will be
no scheduling issue.
-- NIO comes with a price in usability, since you have to fit several
conceptual execution threads on the same stack.
--Thomas Pornin