Account Options

  1. Sign in
The old Google Groups will be going away soon, but your browser is incompatible with the new version.
Google Groups Home
« Groups Home
jsr203-interest
Home
About this group
Join this group
Direct IO
Options
There are currently too many topics in this group that display first. To make this topic appear first, remove this option from another topic.
There was an error processing your request. Please try again.
flag
   8 messages - Collapse all  -  Translate all to Translated (View all originals)
The group you are posting to is a Usenet group. Messages posted to this group will make your email address visible to anyone on the Internet.
Your reply message has not been sent.
Your post was successful
 
From:
To:
Cc:
Followup To:
Add Cc | Add Followup-to | Edit Subject
Subject:
Validation:
For verification purposes please type the characters you see in the picture below or the numbers you hear by clicking the accessibility icon. Listen and type the numbers you hear
 
Clebert Suconic  
View profile  
 More options Mar 18 2008, 7:02 pm
From: Clebert Suconic <clebert.suco...@gmail.com>
Date: Tue, 18 Mar 2008 16:02:25 -0700 (PDT)
Local: Tues, Mar 18 2008 7:02 pm
Subject: Direct IO
Print | Individual message | Show original | Report this message | Find messages by this author
Are you guys supporting Direct IO on Linux?

I'm currently developing a native layer for LibAIO.. and for dealing
with Direct IO i had to align buffers before writing them to the OS
layer.

Also.. Is the implementation available anywhere yet? I have seen some
posts about pushing it to OpenJDK. Is it already available there?

Best regards,

Clebert


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Clebert Suconic  
View profile  
 More options Mar 18 2008, 7:05 pm
From: Clebert Suconic <clebert.suco...@gmail.com>
Date: Tue, 18 Mar 2008 16:05:21 -0700 (PDT)
Local: Tues, Mar 18 2008 7:05 pm
Subject: Re: Direct IO
Print | Individual message | Show original | Report this message | Find messages by this author

On Mar 18, 6:02 pm, Clebert Suconic <clebert.suco...@gmail.com> wrote:


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Clebert Suconic  
View profile  
 More options Mar 18 2008, 7:07 pm
From: Clebert Suconic <clebert.suco...@gmail.com>
Date: Tue, 18 Mar 2008 16:07:17 -0700 (PDT)
Local: Tues, Mar 18 2008 7:07 pm
Subject: Re: Direct IO
Print | Individual message | Show original | Report this message | Find messages by this author

I meant.. that is on Files.

Also.. what library are you guys using... LibAIO or Posix?

If you guys are using Posix, are you guys interested in using LibAIO
on Linux?

From what I have seen on the kernel code, LibAIO is just a forward
directly to the kernel.. and it would work pretty fast.

Thanks again,

Clebert Suconic
Jboss Inc / RedHat

On Mar 18, 6:02 pm, Clebert Suconic <clebert.suco...@gmail.com> wrote:


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Alan Bateman  
View profile  
 More options Mar 19 2008, 6:11 am
From: Alan Bateman <Alan.Bate...@Sun.COM>
Date: Wed, 19 Mar 2008 10:11:54 +0000
Local: Wed, Mar 19 2008 6:11 am
Subject: Re: [jsr203-interest] Direct IO
Print | Individual message | Show original | Report this message | Find messages by this author
Clebert Suconic wrote:
> Are you guys supporting Direct IO on Linux?

> I'm currently developing a native layer for LibAIO.. and for dealing
> with Direct IO i had to align buffers before writing them to the OS
> layer.

As I'm sure you agree, direct I/O and bypassing the file system cache is
an advanced topic that a database might use but not the average
developer. Some time ago I prototyped an API for this so that alignment
of the file position (often the size of the disk sector) was exposed and
also a method to allocate a buffer that was appropriately aligned for a
given file system. It's not in the JSR-203 API at this time but could be
if it avoids needing to resort to native code for important cases  (and
also doesn't complicate things for the average developer). Can you say
anything about your needs? Is is intended to run everywhere?

As you mention alignment I'm curious if you make any assumptions on the
alignment of direct buffers? Just asking because they have been page
aligned since 1.4 but we need to re-visit that for big applications that
enable large pages.

> Also.. Is the implementation available anywhere yet? I have seen some
> posts about pushing it to OpenJDK. Is it already available there?

All the main areas of NIO2 have been implemented for some time and the
original plan was to make it available as source + early access binaries
to allow people try it, provide feedback, contribute, etc. I apologize
but I completely underestimated how long it takes to get all the
mandatory approvals (export, trademarks, legal, ...) to release
something standalone. In the mean-time, the jdk7/OpenJDK repositories
have re-opened so we can also proceed with pushing the changes there
(which may be preferable to many as it avoids needing to build from the
source or using special binaries).

-Alan.


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Alan Bateman  
View profile  
 More options Mar 19 2008, 6:27 am
From: Alan Bateman <Alan.Bate...@Sun.COM>
Date: Wed, 19 Mar 2008 10:27:28 +0000
Local: Wed, Mar 19 2008 6:27 am
Subject: Re: [jsr203-interest] Re: Direct IO
Print | Individual message | Show original | Report this message | Find messages by this author
Clebert Suconic wrote:
> I meant.. that is on Files.

> Also.. what library are you guys using... LibAIO or Posix?

> If you guys are using Posix, are you guys interested in using LibAIO
> on Linux?

> From what I have seen on the kernel code, LibAIO is just a forward
> directly to the kernel.. and it would work pretty fast.

The last time I checked, Linux's kernel AIO was limited to raw devices
or files opened for direct I/O. For that reason, AsynchronousFileChannel
is implemented to uses a thread pool + synchronous I/O on that platform
(we don't use PAIOL).  If I've got this wrong (or the underlying I/O
facility improves) then we can switch the implementation as it's
important for NIO to make use of best performing facility on each platform.

-Alan.


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Clebert Suconic  
View profile  
 More options Mar 19 2008, 10:13 am
From: "Clebert Suconic" <clebert.suco...@gmail.com>
Date: Wed, 19 Mar 2008 09:13:35 -0500
Local: Wed, Mar 19 2008 10:13 am
Subject: Re: [jsr203-interest] Re: Direct IO
Print | Individual message | Show original | Report this message | Find messages by this author
Thanks Adam...

My comments bellow:

>As I'm sure you agree, direct I/O and bypassing the file system cache

is an advanced topic that a database might use but not the average
developer.

Well.. It is today... but I was hoping it would make it easier for
developers, as you may find better apis.
And I also think that a regular developer would have an indirect
benefit of this. say... if using a higher level API making advantage
of AIO. (JMS, Mina. or any other layer in top of NIO2).

>The last time I checked, Linux's kernel AIO was limited to raw devices

or files opened for direct I/O.

Yes.. but shouldn't the AsynchronousFileChannel making use of it?

>As you mention alignment I'm curious if you make any assumptions on

the alignment of direct buffers? Just asking because they have been
page aligned since 1.4 but we need to re-visit that for big
applications that enable large pages.

I assumed the newDirectBuffer would call a regular malloc method
underneath, as I needed an aligned buffer. Maybe my assumption was
wrong. I will revisit this. (Maybe I should have read the code :-) )


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Alan Bateman  
View profile  
 More options Mar 19 2008, 11:26 am
From: Alan Bateman <Alan.Bate...@Sun.COM>
Date: Wed, 19 Mar 2008 15:26:49 +0000
Local: Wed, Mar 19 2008 11:26 am
Subject: Re: [jsr203-interest] Re: Direct IO
Print | Individual message | Show original | Report this message | Find messages by this author

I'll change our implementation to always AIO on Linux if/when it works
for regular files that aren't opened for direct I/O. I prototyped using
this interface a while back but as I recall,  and correct me if I'm
wrong, io_submit fails or quietly waits when the file isn't opened for
direct I/O. Performance is always a moving target and we regularly swap
in better implementations so it shouldn't be a problem (replacing the
poll based Selector with the epoll based Selector is a good example). If
we supported the direct I/O option (either in the spec or as an
implementation specific option as the spec allows) then we could use AIO
for file channels opened with that option. As the issues are linked then
the reason we don't always open files for direct I/O is because it
bypasses the file system cache and so any application that doesn't do
its own caching can suffer. That and the usability issue as the
alignment of the file position, buffer alignment and size, need to be
exposed.

> :
> I assumed the newDirectBuffer would call a regular malloc method
> underneath, as I needed an aligned buffer. Maybe my assumption was
> wrong. I will revisit this. (Maybe I should have read the code :-) )

Direct buffers are allocated from the native heap but the alignment
isn't specified. Our implementation has (since 1.4) returned a page
aligned buffer but this is problematic when large pages are used. It
seems to be common now to configure large pages so as to reduce TLS
misses (Sun's CMT processors support 256MB pages so page aligned buffers
can be a killer).

-Alan.


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Alan Bateman  
View profile  
 More options Mar 19 2008, 11:30 am
From: Alan Bateman <Alan.Bate...@Sun.COM>
Date: Wed, 19 Mar 2008 15:30:48 +0000
Local: Wed, Mar 19 2008 11:30 am
Subject: Re: [jsr203-interest] Re: Direct IO
Print | Individual message | Show original | Report this message | Find messages by this author
Alan Bateman wrote:
> reduce TLS misses

That should be TLB of course - typo in my mail.

 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
End of messages
« Back to Discussions « Newer topic     Older topic »

Create a group - Google Groups - Google Home - Terms of Service - Privacy Policy
©2013 Google