ERROR:: Invalid argument using O_DIRECT flag when opening a file

[Javier Narro]

Getting below error when opening a file using Direct I/O flag on, the
code works OK if I remove the O_DIRECT flag. Please advise. Thank
you !!!


Mounting file system with:
nohup /sbin/mount.sdfs -o
use_ino,direct_io,big_writes,allow_other,fsname=sdfs:/etc/sdfs/
orabackup-volume-cfg.xml:6442 -v orabackup


Installation information
Red Hat 6.2 (2.6.32-220.el6.x86_64)

RPM Fuse installed:
---------------------
fuse-libs-2.8.3-4.el6.x86_64
fuse-devel-2.8.3-4.el6.x86_64
fuse-2.8.3-4.el6.x86_64

SDFS:
------
SDFS-1.1.2-2.x86_64



Source code example
---------------------

#define _GNU_SOURCE
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <stdio.h>
#include <stdlib.h>
#include <errno.h>

int main(int argc, char *argv[])
{
int ret;

// create file
ret=open(argv[1], O_RDWR|O_CREAT|O_EXCL|O_DSYNC, 0660);
if ( ret < 0 ) {
perror("ERROR creating:");
printf("errno creating= %d.\n", errno);
} else {
printf ("create OK : %d\n" , ret);
close(ret);
}

// open file with O_DIRECT flag
ret=open(argv[1], O_RDWR|O_DSYNC|O_DIRECT);
if ( ret < 0 ) {
perror("ERROR:");
printf("errno = %d.\n", errno);
} else {
printf ("Open O_DIRECT OK : %d\n" , ret);
close(ret);
}
}


./a.out /media/test/20120501_233529/test.txt
create OK : 3
ERROR:: Invalid argument
errno = 22.


Log file from SDFS:
INFO [main] (Config.java:165) - Running SDFS Version 1.1.2
INFO [main] (Config.java:167) - Parsing subsystem-config version 1.1.2
INFO [main] (Config.java:172) - parsing folder locations
INFO [main] (Config.java:189) - Setting hash size to 16
INFO [main] (Volume.java:66) - Mounting volume /sdfs01/files
INFO [main] (Volume.java:91) - Setting volume size to 483183820800
INFO [main] (Volume.java:96) - Setting maximum capacity to infinite
INFO [main] (Config.java:278) - ######### Will allocate 483183820800
in chunkstore ##############
INFO [main] (MgmtWebServer.java:355) - ###################### SDFSCLI
Management WebServer Started at localhost/127.0.0.1:6442
#########################
INFO [main] (FileChunkStore.java:52) - Opening Chunk Store
INFO [main] (FileChunkStore.java:67) - ChunkStore /sdfs01/chunkstore/
chunks/chunks.chk created
INFO [main] (HashStore.java:162) - Loading hashdb class
org.opendedup.collections.CSByteArrayLongMap

INFO [main] (CSByteArrayLongMap.java:306) - ########## Finished
Loading Hash Database in [0] seconds ###########
INFO [main] (CSByteArrayLongMap.java:310) - loaded [0] into the
hashtable [/sdfs01/chunkstore/hdb/hashstore-sdfs] free slots available
are [0] free slots added [0] end file position is [0]!
INFO [main] (HashStore.java:83) - Cache Size = 131072 and Dirty
Timeout = 1000
INFO [main] (HashStore.java:87) - Total Entries 0
INFO [main] (HashStore.java:88) - Added sdfs
INFO [main] (PFullGC.java:18) - Current DSE Percentage Full is [0.0]
will run GC when [0.0]
INFO [main] (StandAloneGCScheduler.java:15) - Using
org.opendedup.sdfs.filestore.gc.PFullGC for DSE Garbage Collection
INFO [main] (MountSDFS.java:160) - Mount Option : -f
INFO [main] (MountSDFS.java:160) - Mount Option : /media
INFO [main] (MountSDFS.java:160) - Mount Option : -o
INFO [main] (MountSDFS.java:160) - Mount Option :
use_ino,direct_io,big_writes,allow_other,fsname=sdfs:/etc/sdfs/
orabackup-volume-cfg.xml:6442
INFO [Thread-7] (FDisk.java:20) - Starting FDISK
INFO [main] (SDFSFileSystem.java:47) - mounting /sdfs01/files to /
media
INFO [Thread-7] (FDisk.java:25) - took [0] seconds to check [0]. Found
[0] corrupt files
INFO [Thread-7] (CSByteArrayLongMap.java:176) - claiming records
INFO [Thread-7] (CSByteArrayLongMap.java:215) - processed [0] claimed
records in [39] ms
22:51:37.601 main INFO [fuse.FuseMount]: Mounted filesystem
INFO [Thread-7] (CSByteArrayLongMap.java:344) - Garbage collection
starting for records older than Tue May 01 22:46:37 CDT 2012
INFO [Thread-7] (CSByteArrayLongMap.java:415) - Removed [0] records.
Free slots [0]
INFO [Thread-7] (FDisk.java:20) - Starting FDISK
INFO [Thread-7] (FDisk.java:25) - took [0] seconds to check [0]. Found
[0] corrupt files
INFO [Thread-7] (CSByteArrayLongMap.java:176) - claiming records
INFO [Thread-7] (CSByteArrayLongMap.java:215) - processed [0] claimed
records in [1] ms
INFO [Thread-7] (CSByteArrayLongMap.java:344) - Garbage collection
starting for records older than Tue May 01 22:46:44 CDT 2012
INFO [Thread-7] (CSByteArrayLongMap.java:415) - Removed [0] records.
Free slots [0]
INFO [Thread-7] (FDisk.java:20) - Starting FDISK
INFO [Thread-7] (FDisk.java:25) - took [0] seconds to check [0]. Found
[0] corrupt files
INFO [Thread-7] (CSByteArrayLongMap.java:176) - claiming records
INFO [Thread-7] (CSByteArrayLongMap.java:215) - processed [0] claimed
records in [2] ms
INFO [Thread-7] (CSByteArrayLongMap.java:344) - Garbage collection
starting for records older than Tue May 01 22:46:50 CDT 2012
INFO [Thread-7] (CSByteArrayLongMap.java:415) - Removed [0] records.
Free slots [0]
INFO [Thread-7] (PFullGC.java:37) - Current DSE Percentage Full is
[0.0] will run GC when [0.1]
INFO [Thread-7] (FDisk.java:20) - Starting FDISK
INFO [Thread-7] (FDisk.java:25) - took [1] seconds to check [4]. Found
[0] corrupt files
INFO [Thread-7] (CSByteArrayLongMap.java:176) - claiming records
INFO [Thread-7] (CSByteArrayLongMap.java:215) - processed [373849]
claimed records in [18066] ms
INFO [Thread-7] (CSByteArrayLongMap.java:344) - Garbage collection
starting for records older than Wed May 02 00:11:58 CDT 2012
INFO [Thread-7] (CSByteArrayLongMap.java:415) - Removed [0] records.
Free slots [0]
INFO [Thread-7] (PFullGC.java:37) - Current DSE Percentage Full is
[0.10174480294499784] will run GC when [0.20174480294499786]
INFO [Thread-7] (FDisk.java:20) - Starting FDISK
INFO [Thread-7] (FDisk.java:25) - took [5] seconds to check [6]. Found
[0] corrupt files
INFO [Thread-7] (CSByteArrayLongMap.java:176) - claiming records
INFO [Thread-7] (CSByteArrayLongMap.java:215) - processed [760142]
claimed records in [34481] ms
INFO [Thread-7] (CSByteArrayLongMap.java:344) - Garbage collection
starting for records older than Wed May 02 00:47:26 CDT 2012
INFO [Thread-7] (CSByteArrayLongMap.java:415) - Removed [4] records.
Free slots [0]
INFO [Thread-7] (PFullGC.java:37) - Current DSE Percentage Full is
[0.20627436119532266] will run GC when [0.3062743611953227]
INFO [Thread-7] (FDisk.java:20) - Starting FDISK
INFO [Thread-7] (FDisk.java:25) - took [5] seconds to check [8]. Found
[0] corrupt files
INFO [Thread-7] (CSByteArrayLongMap.java:176) - claiming records
INFO [Thread-7] (CSByteArrayLongMap.java:215) - processed [1138151]
claimed records in [26183] ms
INFO [Thread-7] (CSByteArrayLongMap.java:344) - Garbage collection
starting for records older than Wed May 02 01:19:14 CDT 2012
INFO [Thread-7] (CSByteArrayLongMap.java:415) - Removed [4] records.
Free slots [0]
INFO [Thread-7] (PFullGC.java:37) - Current DSE Percentage Full is
[0.3088041359896059] will run GC when [0.40880413598960585]
INFO [Thread-7] (FDisk.java:20) - Starting FDISK
INFO [Thread-7] (FDisk.java:25) - took [7] seconds to check [11].
Found [0] corrupt files
INFO [Thread-7] (CSByteArrayLongMap.java:176) - claiming records
INFO [Thread-7] (CSByteArrayLongMap.java:215) - processed [1519087]
claimed records in [41157] ms
INFO [Thread-7] (CSByteArrayLongMap.java:344) - Garbage collection
starting for records older than Wed May 02 01:49:55 CDT 2012
INFO [Thread-7] (CSByteArrayLongMap.java:415) - Removed [10] records.
Free slots [0]
INFO [Thread-7] (PFullGC.java:37) - Current DSE Percentage Full is
[0.4120899198787354] will run GC when [0.5120899198787354]
INFO [Thread-7] (FDisk.java:20) - Starting FDISK
INFO [Thread-7] (FDisk.java:25) - took [8] seconds to check [15].
Found [0] corrupt files
INFO [Thread-7] (CSByteArrayLongMap.java:176) - claiming records
INFO [Thread-7] (CSByteArrayLongMap.java:215) - processed [1910604]
claimed records in [54780] ms
INFO [Thread-7] (CSByteArrayLongMap.java:344) - Garbage collection
starting for records older than Wed May 02 02:25:52 CDT 2012
INFO [Thread-7] (CSByteArrayLongMap.java:415) - Removed [7] records.
Free slots [0]
INFO [Thread-7] (PFullGC.java:37) - Current DSE Percentage Full is
[0.5181306842789086] will run GC when [0.6181306842789086]
INFO [Thread-7] (FDisk.java:20) - Starting FDISK
INFO [Thread-7] (FDisk.java:25) - took [11] seconds to check [18].
Found [0] corrupt files
INFO [Thread-7] (CSByteArrayLongMap.java:176) - claiming records
INFO [Thread-7] (CSByteArrayLongMap.java:215) - processed [2292624]
claimed records in [41187] ms
INFO [Thread-7] (CSByteArrayLongMap.java:344) - Garbage collection
starting for records older than Wed May 02 03:03:04 CDT 2012
INFO [Thread-7] (CSByteArrayLongMap.java:415) - Removed [12] records.
Free slots [0]
INFO [Thread-7] (PFullGC.java:37) - Current DSE Percentage Full is
[0.6212004655695106] will run GC when [0.7212004655695106]
INFO [Thread-7] (FDisk.java:20) - Starting FDISK
INFO [Thread-7] (FDisk.java:25) - took [15] seconds to check [23].
Found [0] corrupt files
INFO [Thread-7] (CSByteArrayLongMap.java:176) - claiming records
INFO [Thread-7] (CSByteArrayLongMap.java:215) - processed [2680298]
claimed records in [139051] ms
INFO [Thread-7] (CSByteArrayLongMap.java:344) - Garbage collection
starting for records older than Wed May 02 03:52:06 CDT 2012
INFO [Thread-7] (CSByteArrayLongMap.java:415) - Removed [13] records.
Free slots [0]
INFO [Thread-7] (PFullGC.java:37) - Current DSE Percentage Full is
[0.7262984517106973] will run GC when [0.8262984517106973]
INFO [Thread-7] (FDisk.java:20) - Starting FDISK
INFO [Thread-7] (FDisk.java:25) - took [20] seconds to check [30].
Found [0] corrupt files
INFO [Thread-7] (CSByteArrayLongMap.java:176) - claiming records
INFO [Thread-7] (CSByteArrayLongMap.java:215) - processed [3080823]
claimed records in [325150] ms
INFO [Thread-7] (CSByteArrayLongMap.java:344) - Garbage collection
starting for records older than Wed May 02 04:56:49 CDT 2012
INFO [Thread-7] (CSByteArrayLongMap.java:415) - Removed [15] records.
Free slots [15]
INFO [Thread-7] (PFullGC.java:37) - Current DSE Percentage Full is
[0.8339581528800346] will run GC when [0.91]

[Sam Silverberg]

Javier,


Fuse does not support o_direct as far as I know. The is a patch that adds support and it is found here http://lwn.net/Articles/476978/. You can try to compile fuse with this patch but I have not tested it.

Sam

[vlaero]

I thought I'd reply to this post with some updated information.  EL6.3 does support O_direct in fuse now.  (The poster below was using EL6.2).
See the 'File Systems' section here: http://www.redhat.com/about/news/archive/2012/6/red-hat-enterprise-linux-63-globally-available

Sam,  I notice that you have Debian packages for your custom Fuse but no RPMs for EL.  I ran into a problem where even when I compiled and installed the custom fuse on EL the package manager still wanted to pull in the disto version of Fuse when I installed the SDFS rpms.
Would you be able to update the doco with some notes of whether the current Fuse supplied with distros is good enough? If not, could you look at producing RPMs for people using EL (EL6 in particular)?