[HLFS] [BUG] Read in sequence with fio

[harryxiyou]

Tester: 贾威威
Time: 2013, 5, 14

测试过程（恢复场景）
================
1, Download my branch nbd source codes
http://cloudxy.googlecode.com/svn/branches/hlfs/person/harry/hlfs/

2, $ sudo fio -filename=/dev/nbd6 -direct=1 -rw=write -bs=1G -size=10G
-numjobs=1 -runtime=20 -group_reporting -name=test-seq-write

3, $ sudo fio -filename=/dev/nbd6 -direct=1 -rw=read -bs=1G -size=10G
-numjobs=1 -runtime=20 -group_reporting -name=test-seq-read


期望结果
==========
看到正常的I/O顺序读速度


实际结果
========
$ sudo fio -filename=/dev/nbd6 -direct=1 -rw=read -bs=1G -size=10G
-numjobs=1 -runtime=20 -group_reporting -name=test-seq-read
[sudo] password for hadoop1:
test-seq-read: (g=0): rw=read, bs=1G-1G/1G-1G, ioengine=sync, iodepth=1
fio 1.59
Starting 1 process
fio: pid=14018, err=5/file:engines/sync.c:62, func=xfer,
error=Input/output error8s]

test-seq-read: (groupid=0, jobs=1): err= 5 (file:engines/sync.c:62,
func=xfer, error=Input/output error): pid=14018
cpu : usr=0.00%, sys=0.08%, ctx=318, majf=0, minf=262200
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=50.0%, 4=50.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued r/w/d: total=1/0/0, short=0/0/0



Run status group 0 (all jobs):

Disk stats (read/write):
nbd6: ios=8456/0, merge=0/0, ticks=190225244/0, in_queue=190247140,
util=100.00%


could not find method loadFileSystems from class
org/apache/hadoop/fs/FileSystem with signature ()V
loadFileSystems error:
java.lang.NoSuchMethodError: loadFileSystems
could not find method read from class
org/apache/hadoop/fs/FSDataInputStream with signature
(Ljava/nio/ByteBuffer;)I
readDirect: FSDataInputStream#read error:
java.lang.NoSuchMethodError: read
hdfsOpenFile(/tmp/testenv/testfs/superblock): WARN: Unexpected error
255 when testing for direct read compatibility
could not find method read from class
org/apache/hadoop/fs/FSDataInputStream with signature
(Ljava/nio/ByteBuffer;)I
readDirect: FSDataInputStream#read error:
java.lang.NoSuchMethodError: read
hdfsOpenFile(/tmp/testenv/testfs/0.seg): WARN: Unexpected error 255
when testing for direct read compatibility
could not find method read from class
org/apache/hadoop/fs/FSDataInputStream with signature
(Ljava/nio/ByteBuffer;)I
readDirect: FSDataInputStream#read error:
java.lang.NoSuchMethodError: read
hdfsOpenFile(/tmp/testenv/testfs/1.seg): WARN: Unexpected error 255
when testing for direct read compatibility
could not find method read from class
org/apache/hadoop/fs/FSDataInputStream with signature
(Ljava/nio/ByteBuffer;)I
readDirect: FSDataInputStream#read error:
java.lang.NoSuchMethodError: read
hdfsOpenFile(/tmp/testenv/testfs/2.seg): WARN: Unexpected error 255
when testing for direct read compatibility
could not find method read from class
org/apache/hadoop/fs/FSDataInputStream with signature
(Ljava/nio/ByteBuffer;)I
readDirect: FSDataInputStream#read error:
java.lang.NoSuchMethodError: read
hdfsOpenFile(/tmp/testenv/testfs/3.seg): WARN: Unexpected error 255
when testing for direct read compatibility
could not find method read from class
org/apache/hadoop/fs/FSDataInputStream with signature
(Ljava/nio/ByteBuffer;)I
readDirect: FSDataInputStream#read error:
java.lang.NoSuchMethodError: read
hdfsOpenFile(/tmp/testenv/testfs/4.seg): WARN: Unexpected error 255
when testing for direct read compatibility
could not find method read from class
org/apache/hadoop/fs/FSDataInputStream with signature
(Ljava/nio/ByteBuffer;)I
readDirect: FSDataInputStream#read error:
java.lang.NoSuchMethodError: read
hdfsOpenFile(/tmp/testenv/testfs/5.seg): WARN: Unexpected error 255
when testing for direct read compatibility
could not find method read from class
org/apache/hadoop/fs/FSDataInputStream with signature
(Ljava/nio/ByteBuffer;)I
readDirect: FSDataInputStream#read error:
java.lang.NoSuchMethodError: read
hdfsOpenFile(/tmp/testenv/testfs/6.seg): WARN: Unexpected error 255
when testing for direct read compatibility
could not find method read from class
org/apache/hadoop/fs/FSDataInputStream with signature
(Ljava/nio/ByteBuffer;)I
readDirect: FSDataInputStream#read error:
java.lang.NoSuchMethodError: read
hdfsOpenFile(/tmp/testenv/testfs/7.seg): WARN: Unexpected error 255
when testing for direct read compatibility
could not find method read from class
org/apache/hadoop/fs/FSDataInputStream with signature
(Ljava/nio/ByteBuffer;)I
readDirect: FSDataInputStream#read error:
java.lang.NoSuchMethodError: read
hdfsOpenFile(/tmp/testenv/testfs/8.seg): WARN: Unexpected error 255
when testing for direct read compatibility
could not find method read from class
org/apache/hadoop/fs/FSDataInputStream with signature
(Ljava/nio/ByteBuffer;)I
readDirect: FSDataInputStream#read error:
java.lang.NoSuchMethodError: read
hdfsOpenFile(/tmp/testenv/testfs/9.seg): WARN: Unexpected error 255
when testing for direct read compatibility
could not find method read from class
org/apache/hadoop/fs/FSDataInputStream with signature
(Ljava/nio/ByteBuffer;)I
readDirect: FSDataInputStream#read error:
java.lang.NoSuchMethodError: read
hdfsOpenFile(/tmp/testenv/testfs/10.seg): WARN: Unexpected error 255
when testing for direct read compatibility
could not find method read from class
org/apache/hadoop/fs/FSDataInputStream with signature
(Ljava/nio/ByteBuffer;)I
readDirect: FSDataInputStream#read error:
java.lang.NoSuchMethodError: read
hdfsOpenFile(/tmp/testenv/testfs/11.seg): WARN: Unexpected error 255
when testing for direct read compatibility
could not find method read from class
org/apache/hadoop/fs/FSDataInputStream with signature
(Ljava/nio/ByteBuffer;)I
readDirect: FSDataInputStream#read error:
java.lang.NoSuchMethodError: read
hdfsOpenFile(/tmp/testenv/testfs/12.seg): WARN: Unexpected error 255
when testing for direct read compatibility
could not find method read from class
org/apache/hadoop/fs/FSDataInputStream with signature
(Ljava/nio/ByteBuffer;)I
readDirect: FSDataInputStream#read error:
java.lang.NoSuchMethodError: read
hdfsOpenFile(/tmp/testenv/testfs/13.seg): WARN: Unexpected error 255
when testing for direct read compatibility
could not find method read from class
org/apache/hadoop/fs/FSDataInputStream with signature
(Ljava/nio/ByteBuffer;)I
readDirect: FSDataInputStream#read error:
java.lang.NoSuchMethodError: read
hdfsOpenFile(/tmp/testenv/testfs/14.seg): WARN: Unexpected error 255
when testing for direct read compatibility
could not find method read from class
org/apache/hadoop/fs/FSDataInputStream with signature
(Ljava/nio/ByteBuffer;)I
readDirect: FSDataInputStream#read error:
java.lang.NoSuchMethodError: read
hdfsOpenFile(/tmp/testenv/testfs/15.seg): WARN: Unexpected error 255
when testing for direct read compatibility
could not find method read from class
org/apache/hadoop/fs/FSDataInputStream with signature
(Ljava/nio/ByteBuffer;)I
readDirect: FSDataInputStream#read error:
java.lang.NoSuchMethodError: read
hdfsOpenFile(/tmp/testenv/testfs/16.seg): WARN: Unexpected error 255
when testing for direct read compatibility
**
ERROR:/home/hadoop1/hadoop-0.20.2-cdh3u6/hlfs/nbd-2.9.15/nbd-server.c:1176:expread:
assertion failed: (0)


[...]
20130514 14:33:02.094 DEBUG hlfslog-
[0xd71200][/home/hadoop1/hadoop-0.20.2-cdh3u6/hlfs/src/icache/icache_query.c][icache_query][20]iblock_no
128 will be queried
20130514 14:33:02.094 DEBUG hlfslog-
[0xd71200][/home/hadoop1/hadoop-0.20.2-cdh3u6/hlfs/src/icache/icache_query.c][icache_query_iblock][36]--Entering
func

icache_query_iblock,icache_ctrl:0x32ce850,iblock_size:8192
20130514 14:33:02.094 DEBUG hlfslog-
[0xd71200][/home/hadoop1/hadoop-0.20.2-cdh3u6/hlfs/src/icache/icache_query.c][icache_query_iblock][52]--read
iblock

no:128,iblock->iblock:0x39d7090,iblock_size:8192
20130514 14:33:02.095 DEBUG hlfslog-
[0xd71200][/home/hadoop1/hadoop-0.20.2-cdh3u6/hlfs/src/icache/icache_query.c][icache_query_iblock][56]--Leaving
func

icache_query_iblock
20130514 14:33:02.095 DEBUG hlfslog-
[0xd71200][/home/hadoop1/hadoop-0.20.2-cdh3u6/hlfs/src/logger/block_ops.c][__load_block_by_no][263]storage
address:1116460551
20130514 14:33:02.095 DEBUG hlfslog-
[0xd71200][/home/hadoop1/hadoop-0.20.2-cdh3u6/hlfs/src/logger/segfile_handler_optmize.c][read_block_fast][150]enter
func

read_block_fast
20130514 14:33:02.095 DEBUG hlfslog-
[0xd71200][/home/hadoop1/hadoop-0.20.2-cdh3u6/hlfs/src/logger/segfile_handler_optmize.c][read_block_fast][159]offset
:42718727,

segno:16, last_offset:42759839,
last_rsegfile_offset:42759839
20130514 14:33:02.095 DEBUG hlfslog-
[0xd71200][/home/hadoop1/hadoop-0.20.2-cdh3u6/hlfs/src/logger/segfile_handler_optmize.c][prev_open_rsegfile][99]using
pre open

read file handler
20130514 14:33:02.098 WARN hlfslog-
[0xd71200][/home/hadoop1/hadoop-0.20.2-cdh3u6/hlfs/src/logger/segfile_handler_optmize.c][read_block_fast][191]can
not read

block from seg:16#42718727 :ret :2041
20130514 14:33:02.103 WARN hlfslog-
[0xd71200][/home/hadoop1/hadoop-0.20.2-cdh3u6/hlfs/src/logger/segfile_handler_optmize.c][read_block_fast][191]can
not read

block from seg:16#42718727 :ret :2041
20130514 14:33:02.108 WARN hlfslog-
[0xd71200][/home/hadoop1/hadoop-0.20.2-cdh3u6/hlfs/src/logger/segfile_handler_optmize.c][read_block_fast][191]can
not read

block from seg:16#42718727 :ret :2041
20130514 14:33:02.109 ERROR hlfslog-
[0xd71200][/home/hadoop1/hadoop-0.20.2-cdh3u6/hlfs/src/storage/hlfs_read.c][hlfs_read][112]fail
to load block for no 131069
20130514 14:33:02.109 ERROR hlfslog-
[0xd71200][/home/hadoop1/hadoop-0.20.2-cdh3u6/hlfs/nbd-2.9.15/nbd-server.c][expread][1175]read
failed




--
Thanks
Harry Wei