Loss of data on logout
Joachim
I am currently trying to implement an iscsi target driver for a remote
storage space with tgt.
At first, the driver was slow (quite normal), but worked quite well,
with a testing script i wrote.
Then, I added some features for it to get faster, and thus the problem
appeared.
Actually, after the login to the target with iscsiadm, I partition the
disk, create some filesystem, and then make a copy of a random file
tree, before umounting the partition, and logout.
When i try to mount the partition again (after all required
manipulations with iscsiadm), i stumble upon a series of IO Errors,
meaning that my filesystem is totally corrupted, and that some of the
data weren't saved by my driver (be it that i didnt receive it or that
i could not save it)
When I use the sync command before umounting, I actually succeed to
read the whole file system again afterwards.
While searching a solution, I found this in another thread of this
mailing list (http://groups.google.com/group/open-iscsi/browse_thread/
thread/e445471fab2af459/b8fae3dfb6e6c68f?pli=1) :
>> is the Sync cache that occurs at logout actually sending data? or is
>> it just sending a scsi msg to flush/checkpoint the target side?
>
>No data. It is just telling the target side to basically flush what it
>has in its cache. For example tgt does a sync() to write out data to the
>backing store. I believe IET does the equivalent of a sync() but it is
>done from the kernel since IET is implemented there. Look at the source
>for IET or just fire off a mail to that list if you want to know all the
>details of what it does.
If i'm not misunderstanding what's written here, That means that when
i type the command
$ iscsiadm --mode node --targetname mytarget --logout
All that is done is actually sending a sync command to the target (tgt
in my case) without sending any data relating to the system/iscsid
cache ?
My hypothesis is thus that my problem originates from this behavior of
the iscsiadm logout action. If I'm not wrong about that, how can i
make it so that a synchronization is done before the iscsi logout ? Or
I am completely wrong, and then do you have any insight as to what my
problem really is ?
Thanks for your help in advance !
Mike Christie
Right. The iscsi layer removes the devices that were getting accessed by
the session and this causes the scsi layer to send a sync cache command
if the device had caching on.
But the iscsi layer does not do any caching of data. It passes/sends
whatever it has when it gets it. Upper layers like filesystems or memory
layer cache data. If you were writing out data and ran the logout
command, the logout and sync cache can occur before all the writing is
done. The iscsi layer does not know what the layers above have to write
out, so the iscsi layer just kills the devices and has the scsi layer
send the sync cache when ever you run the logout command.
The user needs to make sure they have unmounted FSes or had they apps
sync their data before running the logout command.
>
> My hypothesis is thus that my problem originates from this behavior of
> the iscsiadm logout action. If I'm not wrong about that, how can i
> make it so that a synchronization is done before the iscsi logout ? Or
> I am completely wrong, and then do you have any insight as to what my
> problem really is ?
I do not think it is really a iscsiadm issue or at least there is
nothing the iscsi layer can do. It just does not know what the upper
layers are doing. The iscsi layer just handles iscsi stuff. What is in
the FS/VM/block/userspace buffers/cache at the time of the logout
command is not known to us. You need to sync your data, unmount
filesystems, etc, before you logout a session. For example when the
system shutsdown, the FSes will be unmounted and data written out, then
the iscsi service is shutdown which does the logout command.
Did you check if the target is even getting the sync cache command btw?
Some older kernels had a bug in the scsi or driver model layer where it
never got sent. Also check your cache settings and make sure the cache
is on or the sync cache does not get sent.
Mike Christie
> What seemed strange to me is that without the sync command, my driver
> did not update many blocs/sectors on my storage, even though I did
> properly umount and logout. To sum it up, I could not find where were
> the lost data, and that is the reason why i came here with my
> question.
What FS and what mount options are you using?
Mike Christie
> What seemed strange to me is that without the sync command, my driver
> did not update many blocs/sectors on my storage, even though I did
> properly umount and logout. To sum it up, I could not find where were
> the lost data, and that is the reason why i came here with my
> question.
>
If you do
- umount fs
- iscsiadm -m .... -logout
- sync
does the sync cause IO errors to get printed in /var/log/messages for
the devices that just got deleted by the iscsiadm logout command?
Ulrich Windl
<eec86689-0a67-40f6...@x12g2000yql.googlegroups.com>:
> Hello,
>
> I am currently trying to implement an iscsi target driver for a remote
> storage space with tgt.
>
> At first, the driver was slow (quite normal), but worked quite well,
> with a testing script i wrote.
> Then, I added some features for it to get faster, and thus the problem
> appeared.
>
> Actually, after the login to the target with iscsiadm, I partition the
> disk, create some filesystem, and then make a copy of a random file
> tree, before umounting the partition, and logout.
> When i try to mount the partition again (after all required
> manipulations with iscsiadm), i stumble upon a series of IO Errors,
> meaning that my filesystem is totally corrupted, and that some of the
> data weren't saved by my driver (be it that i didnt receive it or that
> i could not save it)
>
> When I use the sync command before umounting, I actually succeed to
> read the whole file system again afterwards.
Usually "umount" should sync that filesystem's data, and probably wait until that finished. I don't thin that iSCSI adds another buffering layer that could hold back data between unmount and iSCSI logout.
Ulrich
Ulrich Windl
Nachricht <4E14DB30...@cs.wisc.edu>:
> On 07/06/2011 04:01 PM, David Pineau wrote:
> > What seemed strange to me is that without the sync command, my driver
> > did not update many blocs/sectors on my storage, even though I did
> > properly umount and logout. To sum it up, I could not find where were
> > the lost data, and that is the reason why i came here with my
> > question.
> >
>
> If you do
>
> - umount fs
> - iscsiadm -m .... -logout
> - sync
What comes to my mind: Do you logout after unmount, or do you logout after SUCCESSFUL unmount? I mean: if the filesystem is busy, it won't unmount, and there will be no sync then.
if umount fs; then
iscsiadm ...
else
error
fi
>
> does the sync cause IO errors to get printed in /var/log/messages for
> the devices that just got deleted by the iscsiadm logout command?
Regards,
Ulrich
David Pineau
Yeah, I checked that my tgt driver received the sync when i do the
logout command, and that is the case . No problem over this side.
By the way, I work on an ubuntu 10.4 LTS, so I don't think that could
come from an old kernel version (or a too old one).
What seemed strange to me is that without the sync command, my driver
did not update many blocs/sectors on my storage, even though I did
properly umount and logout. To sum it up, I could not find where were
the lost data, and that is the reason why i came here with my
question.
Anyways, thanks for the answer, it seems i'll have to dig a little bit
more in how the FS side works to understand where is my shortcoming
then.
Thanks again :)
--
David Pineau, Etudiant Epitech -Promotion 2012-
Assistant C/Unix, Assistant Langages Avancés
Membre LSE (Laboratoire Système Epita/Epitech).
David Pineau
I am currently using ext2 for my tests (i tried with fat32 too, and if
i remember correctly, that was about the same for this cache matter).
About the mount options, i do not give any peculiar one, as I didnt
see the need. When i came across this problem i went through the
manual to try and find some fs-independant options that could help but
did not manage to identify any (maybe did i miss some).