ReadFile/WriteFile on PhysicalDriveXX
K@discussions.microsoft.com David K
where else I would post it. Let me know if it needs to be posted in a
different location.
I am creating an application for managing mass storage devices that are
plugged into a certain USB duplication hub (which is identified by it's VID
and PID). It enumerates all the mass storage devices on the hub and allows
the user to perform file copies, delete, format, eject, etc on the drives. A
feature that this application needs to have is being able to do a raw copy at
the device level from one USB drive to another. The idea is to get a mirror
image from the source device and write it onto one or more destination
devices.
Now, being very new to the technologies used to interact with USB devices
(beyond just the basic .NET framework file copy and delete and such), I have
very little knowledge about the plausibility of this feature or how
complicated it would be. From the knowledge I do have, I planned on opening
a device handle to the source and destination drives using the CreateFile API
function and then using ReadFile and WriteFile to copy the drive bit for bit
to the new drive.
I wrote the code for this and ran it successfully only to find that my
destination USB drive has been corrupted beyond repair. Obviously it is not
quite as simple as I had anticipated. I used the same code, substituting
files for the path when calling the CreateFile functions (instead of the
"\\.\PhysicalDeviceXX" path) and my code copied the files just fine so I know
my code runs correctly and isn't corrupting the data.
I have searched all over the web trying to find examples of a bit-level copy
of USB drives (or any drives for that matter) using ReadFile/WriteFile and
have come up with nothing. Can anyone provide some help and/or direction on
this issue? My time on this project is running VERY short and I need to get
this completed asap.
Any help is greatly appreciated!
David K.
David Craig
It is a rather simple task to do copies as long as the 'new' drives can only
be mounted by the RAW filesystem. If they have a filesystem on them, or it
is the source drive which may have a filesystem that Windows would recognize
it requires they be dismounted so the RAW filesystem will claim them. This
also implies that they are not either the boot or system drive for the
Windows that is currently running.
Dismounting and locking of drives/volumes appears in some of the header
files that come with the WDK. I believe most of them are also present in
the SDK headers. Using CreateFile for the physical drive and not one of the
volumes is the correct way to begin. Be careful to follow the rules for the
appropriate bits to be set in the CreateFile parameters.
You neglected to mention the Windows version you are required to support.
It gets a little more difficult with Vista and later because of UAC. If
they have to be supported, I would recommend that you add a manifest
indicating you need administrator rights.
Read some of the documents about formatting/partitioning a drive from
Sysinternals and in the MSDN Library. Your basic task is to partition and
then format the drive, but instead of just writing the system areas you will
be writing most or all of the drive.
If time is a major requirement, there are several who participate in the
newsgroups who can be hired. I suspect some have a toolkit that can speed
their delivery of the product, but don't forget that they get paid for not
only their time but their knowledge. It will take you some time to
understand all the implications of the various options in CreateFile,
ReadFile, WriteFile, and the IoCtls required to ensure you have exclusive
access to the drive(s).
Using USB is easy, but not nearly as fast as SATA 300 or 1394. Doing a copy
from one master to multiple targets will speed the duplicating process but
USB is a highly sharable design and not designed primarily for mass storage.
"David K" <David K...@discussions.microsoft.com> wrote in message
news:D5C197FC-66F6-406F...@microsoft.com...
David K
To answer your question about operating system support, I only need the
application to run on Windows XP and I already expect that the user of the
application will need administrative access to the computer due to the
methods I am using for enumeration of the USB drives and the eject
functionality as well.
Also, I can give you a little more detail about the reason for wanting this
functionality. The software I am writing is for management of a USB
duplicator. Many of the potential users of this duplicator want to be able
to duplicate very quickly to multiple drives at once and with this method I
should be able to use multiple threads to write to each USB drive from a FIFO
read buffer asynchronously. This way I only have to read through once and
yet I can be copying all 20 USB drives at once.
Another reason for this functionality is that some USB drives have more than
just a basic partition and filesystem on them. For instance, there are USB
drives which have a second partition that Windows recognizes as a CD drive
and this allows them to get around the "AutoRun" security restrictions that
have been placed on USB pen drives.
We figured that if we could get a bit-level duplication of such a drive, the
destination drive(s) would all be partitioned the same as the source drive
and contain the same files as well. I imagine it would be much more
difficult (and in some cases slower) to try and duplicate the partitioning
information first and then copy each partition's contents.
Anyway, It sounds like what I'm missing then is the dismounting and locking
of the destination drive before I try to copy the RAW data (and then an
unlock and remount afterwards I suppose as well). I will definitely check
out the Sysinternals documents on formatting/partitioning.
Also is it possible you know a good link to start with on the MSDN library
as my searches on formatting/partitioning have not been very productive to
this point.
I have one other question, and that is about "dd for windows". It is the
Windows version of the GNU dd project which enables RAW image creation and
device duplication from a command line interface. It sounds like it already
has the feature set I need, in which case I could just include that with the
application and use it to perform just what I am currently trying to do
myself. If anyone has experience with using this, I'd like to hear what they
have to say...
Thanks again for the quick reply and useful information...
- David K
David Craig
grateful that I can just wipe the U3 stuff out including the CD. You can
try a copy of WinHex to see if it can copy the entire media to a file. It
can also be used to write it to one drive at a time. If it works, then you
can expect success with a good program.
Look at ntdddisk.h & ntddstor.h in the WDK. Also look at the full SDK.
I will reply to this message later with a few hints.
"David K" <Dav...@discussions.microsoft.com> wrote in message
news:523522EA-8793-41EA...@microsoft.com...
David Craig
1. m_hDevice = CreateFile(
m_csDeviceName,
GENERIC_READ | GENERIC_WRITE, // Need full access
FILE_SHARE_READ | FILE_SHARE_WRITE,
NULL,
OPEN_EXISTING,
FILE_FLAG_NO_BUFFERING | FILE_FLAG_WRITE_THROUGH,
NULL
);
2. bStatus = DeviceIoControl(
m_hDevice,
FSCTL_LOCK_VOLUME,
NULL,
0,
NULL,
0,
&BytesReturned,
NULL
);
3. if (!(WriteFile(
m_hDevice,
pBuffer,
dwNumSects * dwSectorSize,
&dwBytesWritten,
&ovl
)))
{
m_dwStatus = GetLastError();
}
else if (dwBytesWritten != (dwNumSects * dwSectorSize))
{
m_dwStatus = ERROR_WRITE_FAULT;
}
4. bStatus = DeviceIoControl(
m_hDevice,
FSCTL_DISMOUNT_VOLUME,
NULL,
0,
NULL,
0,
&BytesReturned,
NULL
);
5. bStatus = DeviceIoControl(
m_hDevice,
FSCTL_UNLOCK_VOLUME,
NULL,
0,
NULL,
0,
&BytesReturned,
NULL
);
6. bStatus = DeviceIoControl(
m_hDevice,
IOCTL_DISK_UPDATE_PROPERTIES,
NULL,
0,
NULL,
0,
&BytesReturned,
NULL
);
7. // This function, available with Windows XP & 2003 does
// work correctly and the physical drive is properly
// remounted with the correct file system driver.
//
bStatus = DeviceIoControl(
m_hDevice,
IOCTL_DISK_GET_DRIVE_LAYOUT,
NULL,
0,
&outBuf,
sizeof(outBuf),
&BytesReturned,
NULL
);
bStatus = DeviceIoControl(
m_hDevice,
IOCTL_DISK_SET_DRIVE_LAYOUT,
&outBuf,
BytesReturned,
NULL,
0,
&BytesReturned,
NULL
);
Here are a lot of hints. I leave it to you to supply the error code
handling which I omitted. I would read about each function call and IoCtl
used. Items 6 & 7 are complementary and only one should be used, though I
think I used both. I only tested on XP Pro and I am certain it won't
function properly under Vista because of UAC though invoking from an
elevated cmd.exe prompt could solve that problem.
"David Craig" <dri...@nowhere.us> wrote in message
news:OQzhDPez...@TK2MSFTNGP02.phx.gbl...
David K
and also done some more searching about formatting/partitioning using
CreateFile/WriteFile. What I am doing is very similar except I am opening a
handle to the physical device instead of just the volume (which is what a
format operation would be opening up) and thus overwriting the entire drive
instead of just the part for an individual volume. Now, I have a couple
questions that stem from that difference...
As you have suggested and as I have read in multiple other locations, the
volume needs to be locked and dismounted as part of the process. I have two
questions on this:
First of all, you numbered the steps you sent which seems to indicate they
are ordered. In that case, you are saying that the FSCTL_DISMOUNT_VOLUME
control code should be sent after the writing via WriteFile has been
performed. My impression however from reading other locations and from my
understanding of the process is that the dismounting of the volume allows the
RAW filesystem to take over and this is necessary to be able to write to that
drive correctly? In my case, both the source and destination drives will
already be using the FAT/FAT32 filesystem and so, in this case, wouldn't I
need to dismount the volume on the drive I am about to write to before the
writing takes place?
My second question is regarding sending the lock/unlock and dismount
commands. Wouldn't I need to first enumerate the volume(s) associated with
the device and then perform a lock, dismount, and unlock individually on the
handle(s) to each one instead of the PhysicalDrive handle? The MSDN
documentation for the FSCTL lock/dismount/unlock codes indicate that the
handle passed to DeviceIoControl must be a handle to a volume.
Anyway, I think I am about ready to test my revised code out and see if I
can do this without corrupting another drive. Thanks again for all your help!
David Craig
for Volumes, and harddrives. Use DeviceTree from OSR and their harddisk
utility. Also look at the various utilities written by Sysinternals. There
are a few bugs in the DiskView utility from OSR in that it keeps serial
numbers and busses from earlier drives when they really don't apply.
My program always worked on one physical drive creating only one FAT
partition. It was designed to solve the limit of 32GB the Microsoft
formatter has for FAT32. It also handled properly formatting SmartMedia
since they require non-standard FAT12/16 formats. If you want multiple
volumes on a single physical drive I would suggest you look at the
partitioning IoCtls located in some of the same headers (Storage or disk).
Depending upon which version of Windows your program runs some of the
storage IoCtls may fail and you will have to fall back to the disk IoCtls
which were the first.
"David K" <Dav...@discussions.microsoft.com> wrote in message
news:974F5D74-D9B0-40C7...@microsoft.com...
David K
now I'm continuing my tests to try other drives and such.
While running the copy on a Kingston DTI/1GB (source drive and destination
drive were both this same type of drive), ReadFile terminated at the end with
a return code of FALSE and GetLastError code 0x1b (ERROR_SECTOR_NOT_FOUND).
According to the ReadFile documentation, it should end with a return of TRUE
and BytesRead = 0 when it has reached the end of the drive. Regardless, the
drive being written came out just fine. My question is, should I be using
the drive geometry that I get with IOCTL_DISK_GET_DRIVE_GEOMETRY to determine
when to stop reading instead?
I noticed that the disk size that I calculate using the information in the
DISK_GEOMETRY structure is smaller than the number of bytes that I am able to
read from the drive. So, if I were to use the calculated disk size for my
read, I would be missing approximately 6000 KB of data at the end of the
drive (which I I believe it to just be all filled with 0xFF bytes anyway).
Interestingly enough, using the application Disk Probe that comes with
Windows Support Tools, I am unable to read any of these sectors even though I
can read them with ReadFile and a handle to the disk.
One other issue I am having is that after completing the read/write,
unlocking the volume and using IOCTL_DISK_UPDATE_PROPERTIES, the computer on
which I perform the operation will no longer enumerate volumes/partitions of
drives that are inserted. They are recognized and Windows makes the normal
sound it makes when you connect a removable drive, and I am able to access
the drive via my application which also recognizes them and enumerates them,
however Windows Explorer will not show the drive letter of any newly inserted
drives. Rebooting the system clears this problem up.
Thanks again for all the time you've already put into helping me out with
this. It is greatly appreciated!
David Craig
properties in my class destructor.
Have you read the fatgen103.docx file? It contains the rules for formatting
FAT partitions. After the MBR, slack space, partition boot record(s), FATs,
and a possible root directory (not FAT32) the remaining space is allocated
until there are some number of sectors remaining that will not make up a
cluster. There may also be enough sectors to reserve the last cylinder
using the CHS info, but I forget if that applies any more.
Use IOCTL_DISK_GET_PARTITION_INFO to determine the number of sectors on the
media. Use the Physical Drive handle as this won't work properly on a
volume.
"David K" <Dav...@discussions.microsoft.com> wrote in message
news:4F7B911D-257B-4251...@microsoft.com...
David K
handles to the physical devices were opened. Now, I am performing the
dismount after the write has complete, right before unlocking the volume and
I think it is working better now.
As for IOCTL_DISK_GET_PARTITION_INFORMATION, from everything I have read
about it, I don't see any way that it returns the number of sectors on the
drive. Am I missing something?
It gives the starting offset (which should be 0 for a PhysicalDevice
handle), the total number of bytes, and the number of "hidden" sectors (which
again should be 0 since that refers to the number of sectors before this
"partition" begins, which in this case should be none). Is there some way to
calculate the total number of sectors from this without having the bytes per
sector value returned in the DISK_GEOMETRY structure?
Tim Roberts
>
>Also, I can give you a little more detail about the reason for wanting this
>functionality. The software I am writing is for management of a USB
>duplicator. Many of the potential users of this duplicator want to be able
>to duplicate very quickly to multiple drives at once and with this method I
>should be able to use multiple threads to write to each USB drive from a FIFO
>read buffer asynchronously. This way I only have to read through once and
>yet I can be copying all 20 USB drives at once.
"Quickly" is a very relative term in this case. The best you can do to a
single USB device is about 45 MB/s, but remember that USB bandwidth is
shared among ALL devices on a bus. You won't really get any
parallelization. It takes an hour to make one copy of an 80GB drive, but
it's going to take 11 hours to make 20 copies.
--
Tim Roberts, ti...@probo.com
Providenza & Boekelheide, Inc.
Ben Voigt [C++ MVP]
The described system should asymptotically approach a 2x speedup even if a
single device could saturate the bus, simply because you change from N
reads, N writes to 1 read, N writes.
David K
The drives that are going to be copied, although they are hi-speed USB 2.0
devices, most of the time will not function at the full 40-45 MB/s. This
should mean that if I make 20 copies at once, and each has a max transfer
rate of 2 MB/s then I should be able to copy to all 20 devices at once using
their full transfer rate and thus an asynchrounous copy will drastically
speed up the process. Obviously, the faster the transfer rate of each drive,
the more that my returns will be diminishing.
Paul Romero
I think what you are doing is similar to what
I need to do.
What I would like to do in applies particularly
to the following equipment used under Windows XP:
1) Driverless SmartMedia card reader/writer for USB.
2) Driverless USB hub--self powered.
3) 8 MB 8 bit Toshiba SmartMedia card.
Note: Multiple reader/writers are connected to the
hub and the hub is connected to a single USB port.
However, I think the required techniques are
the same for all driverless USB removable media.
I need to to perform the following operations
from a C or C++ program on each reader/writer
attached to the hub.
1) Format the media in the standard way.
2) Write, Read, and Erase binary data sequentially
to the entire user data portion of the fomatted
media.
What libary, API, etc. would you recommend for
performing the operations I have described ?
Also, what are essential operations involved ?
Best Regards,
Paul Romero