Is there a Linux version of fsutil to createnew empty file?
Bill David
empty file by:
fsutil file createnew b.txt 100000000000
Is there a Linux version of fsutil or how could we can create a empty
file with the specified size?
Note:
with the following command:
dd if=/dev/zero of=file.txt count=1000000
we can only create file with the size of n * 512 instead of the
specified size. And it's much slower than fsutil.
Bill David
LONG lDistance = 102; // file length is 102 bytes
DWORD dwPtr = SetFilePointer(hf, lDistance, NULL, FILE_BEGIN);
SetEndOfFile(hf);
But how to do it on Linux?
Josef Moellers
You could either divide your count by 512, but that would leave some
residue. However, the block size is not restricted to a power of 2, so
dd if=/dev/zero of=file.txt bs=1000 count=1000
would create a file of exactly 1000000 bytes, which is filled with bytes
of value 0.
I don't know of any standard program that uses the technique you
describe in your follow-up artice:
LONG lDistance = 102; // file length is 102 bytes
DWORD dwPtr = SetFilePointer(hf, lDistance, NULL, FILE_BEGIN);
SetEndOfFile(hf);
However, if you aren't afraid of some little programming, truncate() (or
ftruncate()) is your friend:
man truncate
This snippet of code will use truncate to do what you need (note: no
error checking done):
off_t lDistance = 102;
ftruncate(fd, lDistance);
This will create a sort of empty shell which consists more or less just
of the meta data describing a file of size 102 bytes but no data is
actually stored in that file. If you then attempt to read that file, the
kernel will return zeroes.
HTH,
Josef
--
These are my personal views and not those of Fujitsu Siemens Computers!
Josef Möllers (Pinguinpfleger bei FSC)
If failure had no penalty success would not be a prize (T. Pratchett)
Company Details: http://www.fujitsu-siemens.com/imprint.html
mblume
> Note:
> with the following command:
> dd if=/dev/zero of=file.txt count=1000000 we can only create file with
> the size of n * 512 instead of the specified size. And it's much slower
> than fsutil.
>
will create a file of exactly 100 bytes.
HTH
Martin
mblume
>
> I don't know of any standard program that uses the technique you
> describe in your follow-up artice:
>
will create a file f.txt composed of 9999 zeroes and 1 random byte.
dunno if it's faster than fsutil and whether it will actually create a
sparse file.
HTH
Martin
Rainer Weikusat
The file pointer can be set with lseek and at least a single byte of
data needs to be written to actually record the expanded filesize in
the filesystem. With dd, this could be expressed as
dd if=/dev/zero seek=101 bs=1 count=1 of=<filename>
bs ::= block size
count ::= # of blocks to copy
seek ::= seek to <position> before writing anything
Josef Moellers
I didn't think of that ...
I tried (the info was generated by "stat <filename>":
1. trunc bla 1000000 (a simple program which uses the ftruncate() call):
File: `bla'
Size: 1000000 Blocks: 0 IO Block: 4096 regular file
Device: fe00h/65024d Inode: 1146883 Links: 1
Access: (0644/-rw-r--r--) Uid: ( 1000/ josef) Gid: ( 1000/ josef)
Access: 2008-08-01 11:53:18.000000000 +0200
Modify: 2008-08-01 11:53:18.000000000 +0200
Change: 2008-08-01 11:53:18.000000000 +0200
2. dd if=/dev/zero of=blabla bs=1 seek=999999 count=1
File: `blabla'
Size: 1000000 Blocks: 16 IO Block: 4096 regular file
Device: fe00h/65024d Inode: 1146885 Links: 1
Access: (0644/-rw-r--r--) Uid: ( 1000/ josef) Gid: ( 1000/ josef)
Access: 2008-08-01 11:54:02.000000000 +0200
Modify: 2008-08-01 11:54:02.000000000 +0200
Change: 2008-08-01 11:54:02.000000000 +0200
3. dd if=/dev/zero bs=1000 count=1000 of=blablabla
File: `blablabla'
Size: 1000000 Blocks: 1968 IO Block: 4096 regular file
Device: fe00h/65024d Inode: 1146886 Links: 1
Access: (0644/-rw-r--r--) Uid: ( 1000/ josef) Gid: ( 1000/ josef)
Access: 2008-08-01 11:54:37.000000000 +0200
Modify: 2008-08-01 11:54:37.000000000 +0200
Change: 2008-08-01 11:54:37.000000000 +0200
Test were run on a Kubuntu 7.10 x86_64 2.6.22-15-generic kernel.
So:
1. ftruncate() generates a truly sparse file with absolutely no data
2. dd with seek generates a sparse file with some (16x512 bytes) data in it
3. dd without seek generates a whole lot of data
YMMV,
Jasen Betts
you can chose the block size to use, 512 is just the default block size.
it's not an empty file either, that one's full of zeros!
with dd you can specify the block size to use.
try this for an empty (that is "sparse") file.
dd if=/dev/null of=file.txt bs=1 seek=100000000000 count=0
jasen@gonzo:/tmp$ dd if=/dev/null of=file.txt bs=1 seek=100000000000
count=0
0+0 records in
0+0 records out
0 bytes (0 B) copied, 6.18e-06 s, 0.0 kB/s
Wow! that was quick! 6.18 milliseconds!
jasen@gonzo:/tmp$ ls -l file.txt
-rw-r--r-- 1 jasen jasen 100000000000 2008-08-01 21:43 file.txt
hmm, it seems to have worked,
jasen@gonzo:/tmp$ du file.txt
0 file.txt
But it's a sparse file. it contains address space for 100000000000 bytes
but it isn't that large on disk. (good thing too - /tmp is only 10GiB here)
if you want a full file instead of an sparse file do.
dd if=/dev/null of=file.txt bs=1 count=100000000000
100000000000 happens to be 2^11 * 5^11 so you could do
dd if=/dev/null of=file.txt bs=51200 count=1953125
which would probably run faster.
or even do
dd if=/dev/null of=file.txt bs=1000000 count=100000
etc... large block sizes will be faster, but block sizes that are
divisible by 4096 may be preferable.
Does the fsutil create sparse files (does windows support sparse files yet?)
or does it creat a file full of garbage sectors (the contents of
deleted files) ? or full of zeros?
AFAIK Linux provides no way for a normal user to map garbage sectors
into a file the drive, you can create normal files by writing them and
sparse files by skipping when writing.
In this way it provides a level of inter-user security when one user
deletes a file another can't easily recover the old data. and
confidential data won't be accidentally included in slack space in a
file.
there may be tool that can combine garbage sectors into a file without
overwriting them but such a tool would require root priviledges to run
and be a privacy risk, so I wouldn't reccomend its use except in
controlled circumstances.
Bye.
Jasen
Jasen Betts
ftruncate
or ftruncate64 if the numbers in your previous post are any indication.
what are you trying to achieve?
Bye.
Jasen
Jasen Betts
I got it to work without writing any bytes (see previous)
filesystem is EXT3
kernel is 2.6.24-1-686
dd is dd (coreutils) 6.10
(hmm I forgot to mount my /tmp partition - should be ext2 /tmp doesn't
need a journal) :)
Bye.
Jasen
Bill David
a file for local reading. A little strange, right? :)
Andrew Halliwell
> On Windows, there is a tool fsutil with which we can create a new
> empty file by:
> fsutil file createnew b.txt 100000000000
dd if=inputfilename of=outputfilename bs=blocksize count=numberofblocks
If the input file is /dev/zero and the output file is /home/wibble/output
and the bs=1M and count=100
Then you just created yourself a blank (full of zeroes) file called
/home/wibble/output that is 100 megabytes in size
dd is more flexible than that, anything can be an input or output file
(within reason)
so it can be used to copy partitions or data cds
or even rescue data from damaged drives
> Is there a Linux version of fsutil or how could we can create a empty
> file with the specified size?
>
> Note:
> with the following command:
> dd if=/dev/zero of=file.txt count=1000000
> we can only create file with the size of n * 512 instead of the
> specified size. And it's much slower than fsutil.
You need to specify blocksize and count
root@ponder:/media/sda8# dd if=/dev/zero of=testfile bs=100 count=1000
1000+0 records in
1000+0 records out
100000 bytes (100 kB) copied, 0.00578659 s, 17.3 MB/s
root@ponder:/media/sda8# ls -l testfile
-rw-r--r-- 1 root 501 100000 2008-08-01 11:57 testfile
root@ponder:/media/sda8# dd if=/dev/zero of=testfile bs=1000 count=1000
1000+0 records in
1000+0 records out
1000000 bytes (1.0 MB) copied, 0.00977888 s, 102 MB/s
root@ponder:/media/sda8# ls -l testfile
-rw-r--r-- 1 root 501 1000000 2008-08-01 11:58 testfile
--
| spi...@freenet.co.uk | |
| Andrew Halliwell BSc | "ARSE! GERLS!! DRINK! DRINK! DRINK!!!" |
| in | "THAT WOULD BE AN ECUMENICAL MATTER!...FECK!!!! |
| Computer Science | - Father Jack in "Father Ted" |
Jasen Betts
> a file for local reading. A little strange, right? :)
ok, then you definately don't want to use a sparse file for that. as
they don't use (or reserve) any disk space. (until you start writing
to them) Earlier I created a 100GB sparse file on a 3Gb partition...
This means you need to write the whole file and that will be slow as
it involves the drive heads passing over each sector you want to have
in the file.
As I said earlier while it is theoretically possible I know of no way
to create a solid file without writing it. (well, I know of one
special case - an undelete tool - but you don't get to choose the
size.)
Hmm there is a file system tool called debugfs it's a low-level thing
- you should unmount the partition before starting debugfs. That can be
inconvenient in many circumstances, also you need to have write perms
to the drive itself (this usually means you need to be root)
but I'm pretty sure it can be used to combine arbitrary sectors into a
file.... but this seems to be the wrong approach on many levels...
A google for "linux reserve file space" (sans quotes)
found me this article.
Which mentions a way to do it with XFS 3 years ago, that may be the
answer you want. it may be possible to do it with EXT3 now. this
question may be better answered by the people in one of the linux
kernel newsgroups.
be sure to mention what you are trying to achieve any why.
Bye.
Jasen
Jasen Betts
> dd is more flexible than that, anything can be an input or output file
> (within reason)
within reason?
# don't try this at home (or especially at work)
dd if=/dev/input/mice of=/dev/mem
anything that exists in the filesystem can be an endpoint for dd.
(this means you can't dd to network sockets, but pretty-much all else
is in the filesystem)
eg: if you're root you can view the strings in the BIOS ROM with
dd if=/dev/mem bs=16 skip=$(( 0xfe00 )) count=8192 | strings | less
Bye.
Jasen
Rainer Weikusat
> On 2008-08-01, Andrew Halliwell <spi...@ponder.sky.com> wrote:
>
>> dd is more flexible than that, anything can be an input or output file
>> (within reason)
>
> within reason?
>
> # don't try this at home (or especially at work)
> dd if=/dev/input/mice of=/dev/mem
dd if=/dev/input/mouse of=/dev/mixer?
Armel
"Jasen Betts" <ja...@xnet.co.nz> a écrit dans le message de news:
g6unk9$54c$1@gonzo...[...]
> Does the fsutil create sparse files (does windows support sparse files
> yet?)
> or does it creat a file full of garbage sectors (the contents of
> deleted files) ? or full of zeros?
>
> AFAIK Linux provides no way for a normal user to map garbage sectors
> into a file the drive, you can create normal files by writing them and
> sparse files by skipping when writing.
>
> In this way it provides a level of inter-user security when one user
> deletes a file another can't easily recover the old data. and
> confidential data won't be accidentally included in slack space in a
> file.
zeroes).
> there may be tool that can combine garbage sectors into a file without
> overwriting them but such a tool would require root priviledges to run
> and be a privacy risk, so I wouldn't reccomend its use except in
> controlled circumstances.
>
> Bye.
> Jasen
Armel
Chris Cox
Personally I would do something like:
dd if=/dev/zero of=file.txt bs=1000000000 count=1
(of course that's just 1G... adjust count appropriately
or increase bs if you have the memory)
Grant Edwards
>> fsutil file createnew b.txt 100000000000
> Personally I would do something like:
>
> dd if=/dev/zero of=file.txt bs=1000000000 count=1
>
> (of course that's just 1G... adjust count appropriately
> or increase bs if you have the memory)
That's going to take a long time. You'd be far better off doing
somthign like this:
dd if=/dev/zero of=file.txt bs=1 seek=99999999999 count=1
That only takes 69 milliseconds (of real time).
--
Grant Edwards grante Yow! YOW!!! I am having
at fun!!!
visi.com
Mark Hobley
> dd if=/dev/zero of=file.txt count=1000000
Hmmm, I would use head to truncate the output to the appropriate number
of bytes:
dd if=dev/zero count=`expr 1000000 / 512` | head -c 1000000 >file.txt
This is a bit of a kludge using a presumed block size.
You might be able to find an alternative data dumper which allows a count in
bytes, rather than in blocks, but I haven't looked at this. However a
count in bytes sounds useful to me.
Mark.
--
Mark Hobley,
393 Quinton Road West,
Quinton, BIRMINGHAM.
B32 1QE.
Grant Edwards
> Bill David <billd...@gmail.com> wrote:
>
>> dd if=/dev/zero of=file.txt count=1000000
>
> Hmmm, I would use head to truncate the output to the appropriate number
> of bytes:
>
> dd if=dev/zero count=`expr 1000000 / 512` | head -c 1000000 >file.txt
Sorry, but I'm a bit baffled why
1) that's better than just producing the proper number of
bytes to start with.
2) that's better than just seeking to the position of the
desired last byte and writing one byte.
--
Grant Edwards grante Yow! Mr and Mrs PED, can I
at borrow 26.7% of the RAYON
visi.com TEXTILE production of the
INDONESIAN archipelago?
Mark Hobley
>> dd if=dev/zero count=`expr 1000000 / 512` | head -c 1000000 >file.txt
>
> Sorry, but I'm a bit baffled why
>
> 1) that's better than just producing the proper number of
> bytes to start with.
Its not. If you could produce the proper number of bytes from the
dumper, that would be a better method. However the count is in blocks.
It is not advisable to use a block size of one, this causes a device to
re-read the same block 512 times. It is better to read whole blocks,
then truncate what is not needed. (This wouldn't really matter for the
/dev/zero device, but could really hammer a diskette type input device.)
>
> 2) that's better than just seeking to the position of the
> desired last byte and writing one byte.
Hmmm, I must have misread your question. I presumed that you wanted the
first so many bytes, rather than a single byte.
Regards,
Francis Litterio
> Grant Edwards <gra...@visi.com> wrote:
>>> dd if=dev/zero count=`expr 1000000 / 512` | head -c 1000000 >file.txt
>>
>> Sorry, but I'm a bit baffled why
>>
>> 1) that's better than just producing the proper number of
>> bytes to start with.
>
> Its not. If you could produce the proper number of bytes from the
> dumper, that would be a better method. However the count is in blocks.
> It is not advisable to use a block size of one, this causes a device to
> re-read the same block 512 times.
I think the block is read from the disk just once. Subsequent reads
are satisfied from the in-memory disk buffer cache. What you describe
would only happen if you made dd read from an unbuffered disk device.
Even /dev/sda is buffered.
--
Fran
Grant Edwards
The "first so many bytes" of /dev/zero are all 0x00's.
Seeking to the last byte of an empty file and writing a 0x00
results in a file that's all 0x00's.
You end up with the same thing, except the seek approach uses
much less disk space.
--
Grant Edwards grante Yow! Do I hear th'
at SPINNING of various
visi.com WHIRRING, ROUND, and WARM
WHIRLOMATICS?!
Chris Cox
> On 2008-08-02, Chris Cox <ccox_nop...@airmail.net> wrote:
>
>>> fsutil file createnew b.txt 100000000000
>
>> Personally I would do something like:
>>
>> dd if=/dev/zero of=file.txt bs=1000000000 count=1
>>
>> (of course that's just 1G... adjust count appropriately
>> or increase bs if you have the memory)
>
> That's going to take a long time. You'd be far better off doing
> somthign like this:
>
> dd if=/dev/zero of=file.txt bs=1 seek=99999999999 count=1
>
> That only takes 69 milliseconds (of real time).
>
yes... as others have already pointed out.. nothing was actually
WRITTEN in your example. So yes, creating a sparse file
(duh) is obviously faster... thanks for pointing it out
(again).
Tim Smith
<dd179217-f29c-4496...@r15g2000prd.googlegroups.com>,
Bill David <billd...@gmail.com> wrote:
> On Windows, there is a tool fsutil with which we can create a new
> empty file by:
> fsutil file createnew b.txt 100000000000
>
> Is there a Linux version of fsutil or how could we can create a empty
> file with the specified size?
$ perl > b.txt
seek STDOUT, 100000000000-1, 0; print 'a';
(hit control-D)
$ ls -ls b.txt
16 -rw-r--r-- 1 tzs tzs 100000000000 Aug 2 22:09 b.txt
On filesystems that support holes, the file will not actually take up
100000000000 bytes. It will appear to applications as if it has
100000000000-1 bytes of nulls, followed by a byte of 'a'. Note that
b.txt is only actually using 16 blocks.
If, for some reason, you do not want that, but really want ALL the
blocks allocated, you can do this:
$ perl > b.tmp
seek STDOUT, 100000000000-1, 0; print 'a';
(hit control-D)
$ cat b.tmp > b.txt
$ rm b.tmp
If you were to do that, and do "ls -ls b.txt", you'd see b.txt has all
the blocks allocated.
--
--Tim Smith
Jasen Betts
>> dd if=dev/zero count=`expr 1000000 / 512` | head -c 1000000 >file.txt
>
> Sorry, but I'm a bit baffled why
>
> 1) that's better than just producing the proper number of
> bytes to start with.
Disk operations are marginally faster if the data is block aligned
the original post suggested sizes in the order of 100GB
> 2) that's better than just seeking to the position of the
> desired last byte and writing one byte.
The OP's goal is to use space on the disk and a sparse file won't do
that.
Bye.
Jasen
Grant Edwards
> Grant Edwards wrote:
>
>> dd if=/dev/zero of=file.txt bs=1 seek=99999999999 count=1
>>
>> That only takes 69 milliseconds (of real time).
>>
>
> yes... as others have already pointed out.. nothing was actually
> WRITTEN in your example.
One byte was written -- which is pretty close to nothing
compared with 99999999999.
--
Grant Edwards grante Yow! Go on, EMOTE! I
at was RAISED on thought
visi.com balloons!!
David Schwartz
> That's going to take a long time. You'd be far better off doing
> somthign like this:
>
> dd if=/dev/zero of=file.txt bs=1 seek=99999999999 count=1
>
> That only takes 69 milliseconds (of real time).
The OP's goal was to reserve space. Of what use is a method that
doesn't reserve any space?
DS
David Schwartz
> > Is there a Linux version of fsutil or how could we can create a empty
> > file with the specified size?
The answer, sadly, is no. I had precisely the same problem and
ultimately was forced to accept that this is going to take awhile on
Linux.
> $ perl > b.txt
> seek STDOUT, 100000000000-1, 0; print 'a';
> (hit control-D)
> $ ls -ls b.txt
> 16 -rw-r--r-- 1 tzs tzs 100000000000 Aug 2 22:09 b.txt
>
> On filesystems that support holes, the file will not actually take up
> 100000000000 bytes. It will appear to applications as if it has
> 100000000000-1 bytes of nulls, followed by a byte of 'a'. Note that
> b.txt is only actually using 16 blocks.
Right, so this does less than the OP wants. (The space is not actually
reserved.)
> If, for some reason, you do not want that, but really want ALL the
> blocks allocated, you can do this:
>
> $ perl > b.tmp
> seek STDOUT, 100000000000-1, 0; print 'a';
> (hit control-D)
> $ cat b.tmp > b.txt
> $ rm b.tmp
>
> If you were to do that, and do "ls -ls b.txt", you'd see b.txt has all
> the blocks allocated.
Right, but this does more than the OP wants. (So it takes a long
time.)
DS