Why is MOVEFILE so slow?
Kevin
I use MOVEFILE to rename file but it seems to me that it takes some time if the file is large. This should not happen.
Am I missing something?
Rune Allnor
Yes. The speed of disk operations depend on the
operating system. Some OSes allow you to rename
a file by updating the name property of some
look-up table in the file system, while other
OSes do not. Instead, they require the file to
actually be moved, in order to create a new entry
in the table and delete the old one.
If you happen to work with an OS that does not
allow users to manipulate the name properties,
you need to take the slow path.
Rune
Kevin
Or this is more related to Java. Does MOVEFILE eventually call some JAVA functions to do the actual work and to make this portable for all OS that Matlab supports?
Rune Allnor <all...@tele.ntnu.no> wrote in message <12e8aec7-ae45-40ce...@r5g2000yqb.googlegroups.com>...
Steven Lord
"Kevin" <kh...@fake.com> wrote in message
news:hf1hv8$2dn$1...@fred.mathworks.com...
Why not? If you've got a lot of bits to move, why shouldn't it take a while
to move them?
Are you using MOVEFILE with the source and destination file both on the same
local disk, or is one or both of the files on a network location?
Do you have a (potentially overzealous) virus scanner that needs to scan the
file before or as it's moving?
Is the disk to which you're moving the file heavily fragmented, so the OS
needs to search around a bit for places to put the pieces of the file?
Is some other process accessing the disk at the same time, thus causing
contention for who gets to read/write at a given time?
etc.
> Am I missing something?
Yes. There are lots of reasons why MOVEFILE could take some time to move a
file. Now that being said, if it's taking an EXTREMELY long time (say
minutes for a few meg file) then you should contact whomever is in charge of
maintenance for the system on which you're running to make sure that the
machine is operating correctly, and if they give it a thumbs-up contact
MathWorks Technical Support.
--
Steve Lord
sl...@mathworks.com
comp.soft-sys.matlab (CSSM) FAQ: http://matlabwiki.mathworks.com/MATLAB_FAQ
Kevin
But you have raised a good point about the virus scanner. I use AVG and I would not know what AVG does.
"Steven Lord" <sl...@mathworks.com> wrote in message <hf3ar4$p21$1...@fred.mathworks.com>...
Kevin
Firstly, I created a file called "tmp.bin" in my local hard-drive and the size of this file is 20,000,000 bytes.
Then I do this in the Matlab command line:
datestr(now),
for n = 1:10,
movefile('tmp.bin', 'tmp1.bin');
movefile('tmp1.bin', 'tmp.bin');
end;
datestr(now)
ans =
01-Dec-2009 14:27:37
ans =
01-Dec-2009 14:27:48
So this means that it takes 10 seconds to do 20 file renames.
I am using P-4 with 2 GB memory and Windows XP.
I agree that I don't have a fast PC (in today standard) but I don't expect it takes that long to rename a file.
I don't think I have virus scan running on my PC.
I have done one more experiment. Reduce the file size to 200,000 (i.e. 100 times less) and now it takes less than 1 sec to do 20 renames.
Just think of one more thing to try. Instead of using MOVEFILE, I use dos('rename ...').
datestr(now),
for n = 1:10,
dos('rename tmp.bin tmp1.bin');
dos('rename tmp1.bin tmp.bin');
end;
datestr(now)
This takes 1 sec to do 20 renames for file szie = 20,000,000 bytes.
I also use the dos method on smaller file (200,000 bytes). It also takes 1 sec to do 20 renames.
So it is probably not Windows. It is somewhere else inside MOVEFILE.
In conclusion, it is faster to use dos('rename ....') than MOVEFILE to rename file. On Unix systems, we can use unix('mv ...').
"Kevin" <kh...@fake.com> wrote in message <hf4217$j7h$1...@fred.mathworks.com>...
dpb
...
> So it is probably not Windows. It is somewhere else inside MOVEFILE.
That does seem peculiar -- I at first was going to suggest that perhaps
movefile() is internally spawning a DOS session but the test of using
the system call seems to indicate that even that should be much quicker.
Seems like only recourse would be to follow S Lord's suggestion and send
this to TMW formal support for comment/action...
--
Kevin
Right?
dpb <no...@non.net> wrote in message <hf47bu$msl$2...@news.eternal-september.org>...
Rune Allnor
> Really!!! I am quite surprised any today OS would do that. I am using Windows Vista. I would think that even Windows would not even do that.
Run the script below with the file browser open.
The script stores 32MBytes of data in the file
test01.mat, and renames the file to test02.mat,
test03.mat, etc., up to test21.mat.
Under winXP and R2006a you can see that there are
two files present at any time; the source file
and destination file. In this configuration there
is in fact a file copy taking place.
Rune
Rune Allnor
It's well after midnight - forgot the code...:
A = randn(1000000,4);
save test01.mat A
tic
for n=1:20
sname = sprintf('test%02d.mat',n);
dname = sprintf('test%02d.mat',n+1);
movefile(sname,dname);
end
toc
delete(dname);
Kevin
After MOVEFILE is executed, the source file should not exist anymore. Right?
Rune Allnor <all...@tele.ntnu.no> wrote in message <89dd7d3f-2020-461c...@k19g2000yqc.googlegroups.com>...
Rune Allnor
> Hmm. I am missing something.
>
> After MOVEFILE is executed, the source file should not exist anymore. Right?
Don't top post!
Both the source and destination files exist *while* the
file is moved. Which is why one can confidently state
that there is a copy taking place.
Rune
Kevin
Do you have another Matlab or window apps running to do directory listing all the time?
Rune Allnor <all...@tele.ntnu.no> wrote in message <cfeb78fd-d324-4f39...@o10g2000yqa.googlegroups.com>...
Chris
> I have seen this before but just don't have time to look into it. Actually there is not much that I can do since MOVEFILE is a built-in function.
>
> I use MOVEFILE to rename file but it seems to me that it takes some time if the file is large. This should not happen.
>
> Am I missing something?
I suspect there is some dependence on the number of files in the directory you are moving files from. I'm trying to use matlab to move images that meet a certain criteria, & although none over 100kb in size, the movefile function takes over a minute per file. The dos('move ...') takes 10s (including loading and some arithmatic) per file, which is considerably quicker, but still slow!
Catherine
I found a similiar issue - when I did a practice run on a folder with <100 .wav files, it went very fast (~.1s/file), but on the full folder with over 4000 .wav files it slowed down to around 5 seconds a file. BUT, the interesting thing is that if I wavread in the file, then wavwrite it back out as a new filename - it only takes .3s/file, even with the 4000 file folder. That doesn't seem right. I tried the following renaming lines (ofn: oldfilename, nfn: newfilename):
movefile(ofn, nfn)
dos(['rename ' ofn ' ' nfn]);
copyfile(ofn,nfn)
system([localmovename ' ' ofn ' ' nfn]);
Does anyone understand this?
Marc
try the following function in Matlab I went from 4.6s for rename 1 file to 0.0016s. I had 10,000 files to rename and went from 11hr to about 10s to rename everything!!!
java.io.File(filename).renameTo(java.io.File(newFilename));
source and explanation
http://laiso1983.blogspot.com/2010/11/how-to-quickly-rename-file-in-matlabhow.html
Jan Simon
> java.io.File(filename).renameTo(java.io.File(newFilename));
BTW: Split this for Matlab 6.5:
A = java.io.File(filename);
A.renameTo(java.io.File(newFilename));
I've implemented a call to _wrename as C-Mex FileRename, which is 20% to 50% faster than this java method:
http://www.mathworks.com/matlabcentral/fileexchange/29569
The only drawback compared to MOVEFILE is, that _wrename cannot move a folder to another drive, and that the destination folder must exist.
MOVEFILE of Matlab 6.5 called DOS('move'), which was really ugly slow: the C-Mex is 1600 times (not percent!) faster. Fortunately Matlab uses a built-in function now, but even considering the additional features I cannot imagine, why the C-Mex could be ~50 times faster than MOVEFILE for renaming 800 files.
After including the java renameTo method in the text function of the FileRename, the system memory was sometimes exhausted and the OS started disk swapping. Of course I cannot be sure, if this was caused by calling java, and such effects are not perfectly reproducible.
I'd be curious for some comparisons under MacOS and Linux.
Kind regards, Jan