[erlang-questions] Newbie questions about file module
Ville Silventoinen
I've written a small Erlang program that copies directory trees. I cannot figure out how to set the owner (Unix UID and GID) for symbolic links, because file:write_file_info/2 modifies the target file, not the symlink. Unix command chown has -h option for this purpose. Is there any way of doing this in Erlang? Why isn't there file:write_link_info/2? Perhaps I've missed something, is there a way to copy the symlink as it is? (I do use read_link and read_link_info.)
My second question relates to eprof. I analysed the execution of my program (copying /usr/local/lib to NFS directory). The eprof:total_analyse/0 results are below ("mirror11" is my module). It shows that 82% of the time goes in file_name_1/1, which "Generates a flat file name from a deep list of atoms and characters (integers)". Does this really mean that 82% of time is spent here, instead of doing disk I/O? I would have expected file:copy/2 to be on the top of the list. (There are 481 directories and 6744 files in my /usr/local/lib.)
Any comments would be appreciated!
Thanks,
Ville
7> eprof:total_analyse().
FUNCTION CALLS TIME
file:file_name_1/1 3086234 82 %
gen:do_call/4 29869 2 %
file:check_args/1 93941 2 %
gen:wait_resp_mon/3 29869 1 %
mirror11:get_file_info/2 21195 1 %
file:check_and_call/2 29869 1 %
file:call/2 29869 1 %
gen:call/4 29869 1 %
file:file_name/1 36615 1 %
gen_server:call/3 29869 1 %
mirror11:do_file/4 6744 1 %
erlang:'++'/2 21680 0 %
erlang:monitor/2 29869 0 %
file:read_link_info/1 14451 0 %
file:copy_int/3 13488 0 %
erlang:whereis/1 29869 0 %
erlang:demonitor/1 29869 0 %
erlang:list_to_tuple/1 29869 0 %
erlang:list_to_binary/1 21195 0 %
mirror11:by_type/6 7707 0 %
lists:foreach/2 7708 0 %
ets:lookup/2 21195 0 %
mirror11:log_debug/2 6748 0 %
file:write_file_info/2 7225 0 %
ets:insert/2 7707 0 %
mirror11:do_dir/3 481 0 %
mirror11:'-do_dir/3-fun-5-'/4 6744 0 %
file:copy/2 6744 0 %
mirror11:start_dir_workers/5 963 0 %
ets:delete/1 481 0 %
file:list_dir/1 962 0 %
mirror11:rendezvous/1 963 0 %
erlang:apply/2 481 0 %
erlang:spawn/3 481 0 %
file:make_dir/1 481 0 %
ets:new/2 481 0 %
erlang:spawn/1 481 0 %
mirror11:'-start_dir_workers/5-fun-0-'/3 481 0 %
mirror11:do_symlink/4 2 0 %
mirror11:master_loop/0 1 0 %
file:read_link/1 4 0 %
file:make_symlink/2 2 0 %
mirror11:'-do_dir/3-fun-6-'/4 2 0 %
Total time: 51.30
Measurement overhead: 13.48
ok
Scott Lystig Fritchie
vs> [...] It shows that 82% of the time goes in file_name_1/1,
vs> which "Generates a flat file name from a deep list of atoms and
vs> characters (integers)".
Ville, I don't have a good reply (other than, yes, 82% of the app's user
time is probably being used there ... but eprof can't measure
system/in-kernel time well). But I do have a "yeah, me too!" message
for the OTP gang.
I've got an app that does a *lot* of file opening & closing.
1. Figure out what file a key is stored in.
2. Open that file.
3. Seek to some position within the file
4. Read some bytes.
5. Close the file
6. Lather, rinse, repeat...
... across tens of thousands of files in the local file system
(representing gazillions of smaller objects). After I discovered the
cost of erlang:md5/1 vs. crypto:md5/1 (see message to the list a little
while ago), the 2nd biggest time sucker was the same function that Ville
is complaining about: file:file_name_1/1.
How frequent is it to use a deep I/O list as a file name? (Perhaps I've
been in C land for too long to think that using a deep I/O list like
that is actually a good idea?) Doesn't an argument like that get
automatically flattened when passed to the efile_drv driver? If yes,
why bother flattening on the Erlang side?
-Scott
_______________________________________________
erlang-questions mailing list
erlang-q...@erlang.org
http://www.erlang.org/mailman/listinfo/erlang-questions
Bjorn Gustavsson
<frit...@snookles.com> wrote:
> Ville Silventoinen <ville.sil...@googlemail.com> wrote:
>
> How frequent is it to use a deep I/O list as a file name? (Perhaps I've
> been in C land for too long to think that using a deep I/O list like
> that is actually a good idea?) Doesn't an argument like that get
> automatically flattened when passed to the efile_drv driver? If yes,
> why bother flattening on the Erlang side?
Unfortunately, for historical reasons, filenames are allowed to
contain atoms and atoms are not allowed in I/O lists. At one time
in OTP's history, we wanted to disallowed atoms in filenames, but
it turned out that many applications used atoms in filenames.
/Bjorn
--
Björn Gustavsson, Erlang/OTP, Ericsson AB
Bjorn Gustavsson
> Hi,
>
> My second question relates to eprof. I analysed the execution of my program
> (copying /usr/local/lib to NFS directory). The eprof:total_analyse/0 results
> are below ("mirror11" is my module). It shows that 82% of the time goes in
> file_name_1/1, which "Generates a flat file name from a deep list of atoms
> and characters (integers)". Does this really mean that 82% of time is spent
> here, instead of doing disk I/O? I would have expected file:copy/2 to be on
> the top of the list. (There are 481 directories and 6744 files in my
> /usr/local/lib.)
>
Unless you specifically included the file_server_2 process in the set
of processes
to profile, you are only profiling the time spent in your own
processes and not code
that is executed in other processes. (And since I don't see no calls
from the file_server or prim_file
module in the output from eprof, I assume that you did not include the
file_server_2 process.)