Automatically rename a downloaded file by curl to avoid filename conflict.
Hongyi Zhao
When I download a file by curl, I use -o switch to write the file as a
local file, say, myfile. The issue is: if a file with the same
filename already exists in the target folder, the original file will
be overwrited. So, I want to rename this downloaded file to something
like this: myfile.1 in order to avoid filename conflict in the above
case; in gerneral, if myfile.1 also has already exists in the
destination folder, use myfile.2 as the filename for the curl's
output; and so on ...
Any hints for the above purpose?
Thanks in advance.
--
.: Hongyi Zhao [ hongyi.zhao AT gmail.com ] Free as in Freedom :.
Chris F.A. Johnson
> Hi all,
>
> When I download a file by curl, I use -o switch to write the file as a
> local file, say, myfile. The issue is: if a file with the same
> filename already exists in the target folder, the original file will
> be overwrited. So, I want to rename this downloaded file to something
> like this: myfile.1 in order to avoid filename conflict in the above
> case; in gerneral, if myfile.1 also has already exists in the
> destination folder, use myfile.2 as the filename for the curl's
> output; and so on ...
dlfile=myfile
dfnum=1
while [ -e "$myfile.$dfnum" ]
do
dfnum=$(( $dfnum + 1 ))
done
dlfile=$dlfile.$dfnum
curl -o "$dlfile" ...
--
Chris F.A. Johnson, author <http://shell.cfajohnson.com/>
===================================================================
Shell Scripting Recipes: A Problem-Solution Approach (2005, Apress)
Pro Bash Programming: Scripting the GNU/Linux Shell (2009, Apress)
===== My code in this post, if any, assumes the POSIX locale =====
===== and is released under the GNU General Public Licence =====
Hongyi Zhao
<cfajo...@gmail.com> wrote:
>dlfile=myfile
>dfnum=1
>while [ -e "$myfile.$dfnum" ]
>do
> dfnum=$(( $dfnum + 1 ))
>done
>dlfile=$dlfile.$dfnum
>
>curl -o "$dlfile" ...
Dood, thanks a lot, I've done some revision based on your version:
dlfile=myfile
dfnum=1
if [ ! -e "$dlfile" ]
then
curl -o "$dlfile" ...
else
while [ -e "$dlfile.$dfnum".pdf ]
do
dfnum=$(( $dfnum + 1 ))
done
dlfile=$dlfile.$dfnum
curl -o "$dlfile" ...
fi
Thanks again.
Hongyi Zhao
<hongy...@gmail.com> wrote:
>while [ -e "$dlfile.$dfnum".pdf ]
Should be
while [ -e "$dlfile.$dfnum" ]
Sorry.
Mart Frauenlob
> Hi all,
>
> When I download a file by curl, I use -o switch to write the file as a
> local file, say, myfile. The issue is: if a file with the same
> filename already exists in the target folder, the original file will
> be overwrited. So, I want to rename this downloaded file to something
> like this: myfile.1 in order to avoid filename conflict in the above
> case; in gerneral, if myfile.1 also has already exists in the
> destination folder, use myfile.2 as the filename for the curl's
> output; and so on ...
>
> Any hints for the above purpose?
>
> Thanks in advance.
Example in bash, x is undeclared or '0' at first call:
#!/bin/bash
filename="/tmp/file"
newfile="${filename}"
x=""
until [[ ! -e $newfile ]]; do newfile="${filename}.$((++x))"; done
curl -o "$newfile"
best regards
Mart
Stephane CHAZELAS
> On Thu, 04 Feb 2010 16:12:30 +0800, Hongyi Zhao
> <hongy...@gmail.com> wrote:
>
>>while [ -e "$dlfile.$dfnum".pdf ]
>
> Should be
>
> while [ -e "$dlfile.$dfnum" ]
No, those are subject to race conditions, use set -C,
redirections (no -o in curl or -O- in wget).
(
set -C
ext= n=1
until command exec 3> "$file$ext"
ext=.$((++n))
done
exec curl ... >&3
)
--
Stï¿œphane
Hongyi Zhao
<hongy...@gmail.com> wrote:
>When I download a file by curl, I use -o switch to write the file as a
>local file, say, myfile. The issue is: if a file with the same
>filename already exists in the target folder, the original file will
>be overwrited. So, I want to rename this downloaded file to something
>like this: myfile.1 in order to avoid filename conflict in the above
>case; in gerneral, if myfile.1 also has already exists in the
>destination folder, use myfile.2 as the filename for the curl's
>output; and so on ...
Let me be more precise in descripting my requirements as follows:
1- If myfile doesn't exist in the target folder, use myfile as the
downloaded filename.
2- If myfile already exists in the target folder and at the same time
myfile.1 doesn't exist in the target folder, use myfile.1 as the
downloaded filename.
3- If myfile and myfile.1 already exist in the target folder and at
the same time myfile.2 doesn't exist in the target folder, use
myfile.2 as the downloaded filename.
4- Extending the above rules to deal with the remaining cases.
Regards.
Hongyi Zhao
<stephane...@yahoo.fr> wrote:
>No, those are subject to race conditions, use set -C,
I'm a newbie of bash/shell programming. What do you mean by saying
*race conditions*? Could you please give me some more hints?
>redirections (no -o in curl or -O- in wget).
>
>(
> set -C
> ext= n=1
> until command exec 3> "$file$ext"
> ext=.$((++n))
> done
> exec curl ... >&3
>)
Could you please give the complete code segment for my issue?
Thanks in advance.
Stephane CHAZELAS
> On Thu, 4 Feb 2010 10:57:07 +0000 (UTC), Stephane CHAZELAS
> <stephane...@yahoo.fr> wrote:
>
>>No, those are subject to race conditions, use set -C,
>
> I'm a newbie of bash/shell programming. What do you mean by saying
> *race conditions*? Could you please give me some more hints?
I mean when you do
if (can I do-that) then do-that
There's always a chance that in between the test (can I do-that)
and the action (do-that), the condition has changed. For
instance, if two instances of your script where run at the same
time, it could happen. It's probably not critical here, but this
kind of coding is known to have introduced countless security
issues, so it's a good habit to avoid it.
>>redirections (no -o in curl or -O- in wget).
>>
>>(
>> set -C
>> ext= n=1
>> until command exec 3> "$file$ext"
>> ext=.$((++n))
>> done
>> exec curl ... >&3
>>)
>
> Could you please give the complete code segment for my issue?
[...]
url=http://www.google.com/
file=myfile
(
set -C
ext= n=0
until command exec 3> "$file$ext"; do
ext=.$((++n))
done
exec curl "$url" >&3
)
--
Stï¿œphane
Seebs
> I'm a newbie of bash/shell programming. What do you mean by saying
> *race conditions*? Could you please give me some more hints?
A "race condition" is a case where other things happening at the same
time could cause something to fail. For instance, consider the simple
test:
if [ ! -e mydir ]; then
mkdir mydir
fi
This seems like it should create "mydir" only if there is not currently
anything named "mydir". However, it is not so. The [ ! -e mydir ] executes,
and then the shell executes mkdir. Other programs can do things between
them; for instance, another program could create a file named "mydir" after
the [ command has run, but before the mkdir has run, and then mkdir will
fail.
Similarly:
if [ -d mydir ]; then
cd mydir
fi
may fail unexpectedly, because "mydir" could be deleted between the [ and
the cd.
-s
--
Copyright 2010, all wrongs reversed. Peter Seebach / usenet...@seebs.net
http://www.seebs.net/log/ <-- lawsuits, religion, and funny pictures
http://en.wikipedia.org/wiki/Fair_Game_(Scientology) <-- get educated!
Rikishi42
> When I download a file by curl, I use -o switch to write the file as a
> local file, say, myfile. The issue is: if a file with the same
> filename already exists in the target folder, the original file will
> be overwrited. So, I want to rename this downloaded file to something
> like this: myfile.1 in order to avoid filename conflict in the above
> case; in gerneral, if myfile.1 also has already exists in the
> destination folder, use myfile.2 as the filename for the curl's
> output; and so on ...
>
> Any hints for the above purpose?
Why bother ? Why not just put date and time in the file's name?
get download date in a variable (not technically indispensable).
OUTPUT_NAME=`date "+%Y%m%d_%H%M%S"`
Then, eighter use:
curl -o $OUTPUT_NAME
or, if you can't specify a download name:
curl -o
mv myfile $OUTPUT_NAME
--
Any time things appear to be going better, you have overlooked
something.
Hongyi Zhao
>A "race condition" is a case where other things happening at the same
>time could cause something to fail. For instance, consider the simple
>test:
[snipped]
Good, thanks a lot.
Hongyi Zhao
<skunk...@rikishi42.net> wrote:
>Why bother ? Why not just put date and time in the file's name?
>
>get download date in a variable (not technically indispensable).
> OUTPUT_NAME=`date "+%Y%m%d_%H%M%S"`
>
>Then, eighter use:
> curl -o $OUTPUT_NAME
>
>or, if you can't specify a download name:
> curl -o
> mv myfile $OUTPUT_NAME
If the system's time has happened to be changed by some virus, this
method maybe fail to generate the _unique_ OUTPUT_NAME.
Kaz Kylheku
>
> When I download a file by curl, I use -o switch to write the file as a
> local file, say, myfile. The issue is: if a file with the same
> filename already exists in the target folder, the original file will
> be overwrited. So, I want to rename this downloaded file to something
> like this: myfile.1 in order to avoid filename conflict in the above
> case; in gerneral, if myfile.1 also has already exists in the
> destination folder, use myfile.2 as the filename for the curl's
> output; and so on ...
>
> Any hints for the above purpose?
With all these .1 .2 .3 ... files littering a single directory, how will you
know which is which? File foo.2 might be last week's file, whereas bar.3 is
yesterday's. Moreover, foo.1 and foo.2 might not even be related; what if they
come from different sites, and just coincidentally have the same name?
Maybe you should start a new directory for each separate download session,
to keep things cleanly separated and organized.
If the files are related together (.1 .2 ... are in fact versions of the same
file), then download into a version control sandbox, allowing the download to
overwrite, and check in the revisions.
GIT is good for this purpose because it's fast, and the working copy /is/ the
repository. No separate repo to set up.
Rikishi42
>
>
> On Fri, 5 Feb 2010 01:24:10 +0100, Rikishi42
><skunk...@rikishi42.net> wrote:
>
>>Why bother ? Why not just put date and time in the file's name?
>>
>>get download date in a variable (not technically indispensable).
>> OUTPUT_NAME=`date "+%Y%m%d_%H%M%S"`
>>
>>Then, eighter use:
>> curl -o $OUTPUT_NAME
>>
>>or, if you can't specify a download name:
>> curl -o
>> mv myfile $OUTPUT_NAME
>
> If the system's time has happened to be changed by some virus, this
> method maybe fail to generate the _unique_ OUTPUT_NAME.
Sigh... I forgot I was in nitpicker's county, here.
Yes, you're right. But don't you think the user has more urgent problems to
deal with, then?
Besides, there is a much bigger problem with the script as it is: DST. Once
a year, the files will overwrite themselves for an hour.
... and even that is not an issue. Using UTC time, instead of local time, will
solve that.
Hongyi Zhao
The above code will give annoying output like this:
line 15: myfile: cannot overwrite existing file.
How can I suppress these output?
Hongyi Zhao
<stephane...@yahoo.fr> wrote:
> ext= n=0
Why this line must has a white space after *ext=*?
Best regards.
Chris F.A. Johnson
> On Thu, 4 Feb 2010 12:18:29 +0000 (UTC), Stephane CHAZELAS
><stephane...@yahoo.fr> wrote:
>
>> ext= n=0
>
> Why this line must has a white space after *ext=*?
If it didn't, it would assign 'n=0' to ext.
Stephane CHAZELAS
> On Thu, 4 Feb 2010 12:18:29 +0000 (UTC), Stephane CHAZELAS
> <stephane...@yahoo.fr> wrote:
>
>>url=http://www.google.com/
>>file=myfile
>>(
>> set -C
>> ext= n=0
>> until command exec 3> "$file$ext"; do
>> ext=.$((++n))
>> done
>> exec curl "$url" >&3
>>)
>
> The above code will give annoying output like this:
>
> line 15: myfile: cannot overwrite existing file.
>
> How can I suppress these output?
until { command exec 3> "$file$ext"; } 2> /dev/null; do
--
Stï¿œphane
Hongyi Zhao
<stephane...@yahoo.fr> wrote:
>until { command exec 3> "$file$ext"; } 2> /dev/null; do
Good, thanks a lot. It does the trick.
BTW, why cann't I use the following one:
{until command exec 3> "$file$ext"; } 2> /dev/null; do
Thanks again.
Stephane CHAZELAS
> On Sun, 7 Feb 2010 09:45:21 +0000 (UTC), Stephane CHAZELAS
> <stephane...@yahoo.fr> wrote:
>
>>until { command exec 3> "$file$ext"; } 2> /dev/null; do
>
> Good, thanks a lot. It does the trick.
>
> BTW, why cann't I use the following one:
>
> {until command exec 3> "$file$ext"; } 2> /dev/null; do
First, because, to be recognised as such the "{" keyword must be
followed by a blank, and then, the until is part of a "until,
do, done" sequence. {...} is group of commands, the
until,do,done would be unfinished in the {...}.
You could do:
{
until ...; do
wget...
done
} 2> /dev/null
or simply:
until ...; do
wget...
done 2> /dev/null
but then, the stderr of wget would also be redirected to
/dev/null, which is probably not what you want.
--
Stï¿œphane
Hongyi Zhao
Thanks a lot, I've got it.
Hongyi Zhao
I've another issue on the above code:
When I run it for the first time, I'll obtain a file with the name
myfile. My issue is: the initial value of ext should be _.0_, so I
think the above script should name the output as myfile.0. Any hints
on this?
Thanks again.
Stephane CHAZELAS
> On Thu, 4 Feb 2010 12:18:29 +0000 (UTC), Stephane CHAZELAS
> <stephane...@yahoo.fr> wrote:
>
>>url=http://www.google.com/
>>file=myfile
>>(
>> set -C
>> ext= n=0
>> until command exec 3> "$file$ext"; do
>> ext=.$((++n))
>> done
>> exec curl "$url" >&3
>>)
>
> I've another issue on the above code:
>
> When I run it for the first time, I'll obtain a file with the name
> myfile. My issue is: the initial value of ext should be _.0_, so I
> think the above script should name the output as myfile.0. Any hints
> on this?
It shouldn't be that hard to figure it out by yourself.
url=http://www.google.com/
file=myfile
(
set -C
n=0
until { command exec 3> "$file.$n"; } 2> /dev/null; do
n=$(($n + 1))
done
exec curl "$url" >&3
)
--
Stï¿œphane
Hongyi Zhao
<stephane...@yahoo.fr> wrote:
>>>url=http://www.google.com/
>>>file=myfile
>>>(
>>> set -C
>>> ext= n=0
>>> until command exec 3> "$file$ext"; do
>>> ext=.$((++n))
>>> done
>>> exec curl "$url" >&3
>>>)
>>
>> I've another issue on the above code:
>>
>> When I run it for the first time, I'll obtain a file with the name
>> myfile. My issue is: the initial value of ext should be _.0_, so I
>> think the above script should name the output as myfile.0. Any hints
>> on this?
>
>
>It shouldn't be that hard to figure it out by yourself.
My poor English. In fact, I want to know why your original code will
name the first downloaded file as myfile instead of myfile.0.
According to my understanding, it should also name the first
downloaded file as myfile.0 just as the below code does. Any hints?
>
>url=http://www.google.com/
>file=myfile
>(
> set -C
> n=0
> until { command exec 3> "$file.$n"; } 2> /dev/null; do
> n=$(($n + 1))
> done
> exec curl "$url" >&3
>)
Regards.
Stephane CHAZELAS
[...]
>>>>url=http://www.google.com/
>>>>file=myfile
>>>>(
>>>> set -C
>>>> ext= n=0
>>>> until command exec 3> "$file$ext"; do
>>>> ext=.$((++n))
>>>> done
>>>> exec curl "$url" >&3
>>>>)
> My poor English. In fact, I want to know why your original code will
> name the first downloaded file as myfile instead of myfile.0.
> According to my understanding, it should also name the first
> downloaded file as myfile.0 just as the below code does. Any hints?
You could try to add a "set -x" at the top, to see what happens.
--
Stï¿œphane
Hongyi Zhao
<stephane...@yahoo.fr> wrote:
>You could try to add a "set -x" at the top, to see what happens.
Thanks again, I've seen.