Tool for uploading files (FTP)

[A.D. Fundum]

What's a recommended tool to upload files to an ISP webserver? The
number of files is between 600 and 1000, the total size is about 5 MB.
All file name masks of files to be uploaded or updated are known. FTP
script? NetDrive?

The script which creates or updates the is using ORexx, because of the
newer Date() BIF, and calls EXEs. Using a "diff" isn't required,
because that could also upload a few other files (again with known
file masks) for local use. Albeit a "diff" may be needed in the
future. The local files are stored in an organized way so the number
of file masks is limited, even if the upload utility wouldn't support
subdirectories.


--

[tho...@antispam.ham]

"A.D. Fundum" <what...@neverm.ind> writes:

> What's a recommended tool to upload files to an ISP webserver? The
> number of files is between 600 and 1000, the total size is about 5 MB.
> All file name masks of files to be uploaded or updated are known. FTP
> script? NetDrive?

Are the masks unique? If so, use the "prompt" command on FTP to
turn off interactive (if already on; it's a toggle), and then use
the "mput" command:

ftp> prompt
ftp> mput *mask*

[Marcel Müller]

On 25.09.12 23.07, A.D. Fundum wrote:
> The script which creates or updates the is using ORexx, because of the
> newer Date() BIF, and calls EXEs. Using a "diff" isn't required,
> because that could also upload a few other files (again with known
> file masks) for local use. Albeit a "diff" may be needed in the
> future. The local files are stored in an organized way so the number
> of file masks is limited, even if the upload utility wouldn't support
> subdirectories.

I use a Perl script for this purpose. It performs delta uploads
(preferable if you have limited upstream bandwidth and/or large files).

http://www.maazl.de/project/misc/dsync.html
Example:
call perl -S dsync.pl -y2 -s -i -d "-x*.svn/*" "-x*/src/*"
yourlocalhompagedir "ftp://user:pass...@yourprovider.com/targetpath/"
Most likely you want to adjust the exclude patterns.

There is also a REXX version, but it does not support ftp, since rxftp
is incomplete. For testing you might start without -d and -y2 to see
what is going to be done without making any changes.


Marcel

[A.D. Fundum]

> I use a Perl script for this purpose.

Just wondering, before having looked at it: does it upload 600 (1*600)
files in one happy go, or does it upload 600*1 file?

Uploading isn't critical, but it would be nice if it would fail just
once, due to e.g. a problem with the FTP server, instead of 600 times.

> It performs delta uploads

That's a nice-to-have. Both ways? It's possible that uploaded files
became obsolete, and are skipped/ missing in the new situation (delete
* at the server's side would work too).

> Most likely you want to adjust the exclude patterns.

Yes, no problem. Both excluding (e.g. *.EXE, *.LOG) and including
(*.HTM,*.PNG) is fine.


--

[A.D. Fundum]

> Are the masks unique?

Yes, and known, and fixed (that may change, but in a new situation
those conditions will still apply).

> ftp> prompt
> ftp> mput *mask*

Thanks, 1995's simplicity.

In the end I'll probably pick the tool which is the fastest in an
average situation. The whole updating process already is in my top-3
of CPU- and disktorturing activities, e.g. it already invokes
gDiagramm.EXE over 600 times to create all thumbnails and graphs.


--

[Steve Wendt]

On 09/26/12 06:33 am, A.D. Fundum wrote:

>> It performs delta uploads
>
> That's a nice-to-have. Both ways? It's possible that uploaded files
> became obsolete, and are skipped/ missing in the new situation (delete
> * at the server's side would work too).

rsync works nicely, if you can use that on the server side.

[A.D. Fundum]

> rsync works nicely, if you can use that on the server side.

It's my ISP's (Warp's former local ibm.net-infrastructure) FTP server,
so it'll essentially be a private website without an ability to
control the server.

In theory I can reconvert EXEs to Rexx and use Rexx2Nrx to end up with
Java creating the files, but that's about the highest level of control
with a server-side solution. And perhaps the slowest way to create the
files, which still have to be downloaded then. IRL it are local files
for private use, to be made available to the public('s private use).
Having an "own" server isn't my intention.


--

[Shmuel Metz]

In <o8uYFJ3iqTdG-pn2-Ilpws0O9dNH2@localhost>, on 09/25/2012

at 11:07 PM, "A.D. Fundum" <what...@neverm.ind> said:

>The script which creates or updates the is using ORexx,

Then why not use RxFTP?

--
Shmuel (Seymour J.) Metz, SysProg and JOAT <http://patriot.net/~shmuel>

Unsolicited bulk E-mail subject to legal action. I reserve the
right to publicly post or ridicule any abusive E-mail. Reply to
domain Patriot dot net user shmuel+news to contact me. Do not
reply to spam...@library.lspace.org

[Marcel Müller]

On 26.09.12 15.33, A.D. Fundum wrote:
> Just wondering, before having looked at it: does it upload 600 (1*600)
> files in one happy go, or does it upload 600*1 file?

The ftp connection is kept open.
First, the source and the target directory structure is scanned.
Secondly, the differences are calculated.
Thirdly, the configured actions are taken on the differences.

> Uploading isn't critical, but it would be nice if it would fail just
> once, due to e.g. a problem with the FTP server, instead of 600 times.

You will get an error that the ftp connection cannot be established.

> > It performs delta uploads
>
> That's a nice-to-have. Both ways?

Yes, if you want to do that. But there are ambiguities, e.g. whether a
file is new in tree 1 or deleted in tree 2.

> It's possible that uploaded files
> became obsolete, and are skipped/ missing in the new situation (delete
> * at the server's side would work too).

If you synchronize in one direction only then there are no ambiguities.
Delete operations are transported too.

> > Most likely you want to adjust the exclude patterns.
>
> Yes, no problem. Both excluding (e.g. *.EXE, *.LOG) and including
> (*.HTM,*.PNG) is fine.

Hmm, AFAIR there is no option to have more than one /include/ pattern.


Marcel

[Dave Saville]

On Fri, 28 Sep 2012 19:23:15 UTC, Marcel M〕ler
<news.5...@spamgourmet.org> wrote:

<snip>

> Yes, if you want to do that. But there are ambiguities, e.g. whether a
> file is new in tree 1 or deleted in tree 2.
>

Not a problem. The script produces a file that reflects the state as
of the last sync. Thus:

In tree 1 not in tree 2 and not in master file - New.
In tree 1 not in tree 2 and in master file - Deleted.

1 & 2 are interchangable. ie it works both ways.

HTH
--
Regards
Dave Saville

[A.D. Fundum]

>> The script ... is using ORexx,

FTR: essentially it's a regular CMD batch file. The only remaining use
of Rexx is the newer Date() BIF, because dates are used as arguments
w.r.t. the EXEs it's launching. I actually could avoid the use of
Rexx.

PULL lastworkingdayofthisweek /* 20120928 */
lastweekguess=Date('B',lastworkingdayofthisweek,'S')-7
lastweekguess=Date('S',lastweekguess,'B')
CALL CharOut '','Last week (<'||lastweekguess||'>)? '
PULL lastweek
IF lastweek='' THEN lastweek=lastweekguess

So typically I just have to press <Enter> to select a date.

> Then why not use RxFTP?

Yet another possibily. I've just looked at it, and its functions seem
to require re-invented wheels regarding management of the files.

Most, if not all, files are updates. Or new. So deleting *MASK* is
fine. Apparently FtpDelete() deletes one file, so a remote driectory
has to be PARSE'd and FtpDelete() has to be used over 600 times.

Another option is to not delete to be updated files. So you just have
to delete a few orphans (file exists on the server, but no longer
locally, e.g. due to an implied rename). This also requires a PARSE'd
remote directory, in order to delete a few files.

Regarding the implied rename: I've seen FtpRename(), but that has no
use in this case. Unless FtpRename() also changes the title of a graph
by modifying the file. The underlying action or event is a rename, but
it's processed as a delete&recreate.

FtpPut() also seems to work with individual files. SysFileTree() would
help, but "mput" is easier (albeit it doesn't preserve the case, the
server support case-sensitivity).

Perhaps RxFTP() still is a candidate, for example when FTP.EXE doesn't
support scripts.


--

[A.D. Fundum]

> The ftp connection is kept open.

Thanks.

>>> It performs delta uploads

>> That's a nice-to-have. Both ways?

> Yes, if you want to do that. But there are ambiguities,
> e.g. whether a file is new in tree 1 or deleted in tree 2.

I think I agree with Dave Saville, and the situation isn't special. If
tree 1 is local and 2 is remote, I've to deal with new(er) files in
tree 1 and possible orphans in tree 2. So tree 2 is a recent copy of
tree 1, without tree 2 having to containing files that used to be part
of tree 1.

> AFAIR there is no option to have more than one /include/ pattern.

Including more than one pattern seems to make sense (e.g.
holiday*.html and pict*.jpg), albeit I can use more than one "exclude"
(e.g. *.C, *.OBJ, *.EXE, *.LOG, *.GIF, *.BCK) as well.

Calling it twice is also an option, so long as "XFER2 PICT*.JPG"
doesn't delete the mismatching files "HOLIDAY*.HTML", transfered with
an earlier "XFER1 HOLIDAY*.HTML".


--

[A.D. Fundum]

> http://www.maazl.de/project/misc/dsync.html

"ZIP file ... guide: Release 0.21" should read 0.3 instead of 0.21

"wired things": ignore use the rule "I before E except after C"
(http://www.youtube.com/watch?v=duqlZXiIZqA) ;-)


--

[ba...@baden.nu]

Hi:

I didn't examine the whole thread in detail, but I
previously experienced similar ftp problems until I switched
to Fireftp which runs on Mozilla based browsers. If you can
adapt a solution to this, IMHO, you will be best off.

What I do for changed/new files, is copy them to a new
directory, and then use Fireftp to upload that whole
directory to the server.

thanks,
Baden

[A.D. Fundum]

> I didn't examine the whole thread in detail

There are a few disadvantages. One disadvantage is that the whole
process may take a while (on the OS/2 scale), so an unattended
solution has a slight advantage. Nevertheless I'll give it a try too.
I haven't picked a final solution due to yet another redesigned
website, which screws up the data used to produce the files. BTW, I
also found another (Russian?) dir synchronizer.

> I switched to Fireftp which runs on Mozilla based browsers.

> What I do for changed/new files, is copy them to a new
> directory, and then use Fireftp to upload that whole
> directory to the server.

What about its synchronization option? I hope FF's overhead is a
showstopper, just because that would mean I've found a rather fast
solution. (Deleting and) copying *.* from and to a single directory is
what I've done so far, manually, with NetDrive, to test the website.
Such a strategy works, but for one it copies more files than needed.
Not an enormous number of files, but it's possible to optimize the
time it takes.


--

[ba...@baden.nu]

On Oct 5, 1:49 am, "A.D. Fundum" <what.e...@neverm.ind> wrote:
> > I didn't examine the whole thread in detail
>
> There are a few disadvantages. One disadvantage is that the whole
> process may take a while (on the OS/2 scale), so an unattended

The two big advantages with Fireftp I found were that it
didn't corrupt file or path names (like mput does), and that
it's connectivity is very robust, as it will attempt to
reconnect all night when there are problems. This is
valuable, as I usually never have to check for completion.

> solution has a slight advantage. Nevertheless I'll give it a try too.
> I haven't picked a final solution due to yet another redesigned
> website, which screws up the data used to produce the files. BTW, I
> also found another (Russian?) dir synchronizer.
>
> > I switched to Fireftp which runs on Mozilla based browsers.
> > What I do for changed/new files, is copy them to a new
> > directory, and then use Fireftp to upload that whole
> > directory to the server.
>
> What about its synchronization option? I hope FF's overhead is a

I just checked the "options" menu, and there is a lot more
available on the latest version than initially. I think I
tried once the synchronisation option, but there may have
been problems with deeper directory paths.

> showstopper, just because that would mean I've found a rather fast
> solution. (Deleting and) copying *.* from and to a single directory is
> what I've done so far, manually, with NetDrive, to test the website.
> Such a strategy works, but for one it copies more files than needed.
> Not an enormous number of files, but it's possible to optimize the
> time it takes.

I found that transferring many smaller files takes an
inordinate amount of time compared to large files, so it
depends what the upload contains.

thanks,
Baden