Hello,

I have been using the BOTS program for a while but I am not going to
pretend I have full knowledge so please bear with me.

When I first looked at BOTS, some of the requirements for extra
functionality that I had were coded as enhancments to the core
software.  However, recently I have been able to rework almost
everything into external scripts and leave the core product untouched.

The one thing I have been wrestling with that BOTS appears to be
missing is the ability to FTP an output file with a temporary name,
and then have it renamed at the end of the successful file transfer.

This could be a file that is sent as "ORDERS000123.xml.tmp" and then
renamed to "ORDERS000123.xml" at the completion of transfer as
discussed in previous posts.

As I understand it, I could take control of the FTP communication
channel (by overriding the main and connect functions) and take full
control - but it appears that this channel is executed one file at a
time.   I don't want to be connecting and disconnecting for every file
sent.

So my goals here are:

a) I don't want to modify the core product if at all possible.
b) Want FTP out channel to support a rename function.

My questions are:

a) Can I achieve this functionality right now?  Is there something I
am missing?
b) Have you already put this in a future version?  Perhaps it could be
backported into my local scripts.

I had previously programmed this as a user exit function in
communication.py but I really want to avoid custom modifications.   We
have had such significant problems over the years with EDI transfers
because of this issue that it just seems like an essential part of the
FTP scripting imo.

Best Regards,
Rohan.

hi Rohan,

On 02/07/2012 11:34 PM, Rohan wrote:

yes, I have seen this sort of ftp communication.
I not sure what you have in mind here.
bots will transfers all edi file in one session.

send me the user user function, if it fits I will use it in bots.

henk-jan

Hi Henk-Jan,

Thanks for your swift response.

I simply had these two lines in communication.py in the FTP
outcommunicate() after fromfile.close()

if self.userscript and hasattr(self.userscript,'rename'):
    tofilename =
botslib.runscript(self.userscript,self.scriptname,'rename',
channeldict=self.channeldict,filename=tofilename,ta=ta_from,session=self.se ssion)

The idea being that in my channel script:

def filename(channeldict,ta,filename):
    return filename + '.tmp'

def rename(channeldict,ta,filename,session):
    assert isinstance(session, ftplib.FTP)

    # filename in format "path\filename.xml.tmp"  - remove the tmp
extension
    tofilename, extension = os.path.splitext(filename)

    # if for some reason our file does not have a .tmp extension just
return
    if unicode.lower(extension) != '.tmp':
        return filename

    session.rename(filename, tofilename)
    return tofilename

I feel it would be better though, if the session.rename() function
remained in communication.py.

I can't think of any reason why you wouldn't want to send it without
an extension.  There is no way to lock the destination file during
transfer and there is no way to know on the remote end that the
transfer has completed.

A self-contained outcommunicate() function would have something like:

try:
    ta_from = botslib.OldTransaction(row['idta'])
    ta_to = ta_from.copyta(status=EXTERNOUT)
    sendfilename = row['sendfilename']
    unique = str(botslib.unique(self.channeldict['idchannel']))
#create unique part for filename
    if sendfilename and self.channeldict['filename']:
        tofilename =
self.channeldict['filename'].replace('*',sendfilename) #filename is
filename in channel where '*' is replaced by idta
    elif self.channeldict['filename']:
        tofilename = self.channeldict['filename'].replace('*',unique)
#filename is filename in channel where '*' is replaced by idta
    else:
        tofilename = unique
    if self.userscript and hasattr(self.userscript,'filename'):
        tofilename =
botslib.runscript(self.userscript,self.scriptname,'filename',channeldict=se lf.channeldict,filename=tofilename,ta=ta_from)
    tofilenamewithtmpext = tofilename +
'.TMP'
######  Create file with .TMP extension
    if self.channeldict['ftpbinary']:

botslib.checkcodeciscompatible(row['charset'],self.channeldict['charset'])
        fromfile = botslib.opendata(row['filename'], 'rb')
        self.session.storbinary(mode + tofilenamewithtmpext,
fromfile)                                      ######  Transfer file
with .TMP extension
    else:
        #~ self.channeldict['charset'] = 'us-ascii'

botslib.checkcodeciscompatible(row['charset'],self.channeldict['charset'])
        fromfile = botslib.opendata(row['filename'], 'r')
        self.session.storlines(mode + tofilenamewithtmpext,
fromfile)                                        ######  Transfer file
with .TMP extension
    fromfile.close()
    self.session.rename(tofilenamewithtmpext,
tofilename)
######  Rename the file back to its original name
except:
    txt=botslib.txtexc()

ta_to.update(statust=ERROR,errortext=txt,filename='ftp:/'+posixpath.join(se lf.dirpath,tofilename))
else:
    ta_from.update(statust=DONE)

ta_to.update(statust=DONE,filename='ftp:/'+posixpath.join(self.dirpath,tofi lename))

This section above is untested code.  If you like I can modify and run
a few test to make sure although you would probably want to test
yourself anyway.

Regards,
Rohan.

look ok to me.
bots has to deal with a lot of ftp-servers.
some are really weird (fetch a file, delete it, look again if there is there is still a file with the same name, fetch it again, delete it, look again, etc etc
sort of a file queue ;-))

your enhancement is nice!
but I will not use this in the main ftp flow (but will add it as user scripting), as it might break other server implementations.
hope you understand that I try to be very cautious here.

thank you!

henk-jan

You could always add a switch - "Add temp extension during transfer?"
in the FTP options ;-)

Anyway, to be totally clear, can I just confirm that you are adding:

if self.userscript and hasattr(self.userscript,'rename'):
    tofilename =
botslib.runscript(self.userscript,self.scriptname,'rename',
channeldict=self.channeldict,filename=tofilename,ta=ta_from,session=self.se ssion)

If so, is it worth considering the function name?  It's FTP specific
and you could really do anything at that point.  So rather than
"rename" it could be called "postfilewrite" or something of that
nature?

Cheers,
Rohan.

On 02/08/2012 01:47 AM, Rohan wrote:

the server apparently has implemented the file naming convention that *.tmp'-files are treated differently.
this seems to be very specific to this ftp-server.
if most or many ftp-servers would have such a file name convention, that would be great (convince me of that ;-)).
(I guess reason for all this is to have avoid reading of a file that is not completely written. There seems to be many conventions & solutions for this, all implementation dependent. Some ftp-servers
do not need such a mechanism at all, they know when a file is completly written)
I will check this out and let you know.

henk-jan

Hi Rohan, Henk-jan,

I agree that this would be nice to have built into Bots as a user
scriptable option.
I have just started setting up a new partner, and they recommend the
same approach.

To quote directly from their document...

6.      Basic Requirements
The file is generated by the customer and sent to Schenker via a
mutually-agreed transmission method.
File Transfer Protocol (FTP) is the preferred transmission method. The
following process is recommended during the FTP.

Step 1: Send the file with a _tmp while transmitting (e.g) say file
custname_00002.txt will be transmitted as custname_00002.txt_tmp
Step 2: Rename the file to the actual file name, without _tmp (e.g.)
Rename custname_00002.txt_tmp to custname_00002.txt

This is to ensure that Customer process will always get the complete
data file and not attempt to load while it is being partially
transmitted.

hi Mike,

yes I agree.
thanks for reminding me, I had not added this to my list.

henk-jan

On 03/09/2012 06:28 AM, BikeMike wrote:

I have developed a method to achieve this without modifying bots.

Firstly I have a set of routescript functions (routescript.py) which
includes open_ftp() and close_ftp(). These are based on code in
communications.py and allow you to open an ftp session by just
specifying the channel. Then you can code other ftp commands to do
whatever you need. I also use this method to send remote commands to our
AS/400 system, delete files from servers, and monitor the number of
files queued on various servers, etc.

I create a routescript (Imports_Schenker.py) for the route, and in
postoutcommunication exit point, I establish a new ftp session and do
the renaming. You could just change the extension, or move them to
another folder, whatever is required.

An exit point built into bots might still be better, as you could use
the existing ftp session.

Kind Regards,
Mike

routescript.py 4K Download

Imports_Schenker.py 1K Download