FTP Rename on send

[Rohan]

Hello,

I have been using the BOTS program for a while but I am not going to
pretend I have full knowledge so please bear with me.

When I first looked at BOTS, some of the requirements for extra
functionality that I had were coded as enhancments to the core
software. However, recently I have been able to rework almost
everything into external scripts and leave the core product untouched.

The one thing I have been wrestling with that BOTS appears to be
missing is the ability to FTP an output file with a temporary name,
and then have it renamed at the end of the successful file transfer.

This could be a file that is sent as "ORDERS000123.xml.tmp" and then
renamed to "ORDERS000123.xml" at the completion of transfer as
discussed in previous posts.

As I understand it, I could take control of the FTP communication
channel (by overriding the main and connect functions) and take full
control - but it appears that this channel is executed one file at a
time. I don't want to be connecting and disconnecting for every file
sent.

So my goals here are:

a) I don't want to modify the core product if at all possible.
b) Want FTP out channel to support a rename function.

My questions are:

a) Can I achieve this functionality right now? Is there something I
am missing?
b) Have you already put this in a future version? Perhaps it could be
backported into my local scripts.

I had previously programmed this as a user exit function in
communication.py but I really want to avoid custom modifications. We
have had such significant problems over the years with EDI transfers
because of this issue that it just seems like an essential part of the
FTP scripting imo.

Best Regards,
Rohan.

[hjeb...@gmail.com]

hi Rohan,

On 02/07/2012 11:34 PM, Rohan wrote:
> Hello,
>
> I have been using the BOTS program for a while but I am not going to
> pretend I have full knowledge so please bear with me.
>
> When I first looked at BOTS, some of the requirements for extra
> functionality that I had were coded as enhancments to the core
> software. However, recently I have been able to rework almost
> everything into external scripts and leave the core product untouched.
>
> The one thing I have been wrestling with that BOTS appears to be
> missing is the ability to FTP an output file with a temporary name,
> and then have it renamed at the end of the successful file transfer.
>
> This could be a file that is sent as "ORDERS000123.xml.tmp" and then
> renamed to "ORDERS000123.xml" at the completion of transfer as
> discussed in previous posts.

yes, I have seen this sort of ftp communication.

> As I understand it, I could take control of the FTP communication
> channel (by overriding the main and connect functions) and take full
> control - but it appears that this channel is executed one file at a
> time. I don't want to be connecting and disconnecting for every file
> sent.

I not sure what you have in mind here.
bots will transfers all edi file in one session.

> So my goals here are:
>
> a) I don't want to modify the core product if at all possible.
> b) Want FTP out channel to support a rename function.
>
> My questions are:
>
> a) Can I achieve this functionality right now? Is there something I
> am missing?
> b) Have you already put this in a future version? Perhaps it could be
> backported into my local scripts.
>
> I had previously programmed this as a user exit function in
> communication.py but I really want to avoid custom modifications.

send me the user user function, if it fits I will use it in bots.

henk-jan

[Rohan]

Hi Henk-Jan,

Thanks for your swift response.

I simply had these two lines in communication.py in the FTP
outcommunicate() after fromfile.close()

if self.userscript and hasattr(self.userscript,'rename'):
tofilename =
botslib.runscript(self.userscript,self.scriptname,'rename',
channeldict=self.channeldict,filename=tofilename,ta=ta_from,session=self.session)

The idea being that in my channel script:

def filename(channeldict,ta,filename):
return filename + '.tmp'

def rename(channeldict,ta,filename,session):
assert isinstance(session, ftplib.FTP)

# filename in format "path\filename.xml.tmp" - remove the tmp
extension
tofilename, extension = os.path.splitext(filename)

# if for some reason our file does not have a .tmp extension just
return
if unicode.lower(extension) != '.tmp':
return filename

session.rename(filename, tofilename)
return tofilename


I feel it would be better though, if the session.rename() function
remained in communication.py.

I can't think of any reason why you wouldn't want to send it without
an extension. There is no way to lock the destination file during
transfer and there is no way to know on the remote end that the
transfer has completed.

A self-contained outcommunicate() function would have something like:

try:
ta_from = botslib.OldTransaction(row['idta'])
ta_to = ta_from.copyta(status=EXTERNOUT)
sendfilename = row['sendfilename']
unique = str(botslib.unique(self.channeldict['idchannel']))
#create unique part for filename
if sendfilename and self.channeldict['filename']:
tofilename =
self.channeldict['filename'].replace('*',sendfilename) #filename is
filename in channel where '*' is replaced by idta
elif self.channeldict['filename']:
tofilename = self.channeldict['filename'].replace('*',unique)
#filename is filename in channel where '*' is replaced by idta
else:
tofilename = unique
if self.userscript and hasattr(self.userscript,'filename'):
tofilename =
botslib.runscript(self.userscript,self.scriptname,'filename',channeldict=self.channeldict,filename=tofilename,ta=ta_from)
tofilenamewithtmpext = tofilename +
'.TMP'
###### Create file with .TMP extension
if self.channeldict['ftpbinary']:

botslib.checkcodeciscompatible(row['charset'],self.channeldict['charset'])
fromfile = botslib.opendata(row['filename'], 'rb')
self.session.storbinary(mode + tofilenamewithtmpext,
fromfile) ###### Transfer file
with .TMP extension
else:
#~ self.channeldict['charset'] = 'us-ascii'

botslib.checkcodeciscompatible(row['charset'],self.channeldict['charset'])
fromfile = botslib.opendata(row['filename'], 'r')
self.session.storlines(mode + tofilenamewithtmpext,
fromfile) ###### Transfer file
with .TMP extension
fromfile.close()
self.session.rename(tofilenamewithtmpext,
tofilename)
###### Rename the file back to its original name
except:
txt=botslib.txtexc()

ta_to.update(statust=ERROR,errortext=txt,filename='ftp:/'+posixpath.join(self.dirpath,tofilename))
else:
ta_from.update(statust=DONE)

ta_to.update(statust=DONE,filename='ftp:/'+posixpath.join(self.dirpath,tofilename))


This section above is untested code. If you like I can modify and run
a few test to make sure although you would probably want to test
yourself anyway.

Regards,
Rohan.

[hjeb...@gmail.com]

look ok to me.

> I can't think of any reason why you wouldn't want to send it without
> an extension. There is no way to lock the destination file during
> transfer and there is no way to know on the remote end that the
> transfer has completed.

bots has to deal with a lot of ftp-servers.
some are really weird (fetch a file, delete it, look again if there is there is still a file with the same name, fetch it again, delete it, look again, etc etc
sort of a file queue ;-))

your enhancement is nice!
but I will not use this in the main ftp flow (but will add it as user scripting), as it might break other server implementations.
hope you understand that I try to be very cautious here.

thank you!

henk-jan

[Rohan]

You could always add a switch - "Add temp extension during transfer?"
in the FTP options ;-)

Anyway, to be totally clear, can I just confirm that you are adding:

if self.userscript and hasattr(self.userscript,'rename'):
tofilename =
botslib.runscript(self.userscript,self.scriptname,'rename',
channeldict=self.channeldict,filename=tofilename,ta=ta_from,session=self.session)

If so, is it worth considering the function name? It's FTP specific
and you could really do anything at that point. So rather than
"rename" it could be called "postfilewrite" or something of that
nature?

Cheers,
Rohan.

[hjeb...@gmail.com]

On 02/08/2012 01:47 AM, Rohan wrote:
> You could always add a switch - "Add temp extension during transfer?"
> in the FTP options ;-)

the server apparently has implemented the file naming convention that *.tmp'-files are treated differently.
this seems to be very specific to this ftp-server.
if most or many ftp-servers would have such a file name convention, that would be great (convince me of that ;-)).
(I guess reason for all this is to have avoid reading of a file that is not completely written. There seems to be many conventions & solutions for this, all implementation dependent. Some ftp-servers
do not need such a mechanism at all, they know when a file is completly written)

> Anyway, to be totally clear, can I just confirm that you are adding:
>
> if self.userscript and hasattr(self.userscript,'rename'):
> tofilename =
> botslib.runscript(self.userscript,self.scriptname,'rename',
> channeldict=self.channeldict,filename=tofilename,ta=ta_from,session=self.session)

I will check this out and let you know.

henk-jan

[BikeMike]

Hi Rohan, Henk-jan,

I agree that this would be nice to have built into Bots as a user
scriptable option.
I have just started setting up a new partner, and they recommend the
same approach.

To quote directly from their document...

6. Basic Requirements
The file is generated by the customer and sent to Schenker via a
mutually-agreed transmission method.
File Transfer Protocol (FTP) is the preferred transmission method. The
following process is recommended during the FTP.

Step 1: Send the file with a _tmp while transmitting (e.g) say file
custname_00002.txt will be transmitted as custname_00002.txt_tmp
Step 2: Rename the file to the actual file name, without _tmp (e.g.)
Rename custname_00002.txt_tmp to custname_00002.txt

This is to ensure that Customer process will always get the complete
data file and not attempt to load while it is being partially
transmitted.

[hjeb...@gmail.com]

hi Mike,

yes I agree.
thanks for reminding me, I had not added this to my list.

henk-jan

[BikeMike]

I have developed a method to achieve this without modifying bots.

Firstly I have a set of routescript functions (routescript.py) which
includes open_ftp() and close_ftp(). These are based on code in
communications.py and allow you to open an ftp session by just
specifying the channel. Then you can code other ftp commands to do
whatever you need. I also use this method to send remote commands to our
AS/400 system, delete files from servers, and monitor the number of
files queued on various servers, etc.

I create a routescript (Imports_Schenker.py) for the route, and in
postoutcommunication exit point, I establish a new ftp session and do
the renaming. You could just change the extension, or move them to
another folder, whatever is required.

An exit point built into bots might still be better, as you could use
the existing ftp session.

Kind Regards,
Mike

[BikeMike]

Hi Henk-jan,
Do you still plan to add an exit point for this? I have another route that I need to put files in a temp folder, then move (rename) them to the main folder after upload (due to the receiving end sometimes getting partial files). My solution above will work, but you are opening a new session to the ftp server, so not the most efficient way.

Kind Regards,
Mike

[henk-jan ebbers]

hi Mike,

certainly don't mind doing that. just let me know.
did not implement this, as I did not get any response ;-))

in a recent implementation I did this:

class file(communication.file):
def disconnect(self):
filelist = [filename for filename in glob.iglob(self.channeldict['path']) if os.path.isfile(filename)]
for filename in filelist:
os.rename(filename, os.path.splitext(filename)[0])
files are written with extension .tmp; this script remove the extension.
the rename is 'atomic': this avoid the 'reading while still writing' problem.

kind regards,
henk-jan

> --
> You received this message because you are subscribed to the Google Groups "Bots Open Source EDI Translator" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to botsmail+u...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.

[Jan Linders]

I think its good practice to use either a tmp directory or a temp filename or extension to overcome the issue of the system picking up files that are in the process of being transfered. I've come across several operational issues where this turned out to be the root cause.

Op 23 aug. 2013 14:12 schreef "henk-jan ebbers" <eppye...@gmail.com> het volgende:

To unsubscribe from this group and stop receiving emails from it, send an email to botsmail+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "Bots Open Source EDI Translator" group.

To unsubscribe from this group and stop receiving emails from it, send an email to botsmail+unsubscribe@googlegroups.com.

[BikeMike]

Yes... I just tried this and it does what I need. No need for an exit point.
The same code can be used to change the file extension instead.

In my communicationscript

import bots.communication as communication
import bots.botslib as botslib
import bots.botsglobal as botsglobal

class ftp(communication.ftp):
    def disconnect(self):

        # files are in a temporary directory. Move them to final destination (rename)
        for f in self.session.nlst():
            self.session.rename(f,'../TEST2/%s' %f)

        try:
            self.session.quit()
        except:
            self.session.close()
        botslib.settimeout(botsglobal.ini.getint('settings','globaltimeout',10))

Kind Regards,
Mike

[BOTJE]

Wouldn't it be great if this can be driven by parameters from the bots.ini to make it flexible for all bots users

without the use of communicationscripts.

Op vrijdag 23 augustus 2013 16:04:22 UTC+2 schreef BikeMike:

[BikeMike]

On Saturday, August 24, 2013 1:54:56 AM UTC+9:30, BOTJE wrote:

Wouldn't it be great if this can be driven by parameters from the bots.ini to make it flexible for all bots users

without the use of communicationscripts.

I guess it would be possible, but you would need to take into account all requirements. Some people want to change the file extension. In my case that is no good, I need to use another directory and move the completed files. Maybe some need a specific extension, or a specific subdirectory name, or need a directory outside of the upload directory...

Anyway I made a slight change to my communicationscript. Now the Bots channel specifies the "final" destination. A subdirectory is added to filenames, then the files are moved to parent directory in disconnect. You need to first create tempupload on the server manually.

import bots.communication as communication
import bots.botslib as botslib
import bots.botsglobal as botsglobal

def filename(channeldict,ta,filename):
    return 'tempupload/' + filename

class ftp(communication.ftp):
    def disconnect(self):

        # move files from tempupload subdirectory to parent directory (rename)
        self.session.cwd('tempupload')
        for f in self.session.nlst():
            self.session.rename(f,'../%s' %f)

        try:
            self.session.quit()
        except:
            self.session.close()
        botslib.settimeout(botsglobal.ini.getint('settings','globaltimeout',10))

Kind Regards,
Mike

[BikeMike]

Added to the wiki
http://code.google.com/p/bots/wiki/CommunicationScriptsSubclass

[BOTJE]

Thanks Mike.

Do you know if the same can be applied to SFTP ?

Op dinsdag 7 februari 2012 23:34:53 UTC+1 schreef Rohan:

[Mike Griffin]

Yes it should work in the same way for sftp. Just copy the sftp disconnect code from communication.py and modify it.

[Eppye Bots]

hi Mike,

thinking about this.

what if there is field 'filename' (like now) and a field 'tmp filename'.

first use the tmp name, that a rename (this is users responsibility).

think that also works for your use case.

kind regards,

henk-jan

On Sat, Aug 24, 2013 at 12:07 PM, Mike Griffin <mjg...@gmail.com> wrote:

Yes it should work in the same way for sftp. Just copy the sftp disconnect code from communication.py and modify it.

--

You received this message because you are subscribed to the Google Groups "Bots Open Source EDI Translator" group.

To unsubscribe from this group and stop receiving emails from it, send an email to botsmail+u...@googlegroups.com.

[BikeMike]

On Tuesday, August 27, 2013 4:03:18 AM UTC+9:30, eppye wrote:

hi Mike,

thinking about this.

what if there is field 'filename' (like now) and a field 'tmp filename'.

first use the tmp name, that a rename (this is users responsibility).

think that also works for your use case.

kind regards,

henk-jan

Do you mean that bots (communication.py) would handle the rename if both fields were filled in? I think this would solve most issues, tmp filename can be a change of extension, or a change of directory.

But then... to handle all the filename possibilities, overwrite or append, etc... could get complex. Also should this work with just ftp, or other communication (file?)

I think it could be a simple option. [temp extension or directory]. Here you enter for example ".tmp" to add a temporary extension, or "bots-temp" to use a temporary sub-directory. ie. whether the value starts with a dot determines the function.

Example 1.
Path: /edi/in
Filename: *.txt
Temp extension or directory: .tmp

Result - all files are uploaded to /edi/in with a .tmp extension used in place of .txt, then renamed back to .txt extension.

Example 2.
Path: /edi/in
Filename: *.txt
Temp extension or directory: bots-temp

Result - all files are uploaded to /edi/in/bots-temp with a .txt extension, then renamed to the /edi/in directory with no change to extension.

For now I am using the communicationscript method now I understand how to do the ftp subclassing :-) It just means I have to create a few more scripts and import the rename code.

Kind Regards,
Mike

[Eppye Bots]

Yes, that is  a better way. I also noticed this would get more complicated than I tought, eg with the '*' being replaced by counter.
Kind regards,
Henk-Jan

[BikeMike]

Perhaps we can use the "lockname" field on channel. I notice this is only used for file type channels.

[BikeMike]

I coded this to test it using lockname. Only implemented for ftp (ftps,ftpis), will do sftp also if this is accepted.

Usage: Lockname can contain an alternate ftp path for uploads (eg. \edi\bots-temp), or an extension to append (eg. .tmp).
After uploads are completed, files are renamed to the output path, or the extension is removed.
The control for which method to use is that extensions start with a dot.

The changes are as follows.
in set_cwd() use the path from lockname for uploads, if configured. Save both paths.

        self.dirpath = self.session.pwd()
        self.dirpath2 = self.session.pwd()
        # lockname can set a temporary directory for uploads
        if self.channeldict['lockname'] and not self.channeldict['lockname'].startswith('.') and self.channeldict['inorout'] == 'out':
            self.dirpath = posixpath.normpath(posixpath.join(self.dirpath,self.channeldict['lockname']))
            if self.channeldict['path']:
                self.dirpath2 = posixpath.normpath(posixpath.join(self.dirpath2,self.channeldict['path']))
        elif self.channeldict['path']:
            self.dirpath = posixpath.normpath(posixpath.join(self.dirpath,self.channeldict['path']))

outcommunicate() adds temporary file extension if configured

                # append temporary file extension
                if self.channeldict['lockname'] and self.channeldict['lockname'].startswith('.'):
                    tofilename += self.channeldict['lockname']

disconnect() does the renaming

        # rename files if lockname was used
        if self.channeldict['lockname'] and self.channeldict['inorout'] == 'out':
            if self.channeldict['lockname'].startswith('.'):
                # remove temporary extension
                l = 0-len(self.channeldict['lockname'])
                for f in self.session.nlst():
                    if f.endswith(self.channeldict['lockname']):
                        self.session.rename(f,f[:l])
                        botsglobal.logger.debug('rename %s to %s',f,f[:l])
            else:
                # move files from temporary directory to path directory
                for f in self.session.nlst():
                    self.session.rename(f,posixpath.join(self.dirpath2,f))
                    botsglobal.logger.debug('rename %s to %s',f,posixpath.join(self.dirpath2,f))

Kind Regards,
Mike

[BikeMike]

I updated the wiki with a more robust solution using subclassing in a communicationscript. Actually I think this could go into the core bots code as-is. There is no need for any additional configuration fields. You just configure your channel with a path that ends in /tmp, or a filename that ends in .tmp and they will be renamed as needed. Other channels will not change.

http://code.google.com/p/bots/wiki/CommunicationScriptsSubclass?ts=1378448292&updated=CommunicationScriptsSubclass#Example_3

Kind Regards,
Mike

[henk-jan ebbers]

yes, but it uses 'file name magic'.

think I liked the solution wirth a extra tmp-file field more.

kind regards,
henk-jan