plannign for release 3.1

[hjebbers]

planning for bots 3.1:
20130601: release version
20130701: definitive version

current list of changes: http://code.google.com/p/bots/issues/list?can=1&q=label%3Aversion3.1

as always, patches and suggestions are welcome!


kind regards,
henk-jan

[Harold DeWayne]

Is BikeMike's csv import to User Code List included in this version or will it still be an plug-in?  I _LIKE_ this one and can see a BIG use for it.

Thanks.
Harold.

[henk-jan ebbers]

hi Harold,

that is exactly what plugins are for, isn't it?

I learned that it is a bad idea to have grammars, routes etc in a default installation.

but it might be a good idea to have it a more up-to-date version of it, and give it a beter place.

kind regards,
henk-jan

On 05/09/2013 04:05 AM, Harold DeWayne wrote:
> Is BikeMike's csv import to User Code List included in this version or will it still be an plug-in? I _LIKE_ this one and can see a BIG use for it.
>
> Thanks.
> Harold.
>
> On Wednesday, May 8, 2013 7:18:05 PM UTC-4, hjebbers wrote:
>
> planning for bots 3.1:
> 20130601: release version
> 20130701: definitive version
>

> current list of changes: http://code.google.com/p/bots/issues/list?can=1&q=label%3Aversion3.1 <http://code.google.com/p/bots/issues/list?can=1&q=label%3Aversion3.1>

>
> as always, patches and suggestions are welcome!
>
>
> kind regards,
> henk-jan
>

> --
> You received this message because you are subscribed to the Google Groups "Bots Open Source EDI Translator" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to botsmail+u...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>

[BikeMike]

Updated issue 224 - view source in GUI
http://code.google.com/p/bots/issues/detail?id=224#c12

[BikeMike]

On Thursday, May 9, 2013 11:35:39 AM UTC+9:30, Harold DeWayne wrote:

Is BikeMike's csv import to User Code List included in this version or will it still be an plug-in?  I _LIKE_ this one and can see a BIG use for it.

How about something like this? You could download or upload individual ccode tables to csv via the GUI.
 

[Harold DeWayne]

That's exactly what I was hoping for. ("download or upload individual ccode tables to csv via the GUI.  ")

Harold.

[henk-jan ebbers]

btw, what I do with this kind of import: separate route, scheduled eg once a day to do 'refresh' of code via import of ERP.

hje

On 05/16/2013 03:50 PM, Harold DeWayne wrote:
> That's exactly what I was hoping for. ("download or upload individual ccode tables to csv via the GUI. ")
>
> Harold.
>
> On Thursday, May 16, 2013 8:19:06 AM UTC-4, BikeMike wrote:
>
> On Thursday, May 9, 2013 11:35:39 AM UTC+9:30, Harold DeWayne wrote:
>
> Is BikeMike's csv import to User Code List included in this version or will it still be an plug-in? I _LIKE_ this one and can see a BIG use for it.
>

> <https://lh3.googleusercontent.com/-kW34GBcE01E/UZTOdplwYvI/AAAAAAAAADs/naKkmKh94So/s1600/ccodetrigger_2013-05-16_21-44-35.png>

>
>
> How about something like this? You could download or upload individual ccode tables to csv via the GUI.
>

[Harold DeWayne]

I would think it could be done on request (a menu item perhaps) for something like a code list or on a schedule in case of something like a price catalog that would be frequently updated.

HD

[henk-jan ebbers]

yes.
my experience: if is is manual, they forget it ;-)))

henk-jan

> > To unsubscribe from this group and stop receiving emails from it, send an email to botsmail+u...@googlegroups.com <javascript:>.
> > For more options, visit https://groups.google.com/groups/opt_out <https://groups.google.com/groups/opt_out>.

[Harold DeWayne]

BTW:  Looks like this product is able to rival anything I've ever used before.  &^)

[Harold DeWayne]

Looks good... as long as there's a way to specify the codelist name (and the file name if different) to update.

HD

On Thursday, May 16, 2013 8:19:06 AM UTC-4, BikeMike wrote:

[Harold DeWayne]

That is, IF they're even told the table is changing.....   LOL

[henk-jan ebbers]

thank you!


do me favor and tell others (eg EDI-L mailing list)

kind regards,
henk-jan

> > For more options, visit https://groups.google.com/groups/opt_out <https://groups.google.com/groups/opt_out>.

[Harold DeWayne]

Will do, but first I want to do a write-up for EC-BP (E-Cmmerce Best Practices)... I'm a contributor there.

HD

[henk-jan ebbers]

that would be so great!


kind regards,
henk-jan

> > > For more options, visit https://groups.google.com/groups/opt_out <https://groups.google.com/groups/opt_out> <https://groups.google.com/groups/opt_out

> <https://groups.google.com/groups/opt_out>>.
> > >
> > >
> >
> > --
> > You received this message because you are subscribed to the Google Groups "Bots Open Source EDI Translator" group.

> > To unsubscribe from this group and stop receiving emails from it, send an email to botsmail+u...@googlegroups.com <javascript:>.

[BikeMike]

On Thursday, May 16, 2013 11:20:04 PM UTC+9:30, Harold DeWayne wrote:

That's exactly what I was hoping for. ("download or upload individual ccode tables to csv via the GUI.  ")

Harold.

Here is the completed code and screenshots for 3.1. (works ok on 3.0 too)

templates/bots/ccodecsv.html
    new template

models.py
    added download, upload, rowcount to ccodetrigger

admin.py
    added download, upload, rowcount to ccodetrigger

forms.py
    UploadFileForm changed "Plugin to read" to "File to read" (now used for both plugins and csv files)

views.py
    added ccodecsv.     This processes database to csv, and csv to database.

Kind Regards,
Mike

[BikeMike]

I found a bug in sftp communication.
If the received file is the first in a new data subdir, the subdir does not get created and you get a process error.

IOError: [Errno 2] No such file or directory: 'C:\\python27\\lib\\site-packages\\bots\\botssys\\data\\325\\325038'

Fixed by adding one line in class sftp incommunicate...

                tofilename = botslib.abspathdata(str(ta_to.idta))
                botslib.dirshouldbethere(os.path.dirname(tofilename)) # self.session.get will not create new data dirs!
                self.session.get(fromfilename,tofilename)

Kind Regards,
Mike

[BikeMike]

Just realised this bug is only in my local version!

I have removed this...

                tofilename = str(ta_to.idta)
                fromfile = self.session.open(fromfilename, 'r')    # SSH treats all files as binary
                content = fromfile.read()
                filesize = len(content)
                tofile = botslib.opendata(tofilename, 'wb')
                tofile.write(content)
                tofile.close()
                fromfile.close()

and replaced with this simpler code...

                tofilename = botslib.abspathdata(str(ta_to.idta))
                botslib.dirshouldbethere(os.path.dirname(tofilename)) # self.session.get will not create new data dirs!
                self.session.get(fromfilename,tofilename)

                filesize = os.path.getsize(tofilename)

Kind Regards,
Mike

[henk-jan ebbers]

OK!
(errors in communication code are not nice ;-))

kind regards,
henk-jan

On 06/12/2013 03:47 AM, BikeMike wrote:
> Just realised this bug is only in my local version!
>
> I have removed this...
> |
> tofilename =str(ta_to.idta)

> fromfile =self.session.open(fromfilename,'r')# SSH treats all files as binary
> content =fromfile.read()
> filesize =len(content)
> tofile =botslib.opendata(tofilename,'wb')

> tofile.write(content)
> tofile.close()
> fromfile.close()
> |
>
> and replaced with this simpler code...
> |
> tofilename =botslib.abspathdata(str(ta_to.idta))

> botslib.dirshouldbethere(os.path.dirname(tofilename))# self.session.get will not create new data dirs!

> self.session.get(fromfilename,tofilename)
> filesize =os.path.getsize(tofilename)
> |
>
>
> Kind Regards,
> Mike

[BikeMike]

Hi Henk-jan,
I suggest this enhancement in automaticmaintenance.py to send error details in the email reports.
Sometimes I get email reports after hours at home, but cannot easily log in to bots and check (have to VPN in to work). So it is useful to see if the error is worth worrying about or I can leave it until tomorrow.

    botsglobal.logger.info(reporttext)
    # sendreportifprocesserror allows blocking of email reports for process errors
    if (results['lasterror'] or results['lastopen'] or results['lastok'] or
       (results['processerrors'] and botsglobal.ini.getboolean('settings','sendreportifprocesserror',True))):

        # Include error details in the email report (if debug is True this will include trace too!)
        reporttext += '\n\n'
        for filereport in botslib.query('''SELECT errortext
                                           FROM filereport
                                           WHERE reportidta>=%(rootidtaofrun)s''',
                                           {'rootidtaofrun':rootidtaofrun}):
            reporttext += filereport['errortext']

        botslib.sendbotserrorreport(subject,reporttext)
    return int(results['status'])    #return report status: 0 (no error) or 1 (error)

Kind Regards,
Mike

[BikeMike]

slight improvement to add some extra data.

        # Include error details in the email report (if debug is True this will include trace too!)
        reporttext += '\n\n'

        for filereport in botslib.query('''SELECT idroute,frompartner,fromchannel,topartner,tochannel,errortext

                                           FROM filereport
                                           WHERE reportidta>=%(rootidtaofrun)s'''

,
                                           {'rootidtaofrun':rootidtaofrun}):
            for field in filereport.keys():
                reporttext += '%s: %s\n' % (field,filereport[field])

[henk-jan ebbers]

I agree.
What I also am interested in is to see if this can be handled better:
- you disable the processerrors (=communciation-errors), that is right?
- only give a error report is for a fiel the number of automatic retries > x

kind regards,
henk-jan

On 07/01/2013 09:33 AM, BikeMike wrote:
> Hi Henk-jan,
> I suggest this enhancement in automaticmaintenance.py to send error details in the email reports.
> Sometimes I get email reports after hours at home, but cannot easily log in to bots and check (have to VPN in to work). So it is useful to see if the error is worth worrying about or I can leave it
> until tomorrow.
>
> |
> botsglobal.logger.info(reporttext)
> # sendreportifprocesserror allows blocking of email reports for process errors
> if(results['lasterror']orresults['lastopen']orresults['lastok']or

> (results['processerrors']andbotsglobal.ini.getboolean('settings','sendreportifprocesserror',True))):

>
> # Include error details in the email report (if debug is True this will include trace too!)
> reporttext +='\n\n'

> forfilereport inbotslib.query('''SELECT errortext

> FROM filereport
> WHERE reportidta>=%(rootidtaofrun)s''',
> {'rootidtaofrun':rootidtaofrun}):
> reporttext +=filereport['errortext']
>
> botslib.sendbotserrorreport(subject,reporttext)

> returnint(results['status'])#return report status: 0 (no error) or 1 (error)
> |
>
> Kind Regards,
> Mike
>

[BikeMike]

I just added the bit in the middle... the other code is existing, so you can locate it!

[BikeMike]

In order to get text for process errors too, need to check ta not filereport (I don't email those but some might)
Also check that errortext not blank, to only show the errors in run.

        # Include error details in the email report (if debug is True this will include trace too!)
        reporttext += '\n\n'

        for row in botslib.query('''SELECT idroute,frompartner,fromchannel,topartner,tochannel,errortext
                                    FROM ta
                                    WHERE errortext > ""
                                    AND idta>=%(rootidtaofrun)s''',
                                    {'rootidtaofrun':rootidtaofrun}):
            for field in row.keys():
                reporttext += '%s: %s\n' % (field,row[field])

[henk-jan ebbers]

yes, this is a really nice feature!

I assume the whole report is always send?

kind regards,
henk-jan

On 07/02/2013 09:27 AM, BikeMike wrote:
> In order to get text for process errors too, need to check ta not filereport (I don't email those but some might)
> Also check that errortext not blank, to only show the errors in run.
>
> |
> # Include error details in the email report (if debug is True this will include trace too!)
> reporttext +='\n\n'

> forrow inbotslib.query('''SELECT idroute,frompartner,fromchannel,topartner,tochannel,errortext

> FROM ta
> WHERE errortext > ""
> AND idta>=%(rootidtaofrun)s''',
> {'rootidtaofrun':rootidtaofrun}):
> forfield inrow.keys():
> reporttext +='%s: %s\n'%(field,row[field])
> |

[BikeMike]

 Yes. Any time an "error report" email is sent, the partner info, channel info and error text for the error(s) is also sent.

[henk-jan ebbers]

hi Mike,

would it be useful to review the handling/reporting of communcation errors?

I am eg thinking of:
- for incoming channels: do not generate error-report communication failures unless communication failed for eg 5 times.
- for outgoing files: do not generate error-report unless outgoing file is retried eg 5 times. (is now: report for each failed communication)

kind regards,
henk-jan

On 07/02/2013 12:25 PM, BikeMike wrote:
> Yes. Any time an "error report" email is sent, the partner info, channel info and error text for the error(s) is also sent. --

[BikeMike]

On Tuesday, July 2, 2013 9:15:58 PM UTC+9:30, eppye wrote:

- for incoming channels: do not generate error-report communication failures unless communication failed for eg 5 times.  

I think this could be useful. I do not report communication failures because occasional failures happen all the time but 5 in a row would indicate a true problem.
 

- for outgoing files: do not generate error-report unless outgoing file is retried eg 5 times. (is now: report for each failed communication)

ONLY if it is a communication error, right? Do not want to retry other errors. But when would the retry happen? Currently I schedule automaticretrycommunication every hour from 6am to 8pm.

Kind Regards,
Mike

[henk-jan ebbers]

On 07/03/2013 01:15 AM, BikeMike wrote:
>
>
> On Tuesday, July 2, 2013 9:15:58 PM UTC+9:30, eppye wrote:
>
> - for incoming channels: do not generate error-report communication failures unless communication failed for eg 5 times.
>
> I think this could be useful. I do not report communication failures because occasional failures happen all the time but 5 in a row would indicate a true problem.

yes, that is the reason why I am thinking of this.

>
> - for outgoing files: do not generate error-report unless outgoing file is retried eg 5 times. (is now: report for each failed communication)
>
> ONLY if it is a communication error, right? Do not want to retry other errors. But when would the retry happen? Currently I schedule automaticretrycommunication every hour from 6am to 8pm.

yes, ONLY communication errors.
in bots 3.* bots keeps track of the number of resends for outgoing files.
so probably the best place for this is in the automaticmaintenance, when bots create the report for the run.
this would mean that these type of errors would occur in automaticretrycommunication runs.

kind regards,
henk-jan
>
> Kind Regards,
> Mike

[henk-jan ebbers]

hi Mike,

can you please send me the complete change (so I can merge it)?

kind regards,
henk-jan

On 07/02/2013 09:27 AM, BikeMike wrote:

> In order to get text for process errors too, need to check ta not filereport (I don't email those but some might)
> Also check that errortext not blank, to only show the errors in run.
>
> |
> # Include error details in the email report (if debug is True this will include trace too!)
> reporttext +='\n\n'

> forrow inbotslib.query('''SELECT idroute,frompartner,fromchannel,topartner,tochannel,errortext

> FROM ta
> WHERE errortext > ""
> AND idta>=%(rootidtaofrun)s''',
> {'rootidtaofrun':rootidtaofrun}):
> forfield inrow.keys():
> reporttext +='%s: %s\n'%(field,row[field])
> |

[BikeMike]

[henk-jan ebbers]

thank you!

kind regards,
henk-jan

[henk-jan ebbers]

hi Mike,

I checked this in.
What I changed:
- use filereports to extract information. (information about files was already gathered from ta, using same info again)
- if there is a processerrors, check for it in ta.

(in order to avoid using ta, which is the biggest and slowest table)

kind regards,
henk-jan

On 07/12/2013 11:24 AM, BikeMike wrote:

[BikeMike]

cleanup.py _cleanarchive() should check for errors when deleting zip archives. For directories it has ignore_errors=True but for zips any error stops the whole cleanup process.

I added try/except around os.remove(entry)

def _cleanarchive():
    ''' delete all archive directories older than maxdaysarchive days. Errors are ignored!'''
    vanaf = (datetime.date.today()-datetime.timedelta(days=botsglobal.ini.getint('settings','maxdaysarchive',180))).strftime('%Y%m%d')
    for row in botslib.query('''SELECT archivepath FROM channel WHERE archivepath != '' '''):
        vanafdir = botslib.join(row['archivepath'],vanaf)
        for entry in glob.iglob(botslib.join(row['archivepath'],'*')):
            if entry < vanafdir:
                if entry.endswith('.zip'):
                    try:
                        os.remove(entry)
                    except:
                        pass
                else:
                    shutil.rmtree(entry,ignore_errors=True)

[henk-jan ebbers]

I checked that one in.

kind regards,
henk-jan

On 07/29/2013 01:48 AM, BikeMike wrote:
> cleanup.py _cleanarchive() should check for errors when deleting zip archives. For directories it has |ignore_errors=True| but for zips any error stops the whole cleanup process.
>
> I added try/except around os.remove(entry)
>
> |
> def_cleanarchive():
> ''' delete all archive directories older than maxdaysarchive days. Errors are ignored!'''

> vanaf =(datetime.date.today()-datetime.timedelta(days=botsglobal.ini.getint('settings','maxdaysarchive',180))).strftime('%Y%m%d')
> forrow inbotslib.query('''SELECT archivepath FROM channel WHERE archivepath != '' '''):

> vanafdir =botslib.join(row['archivepath'],vanaf)
> forentry inglob.iglob(botslib.join(row['archivepath'],'*')):
> ifentry <vanafdir:

> ifentry.endswith('.zip'):

> try:
> os.remove(entry)
> except:
> pass
> else:
> shutil.rmtree(entry,ignore_errors=True)
>
> |
>
>

[Klaas Feenstra]

Hi Mike,

I tried your modifications in 3.2. It is exactly what I am looking for. I read your instructions and copied your code. The problem is that my upload and download button doesn't work. I would expect the new template to come up, but I get only the base template. Attached you can find my modifications. The problem is that I also don't get any error. Do you have any idea? Could it be not compatible with 3.2?

I just found out.. I had to add also the new view in url.py.

Next problem. With the download options I get an 500 error with the csv download code from views.py. Here I had to change the mime type to content_type. Also Fixed.. 

The attached documents is working for 3.2. Maybe would be good to add this also on the wiki, I think it's very useful. Thanks Mike! As you can see, I am a newbie with Django, but with google I could solve this problems while writing this mail.

--