state of bots development

[hjebbers]

first of all, a lot of contribution to the wiki.
the wiki is containing more and more good and much more up to date
information!

I have made a bots 2.1.1 version, containing some fixes for bugs
found. Will be out soon.

I am working on bots 2.2
Most changes are in the translator part (parsing messages, put, get).
Most important changes:
- bots could not handle all edifact/x12 messages, for most messages a
workaround could be used. These are messages where a segmentgroup
cccured more than once in a message, eg in INVOIC an ALC-group both at
header level and summary level. This works now OK.
- for most errors in an incoming or outgoing message, bots will not
stop anymore, but try to go on and collect all errors. So instead of
getting one errors, remove the error and test again most errors will
come out in one time. (it is not possible to do this for all errors)
- I started using the cdecimal library as a drop-in replacement for
python's own decimal library. AFAICS this works, and is much faster
(in formatting fields, bots uses the decimal library for numerical
fields).
Looks like python itself will adopt the cdecimal library in newer
version.
So basically, when cdecimal is availably, you will get better
performance, if not no problem.
If you want to use this in bots 2.1: easy by just changing some
imports.
- the _parse function, that does the work of matching the records with
the grammars (the 'heart' of parsing a message) is rewritten, and is
much simpler now. The code was quite hard to understand :-(

Nothing has to be changed in grammars, translations or database.
I am still working on some minor things, and busy with testing.
performance is slightly better (when not using cdecimal), but much
better with cdecimals (15-35% faster).

I have been looking at the option of running routes parallel, but this
is a big change.
So I will start with this after bots 2.2. Think this will be bots 3.0

kind regards,
henk-jan

[BikeMike]

Hi henk-jan,
Here are a few of my changes for possible inclusion in 2.1.x or 2.2. (I
have some other changes but they are more specific to my installation
and preferences)

skip_firstline for CSV to allow true/false/integer number of lines
Requested by Bill Butler
http://groups.google.com/group/botsmail/msg/670fa9afb1a13cb8
inmessage.py

Allow comma separated list of email addresses
communication.py (change 1 line)
#addresslist = [x for x in (row['tomail'],row['cc']) if x]
addresslist = row['tomail'].split(',') + row['cc'].split(',')
Requested by Jean-Fran�ois

http://groups.google.com/group/botsmail/browse_thread/thread/5140cfef069d1835#

Layout improvements (suggestions from one of my users)
templates/bots/about.html
Show user and login info here instead of on the home menu
botsinfo: don't show undefined values, use consistent case (neater
look)
Extra spacing above/below the links

templates/bots/menu.html
Remove current user and last login (now on about page)
add help link to wiki

Date/Time popup widget on bots select screens
(description and plugins in this thread)

http://groups.google.com/group/botsmail/browse_thread/thread/9be0c8c69550044e#

Kind Regards,
Mike

[BikeMike]

One more that might be of general interest... print start info when
starting webserver.py. This is useful when testing, I can quickly see
that I have started the correct webserver environment.

Kind Regards,
Mike

[hjeb...@gmail.com]

On 02/04/2012 03:20 AM, BikeMike wrote:
> Hi henk-jan,
> Here are a few of my changes for possible inclusion in 2.1.x or 2.2. (I have some other changes but they are more specific to my installation and preferences)
>
> skip_firstline for CSV to allow true/false/integer number of lines
> Requested by Bill Butler
> http://groups.google.com/group/botsmail/msg/670fa9afb1a13cb8
> inmessage.py

is checked in.

>
> Allow comma separated list of email addresses
> communication.py (change 1 line)
> #addresslist = [x for x in (row['tomail'],row['cc']) if x]
> addresslist = row['tomail'].split(',') + row['cc'].split(',')

> Requested by Jean-François
>
> http://groups.google.com/group/botsmail/browse_thread/thread/5140cfef069d1835#
at the moment these fields are validated.
it seems quite doable to have validation on the list of email-addresses (by subclassing a field-type)
would that be possible?

>
> Layout improvements (suggestions from one of my users)
> templates/bots/about.html
> Show user and login info here instead of on the home menu
> botsinfo: don't show undefined values, use consistent case (neater look)
> Extra spacing above/below the links
>
> templates/bots/menu.html
> Remove current user and last login (now on about page)
> add help link to wiki

is checked in.

>
> Date/Time popup widget on bots select screens
> (description and plugins in this thread)
>
> http://groups.google.com/group/botsmail/browse_thread/thread/9be0c8c69550044e#

very nice, that one!
but in the plugin there is a copy of jquery; while jquery is also in the default bots distribution.
is it possible to have this as one copy?

kind regards,
henk-jan

>
> Kind Regards,
> Mike

[BikeMike]

On Mar 28, 1:04 am, "hjebb...@gmail.com" <hjebb...@gmail.com> wrote:
> at the moment these fields are validated.
> it seems quite doable to have validation on the list of email-addresses (by subclassing a field-type)
> would that be possible?

Yes that would be good, to have validation but still allow a list of
addresses.

> Date/Time popup widget on bots select screens

> very nice, that one!
> but in the plugin there is a copy of jquery; while jquery is  also in the default bots distribution.
> is it possible  to have this as one copy?

Yes I used a new copy, I think it is a later version, and also it
includes the theme files. I wasn't sure where the other jquery was
used so left it alone to be safe ;-) But if the newer version is ok
then one copy would be better, of course.

Kind Regards,
Mike

[hjeb...@gmail.com]

On 03/28/2012 11:24 AM, BikeMike wrote:
> Date/Time popup widget on bots select screens
>> very nice, that one!
>> but in the plugin there is a copy of jquery; while jquery is also in the default bots distribution.
>> is it possible to have this as one copy?
> Yes I used a new copy, I think it is a later version, and also it
> includes the theme files. I wasn't sure where the other jquery was
> used so left it alone to be safe ;-) But if the newer version is ok
> then one copy would be better, of course.
>
> Kind Regards,
> Mike
>

I will try this.

Mike,

if I remember right you are using the 'retry all previous errors'-option?
what is your experience with this?

(I am thinking of skipping this option; AFAIK 'retry' and 'retry communication errors' would be enough. This could simplify a lot of complexity in ta-handeling. )

henk-jan

[BikeMike]

I think the only "overlap" is jquery.min.js. jquery-ui* are all new.
I have tried this, moving all the new js and css to appropriate
folders and renaming to remove the version number part. Then changed
the paths in selectform.html. So far it seems to be ok. I don't know
what else uses jquery, maybe a dependency of django?

Also seems like jquery theme images are not used in the datepicker, so
they can probably be omitted too.

Kind Regards,
Mike

[BikeMike]

Images are required for next/prev month button icons.
They need to be in images sub-dir under css

[BikeMike]

On Mar 28, 7:33 pm, "hjebb...@gmail.com" <hjebb...@gmail.com> wrote:
> if I remember right you are using the 'retry all previous errors'-option?
> what is your experience with this?
>
> (I am thinking of skipping this option; AFAIK 'retry' and 'retry communication errors' would be enough. This could simplify a lot of complexity in ta-handeling. )

From the run menu I use:
Retry all previous errors (after fixing something)
Run only new (in test environment, not in production)
Run user indicated retransmits (only sometimes while testing)
I have never used Retry user indicated comm errors.

From the command line I use:
--new (of course!)
--automaticretrycommunication (scheduled once per hour for any
outgoing ftp timeouts)
--retry (scheduled daily in case I miss/forget some error, it fails
again as a reminder)
--retransmit

I have built an automated retransmit solution, where bots has a route
to pick up a csv file of order numbers and retransmit modes. A mapping
script reads this list, finds the ta records (by botskey) and set to
retransmit, then a --retransmit run is called. The retransmit is
mapped differently depending on retransmit mode requested, to overcome
the original error in our ERP system that required the retransmit.
It's complex but works well for us. For example, an order fails with
an invalid contract number, so we retransmit it with the contract
number in text. Then at least the order is in our ERP and just the
contract issue needs to be sorted out manually.

Kind Regards,
Mike

[hjeb...@gmail.com]

On 03/28/2012 01:31 PM, BikeMike wrote:
>
> On Mar 28, 7:33 pm, "hjebb...@gmail.com"<hjebb...@gmail.com> wrote:
>> if I remember right you are using the 'retry all previous errors'-option?
>> what is your experience with this?
>>
>> (I am thinking of skipping this option; AFAIK 'retry' and 'retry communication errors' would be enough. This could simplify a lot of complexity in ta-handeling. )
> > From the run menu I use:
> Retry all previous errors (after fixing something)

yes, that is also why I sometime use it.
but AFAICS 'rereceive' does the same thing.

I found it very hard to explain the 'retry all previous errors';
rereceive is much easier to understand....and much more controllable....user indicates what happens.

henk-jan

[BikeMike]

For the jquery pop-up calendar, I have moved all the files to the
"standard" directories and modified html and css as required. Only the
icon images from the theme are included as these are used in the
calendar. jquery.min.js is updated to a newer version than in bots by
default, this has not caused any problems I can see.

Plugin attached.

Kind Regards,
Mike

[hjeb...@gmail.com]

hi mike,

I checked this in

kind regards,
henk-jan

[BikeMike]

Hi henk-jan,
I have been doing some testing with the latest changes, I think there
is a problem with botskey queries in grammars. I will try to test more
tomorrow to narrow this down. Translations are ok, but document
numbers are missing, and therefore output filenames normally based on
document number do not work. Switching back to old versions
(envelope.py, message.py, inmessage.py, outmessage.py, node.py) fixes
this.

Kind Regards,
Mike

[BikeMike]

still trying to find this problem, I don't have much understanding of
the code in this area. It seems like QUERIES only work if they are in
the first LEVEL of the structure.

In edifact.py (from the plugins) there is a "reference" query at the
UNB-UNH level, and repeated for UNB-UNG-UNH level. I have also added
"botskey" queries at these levels. They do not work with the latest
test version.

Kind Regards,
Mike

[henk-jan ebbers]

found the bug!

henk-jan

[henk-jan ebbers]

hai Mike,

you have an example of a grammar that worked (I guess with 2.1.0) but does not work in dev version?

kidn regards,

henk-jan

On 04/27/2012 09:04 AM, BikeMike wrote:

[BikeMike]

Hi henk-jan
My enhanced edifact grammar is affected by this.
botskey query (in edifact.py) is the problem.

I guess any grammar that has queries at a nested levels would be
affected. X.12 would be similar, but I don't have any X.12 incoming.


Kind Regards,
Mike

[henk-jan ebbers]

hai mike,

think I found the problem; was a small change, is checked in.
check it out, AFAICS alle queries work OK now.

henk-jan

[BikeMike]

Yes, this now works ok, thanks.

Kind Regards,
Mike