fixed pantheon invoic to edifact gives TypeError: unsupported operand type(s) for +: 'int' and 'list'

[Angelo de Laat]

Good morning all,

For the last two days i have been looking into this format but it`s breaking my head.
The invoice is coming from a supplier and he uses pantheon.
As far as i could see the invoice is based on all the elements of an D03A invoice but customized (e.g. longer botsid "Grp20_LIN", No separtaion characters, complete message on 1 line).

So i used a fixed format for the PANTHEON_INVOIC and created a mapping to script to generate an INVOICD96A file but during the import i get the following error.
Parsed - Error - TypeError: unsupported operand type(s) for +: 'int' and 'list'

I have attached the Grammar for the fixed file and an example.
Could somebody please look it over because i dont the cause of this error.

Kr,
Angelo

[henk-jan ebbers]

hi Angelo,

this is an inhouse file.
Is consists of chunks of 640 positions.
Each chunk has a a edifact segment in it.
Structure is like edifact, but without UNH and UNT.

Bots can not directly use this format (first time I see something like this).
Would need preprocessing:
for each chunk in file:
discard strat until the '_'
write to file, add <CR/LF>

after that, process as fixed file.
Records are as in that edifact directory, same structure as edifact message (but incl UNB, without UNH and UNT).

The error should not occur, I will have a look at it.


hope that helps,
henk-jan

> --
> You received this message because you are subscribed to the Google Groups "Bots Open Source EDI Translator" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to botsmail+u...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>

[Angelo de Laat]

HJ,

Indeed it is missing the UNH and UNT but this can be added as a static value.

Why do you suggest pre-processing?
I was planning to increase the BOTSID up to 8 or 9 characters (in recorddef) and map the 8/9 character ID  to the 3 character id for  EDIFACT INVOICD96A.

PS I'm using bots 3.

kr,
angelo

[henk-jan ebbers]

On 05/02/2013 11:30 AM, Angelo de Laat wrote:
> HJ,
>
> Indeed it is missing the UNH and UNT but this can be added as a static value.

yes.

>
> Why do you suggest pre-processing?

the format is quite....strange.
AFAICS some preprocessing is needed before bots can read it as fixed file.
bots has a preprocessing option; you can write a small script that is called in a route to do this preprocessing.

> I was planning to increase the BOTSID up to 8 or 9 characters (in recorddef) and map the 8/9 character ID to the 3 character id for EDIFACT INVOICD96A.

if I look at the file (ather giving a <CR/LF> after 640 char):
MSG_DTM137201304
Grp1_RFFON test
Grp2_NADBY 20901
Grp25_LIN1
They append some information before the segment-tag.
This information is of variable length (4-6 position)
what is strange is apart from that the chuck is fixed format.
bots does not handle this mixture of variable and fixed format.

kind regards,
henk-jan

>
> PS I'm using bots 3.
>
> kr,
> angelo
>
>
>
>

[Angelo de Laat]

HJ,

i tried to solve this by increasing the BOTSID based on the segment-tag and process it as fixed format.
MSG_DTM137201304
would be
        ['BOTSID','M',7,'AN'],
        ['C002', 'M',[
            ['C002_1001', 'C', 3, 'AN'],
            ['C002_1131', 'C', 3, 'AN'],

Grp1_RFFON test
        ['BOTSID', 'M', 8, 'AN'],
        ['C506', 'C', [
            ['C506.1153', 'M', 3, 'AN'],
            ['C506.1154', 'C', 35, 'AN'],
            ['C506.1156', 'C', 6, 'AN'],
            ['C506.4000', 'C', 35, 'AN'],
        ]],
        ],

But you suggest preprocessing remove "additional" segment tag (e.g. Grp1_) and CR/LF after 640 char. and than process as fixed file.
Should fixed file be based on edifact or custom grammar, there are some changes wrt no. of characters and M/C.

Do you have an example of a pre-processing script this is the first time i need to use it.

kr,
angelo

[BikeMike]

Hi Angelo,
There are a few preprocessing examples on the wiki. http://code.google.com/p/bots/wiki/RouteScriptsExample

Look for "Example of creating your own custom preprocessing function" near the bottom. This example reads input and writes a new file, just need to insert your logic to split into 640 char chunks, strip off prefixes etc.

Kind Regards,
Mike

[henk-jan ebbers]

hi Mike,

edited the wiki page.
there was a lot of info on this page, so I splitted this up over 3 pages.
also added the pages in the main menu.

kind regards,
henk-jan

[henk-jan ebbers]

On 05/02/2013 12:25 PM, Angelo de Laat wrote:
> HJ,
>
> i tried to solve this by increasing the BOTSID based on the segment-tag and process it as fixed format.
> MSG_DTM137201304
> would be
> ['BOTSID','M',7,'AN'],
> ['C002', 'M',[
> ['C002_1001', 'C', 3, 'AN'],
> ['C002_1131', 'C', 3, 'AN'],
>
> Grp1_RFFON test
> ['BOTSID', 'M', 8, 'AN'],
> ['C506', 'C', [
> ['C506.1153', 'M', 3, 'AN'],
> ['C506.1154', 'C', 35, 'AN'],
> ['C506.1156', 'C', 6, 'AN'],
> ['C506.4000', 'C', 35, 'AN'],
> ]],
> ],

this does not work.
bots expects from fixed files that the BOTSID is same position and same size for all segments.

>
> But you suggest preprocessing remove "additional" segment tag (e.g. Grp1_) and CR/LF after 640 char. and than process as fixed file.

remove additional tag, and add the <CR/LF>

> Should fixed file be based on edifact or custom grammar, there are some changes wrt no. of characters and M/C.

start with edifact grammars.
but place these under grammar/fixed
I expect that record definitions of segments can be re-used.
Structure can be re-used, but you will need to change it: add UNB, remove UNH, UNT.

>
> Do you have an example of a pre-processing script this is the first time i need to use it.

Mike pointed you to the wiki.
there is also a plugin that demonstrates preprocessing.

kind regards,
henk-jan

[Angelo de Laat]

hi all,

so based on the plugin and wiki i came up with the following...

import bots.preprocess as preprocess
import bots.botslib as botslib
import bots.botsglobal as botsglobal
from bots.botsconfig import *

def postincommunication(routedict,*args,**kwargs):
    preprocess.preprocess(routedict,add_line_remove_special)

def add_line_remove_special(ta_from,endstatus,*args,**kwargs):
    try:
        # copy ta for preprocessing
        ta_to = ta_from.copyta(status=endstatus)

        # open the files
        infile = botslib.opendata(ta_from.filename,'r')
        tofile = botslib.opendata(str(ta_to.idta),'wb')
       
        lines = infile.readlines()
        for line in lines;
            [line[i:i+n] for i in lines(0, len(line), 640)]
            line.split("_")[1]
            tofile.write(line)
           
        infile.close()
        tofile.close()

        ta_to.update(statust=OK,filename=str(ta_to.idta)) #update outmessage transaction with ta_info;
    except:
        txt=botslib.txtexc()
        botsglobal.logger.error(u'Sort preprocess failed. Error:\n%s',txt)
        raise botslib.InMessageError(u'Sort preprocess failed. Error:\n$error',error=txt)
       



only the scripts keeps generating an IndentationError: unexpected indent

if somebody could provide some more insight it would be very appreciated.

kr,
angelo

[Harold DeWayne]

Angelo...

I'm sorry if you've already addressed this, but in my case I'm a nubee to python and I've seen this error when I used a tab for indent my code instead of four spaces.


Harold.

[BikeMike]

In python, either tabs or spaces may be used for indenting, but not mixed together! Spaces are preferred/recommended by most people.

I have my editor configured so tab inserts 4 spaces.

[henk-jan ebbers]

hi Angelo,

Python is quite strict with indentation.
I myself always use 4 positions, only use spaces, never tabs.
Some editors support this very well. Also some editors support python syntax checks, this is real handy.
(in your case, this would be caught)
I use geany and scite (both open source); set set these to insert sapces instead of tabs.

some comments in code

On 05/03/2013 11:35 AM, Angelo de Laat wrote:
> hi all,
>
> so based on the plugin and wiki i came up with the following...
>
> import bots.preprocess as preprocess
> import bots.botslib as botslib
> import bots.botsglobal as botsglobal
> from bots.botsconfig import *
>
> def postincommunication(routedict,*args,**kwargs):
> preprocess.preprocess(routedict,add_line_remove_special)
>
> def add_line_remove_special(ta_from,endstatus,*args,**kwargs):
> try:
> # copy ta for preprocessing
> ta_to = ta_from.copyta(status=endstatus)
>
> # open the files
> infile = botslib.opendata(ta_from.filename,'r')
> tofile = botslib.opendata(str(ta_to.idta),'wb')
>
> lines = infile.readlines()
> for line in lines;

in your file I saw only one line.

> [line[i:i+n] for i in lines(0, len(line), 640)]

this does not do anything; the statement produces a list which is not used.

kind regards,
henk-jan

> line.split("_")[1]
> tofile.write(line)
>
> infile.close()
> tofile.close()
>
> ta_to.update(statust=OK,filename=str(ta_to.idta)) #update outmessage transaction with ta_info;
> except:
> txt=botslib.txtexc()
> botsglobal.logger.error(u'Sort preprocess failed. Error:\n%s',txt)
> raise botslib.InMessageError(u'Sort preprocess failed. Error:\n$error',error=txt)
>
>
>
>
> only the scripts keeps generating an IndentationError: unexpected indent
>
> if somebody could provide some more insight it would be very appreciated.
>
> kr,
> angelo

[Angelo de Laat]

all,

after 3 days reading and looking for examples i have the following code.

def add_line_remove_special(ta_from,endstatus,*args,**kwargs):
    try:
        ta_to = ta_from.copyta(status=endstatus)

        infile = botslib.opendata(ta_from.filename,'r')
        tofile = botslib.opendata(str(ta_to.idta),'wb')

        start = infile.readline()
        import textwrap
        lines= "\r\n".join(textwrap.wrap(start, 640))
        tofile.write(lines)
        infile.close()
        tofile.close()

this gives me the following output.


Ichg_UNBUNOA3                                   14                2090100000015                      14                1304221445000001
MSG_BGM380                                         610809                             9  NA
MSG_DTM13720130422                           102
Grp1_RFFON test EDI
Grp2_NADBY 2090100000015                         9
Grp2_NADIV 2090100000015                         9
Grp2_NADDP 2090100000015                         9
Grp7_CUX2  EUR4
Grp8_PAT22                                                                                              5  3  D   30
Grp25_LIN1        02090100000022                     EN
Grp25_QTY47               5
Grp25_QTY12               5
Grp26_MOA203             15.00
Grp28_PRIINV        3000.00           1000PCE
Grp33_TAX7  VAT                                                                                 21.00                              S
Grp25_LIN2        02090100000039                     EN
Grp25_QTY47              10
Grp25_QTY12              10
Grp26_MOA203            350.00
Grp28_PRIINV       35000.00           1000PCE
Grp33_TAX7  VAT                                                                                 21.00                              S


i`m looking for a good way to exclude the values before the edifact segments so i can map the file.

Could somebody please provide a solution.

thanks,
angelo




Op vrijdag 3 mei 2013 13:51:58 UTC+2 schreef eppye het volgende:

[BikeMike]

Hi Angelo,
You just need one more line after your lines=

lines = [line.partition('_')[2] for line in lines]

Kind Regards,
Mike

[Angelo de Laat]

hi mike,

thats also what i have tried not but using line.split.

nevertheless it results in the error:
TypeError: must be string or buffer, not list

def add_line_remove_special(ta_from,endstatus,*args,**kwargs):
    try:
        ta_to = ta_from.copyta(status=endstatus)
        infile = botslib.opendata(ta_from.filename,'r')
        tofile = botslib.opendata(str(ta_to.idta),'wb')
        start = infile.readline()
        import textwrap
        lines= "\r\n".join(textwrap.wrap(start, 640))

        lines = [line.partition('_')[2] for line in lines]

        tofile.write(lines)

        infile.close()
        tofile.close()
        ta_to.update(statust=OK,filename=str(ta_to.idta)) #update outmessage transaction with ta_info;

Op dinsdag 7 mei 2013 13:56:19 UTC+2 schreef BikeMike het volgende:

[henk-jan ebbers]

hi Angelo,

this works for your file:

import textwrap

fromfile = open('angelo.txt','rb')
tofile = open('angelo.res','wb')
for line in fromfile:
if not line.strip():
continue
chunks = textwrap.wrap(line, 640)
lines = "\r\n".join([chunk.partition('_')[2] for chunk in chunks])
tofile.write(lines)

fromfile.close()
tofile.close()


henk-jan

[Angelo de Laat]

thanks HJ,

this works!
can you explain why you do.

     if not line.strip():
         continue

the rest i can understand.



Op dinsdag 7 mei 2013 15:47:26 UTC+2 schreef eppye het volgende:

[henk-jan ebbers]

hi Angelo,

I am not sure about the specs of the input file.
can it be empty, can there be multiple lines, etc.
so I try to build in some safety ;-))

what this statement does: if a line in orginal file has only whitespace (space, tab, <CR/LF>, etc): skip the line.
details:
line.strip() strips all whitespace
if the result is an empty string (if line.strip():)
fetch the next line from the file (continue)


kind regards,
henk-jan

> > To unsubscribe from this group and stop receiving emails from it, send an email to botsmail+u...@googlegroups.com <javascript:>.
> > For more options, visit https://groups.google.com/groups/opt_out <https://groups.google.com/groups/opt_out>.

[Angelo de Laat]

aaah, indeed.

this is very good it was just hard for me to translate it to a logical business reason.

The file is preprocessed very nice but unfortunately also some characters are lost.
the original line had 640 - ***_ but the first line now only has 131 characters.
based on the mapping and specifications provided by the supplier it should have (optional + mandatory characters summed up) 206 characters.

i think that why I get the following error
GrammarError: Grammar "/root/bots-3.0.0/bots/bots.usersys/grammars/fixed.pant_invoic", in structure: back-collision detected at record "[['UNB'], ['BGM'], ['ALC'], ['MOA']]".

Do i need to change the preprocess script or do i need to change my grammar?


Op dinsdag 7 mei 2013 16:09:26 UTC+2 schreef eppye het volgende:

[henk-jan ebbers]

On 05/07/2013 04:47 PM, Angelo de Laat wrote:
> aaah, indeed.
>
> this is very good it was just hard for me to translate it to a logical business reason.
>
> The file is preprocessed very nice but unfortunately also some characters are lost.

seems to be something the textwrap.wrap does. see http://docs.python.org/2/library/textwrap.html
according to this doc, you can turn this off.

shouldn't matter though.

> the original line had 640 - ***_ but the first line now only has 131 characters.
> based on the mapping and specifications provided by the supplier it should have (optional + mandatory characters summed up) 206 characters.
>
> i think that why I get the following error
> GrammarError: Grammar "/root/bots-3.0.0/bots/bots.usersys/grammars/fixed.pant_invoic", in structure: back-collision detected at record "[['UNB'], ['BGM'], ['ALC'], ['MOA']]".

the error says the grammar is not correct.
best is to use the standard edifact grammar (but placed in the fixed grammars map.
AFAIK these grammars should all be OK.

kind regards,
henk-jan

> > > For more options, visit https://groups.google.com/groups/opt_out <https://groups.google.com/groups/opt_out> <https://groups.google.com/groups/opt_out

[Angelo de Laat]

HJ,

I know there are some differences between the grammar user in the file and the edifact standard. Mostly the number of characters used 17 in edifact and 3 used in the structure of the file.
nevertheless i have merged the envelope structure in the INVOICD03AUN structure as well as the recorddef in the recorddef file.

after running the first test i get the same error as the error i started with...

TypeError: unsupported operand type(s) for +: 'int' and 'list'

hmmm..



Op dinsdag 7 mei 2013 17:00:08 UTC+2 schreef eppye het volgende:

[henk-jan ebbers]

On 05/07/2013 05:43 PM, Angelo de Laat wrote:
> HJ,
>
> I know there are some differences between the grammar user in the file and the edifact standard. Mostly the number of characters used 17 in edifact and 3 used in the structure of the file.

that is no longer needed (we are talking about the length of the segment tag here?)

> nevertheless i have merged the envelope structure in the INVOICD03AUN structure as well as the recorddef in the recorddef file.

in the regular INVOIC message grammar:
- replace the UNH segment by UNB
- delete the UNT segment.


kind regards,
henk-jan

>
> after running the first test i get the same error as the error i started with...
>
> TypeError: unsupported operand type(s) for +: 'int' and 'list'
>
>
> hmmm..
>
>
>
> Op dinsdag 7 mei 2013 17:00:08 UTC+2 schreef eppye het volgende:
>
>
> On 05/07/2013 04:47 PM, Angelo de Laat wrote:
> > aaah, indeed.
> >
> > this is very good it was just hard for me to translate it to a logical business reason.
> >
> > The file is preprocessed very nice but unfortunately also some characters are lost.

> seems to be something the textwrap.wrap does. see http://docs.python.org/2/library/textwrap.html <http://docs.python.org/2/library/textwrap.html>

> <https://groups.google.com/groups/opt_out>> <https://groups.google.com/groups/opt_out <https://groups.google.com/groups/opt_out>

[Angelo de Laat]

HJ,

i try to follow your advice but run into several problems.
Replacing UNH by UNB and removing UNT in the INVOIC segment (and recorddef.) is no issue but the envelope is still imported.
This results in an TypeError: unsupported operand type(s) for +: 'int' and 'list'

In the envelope there is both UNB as UNH, so what should i do with this one and where do i get the from/topartner?
I have tried to hash out the import of the envelope but that results in TypeError: unsupported operand type(s) for +: 'int' and 'list'

The difference between the EDIFACT and INHOUSE character length is for example

edifact
'PAI': [
    ['BOTSID', 'M', 3, 'AN'],
    ['C534', 'M', [
        ['C534.4439', 'C', 3, 'AN'],
        ['C534.4431', 'C', 3, 'AN'],
        ['C534.4461', 'C', 3, 'AN'],
        ['C534.1131', 'C', 17, 'AN'],
        ['C534.3055', 'C', 3, 'AN'],
        ['C534.4435', 'C', 3, 'AN'],
    ]],
],

INHOUSE
'PAI': [
    ['BOTSID', 'M', 3, 'AN'],
    ['C534', 'M', [
        ['C534.4439', 'C', 3, 'AN'],
        ['C534.4431', 'C', 3, 'AN'],
        ['C534.4461', 'C', 3, 'AN'],
        ['C534.1131', 'C', 3, 'AN'],
        ['C534.3055', 'C', 3, 'AN'],
    ]],
],

[henk-jan ebbers]

On 05/07/2013 08:51 PM, Angelo de Laat wrote:
> HJ,
>
> i try to follow your advice but run into several problems.
> Replacing UNH by UNB and removing UNT in the INVOIC segment (and recorddef.) is no issue but the envelope is still imported.

Why define an envelope? (get if out of the imports)

> This results in an TypeError: unsupported operand type(s) for +: 'int' and 'list'
>
> In the envelope there is both UNB as UNH, so what should i do with this one and where do i get the from/topartner?
> I have tried to hash out the import of the envelope but that results in TypeError: unsupported operand type(s) for +: 'int' and 'list'
>
> The difference between the EDIFACT and INHOUSE character length is for example
>

> _edifact_

> 'PAI': [
> ['BOTSID', 'M', 3, 'AN'],
> ['C534', 'M', [
> ['C534.4439', 'C', 3, 'AN'],
> ['C534.4431', 'C', 3, 'AN'],
> ['C534.4461', 'C', 3, 'AN'],

> * ['C534.1131', 'C', 17, 'AN'],*

> ['C534.3055', 'C', 3, 'AN'],

> * ['C534.4435', 'C', 3, 'AN'],*
> ]],
> ],
>
> _INHOUSE_

> 'PAI': [
> ['BOTSID', 'M', 3, 'AN'],
> ['C534', 'M', [
> ['C534.4439', 'C', 3, 'AN'],
> ['C534.4431', 'C', 3, 'AN'],

> * ['C534.4461', 'C', 3, 'AN'],*

> ['C534.1131', 'C', 3, 'AN'],
> ['C534.3055', 'C', 3, 'AN'],

> * ]],*
> ],
probably. maybe they use a diferent version of edifact. Normally the UNH states the version. But there is not UNH ;-)))
checking all these grammars is painful.

kind regards,
henk-jan

[Angelo de Laat]

Going trough the specifications and modifying the EDIFACT grammar was not that much of work.

I have created a grammar (this file includes, syntax / structure and recorddef) but when i run it i get the following error:
GrammarError: Grammar "/root/bots-3.0.0/bots/bots.usersys/grammars/fixed.pant_invoic", in structure: back-collision detected at record "[['UNB'], ['ALC'], ['MOA']]".

what does is mean and how can i solve it?

i have attached the grammar document. If all works, i will make and upload a plugin for this format.

kr,
angelo


Op dinsdag 7 mei 2013 21:04:51 UTC+2 schreef eppye het volgende:

[henk-jan ebbers]

there are 2 MOA's in the ALC group.

> > To unsubscribe from this group and stop receiving emails from it, send an email to botsmail+u...@googlegroups.com <javascript:>.
> > For more options, visit https://groups.google.com/groups/opt_out <https://groups.google.com/groups/opt_out>.

[Angelo de Laat]

Hmmm,
Knowing the details of an error really helps in lookup the problems.

The ALC group is optional and not present in this file so i used
{ID: 'ALC', MIN: 0, MAX: 1, LEVEL:[

where min: 0 defines option.

There is a MOA but this is on the same level as the alc group.

updating all max values for moa to 5 (max lines following up starting with moa is 3) has no impact on the error message.

what is the best way to approach this?

kr,
angelo

[henk-jan ebbers]

where did you get this grammar?

it has more issues....



kind regards,
henkjan

[Angelo de Laat]

Good evening all,

I have been looking for a couple of hours and found an edifact structure that i can reuse instead of remodeling it.
i only needed to change the recorddef to match M/C and the UNB (UNH look a like).

i only get the same error as mentioned before 

GrammarError: Grammar "/root/bots-3.0.0/bots/bots.usersys/grammars/fixed.INVOICD97BUN", in structure: back-collision detected at record "[['UNB'], ['MOA']]".

Based on previous comment i assume it has multiple MOA's in the UNB group.
based on the structure it can only be the MOA at the end of the structure

    {ID: 'CNT', MIN: 0, MAX: 10},
    {ID: 'MOA', MIN: 0, MAX: 100, LEVEL: [
        {ID: 'RFF', MIN: 0, MAX: 1, LEVEL: [
            {ID: 'DTM', MIN: 0, MAX: 5},
        ]},
    ]},
    {ID: 'TAX', MIN: 0, MAX: 10, LEVEL: [
        {ID: 'MOA', MIN: 0, MAX: 2},
    ]},
    {ID: 'ALC', MIN: 0, MAX: 15, LEVEL: [
        {ID: 'ALI', MIN: 0, MAX: 1},
        {ID: 'MOA', MIN: 0, MAX: 2},
        {ID: 'FTX', MIN: 0, MAX: 1},
    ]},
]},
]


but the MOA is set to 100 so finding multiple MOA`s should not be an issue...








Op woensdag 8 mei 2013 12:42:00 UTC+2 schreef eppye het volgende: