[Bots Open Source EDI Translator] CR-LF after passthrough with communication-script

[A.rian]

Hello together,

I am working with Bots and received the following problem:

Bots is currently working as a pass-through system for CSV-files. The CSV-files we are feeding Bots with all use CRLF as line-separator.
With the usage of a communication script we manage to use the same Input-Filename as Output-Filename.

When using this we get the problem, that the Outcoming-files only contain a LF and the CR is missing.
When we do not use the communication script or we use a "Fix" filename is this script it works fine.

Thanks in advance for your help.

Best regards

[henk-jan ebbers]

hi Adrian,

first thought is of course that something is done in the communication-script?

kind regards,

henk-jan

> --
> You received this message because you are subscribed to the Google Groups "Bots Open Source EDI Translator" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to botsmail+u...@googlegroups.com <mailto:botsmail+u...@googlegroups.com>.
> Visit this group at https://groups.google.com/group/botsmail.
> For more options, visit https://groups.google.com/d/optout.

[A.rian]

The communication script I use uses this code:

---start
def filename(channeldict,filename,ta=None,**argv):
    ''' Receive old filename (as bots made the filename), information about the channel and the current entry in ta table.
        Return the new filename.
        Be aware: filenames are supposed to be unique eg for file, ftp.
        Advise: use the filename as bots made this in your own filename.
    '''
    lijst = botslib.trace_origin(ta=ta,where={'status':EXTERNIN})    #where is a dict; bots returns a list of all ta's where all the conditions in 'where' are True
    #trace_origin returns a list!! (because there can be more than one incoming file)
    #~ for dbta in lijst:
        #~ print dbta.filename
    return  os.path.basename(lijst[0].filename) # + filename      #filename is incl path, only needs filename itself
---end

It seems to happen when using "trace_origin" because it works fine if I just implement "return 'TEST".

I found nothing in the trace_origin function that causes my problems.

[henk-jan ebbers]

hi Adrian,

can you send me a plugin so I can try to replicate?

I assume you are using bots 3.2?

I can not imagine why this would happen...trace_origin does not read or write files.

note that if you run code more than one, the output is appended (as filename is not unique anymore).

also, not sure why you are using communication-script for this. Setting such a filesname can be done in GUI, in channel.

kind regards,

henk-jan

[A.rian]

Thanks for your fast answers!!

We are using an older version (2.2) since we did not update our Bots-servers yet. (never change a running system)
We use the communication script so the outgoing file has the same filename as the incoming file. How could we do that by changing the settings in GUI?

With plugin you mean the settings we use? Can you please tell me what exactly you need, so I can provide it to you? Just "make plugin" and use everything which is checked right now?

[henk-jan ebbers]

On 03/29/2017 11:09 AM, A.rian wrote:
> Thanks for your fast answers!!
>
> We are using an older version (2.2) since we did not update our Bots-servers yet. (never change a running system)

let me guess - it runs on windows 95? ;-)))

most migrations are quite easy. unless you did a lot of digging in the 'intestines' of bots.

> We use the communication script so the outgoing file has the same filename as the incoming file. How could we do that by changing the settings in GUI?

both pass-through and setting file-names is much improved in newer versions.
instead of using communication or routing scripts, both can be done in the GUI.
(as both are popular features, it was good to have both feature as standard stuff. and indeed, that is much easier)

kind regards,
henk-jan

>
> With plugin you mean the settings we use? Can you please tell me what exactly you need, so I can provide it to you? Just "make plugin" and use everything which is checked right now?

[A.rian]

:D the system is not thaat old but it is not the current version of windows servers..^^

I found out some more details after testing a bit further.. so maybe it will be easier to find the problem..

the problem that we only have LF instead of CRLF only occurs if we use the ending ".csv" .. if we use a normal "FILE" with no ending we get the CRLF.. for the csv we only get LF..

the csv we feed bots with contains CRLF.. the different ist that in notepad++ we see the one file with CRLF got the charset "DOS/WINDOWS" the file bots created has "UNIX"..
since we only use bots as passthrough we do not do a translation.. maybe bots does something different when creating a file with .csv ending?

Can you help me to get closer to the problem since from this point I am lost in bots.. ^^

[henk-jan ebbers]

On 03/29/2017 12:00 PM, A.rian wrote:
> :D the system is not thaat old but it is not the current version of windows servers..^^
>
> I found out some more details after testing a bit further.. so maybe it will be easier to find the problem..
>
> the problem that we only have LF instead of CRLF only occurs if we use the ending ".csv" .. if we use a normal "FILE" with no ending we get the CRLF.. for the csv we only get LF..

I really can not see what would cause such a thing.

>
> the csv we feed bots with contains CRLF.. the different ist that in notepad++ we see the one file with CRLF got the charset "DOS/WINDOWS" the file bots created has "UNIX"..
> since we only use bots as passthrough we do not do a translation.. maybe bots does something different when creating a file with .csv ending?
>
> Can you help me to get closer to the problem since from this point I am lost in bots.. ^^

provide me a plugin so I can reproduce the problem.
A bots configuration/setup can be written to a plugin (GUI menu:-> SysTasks->write plugin)

It is great if you can provide only the route/channels/etc that have this problem.



kind regards,
henk-jan

[A.rian]

Hey, sorry for my late reply but I can not focus 100% on this problem beside doing my normal work. Since there are personal and customer data which I can not figure out how to completely remove those from the plugin (also header comments in the scripts etc) I could only tell you how to set up the routes etc to reproduce this problem.

I have done some more tests and figured out that this only comes along when using the ending ".csv" for the file Bots creates. If i create a .txt file or a file without an ending the problem does not occur.
So somewhere in bots there has to be a logic that CRLF does not appear as soon as bots wants to create a .csv file. I checked the grammar.py which has all the parameters set correctly but maybe it doesn't even come to this point since we are not using a translation (?)

I am not quite sure how to continue with that :(

[henk-jan ebbers]

this is a 'pass-through', am I right?

a pass-though file should not be touched...


kind regards,
henk-jan

[A.rian]

Hey, so I am back in the office today..

yes this is pass-through.. usally it should not be touched.. there is something Bots does to our file when saving it as CSV.. maybe there is some different ways of saving / creating csv files instead of .txt for example.. i tried to save it local as well as sending it via ftp.. no differnces..

[A.rian]

Hey, we found the mistake..

the ftp we sent the data to was a UNIX system.. therefore it converted our file when transmitting it onto the ftp.. the CR LF got converted to LF..
after checking every settings again we found out that we have to set binary for the IN and the OUT channel.. I was wondering about that since I only expected that to be set in the out channel
but after setting it for the IN and Out channel it worked..

[henk-jan ebbers]

good to hear!


kind regards,

henk-jan