Can BOTS handle really large messages

[Len Finley]

Hi Henk,

We have a new vendor that's going to start sending us some really large 850 files.  Maybe like 20k PO1 lines.  Can BOTS handle that large of a file?  Likewise we would send 855, 856, and 810 for the PO. 

Regards,

Len

[Tom Westrick]

How big of a file in size (kilobytes/megabytes)? Henk has performance metrics here - https://bots.readthedocs.io/en/latest/overview/performance/.

-Tom

[Eppye Bots]

did you try?

kind regards, Henk-Jan Ebbers

--
You received this message because you are subscribed to the Google Groups "Bots Open Source EDI Translator" group.
To unsubscribe from this group and stop receiving emails from it, send an email to botsmail+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/botsmail/e3ba9cb9-c50f-4692-b0a2-4a7f063fe717n%40googlegroups.com.

[Hans]

BOTS is able to process LARGE files, however it uses a large amount of memory doing so.

This week i had an issue that a CSV 2 CSV translation of a 145MB file with 1.500.000 lines (12 records) which took over 12GB of memory and oom killed the server.

By splitting up the file in parts of 200.000 lines in the routescript and merging it in the out-grammar, it just uses 2GB and it's processed a lot quicker.

Also the use of a Database connection as grammar in/output can save a lot of memory / processtime.

This way we're also processing files with 5-6 million lines.

So 20K shouldn't be an issue :)

Op maandag 28 maart 2022 om 23:19:06 UTC+2 schreef eppye:

[Eppye Bots]

hai Hans,

can you elaborate on 'the use of a Database connection as grammar in/output can save a lot of memory / processtime' ?

what is it you do here?

kind regards, Henk-Jan Ebbers

To view this discussion on the web visit https://groups.google.com/d/msgid/botsmail/18705a59-78db-4231-b58a-2d878195419en%40googlegroups.com.

[Hans]

Hi Henk-Jan,

In the above example i gave, the CSV 2 CSV translation takes 12GB.

the same file with the same amount of records in and out CSV 2 DB takes 7GB of memory.

CSV 2 DB however takes more process time because of inserting / updating per line.

This i tackled by translating CSV 2 CSV (splitup as mentioned) and in the communicationscript (File), using mysqlimport to dump the whole CSV in a table which is a lot quicker.

When using DB as incoming channel it's quicker then a CSV grammar and uses less memory.

I don't realy know the technical reason why it is this way, it's just my experience trying to use bots for processing very large files.

Op donderdag 7 april 2022 om 12:29:49 UTC+2 schreef eppye: