AS2 add on for BOTS

[Abhishek Ram]

Hi All,

I have been using a third party tool just for AS2 (as I think most of you do as well), so I thought it would be much better to add AS2 to bots so that we can complete the tool set. I have so far managed to build the AS2 client/sender and below are the dets:

Dependencies:

• Requests library to transfer messages via http/s
• M2Crypto for encryption and signatures (I know M2Crypto is old and does not support py3 but this is the only complete crypto lib available out there, I will migrate to cryptography api once it is a bit more mature)

Features:

• Send signed, encrypted messages to an AS2 endpoint
• Receive signed synchronous mdn back from AS2 endpoint

Usage:

1. Copy the attached as2lib.py to the bots base folder
2. Update the bots.ini file with the contents as per the attached file.
3. Copy the testas2send.py to communicationscripts folder and you are good to go.

Testing:

• I have only tested this with mendelson as2. I no longer have access to drummond certified tools such as Sterling Integrator, Axway, Seeburger. So if any of you do have access then I request you to test against there tools so that we can be sure about the clients veracity to standards.
• The following scenarios need to be tested.
• No Encryption/ No Signature/ No MDN
• No Encryption/ No Signature/ MDN
• No Encryption/ No Signature/ Signed MDN
• No Encryption/ Signature/ Signed MDN
• Encryption/ Signature/ Signed MDN
• In case of errors, turn on debug mode and the as2 message and mdn will be logged, this will be helpful to fix issues.

Future Releases:

• Should we add received MDNs to confirmations? if so how do we go about it
• Option to allow all message digests for signatures, currenlty only SHA1 is supported
• Enable compression of outbound messages, I have tried zlib and it does not work. Any suggestions???
• Enable ASYNC mdn mode, this will require the server

I am also working on the server/receiver part of the add on and will update once that is done. I request python experts here to do a code reveiw and help make it better. All suggestions for improvement are welcome.

[Abhishek Ram]

Attaching the missing files.

Regards
Abhishek Ram

[henk-jan ebbers]

great!
will have a look at this.
very exiting.

kind regards,
henk-jan

--
You received this message because you are subscribed to the Google Groups "Bots Open Source EDI Translator" group.
To unsubscribe from this group and stop receiving emails from it, send an email to botsmail+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Jean-François Van Snick]

Sounds terrific, indeed !

SHA2 is almost mandatory as support of SHA1 will be ended soon.

I don't know if that's a lot of work to implement this, but SHA2 is needed if I want to test against my partners.

Quick question: how does the reception of message work? How do you expect to make it work?

Kind regards and thanks for this great job

JF

[Abhishek Ram]

I understand what you mean regarding the SHA1, will look at improving to support all message digests.

As far as reception of message is concerned, my plan is to have a folder specified in the ini file for the partner and files received will drop in this folder. The drawback is that there is no way of searching in the gui for received file, only a entry will be made in the web server log, what do you think?

Regards

Abhishek Ram

You received this message because you are subscribed to a topic in the Google Groups "Bots Open Source EDI Translator" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/botsmail/vby7iwQNBLg/unsubscribe.
To unsubscribe from this group and all its topics, send an email to botsmail+u...@googlegroups.com.

[Jean-François Van Snick]

Our situation is a bit specific as we're acting on behalf of clients, so we have 2 kinds of AS2 connections

1 - a single link between ourselves and the operator(s) > all messages for all our clients that are received from the operator(s) go in the same folder. We are only using our own AS2 Certificate.

2 - a dedicated link between our clients and the operator(s) > we can use either our own AS2 Certificate for all the clients, but also the client's AS2 certificate. > all messages are put in specific directory (messages/client1/inbox/operator1) etc.

We do have a routing script that opens the incoming messages, checks if this is EDI or XML or CSV, extracts the message type, the Issuer and the Recipient, and according to those 3 parameters, sends the message to the right Bots application (we're running 4 Bots in parallel), in the right folder (XML, EDI, CSV)...

I think the Mendelson method of folders is a good one. Each party has a name and a AS2 Id. Folders are structured: MyName/inbox/MyPartyName and MyPartyName/outbox/MyName

We could use the PartnerGroups (or Partners?) defined in Bots to set up the parties names.

I'm very interested in testing this feature. I'll make some sending tests and give you feedback. As far as reception is concerned, it's important to be as much flexible as possible.

Regards,

JF

[henk-jan ebbers]

On 04/16/2014 03:04 PM, Abhishek Ram wrote:

As far as reception of message is concerned, my plan is to have a folder specified in the ini file for the partner and files received will drop in this folder. The drawback is that there is no way of searching in the gui for received file, only a entry will be made in the web server log, what do you think?

as Jean-François said, the mendelson AS2 server does this in a clear and comprehensive way.
I know the problem with logging  (in log file and/or database).
Think that database & GUI is unavoidable.
(often in edi 'accountability' is required).

kind regards,
henk-jan

Regards

Abhishek Ram

[henk-jan ebbers]

On 04/16/2014 03:48 PM, Jean-François Van Snick wrote:

Our situation is a bit specific as we're acting on behalf of clients, so we have 2 kinds of AS2 connections

1 - a single link between ourselves and the operator(s) > all messages for all our clients that are received from the operator(s) go in the same folder. We are only using our own AS2 Certificate.

2 - a dedicated link between our clients and the operator(s) > we can use either our own AS2 Certificate for all the clients, but also the client's AS2 certificate. > all messages are put in specific directory (messages/client1/inbox/operator1) etc.

yes, think that is important.
(specify per partner waht certificate to use)

We do have a routing script that opens the incoming messages, checks if this is EDI or XML or CSV, extracts the message type, the Issuer and the Recipient, and according to those 3 parameters, sends the message to the right Bots application (we're running 4 Bots in parallel), in the right folder (XML, EDI, CSV)...

is 4 bots'es that because of volume or other reasons?

I understand that the 'what comes in' problem in AS2 is unavoidable.
(strange for a protocol which is made for edi ;-))

kind regards,
henk-jan

[Abhishek Ram]

I can do it the mendelson way, we can set up multiple as2 organisations and partner. When we receive a message we check to find the organisation based in the AS2 to ID and the partner based on as2 from ID then we save just like MEC as2. All configuration is currently in the bots.ini, I have specified examples of both organisation and partner there.

In case of DB and GUI, what if we just save the sent and received MDNs in the confirmation screen, enough or should we have separate as2 screens on GUI??

[Jean-François Van Snick]

In first instance, I think sent/received MDN's are sufficient, displaying status.

Of course, an in-depth logging seems mandatory as we have to give a correct diagnostic in case of communication/transfer error.

You received this message because you are subscribed to the Google Groups "Bots Open Source EDI Translator" group.

To unsubscribe from this group and stop receiving emails from it, send an email to botsmail+u...@googlegroups.com.

[Jean-François Van Snick]

I do use 4 bots'es because some clients require their own webinterface (3 of them). The 4th is a "main", handling the majority of messages. I do also have a dedicated Bots instance for EDIEL messages, but messages are transferred via a VAN and not AS2.

We will probably work on a more customer-oriented webinterface, based on the bots-db, showing green balls and red/yellow balls if there are problems (this is what the final client wants to know :-)

I'll keep you tuned about this, of course.

The problem is that EDI is now replaced, step by step, by XML EDI and so we have to handle (1) larger files and (2) different kinds of files analysis, routing the files to the correct Bots incoming directory (type UNOA / UTF-8 etc.)

Kind regards,

JF

[henk-jan ebbers]

On 04/17/2014 11:03 AM, Jean-François Van Snick wrote:

I do use 4 bots'es because some clients require their own webinterface (3 of them). The 4th is a "main", handling the majority of messages. I do also have a dedicated Bots instance for EDIEL messages, but messages are transferred via a VAN and not AS2.

We will probably work on a more customer-oriented webinterface, based on the bots-db, showing green balls and red/yellow balls if there are problems (this is what the final client wants to know :-)

I'll keep you tuned about this, of course.

yes, I am very interested.
this is more like a 'dashboard'-function?

The problem is that EDI is now replaced, step by step, by XML EDI and so we have to handle (1) larger files and (2) different kinds of files analysis, routing the files to the correct Bots incoming directory (type UNOA / UTF-8 etc.)

when I started bots an idea I had was to read all edifact messages via one channel, process these and filter in the outgoing cahnnel.
there is an option to have bots 'recognize' is edifact/xml, and (for xml) find out the messagetype.

kind regards,
henk-jan

[Jean-François Van Snick]

Kind of dashboard, yes, giving access to more details about the error if necessary.
We already developped a dashboard querying the different Bots instances and databases showing problems and inactivity (db locked).
This will give our dispatchers the ability to have only one central screen - and via html links, to open the bots interface they need to check.

[David Friday]

Hi Abhishek,

 

I was unable to download the bots.ini file, can you make it available again.

 

Thanks,

 

David

[Abhishek Ram]

I have attached all the files again, let me know if you still face issues.

[henk-jan ebbers]

hi Abhishek,

please let me know if I can be of help for the AS2 server part!

kind regards,
henk-jan

You received this message because you are subscribed to the Google Groups "Bots Open Source EDI Translator" group.

To unsubscribe from this group and stop receiving emails from it, send an email to botsmail+u...@googlegroups.com.

[Abhishek Ram]

Hi Henk,

I got busy so haven’t made further progress on the server part. I should be able to do more in the coming weeks as I will have some free time and as for help will let you know when I face the need.

I am also waiting on cryptography lib to progress further as it would not be a good idea to go live with M2Crypto.

Regards

Abhishek Ram

[henk-jan ebbers]

hi Abhishek,

I fully agree about the cryptography lib.
I took a look at AS2 before, but the lack of a good cryptography library always held me back.

I have been looking at http(s); this is also possible without the REST library.
I do not doubt that the library is OK, but it might be simpler without this library.
REST does a lot of things that are ...less useful for EDI.
It seems to be good in some forms of authentication. But...I have never seen this used in edi. AS2 does not use it (think that the AS2 authentication is much more powerful).
I implemented basic authentication (which is quite basic indeed), that was easy.

(just my thoughts)

kind regards,
henk-jan

[Abhishek Ram]

On 26-May-2014, at 5:17 pm, henk-jan ebbers <eppye...@gmail.com> wrote:

hi Abhishek,

I fully agree about the cryptography lib.
I took a look at AS2 before, but the lack of a good cryptography library always held me back

I have been looking at http(s); this is also possible without the REST library.

The http(s) is not something we need to worry about, this will need to be handled by http server such as Apache, CherryPy

I do not doubt that the library is OK, but it might be simpler without this library.
REST does a lot of things that are ...less useful for EDI.

It seems to be good in some forms of authentication. But...I have never seen this used in edi. AS2 does not use it (think that the AS2 authentication is much more powerful).

Having used the library now, I completely agree with you and this is not necessary for either HTTP or AS2 receives.

I implemented basic authentication (which is quite basic indeed), that was easy.

I think this will be best, need a way to make it optional and with this we should be able to do both HTTP and AS2. 

Did u manage to finish up on the HTTP receive?

[henk-jan ebbers]

On 05/26/2014 01:54 PM, Abhishek Ram wrote:

Having used the library now, I completely agree with you and this is not necessary for either HTTP or AS2 receives.

I implemented basic authentication (which is quite basic indeed), that was easy.

I think this will be best, need a way to make it optional and with this we should be able to do both HTTP and AS2. 

Did u manage to finish up on the HTTP receive?

http as a client works.
for server side: got both send and receive working.
Do this fairly simple now (but simple is often a good way...):
- http server is separate from default web-server.
- start with seperate configuration (via -c option, like -cconfig2)
- most django TEMPLATE_CONTEXT_PROCESSORS and INSTALLED_APPS are disabled (as these are not needed)
- basic authentication works via decorator:

@http_basic_access_authentication(realm='bots')
def server_receive(request):
    if request.method == 'POST':
        payload = request.read()
        print 'server received OK, content:', payload[:30]
        return django.http.HttpResponse('OK server received',status=200)

works with apache, but alos with default cherrypy server.


kind regards,
henk-jan

[Abhishek Ram]

This is cool, we can use this separate server for the AS2 as well, I will be starting work this week and will let you know if I stumble and need help :) 

[Abhishek Ram]

Also I suggest adding the “pk” to the url and the server_receive function which corresponds to a communication script so that users can deal with the files however they wish.

[Abhishek Ram]

Hi Guys,

Quick update on the status of this dev, 

1) I have built the AS2 server but I am facing issue with signing and verification of signatures, it is strange but I did not face this issue when I had to do the same for the client. I will debug and update once done with that.

2) Currently partner and organization information is stored in ini file, my plan is to create models AS2Partner and AS2Organization with a one to one relationship with a Partner. This way all user needs to choose for outgoing comms is comm type AS2, partner and orgnization is decided based on from and to partner fields.

3) Also I think AS2Transaction model needs to be created to log all movements as I dont think its possible to integrate with normal Bots Transactions. This will also make it easier to implement ASYNC MDN mode.

I will post further updates as I progress.

[henk-jan ebbers]

great!

On 06/30/2014 01:52 PM, Abhishek Ram wrote:

Hi Guys,

Quick update on the status of this dev, 

1) I have built the AS2 server but I am facing issue with signing and verification of signatures, it is strange but I did not face this issue when I had to do the same for the client. I will debug and update once done with that.

2) Currently partner and organization information is stored in ini file, my plan is to create models AS2Partner and AS2Organization with a one to one relationship with a Partner. This way all user needs to choose for outgoing comms is comm type AS2, partner and orgnization is decided based on from and to partner fields.

yes.

3) Also I think AS2Transaction model needs to be created to log all movements as I dont think its possible to integrate with normal Bots Transactions. This will also make it easier to implement ASYNC MDN mode.

I agree.
AFAICS AS2 server would include both server and client.
Logging could be to log file or via GUI...

kind regards,
henk-jan

I will post further updates as I progress.

--

You received this message because you are subscribed to the Google Groups "Bots Open Source EDI Translator" group.

To unsubscribe from this group and stop receiving emails from it, send an email to botsmail+u...@googlegroups.com.

[henk-jan ebbers]

hi Abhishek,

what did you do with the problems with the crypto-libraries?

kind regards,
henk-jan

On 06/30/2014 01:52 PM, Abhishek Ram wrote:

[Abhishek Ram]

On 08-Jul-2014, at 10:13 pm, henk-jan ebbers <eppye...@gmail.com> wrote:

hi Abhishek,

what did you do with the problems with the crypto-libraries?

I am using M2Crypto for signing and encryption, the only problem with this library is that it is no longer maintained and it does not support newer message digests i.e. the SHA2 family.

I want to complete the code using this lib so that once Cryptography comes of age all we need to do is change the signing and encryption part of the code.

You received this message because you are subscribed to a topic in the Google Groups "Bots Open Source EDI Translator" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/botsmail/vby7iwQNBLg/unsubscribe.
To unsubscribe from this group and all its topics, send an email to botsmail+u...@googlegroups.com.

[Abhishek Ram]

Hi All,

I know its been a while, but I have finally managed to finish this tool. I took more time since I chose to implement the tool as a seperate Django App. I had to do this as I could not find a simple way to integrate with the Bots app. I have also heavily borrowed from Bots in terms of helper functions and UI to speed up development.

The app implements all the functionality specified in the RFC 4130.

The app still has a few drawbacks though:

• Uses M2Crypto for encryption and signature, the library is not maintained
• Does not support python 3 and M2Crypto does not support it.
• Only supports the sha1 algorith for signing messages and MDNs.

The plan is to migrate from M2Crypto to pyca/cryptography once the necessary features have been implemented in that library.

The github link is https://github.com/abhishek-ram/pyas2 and the pypi link is https://pypi.python.org/pypi/pyAS2.

Regards

Abhishek Ram

[Kurohai]

Perfect timing! I was thinking of starting this project myself. I'll be trying it out this week. Thanks!

[Klaas Feenstra]

Hi Abhishek,

Thanks, it took me some time, but finally I could setup the AS2 server and send some testmessages to the mendelson test server. Tomorrow I will try to get it working on Linux. I also tried to get it working adding this app to bots, but here I didn't succeed, so I made a new django app. I am a beginner with python and django, so I have no idea how to do the migration for the bots app, because I don't find the manage.py

One question: I do receive this warning: Message Integrity check failed. What is this? I also configured xml and send a xml file, but I got also this warning.

Thanks again,

Klaas

--

[Klaas Feenstra]

I installed on windows, but to get M2Crypto working, I had to install openssl. See link.

http://slproweb.com/products/Win32OpenSSL.html

[Abhishek Ram]

Hi Klaas, 

Please go ahead and create an issue on github for this, https://github.com/abhishek-ram/pyas2/issues

I will also update the installation steps i the readme file to reflect the fact that a new django project needs to be created for this app.

Regards

Abhishek Ram

You received this message because you are subscribed to a topic in the Google Groups "Bots Open Source EDI Translator" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/botsmail/vby7iwQNBLg/unsubscribe.
To unsubscribe from this group and all its topics, send an email to botsmail+u...@googlegroups.com.

[Klaas Feenstra]

Hi Abhishek,

I am testing now to receive documents, but what is the receive url?

regards,

Klaas

[Abhishek Ram]

The receive URL is https://ip:port/pyas2/as2receive, I will update the docs.

[Klaas Feenstra]

Thanks, so the receive URL is independent from organization..

[Abhishek Ram]

Yeah organisation is decided based on AS2 specific HTTP headers.

[Kurohai]

I haven't used the Mendelson test server before, so I don't know if it supports exchanging keys. It sounds like one side or the other isn't using the pki keys, or is using the wrong one.

[Kurohai]

Klaas Feenstra  How did you get the Mendleson test server set up? I put keys in for my organization and partner, but I keep getting encryption errors like this.

Failed to send message, error is 140589670004480:error:0906D06C:PEM routines:PEM_read_bio:no start line:pem_lib.c:703:Expecting: ANY PRIVATE KEY

On Thursday, April 30, 2015 at 5:31:03 PM UTC-5, Klaas Feenstra wrote:

[Kurohai]

Abhishek Ram, if I write up a tutorial about how to install, configure, and test with the mendelson server on Linux, would you put a link on the readme?

[Abhishek Ram]

Go ahead and create a pull request and update the readme file on github, i will approve.

[Klaas Feenstra]

HI Kiroshai,

I don't remember what I did with the keys, maybe I imported it with Portecle and exported as PEM.. Afterwards separating the private and the public key and then importing..

Kind regards,

Klaas

[Klaas Feenstra]

Sorry Kurohai,

Made a mistake with your name.. Maybe we can arrange some test sessions to send messages between each other..I already did different test using RSS bus as2 server, freeAS2 and mendelson. The problem is that most are installed local. I have also a Pyas2 server in the cloud installed for testing.. Sending from Rss bus and Mendelson to Pyas2 was giving me some errors. (Rss bus depending of the version, with the latest version I could send correctly to Pyas2)

Kind regards,

Klaas

[Kurohai]

That's a good idea, Klass. I'd like to do some more testing soon.

[Harold DeWayne]

Did anyone ever get this written up??  I _REALLY_ need to see the writeup so I can try to implement this on my end too.

Thanks.

[Kurohai]

I'm still working on it. I got distracted making up a docker image for it.

Do you need Linux or Windows?

[Charles Peterson]

Hi-  Just chiming in here...    I would be interested in a Windows version.  

Thank you for all the work on this.

Charles

[Harold DeWayne]

I am running on a Linux (CentOS) box.

Thanks. for all the work you've already done on this.

[Harold DeWayne]

Hi all...

I have successfully followed the directions from the first two steps above (and from many other sites) to get this ...working (??)... 

Here's how (I am running on a CentOS 6.6 box at this point) I did it:

1. I verified that the "requests" module is available for my python (2.7) installation
2. I installed M2Crypto from the yum repositories
   
3. I installed Mendelson AS2 community edition, correcting the CLASSPATH at the top to
   
• PATH=as2.jar:jetty/start.jar:jetty/lib/servlet-api-2.5-6.1.1.jar
  
4. and used sed as below from the Mendelson install direcory to change the port for Mendelson because both BOTS and Mendelson wanted to use port 8080
•  find . -type f | xargs -I {} grep -l 8080 {} | xargs -I {} sed -i "x/8080/9090/g" {}
  
5. Then I installed Portecle
6. I ran Portecle and opened the certificates.p12 file then deleted the Key1 and Key2 files and created a new "keypair"
   
7. I copied the as2lib.py and testas2send.py to the appropriate BOTS folders and updated the bots.ini file, adapting the file paths for encrypt-keys and sign-keys
8. Then I created a route with outgoing channel testas2send, no incoming channel and translate=Nothing
9. Finally I ran the new route. 
   

The all-runs report has a green check for state, but I have no files or data to test with so I can only assume that the green check means it worked.   I appear to be reaching the Mendelson test server but I cannot be certain.

This is my first foray into AS2, so if anyone can let me know if I'm doing something wrong (or even right ;) )  I would appreciate the help.

Thanks for any help anyone can give.

Harold DeWayne

[Harold DeWayne]

BTW:  Thanks again, Abhishek for all the work you've done on this... I could NEVER have gotten this far without what you started.

HD

[Harold DeWayne]

Has anyone using this had an issue with Drummond Certification as far as your trading partners are concerned?

[Klaas Feenstra]

Hi Harold, 

As I understand, you installed the mendelson AS2 server and not the PYAS2 server of Abhishek Ram. There is a test server that you can check on the mendelson site. We were using this test server for testing the PYAS2 solution.

The mendelson solution is not Drummond Certified, but probably would pass the test if they would.

Kind regards,

Klaas

[Abhishek Ram]

Hi Harold,

The process that is mentioned in this thread(copying files) is an old one. To install and test pyAS2 please go through the instructions at https://github.com/abhishek-ram/pyas2.

I have tested pyAS2 with Sterling B2B Integrator which is drummond certified, so I think we are good in this regard.

But still since pyAS2 has still not been rigorously tested it is still in the alpha release phase.

Rwgards

Abhishek Ram

[Wassily]

Hi all 

Have no experience with AS2, but want to move away from FTP. Is anyone using pyAS2 integrated with BOTS now ? Or still aiming for Mendelson. 

Intention is to have it run on a cloud provider, such as Azure or AWS or even containerised. 

Wassily