Mailarchiva no longer importing or receiving emails from IMAP connection after upgrade

[cafet...@gmail.com]

After updating the Mailarchiva install to version 5.3.12, the server stopped receiving emails from the defined IMAP connectors, and could no longer import messages manually from the IMAP connectors. The import jobs failed with the following error message:

failed to retrieve messages during polling operation:File with name 'v_blob.mgn.sbc' does not exist in storage 'archiva' DB name="archiva"

Updating to version 5.3.14 did not resolve the problem. I have searched for messages re. this error and have not been successful in finding any remediations.

Please advise how to fix this problem.

Thank you.

Duke Fleming
CAFEtech Los Angeles

[jamie]

Orient db is corrupted, mostly likely due to the fact that the server wasn't shutdown correctly. You can stop server, rename the existing data at /var/opt/mailarchiva/ROOT/database to /var/opt/mailarchiva/ROOT/databas_bak, start server. The effect of this will be that folder structures will be lost, but not the emails themselves. After deleting the DB, if you rerun the import the folders will be reconstructed and there should be no duplicates in your archive. If you wish to preserve folder structures, you will need to download Orient Db, run the console, connect to the database at  /var/opt/mailarchiva/ROOT/database/archiva.db (i think), and perform a repair. Orient DB can be downloaded from https://orientdb.com/docs/2.2/Console-Command-Repair-Database.html and the repair instructions are outlined at https://orientdb.com/docs/2.2/Console-Command-Repair-Database.html.

[cafet...@gmail.com]

Thank you for your information. That appears to have solved the problem of importing messages manually from the IMAP connections. Your assitance is greatly appreciated.

Unfortunately, unread messages are still not being auto-imported either by push or by pull from the IMAP connections. In the debug log, there are the repeated message sequences as follows:

com.stimulus.archiva.rn: failed to execute IMAP idle {serverName='outlook.office365.com',port='143',userName='xx...@xxxxxxxx.com'}
at com.stimulus.archiva.fw.p(MailArchiva:640)
at com.stimulus.archiva.fw.f(MailArchiva:308)
at com.stimulus.archiva.fw.run(MailArchiva:143)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: javax.mail.MessagingException: IdleManager is not running
at com.sun.mail.imap.IdleManager.watch(IdleManager.java:198)
at com.stimulus.archiva.fw.p(MailArchiva:633)

unusre what that is all about.

Cheers.

[jamie]

If you wish to import data from Office 365, you should be using the web services import function. If you wish to archive from Office 365, then you ought to be using the SMTP journaling approach. Please stick to the beaten tracks outlined in the online help site and you should take out of trouble.

[jamie]

We changed the imap idle functionality recently to use an IdleManager class designed by Oracle. Previously, our code handled the Imap Idle operation. Clearly, there is an issue there. If you disable IMAP idle in the Connections->IMAP Connection class, then it will use polling and skip this code altogether.

[cafet...@gmail.com]

I appreciate the feedback. Thank you.

Just as an aside. I am testing this product as a proof of concept using my own environment before deploying it into a production environment.

I tried using the FAQ for Office 365 to setup a connection using an impersonation account in Azure Active Directory; that ultimately did not work. I have also tried creating Exchange connections using all of the available options and those have not worked either. The only connector I could actually get to work with Office 365 was the IMAP connector, not ideal, and I most definitely prefer to use a connector that is neither POP nor IMAP.

I already resell email archiving solutions for Intradyn and Arcmail. I am looking for a solution that is a little less costly for the customer with 1 - 5 employees who is using Office 365 for email, and needs an archiving solution for compliance purposes (attorney, CPA, insurance broker, etc). If I could get your solution to work in an SBS/Essentials environment that is a hybrid on premises / cloud architecture, I would be willing to resell it as well. Unfortunately, right now, I am not comfortable deploying the solution in the wild in a production environment for a small business that has no resources for the solution other than my company.

Cheers.

[jamie]

I double checked the IMAP code. I have so far been unable to reproduce the problem in our test environment, but I'll keep trying. Regarding your comment. Setting up Office 365 to connect to MailArchiva via the SMTP protocol is trivial and should not represent any problems. You just

1) create an SMTP listener in MailArchiva to listen on port 25

2) port forward an IP address on port 25 on firewall to the MailArchiva server.

3) create a DNS record in a domain to point say mailarchiva.company.com to MailArchiva external ip address on port 25
4) create a journal rule in MailArchiva with the address arc...@mailarchiva.company.com

Mail will be routed directly to MailArchiva's SMTP server. This setup should take no more than 5-10 minutes to complete. Regarding folder sync via the Exchange connection, this step is more complicated when using the On Premise version of MailArchiva. In contrast, MailArchiva Cloud this connection is setup automatically during the one click signon process. With On Premise, you need to create the mailarchiva app in Azure, etc. There is no way that I know around it unless we embed the MailArchiva Azure keys inside the distribution (insecure!).

Until recently, I haven't thought it a strong use case for on premise customers to sync data in Office 365, but it seems there is demand. Hence, you'll notice from the change log we have been working furiously to clear up all issues. I setup Exchange sync for a customer a few days ago. I came across only one snag which was to consent before performing the lookup.

If you are considering offering MailArchiva to your customers, then I would suggest that you use the MailArchiva ISP Edition product, since it can also be setup such that you have a one click setup signon process with Azure. When setup right, your customer would not have to perform any configuration steps at all. After logging in, it should just work.