MTA question

[Andrew Tereskin]

Hi everyone!

I'm going to start my project for large scale email system. As I see in architecture overview ElasticInbox is MDA (Mail Delivey Agent). What can you advice for MTA (Mail Transfer Agent)?

Is it possible for example use Mailgun (http://www.mailgun.com/) for this purpose?

Thanks!

With best regards,

Andrew

[Rustam Aliyev]

Hi Andrew,

I'm not familiar with Mailgun, but any MTA which supports LMTP will do. We use Postfix which is de facto standard MTA on many systems. With ElasticInbox design you can scale MTAs horizontally.

--

Rustam.

--
You received this message because you are subscribed to the Google Groups "ElasticInbox" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticinbox...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Andrew Tereskin]

Rustam,

As I wrote in first letter, I'm currently researching an architecture for million-user scale email system. I have several question regarding ElasticInbox. 

Can you show some doc where I can read about deployment of Postfix (with scaling and fault-tolerance in mind), that could be used as MTA for ElasticInbox? 

In your talk (http://skillsmatter.com/podcast/nosql/cassandra-storage) you mentioned on "future planc" slide that you had plans for IMAP (as I see, POP3 is implemented now), built-in filtering rules, threading. These are things I'm planning to be working on and so maybe in future it could be contributed back in ElasticInbox. So can you give me an advice what are you thoughts of where (and how) in ElasticInbox that features can be introduced?

Thank you!

With best regards,

Andrew Tereskin

-- Andrew Tereskin

[Rustam Aliyev]

Hi Andrew,

I don't have particular guide for setting up with postfix, but you can easily find tons of tutorials with various postfix deployment scenarios. We may add this kind of tutorial at some point.

Regarding future plans:

IMAP - this is probably most complicated component. IMAP has notion UID which in RFC (http://tools.ietf.org/html/rfc3501#section-2.3.1.1) described as unique within mailbox and "assigned in a strictly ascending fashion in the mailbox". This is really complicated to achieve in distributed system without doing global locking. Current implementation of Cassandra counters is unreliable for this. The good news is that CAS support planned for C* 2.x (https://issues.apache.org/jira/browse/CASSANDRA-5062) which can solve this particular problem. Overall IMAP is very complex and old protocol. I wish there would be more modern, REST based standard.

Having said that, we haven't really investigated IMAP implementation. First option would be James Protocols IMAP (http://james.apache.org/protocols/imap4.html) as we already use LMTP and POP3 modules form this project. If you want to play with, this is good place to start.

Filtering rules - this should be trivial. There's standard protocol called SIEVE (http://en.wikipedia.org/wiki/Sieve_%28mail_filtering_language%29) and Apache James has jSieve implementation (http://james.apache.org/jsieve/). We can add this simply by adding extra filter chain (e.g. SieveMailFilter) in com.elasticinbox.lmtp.filter. Rules can be stored in Cassandra, under user's account for example. We'll need to add REST API for managing SIEVE rules as well.

Threading - to enable threading we'll need to have additional secondary indexes where index key will be based on "Message-ID", "References", "In-Reply-To" headers. When message is stored it can be added to the thread index. This is just initial thoughts. We'll need to take into account IMAP THREADS extension when designing this (http://tools.ietf.org/html/rfc5256)

Search - I was thinking to implement pluggable interface which could be then extended and used with existing distributed search systems such as ElasticSearch or Solr. Again, need to keep in mind IMAP SEARCH (http://tools.ietf.org/html/rfc3501#section-6.4.4)

There are no timelines to any of these features, but for us search has the highest priority among the item listed above.

Hope this was helpful and I would be happy to help with any these.

Many thanks,
Rustam.

[Andrew Tereskin]

Hi Rustam,

What development setup can you advice for working on ElasticInbox? I tried both IntelliJ Idea (my favorite, but seems to have some lack of support OSGi) and Eclipse (installed maven plugin and bndtools, but have large number of dependency resolve issues).

With best regards,

Andrey Tereskin

[Rustam Aliyev]

Hi Andrew,

I personally use Eclipse, but Idea should work equally well. You don't really need extensive OSGi support. In fact in my Eclipse setup I don't use any OSGi plugins. All dependencies are described in Maven. Make sure that when you import project you import it as Maven.

Did you see this document? https://github.com/elasticinbox/elasticinbox/wiki/Development-environment

Let me know if you still have problems.

--

Rustam

[Andrew Tereskin]

Hi Rustam,

Thank you! Simple setup with elasticinbox/modules/* maven modules work in Eclipse, but when import that modules in Idea have dependency resolving problems. What does mean section in elasticinbox/modules/lmtp/pom.xml?

<!-- | uncomment to add all imported (non-local) bundles to your compilation classpath

    <dependency>

      <type>pom</type>

      <groupId>${parent.groupId}</groupId>

      <artifactId>provision</artifactId>

      <optional>true</optional>

    </dependency>

-->

With best wishes,

Andrey Tereskin

[Andrew Tereskin]

Finally got everything working on Idea. 

First of all I removed all previous created Idea module files *.iml: find . -name "*.iml" | xargs rm.

Then: 

// File -> Import Project 

// Select "Import project from external model" - select Maven -> Next

// Select "Search for projects recursively" and "Create IDEA modules for aggregator projects (with pom packaging)" -> Next -> Next

// Select only "com.elasticinbox:elasticinbox-parent"

--

Andrey

-- Andrew Tereskin

[Rustam Aliyev]

Thanks Andrew! I'll add this to the guide.