Some questions I ran into while trying to do implement an IMAP proxy

[Michael Elsdörfer]

Hi everyone,

I've been working on a JMAP server that would be a proxy in front of IMAP, but notably, without storing any data locally, like the existing Perl proxy does. The code is here: https://github.com/miracle2k/jmap-python/blob/master/jmap/server/modules/imap.py

It's very much a work in progress, but it can answer basic queries. So far, so good. 

I am not sure if this kind of proxy (not having all data available locally, just working with an IMAP backend) was an intended use case when designing the spec, but it certainly seems *very close* to feasible, save for some things that seem difficult to me. 

Here are my question. I would appreciate any time people have to look them over.

- When only doing IMAP-passthrough, it does not seem feasable to offer the /changes
  endpoint at all. However, it is possible to offer /queryChanges within a single inbox
  filter. Does that sound about right? I am thinking a client which throws away *all*
  cached emails when /changes fails would be missing out on the ability to get the
  /queryChanges on a per-mailbox basis. But how the client know that?

- In `Mailbox/get` or `Mailbox/query` method, since I cannot really return a mailbox state (I don't think?),
  I use the current timestamp. However, we would not want the client to throw away the full local
  state in this case, right?
  I mean, the state indicates that whatever is in the cache cannot be trusted. But combined with
  the fact that there is no ability to query changes, the client will just have to accept that
  it will have to do poll refreshes whenever it desires. But there isn't really a way for
  the server to tell the client: "I don't do changes, there is no stable state value".

- Implementing threads is though. For collapsing an `Email/query` response, we can use
  `THREADS`. But there isn't really a good way to answer the `Thread/get` resource, except
  in the common case where `Threads/get` is executed as part of a multi-method request. That is,
  it might be feasible to keep the response from `THREADS`, acquired during `Threads/get` in memory
  for some time to answer any `Thread/get` call the client might issue after that.


The common theme here, I think is this: This kind of proxy functions, in effect, as the
"online" model defined in RFC 1733. It seems to me that it is entirely possible to map this
to an "online JMAP" model, that is, the client sends a request for every view it want to display:

- Read the mailboxes on every app start (and occasionally refresh).
- Page through the mails in the open folder (with the right IMAP extension, we can even support
  a limited `queryChanges`).
- If a threaded view is desired, use collapseThreads=true while paging.

This could be a working client. However, maybe the full JMAP feature set cannot be supported: Maybe Thread/get has to be limited.

So, I am not sure. Am I missing something? What would the place for an "online-IMAP" proxy in the JMAP ecosystem be? Would it be forced to remain a non-standardized subset of JMAP that only works with clients that are lucky to not use more than what is provided?

Michael

[Neil Jenkins]

Hi Michael,

I am not sure if this kind of proxy (not having all data available locally, just working with an IMAP backend) was an intended use case when designing the spec, but it certainly seems *very close* to feasible, save for some things that seem difficult to me. 

The spec is designed to allow a consistent JMAP and IMAP interface to the same data, but not constrained to acting as a proxy on top of IMAP (the feasibility of which depends hugely on which IMAP extensions the server supports).

You can probably write something that's technically spec compliant (e.g. return cannotCalculateChanges error for every call to /changes), but it's probably not going to be very efficient without some custom IMAP extensions.

- When only doing IMAP-passthrough, it does not seem feasable to offer the /changes
  endpoint at all. However, it is possible to offer /queryChanges within a single inbox
  filter. Does that sound about right? 

Most IMAP operations operate only on a single mailbox, whereas /changes is across an account. So yes, this is unlikely to be feasible to do sanely using just IMAP operations. With a simple (but common) filter of just inMailbox: Foo, implementing /queryChanges should be feasible (as long as the server supports at CONDSTORE/QRESYNC).

I am thinking a client which throws away *all* cached emails when /changes fails would be missing out on the ability to get the /queryChanges on a per-mailbox basis. But how the client know that?

Because most of the message data is immutable, if /changes fails the client only actually has to refetch keywords/mailboxes. Having said that, it's still not great from an efficiency standpoint.

- In `Mailbox/get` or `Mailbox/query` method, since I cannot really return a mailbox state … But there isn't really a way for the server to tell the client: "I don't do changes, there is no stable state value".

No, because we want this to be mandatory in a compliant JMAP implementation. A server without this is going to be way less efficient for clients, and part of the JMAP idea is guaranteeing a much higher baseline clients can rely on (a server can say it supports "IMAP", but only support RFC3501 with zero extensions, which is practically useless).

- Implementing threads is though.

If you don't have server support, you probably can't do this properly. You could return every message as its own "thread", which would be technically spec compliant but not very useful.

Basically, without being able to guarantee a lot of help from the IMAP server (e.g. RFC8474 so you can get proper ids for your mailboxes/threads/emails) it's going to be very hard to write a generically useful proxy while remaining stateless. So it depends what your goals for the project are really; if it's just learn more about IMAP and JMAP, then great! This will probably be a very interesting exercise. But if it's for real world use I think it might be worth constraining the problem somewhat first.

Neil.

[Michael Elsdörfer]

Thanks - something to chew on while I work on it further.

Michael