Java imap performance

[smartmobili]

I found a post about java perfomance and gmail. The sample code also
shows the use of FetchProfile :
http://stackoverflow.com/questions/2538481/javamail-performance

One suggestion is to use multiple imap connections, maybe this is
something we should study...
For now we use dovecot-2.x as imap server but tomorrow we might allow
user to specify user to retrieve emails
from their preferred providers (gmail, yahoo, ...).

[Victor Kazarinov]

It depends on how much simultaneous users is planned to request data
from server. How much count of simultaneous connection to kairos is
planned?
If there will be really huge amount of users, we even need several
servers and make (develop) or use some existing (and integrate in our
system) load balancers (LB) to spread usage between servers.
If there will be 1000 users simultaneously, we can stay with one
server, and of course we need several connections to imap servers. We
can create a pool of connections. I see some related to pools in
ImapService.scala (e.g. imapConnectionPoolSize constant).

One of caveat of multiple IMAP connections is because an external IMAP
server (e.g. gmail) possible can think that so much simultaneous
connections from one IP address (one server) is an DDOS attack and can
block access from that IP. We should take care of that in some way.

[Victor Kazarinov]

I investigated that http://stackoverflow.com/questions/2538481/javamail-performance
example. By turning debugging on, we can see IMAP commands. To turn
debug, need to call this at beginning: sessionIMAP.setDebug(true);
My thoughts here:
If we download a bunch of message headers at once with one request, it
should be pretty fast. But if we request header for each message id,
it will require big count of requests, and this is slow. Because there
is a delay between client and server, and we talking about requests in
one programming (execution) thread, so all commands/requests is made
one by one, so while one request is not done, it will not follow to
next request. So there will be a delay between requests. And this is
happened in that example in javamail-performance article.
So, as result, just note that making one request for 50 mail headers
is faster that requesting 50 times for 1 header.
But why this is happened in javamail-performance if there is used
FetchProfile class, designed to download headers at once with one
request? Again, I investigated IMAP debug output of that example:
If we comment line "Address[] recipients =
message.getAllRecipients();" then that javamail-performance starting
to work fast, really fast, it returns result immediately after fetch
command. But with message.getAllRecipients() it works slow, because
regarding DEBUG output, it made each time new request to IMAP server
when message.getAllRecipients is called. That recipients should be
already in the header (and I see it in DEBUG OUTPUT when calling
fetch). Regarding debug output, calling message.getAllRecipients each
time calls such IMAP command: FETCH 1 (BODY.PEEK[HEADER.FIELDS
(Newsgroups)]). This is strange. Seems like some bug in java mail
classes? There is used IMAPFolder class. I found java doc at strange
location here: http://javamail.kenai.com/nonav/javadocs/com/sun/mail/imap/IMAPFolder.html
Strange, it tells that this API is EXPERIMENTAL. Perhaps it contains
some bug, or perhaps this is google require that "Newsgroups" header.
This explains why other java developers experience problems with gmail
and not with other IMAP servers. To workaround it in that javamail-
performance example, there is need just simple fix for gmail:
If I explicitly add "Newsgroups" field into fetch command, then it
starting to work fine and fast. So fixed part of code from javamail-
performance article looks like:
fp.add(FetchProfile.Item.ENVELOPE);
fp.add("Newsgroups"); // this line is new
folder.fetch(msgs, fp);
After that fix, code from javamail-performance works fast and there is
no need to make any other optimizations.

[Victor Kazarinov]

Now my thoughts regarding pools. In kairos I think we need keep
several opened connections to imap servers. Not one connection per
server, but an pool of connections regarding count of currently
connected users (e.g. for example if we have 100 users, we can keep 10
connections opened to imap). I will describe, why we need pools. If we
will have just 1 connections per server, it will be slow, because it
will download results sequentially even if we have 1000 users
simultaneously. E.g. 1000 users via 1 connection to IMAP - this is
slow. On the other side, if we will open 1 connection per user, it
will feel like DDOS attack from imap server side view. Because in this
case it will try to open 1000 IMAP connections for 1000 users. So as a
balance between that 2 cases, we should keep an pool. Which will open
a less count of IMAP connections, that currently users is connected,
but enough to feel that all system working fast.

But still need to think what to do with external IMAP servers. Still
open question, how much users there will be per one kairos server? I
mean per one IP address of server. Big count of connections from one
IP to an mail (imap) server can be treated as an kind of attack so
IMAP server can block all connections from that IP, so kairos can fall
into situation when it have no access to IMAP server at all. And no
one knows, how exactly much count of such connections can be treated
as an kind of attack. Perhaps 10 simeltanous connections is fine.
Perhaps 100 is not fine. So pool limit can help here. So, we can limit
size of our pool with for example 10, or 30, for example.

On Jan 8, 4:28 pm, Victor Kazarinov <vic...@gmail.com> wrote:
> I investigated thathttp://stackoverflow.com/questions/2538481/javamail-performance

> location here:http://javamail.kenai.com/nonav/javadocs/com/sun/mail/imap/IMAPFolder...

[Victor Kazarinov]

I just realized that IMAP perhaps is not support to switch to other
users in one physical connection (session) (this need to investigate,
IMAP contains LOGOUT command, but I don't know if it support to LOGIN
right immediately after LOGOUT to switch to other user in the same tcp
session). In this case creating pools is not possible, because for
each user we need unique one connection to IMAP.
So now need to think about architecture of pools/imaps and etc. If we
will have at least 1 IMAP connection for one user, that will really
bad, because IMAP server will block so huge count of connections from
one IP (ip of kairos server). (imagine 1000 simultaneous users, will
have 1000 opened connections from kairos server to imap server). If
this is your imap server (e.g. dovecot), perhaps this is fine, but
with an external IMAP server this is not fine.

[vincent richomme]

Hi,

Thanks for your very good analysis. You are asking very good technical questions and for now I don't have any answers.

About java mail before to use it I hesitated because when I look at open source webmail, they generally redevelop their

imap libraries to have a better control about imap commands. 

I am also concerned about the language choice because I wanted to add only one method to create an imap folder and

it's not that easy because of cardano and scala. scala is really different from usual language like c#, java, ruby, python

because it's functional and I am not very productive with it. I am also afraid that is the case for some other developers that 

could want to join the effort.

Do you feel comfortable with scala ? Wouldn't be easier to rewrite it in java/CP2JavaWS or in groovy/ruby/python/?

I have also heard that author of CP2JavaWS is currently porting it to PHP.

Maybe before working on imap, connections and so on we should implement something simpler like folder creation, it 

would help us to define some generals thing like coding styles, error management (how do we display an error when something failed, ...),

naming convention. For instance folder inside kairos is represented by an object called SMMailbox but maybe we should rename into SMailFolder

because to me a mailbox is the whole mail account but my english is not that good so maybe I am mistaken.

About coding style, I find it sometimes a bit difficult inside javascript code to know if the variable is a member or a local,

maybe we could prepend an _ to signify that it's a member variable except for @outlet variables.

[smartmobili]

About imap libraries I found a good one (http://www.vmime.org/) except
for the license that is GPL or commercial and it would implies
to create a C wrapper to be able to call it from backend. I could be
interested in writing a mail library but now I have used to many
high-level languages I am reluctant to write it in C or C++ with a C
wrapper. Maybe in the future I will try to write something simple in
vala because this language offers
a high level abstraction with a c# syntax and that at the end
generates plain-C.
But for no this is a bit early because what I want is to have a
working webmail, so we will use the existing and we'll see later
if we can optimize/customize.

[smartmobili]

So what I propose is to focus on a simple task to validate that we are
going in the right direction.
COuld you try to implement the method to create a folder ?
For now we are only creating a folder in the UI but nothing is created
on the mail server.
The first thing to do is to implement inside createFolder (I already
started it but I think we cannot use Either return value), so I
propose to
follow existing methods and write something like that (not tested) :

folderName is a string holding a folder "path", ex Folder1/Folder2/
Folder3
so algorithm should split this string and create the all the folders.


def createFolder(folderName: String): List[SMMailbox] = {

}

In mailsourceviewController.j:

- (IBAction)addMailbox:(id)sender
{
if (![self canAddMailbox])
return;

[_imapServer createFolder:[selectedMailbox name]
delegate:@selector(imapCreateFolderDidReceived:)
error:nil];

}

- (void)imapCreateFolderDidReceived:(SMMailbox)mailBox
{
//CPLog.debug(@"%@%@", _cmd, mailBox);
var newMailbox = [[mailController mailAccount] createMailbox:self];
[self reload];
[self selectMailbox:newMailbox];

var view = [self view],
rowIndex = [[view selectedRowIndexes] firstIndex];
if (rowIndex !== nil && rowIndex !== CPNotFound)
[view editColumn:0 row:rowIndex withEvent:nil select:YES];

}

there is also a bug in the current implementation because I think that
when right-clicking on a folder and choosing create folder, the folder
is not created as a child of the folder but on the same level.

[smartmobili]

Ok so finally I managed to write that but this is not working, it just
gives a raw idea of what we need.

def createFolder(folderPath: String): List[SMMailbox] = {
var newFolder: List[SMMailbox] = List()
var curPath = ""

try {
// For each folder from given path => create folder if it doesn't
exist and select it as current folder
val folders = folderPath.split("/").foreach { folder =>


val curFolder: Folder = store.getFolder(folder)
if(! curFolder.exists) {
if(curFolder.create(Folder.READ_WRITE | Folder.HOLDS_FOLDERS |
Folder.HOLDS_MESSAGES)) {
curPath = curPath + folder + "/"
}
else {
// Something bad happened
throw new IllegalStateException("Exception thrown");
}
}
else {
//folder already exists
curPath = curPath + folder + "/"
}

// maybe we should add a new parameter to SMMailbox to separate
its name and its full path
// for instance a folder Folder1/Folder2/Folder3 would have the
following values:
// path = Folder1/Folder2/Folder3 or maybe only Folder1/Folder2
// name = Folder3
// because if we don't separate it we will have to do it in
javascript
newFolder = List(new SMMailbox(folderName, 0, 0))
}
} catch {
// if there was an error we will return an empty list
case e: Exception => e.printStackTrace(); newFolder = List()
}

newFolder
}

Wat I would like to achieve is to implement a way of creating folders
either from the same level or as child of some existing folders.
The easiest is to call createFolder with a string represeting its
path :

createFolder("RootFolder") // Will create a root folder
createFolder("RootFolder/SubFolder1/SubFolder2) // Will create
SubFolder1 as a child of RootFolder and SubFolder2 as a child of
SubFolder1
createFolder("RootFolder/Level1")
createFolder("RootFolder/Level2")

So after those calls I should get the following tree:

.
└── Others
└── RootFolder
├── Level1
├── Level2
└── SubFolder1
└── SubFolder2



What I propose is that you try to implement this feature for the next
week and if it's too complicated because of scala/java api,
maybe the decision to use java or something else will have to be
taken.



Have a good week end.

[Victor Kazarinov]

Quick question: when I right click on folder in kairos, there is not
appear any special menu, so I don't know how to use feature to create
subfolders. Or this part is not yet implemented in UI? But you said
that there is bug with subfolders creation, so I think it should be
implemented, but I can't find it. I tried in 2 browsers: Safari and
Chrome.

[smartmobili]

Really ? Last time I tried it it works. Just hopes I didn't commit
something that break it up.
Will try tonight.
Actually I changed my mind and now it's not a bug but a feature ;-)
Creating a subfolder is not a good idea in th current ui because we
don't have too much space to do it and
I am not sure the CPOutlineView control supports it by default.
Actually you can implement it inside the service method but we will
never pass more that one folder for the moment
so it will always be created at the root level.
However for your tests you can create sublevels folders and check it
works by loging in on mail.smartmobili.com with
webguest/webguest78.

[Victor Kazarinov]

Hi Vincent,

This is big message, you can find here answers to questions from
previous your emails, and other my thoughts.

=Context menu=
Regarding right-click context menu with subfolder creation, renaming
and folder deletion, I found that menu! :) I thought that this
functionality is on the site kairos.smartmobili.com, but now I figured
out, that it is only exists in development version or at new location
(http://bifrost.smartmobili.com/kairos/capp).

=Folder creation=
You asked me to implement folder creation. I made it. It mades folders
in ROOT as you suggested in last email (so no subfolders creation).
First, i made thoughts what we need except "folder creation at server
side" to make folder creation at all. This should be also
implementation at client side (you already showed parts of code for
mailsourceviewController.j) and also folder renaming, because after
folder creation in UI, we starting to edit its name, and it should
reflect changes at imap server.
Next, I investigated, how it was supposed to be in current
realization. I found all places in code and "TODO" marks in comments.
I found already existed code for mail folder creation (it is still
named as box not folder) : createMailbox function in SMMailAccount.j.
But full system was not done, e.g. folders only created in UI and not
at server. Here I describe how it works (here is events flow
described): first, is called code addMailbox() in
MailSourceViewController.j when user click button or menu item to add
folder. That code call createMailbox() function in SMMailAccount.j.
There is comment says that it really do nothing, because we not yet
set name to mailbox (it is OK). Next, UI switch to edit mode, and user
should enter the name for mail folder. After that this function
called:
- (void)outlineView:(CPOutlineView)anOutlineView setObjectValue:
(id)aValue forTableColumn:(CPTableColumn) in
MailSourceViewController.j
which calls "renameTo" function in SMMailbox.j. And there is comments
with "TODO" meaning that it is not implemented. There is commended
call to server function [imapServer renameFolder] which not
implemented. Bellow there in save() function we see commented call to
[imapServer createFolder] which also is not implemented.
So the end of this calling chain is undone. This is what I have
implemented. Function calls for [imapServer renameFolder] and
[imapServer createFolder] was commented but I disliked location of
that calls. "createFolder" call located in "save" function which
called for SMRemoteObject, so it will call createFolder each time when
object is just "saved"? This is not good (I'm not sure how it works,
this is part of cardano code i think). So, I made at server function
"renameOrCreateFolder". It try to rename folder if it exists, or it
will create new folder if previous one is not exists. Now I call this
function each time when editing folder name in UI is ended (e.g. when
renameTo function is called).

=How to test renameOrCreateFolder=
So, In last my commit I made this renameOrCreateFolder function. I
tested both, folder creation and folder renaming is works and after
relaunching browser folders is there (saved). Also note that we should
turn off cache to work. Without turning off, it will download list of
folders from cache and not from IMAP server so you will not see
changes in UI when using cache. But without cache it works slow.
Note: it very slow responds cause of current IMAP realization without
pools.
I suggest you to start server in debugger (Eclipse), and set
breakpoint in "renameOrCreateFolder" function. Then run browser, and
after creating and naming folder, wait. Wait when you will hit
breakpoint. For me it takes around 1-2 minutes. After that you can
continue run code in debugger. It will create folder at IMAP server.
Now you can reload web app in browser, and you will see that folder is
still there, re-downloaded from IMAP server, as expected.

=IMAP, architecture and pools=
I made debugging of "connect" function in ImapScala. Seems that during
just usual login to web app via browser, it makes 2-3 connections to
IMAP. This is what I talking about IMAP pools at server. With pool we
can have only 1 connection for 1 user to IMAP and keep it opened. This
is more efficient for resources at our server, and at imap server(s).
But we should think more deeply about imap and pools architecture. As
I said in previous messages, we should be careful with connections
count to IMAP server, to not over flood it. Even just 1 IMAP
connection per user is a huge count if we talking about 1000 users
simultaneously.

So currently "renameOrCreateFolder" function, and other functions is
responds slow. This implementation is temporary, later when all new
IMAP architecture will be done, we need use "IMAP pool" class (or
something like that) to achieve any IMAP-related functions. It will
work faster, because pool should keep opened connection to IMAP.
Currently we open NEW connection to IMAP right here in any our
function (like "renameOrCreateFolder" function), so user feel some
delay (in case of "renameOrCreateFolder" this is big delay), and also
IMAP server will receive huge count of connections.

=Scala=
I don't feel any problems with scala yet, so currently I feel
comfortable with it. I don't think that language is matters here. I
don't like current realization of ImapService.scala, because it have
useless cache, it open huge count of IMAP connections and etc (so at
end it responds slow for user). But it not depends on language.

=Naming=
I agree with you that Mailbox is not right name for "folder". So I
agree, that it can be renamed from Mailbox to MailFolder or something
like that.

=Syntax and conventions=
Regarding syntax and code styles and conventions. I think in Objective-
J we should use same style as in Objective-C by Apple, so we can refer
to Apple documentation here. So local variables is as is, and member
variables can be as is, or can be accessed via self.nameOfVariable
(e.g. self is added), and member properties is accessed via accessor
methods via sending message e.g. [self nameOfVariable] (which actually
calls getnameOfVariable). Current kairos code is OK for me.
Here is more info:
http://developer.apple.com/library/mac/#documentation/Cocoa/Conceptual/CodingGuidelines/Articles/NamingIvarsAndTypes.html#//apple_ref/doc/uid/20001284-BAJGIIJE

=IMAP libraries=
Perhaps later we can investigate, choose between different libraries
and etc. But currently this is not matters. Base functionality is
exists in current standard javamail. It is enough currently. All slow
responding of system is not because of library, but because it is used
wrong (e.g. call connect() and open many connections).

[smartmobili]

On 10 jan, 12:05, Victor Kazarinov <vic...@gmail.com> wrote:
> Hi Vincent,
>
> This is big message, you can find here answers to questions from
> previous your emails, and other my thoughts.
>
> =Context menu=
> Regarding right-click context menu with subfolder creation, renaming
> and folder deletion, I found that menu! :)  I thought that this
> functionality is on the site kairos.smartmobili.com, but now I figured
> out, that it is only exists in development version or at new location
> (http://bifrost.smartmobili.com/kairos/capp).
>

Yes it's available in the last version hosted on bifrost...

I see your point and I will think more about it but I agree with your
decisions

> =How to test renameOrCreateFolder=
> So, In last my commit I made this renameOrCreateFolder function. I
> tested both, folder creation and folder renaming is works and after
> relaunching browser folders is there (saved). Also note that we should
> turn off cache to work. Without turning off, it will download list of
> folders from cache and not from IMAP server so you will not see
> changes in UI when using cache. But without cache it works slow.
> Note: it very slow responds cause of current IMAP realization without
> pools.
> I suggest you to start server in debugger (Eclipse), and set
> breakpoint in "renameOrCreateFolder" function. Then run browser, and
> after creating and naming folder, wait. Wait when you will hit
> breakpoint. For me it takes around 1-2 minutes. After that you can
> continue run code in debugger. It will create folder at IMAP server.
> Now you can reload web app in browser, and you will see that folder is
> still there, re-downloaded from IMAP server, as expected.
>

I will test it tonight but I am really surprised because dovecot
server is on the same
machine so it shouldn't be that slow. Cannot say more as long as I
didn't test

Ok. If you feel comfortable I am ok with that.

> =Naming=
> I agree with you that Mailbox is not right name for "folder". So I
> agree, that it can be renamed from Mailbox to MailFolder or something
> like that.
>

Actually I found some code where they use mailbox to talk about folder
so maybe it's not
a such a bad name. Anyway I prefer folder terminology I find it more
relevant.

> =Syntax and conventions=
> Regarding syntax and code styles and conventions. I think in Objective-
> J we should use same style as in Objective-C by Apple, so we can refer
> to Apple documentation here. So local variables is as is, and member
> variables can be as is, or can be  accessed via self.nameOfVariable
> (e.g. self is added), and member properties is accessed via accessor
> methods via sending message e.g. [self nameOfVariable] (which actually
> calls getnameOfVariable). Current kairos code is OK for me.

> Here is more info:http://developer.apple.com/library/mac/#documentation/Cocoa/Conceptua...
>
Regarding syntax objective-J is a bit different from objc and I prefer
to use what obective-j generally adopts.
For instance if you look at cappuccino source code here :
https://github.com/cappuccino/cappuccino/blob/master/AppKit/CPWindow/_CPHUDWindowView.j
You will see that member variable use _
Another very good application writtent in cappuccino is available
here : https://github.com/primalmotion/Archipel/blob/master/ArchipelClient/AppController.j
I find the code very clear, outlet are put together and member
variables are suffixed with underscore.

> =IMAP libraries=
> Perhaps later we can investigate, choose between different libraries
> and etc. But currently this is not matters. Base functionality is
> exists in current standard javamail. It is enough currently. All slow
> responding of system is not because of library, but because it is used
> wrong (e.g. call connect() and open many connections).
>

Agree

Now I need to test more before being able to evaluate/answer some
other questions.

[Victor Kazarinov]

>I will test it tonight but I am really surprised because dovecot
>server is on the same
>machine so it shouldn't be that slow. Cannot say more as long as I
>didn't test

This is when testing on smartmobili.com server, then dovecot on the
same machine. But when I testing on my local computer, it makes
connections to dovecot located on remote (smartmobili.com) server, so
it looks like using external IMAP server. So there is delay between
scala/lift server and imap server.
2 minutes it takes because it also download all messages from imap,
because cache is turned off during folder creation tests.

> For instance if you look at cappuccino source code here :https://github.com/cappuccino/cappuccino/blob/master/AppKit/CPWindow/...

> You will see that member variable use _
> Another very good application writtent in cappuccino is available

> here :https://github.com/primalmotion/Archipel/blob/master/ArchipelClient/A...

[vincent richomme]

The question is why does it dowload all messages when creating a folder.

2012/1/10 Victor Kazarinov <vic...@gmail.com>

[Victor Kazarinov]

Because server just started in eclipse and server download all messages to show INBOX :)

--
With best regards,
Victor Kazarinov

[vincent richomme]

OK sorry I know why. Forget my last question.

2012/1/10 vincent richomme <v.ric...@gmail.com>

[vincent richomme]

Ok but when we create a new folder, focus is given to the new folder so it shouldn't select INBOX.

Of course after if I select INBOX and cache is disabled it should download all messages again.

2012/1/10 Victor Kazarinov <vic...@gmail.com>

[vincent richomme]

So maybe  you should work on the paging system first so that it download in the worst case only 50 emails.

2012/1/10 vincent richomme <v.ric...@gmail.com>