VM feature requests
rpgo...@gmail.com
This isn't really meant to be just me whining about things I'd like to
see in VM. I might even do some of these myself. It's just that,
since we don't have a bugzilla or anything, and since Google kindly
makes these usenet groups eternal (at least for an internet time
interpretation of "eternal"), gnu.emacs.vm.info can serve as a proxy
for a trouble ticket system.
Here are some possible areas for improvement:
1. searching folders:
a. ignore encoded binary blocks. I find that I often get bad
false positives when searching for strings in vm folders, because of
random strings in encoded binary (mime attachments). It would be dandy
if we could skip these when doing an isearch over the folder.
b. the ability to search in virtual folders.
2. MIME attachment and encryption: The encryption badness of VM is
primarily due to its reliance on mailcrypt. However, it is also
related to the way MIME attachment is handled, IIUC. In order to do
encryption right, we need to be able to properly stage the encryption
process so that it can happen after all the other attaching and
mimulation is handled. This i s quite different from the way mailcrypt
does things. The mailcrypt technique, AFAICT, is really not up to
handling encrypting of enclosrues, etc.
I note that hooking to gnu anubis *might* be a way to avoid doing
encryption in VM at all, and that handling this completely after email
sending might be preferable (even if not using anubis). I do not have
any sense of how good/bad/live the gnu anubis project is.
3. Make the code easier for others to deal with. A major impediment
to contribution to the VM code base is emacs-lisp's very crude data
structures. This is particuarly true of the mime stuff. I have many
times struggled through trying to figure out just what the mime data
structures look like. For example, here is what is effectively the
declaration of a data-structure:
(defun vm-mm-layout-type (e) (aref e 0))
(defun vm-mm-layout-qtype (e) (aref e 1))
(defun vm-mm-layout-encoding (e) (aref e 2))
(defun vm-mm-layout-id (e) (aref e 3))
(defun vm-mm-layout-description (e) (aref e 4))
(defun vm-mm-layout-disposition (e) (aref e 5))
(defun vm-mm-layout-qdisposition (e) (aref e 6))
(defun vm-mm-layout-header-start (e) (aref e 7))
(defun vm-mm-layout-header-end (e) (aref e 8))
(defun vm-mm-layout-body-start (e) (aref e 9))
(defun vm-mm-layout-body-end (e) (aref e 10))
(defun vm-mm-layout-parts (e) (aref e 11))
(defun vm-mm-layout-cache (e) (aref e 12))
(defun vm-mm-layout-message-symbol (e) (aref e 13))
(defun vm-mm-layout-message (e)
(symbol-value (vm-mm-layout-message-symbol e)))
;; if display of MIME part fails, error string will be here.
(defun vm-mm-layout-display-error (e) (aref e 14))
(defun vm-mm-layout-is-converted (e) (aref e 15))
(defun vm-set-mm-layout-type (e type) (aset e 0 type))
(defun vm-set-mm-layout-qtype (e type) (aset e 1 type))
(defun vm-set-mm-layout-encoding (e encoding) (aset e 2 encoding))
(defun vm-set-mm-layout-id (e id) (aset e 3 id))
(defun vm-set-mm-layout-description (e des) (aset e 4 des))
(defun vm-set-mm-layout-disposition (e d) (aset e 5 d))
(defun vm-set-mm-layout-qdisposition (e d) (aset e 6 d))
(defun vm-set-mm-layout-header-start (e start) (aset e 7 start))
(defun vm-set-mm-layout-header-end (e start) (aset e 8 start))
(defun vm-set-mm-layout-body-start (e start) (aset e 9 start))
(defun vm-set-mm-layout-body-end (e end) (aset e 10 end))
(defun vm-set-mm-layout-parts (e parts) (aset e 11 parts))
(defun vm-set-mm-layout-cache (e c) (aset e 12 c))
(defun vm-set-mm-layout-display-error (e c) (aset e 14 c))
(defun vm-set-mm-layout-is-converted (e c) (asef e 15 c))
I'm more of a Common Lisp programmer than an emacs-lisp programmer, but
this seems like the sort of thing that would really benefit from the
use of something like defstruct (and some documentation strings).
cl-macs would be one way of getting some better structure here (I note
with some dismay that the defstruct in cl-macs does not seem to support
documentation strings). Quite possibly there are others.
Cheers,
R
ew
I'm another "silent" one on the list. I have used VM in times long ago
and really liked it --- have the full power of an editor at my
fingertips and not a IMHO lousy "text widget" or whatever it is named.
For several months already I'm contemplating the idea of using VM
again. However, my playing around with some accounts and different
mailclients found exactly, what the rpgoldman (yet another Robert? ;-)
has stated before:
rpgo...@gmail.com wrote:
-8<---
> 2. MIME attachment and encryption: The encryption badness of VM is
> primarily due to its reliance on mailcrypt. However, it is also
> related to the way MIME attachment is handled, IIUC. In order to do
> encryption right, we need to be able to properly stage the encryption
> process so that it can happen after all the other attaching and
> mimulation is handled. This i s quite different from the way mailcrypt
> does things. The mailcrypt technique, AFAICT, is really not up to
> handling encrypting of enclosrues, etc.
> I note that hooking to gnu anubis *might* be a way to avoid doing
> encryption in VM at all, and that handling this completely after email
> sending might be preferable (even if not using anubis). I do not have
> any sense of how good/bad/live the gnu anubis project is.
>
I herewith offer to contribute some testing, I'm not in a position to
write lisp code at this point. I am using emacs22 from Debian's
experimental branch (emacs-snapshot).
And yes, I would support the idea of Robert maintain a devel branch of VM.
Cheers,
Erich
John J Foerch
> 3. Make the code easier for others to deal with. A major impediment
> to contribution to the VM code base is emacs-lisp's very crude data
> structures. This is particuarly true of the mime stuff. I have many
> times struggled through trying to figure out just what the mime data
> structures look like. For example, here is what is effectively the
> declaration of a data-structure:
>
You are right about the need for docstrings. Another thing I mean to look
into as soon as I get a round tuit is to see how much of the code duplication
can be abstracted.
--John
newsspam5...@robf.de
[...]
> 1. searching folders:
> a. ignore encoded binary blocks. I find that I often get bad
> false positives when searching for strings in vm folders, because of
> random strings in encoded binary (mime attachments). It would be dandy
> if we could skip these when doing an isearch over the folder.
> b. the ability to search in virtual folders.
For some time I have thought and played with the idea of creating
a backend for VM which does all the storage and search work.
As I got used to virtual folders, I felt the desire to get rid of
physical folders. I do not need them at all, they are only a
crude hack for the problem that Emacs gets unusable if a folder
exceeds a certain message count and/or file size.
I played a bit with C++, TCL and now Python+Luceen, but never had
the time to get it really working, but I think this is the way to
go.
> 3. Make the code easier for others to deal with. A major impediment
> to contribution to the VM code base is emacs-lisp's very crude data
> structures. This is particuarly true of the mime stuff. I have many
> times struggled through trying to figure out just what the mime data
> structures look like. For example, here is what is effectively the
> declaration of a data-structure:
>
> (defun vm-mm-layout-type (e) (aref e 0))
> (defun vm-mm-layout-qtype (e) (aref e 1))
[...]
Well there is an API for accessing the fields, it is not that bad
and the function names are pretty self explanatory ;-)
> I'm more of a Common Lisp programmer than an emacs-lisp programmer, but
> this seems like the sort of thing that would really benefit from the
> use of something like defstruct (and some documentation strings).
> cl-macs would be one way of getting some better structure here (I note
> with some dismay that the defstruct in cl-macs does not seem to support
> documentation strings). Quite possibly there are others.
Use of CL is discouraged for Emacs, i.e. only macros should be used.
Unfortunately, there are many undocumented functions and nothing
on the internals of VM.
Robert.
John Stoffel
newsspam5REMOVETHIS> As I got used to virtual folders, I felt the
newsspam5REMOVETHIS> desire to get rid of physical folders. I do not
newsspam5REMOVETHIS> need them at all, they are only a crude hack for
newsspam5REMOVETHIS> the problem that Emacs gets unusable if a folder
newsspam5REMOVETHIS> exceeds a certain message count and/or file size.
This bites me in the ass from time to time as well. So how do virtual
folders get around the buffer size limits in emacs? 128mb is too
small these days!
So if I have a maildir style directory, with each message in it's own
file, can I just turn that entire directory into a virtual folder to
be browsed from time to time?
Thanks,
John
newsspam5...@robf.de
No it is not possible.
But that would be one way to go.
128MB are really much for a summary and a message buffer.
Unfortunately, VM has not generic folder API like Gnus.
Robert.
Paul Smith
> John Stoffel <jo...@stoffel.org> writes:
> > This bites me in the ass from time to time as well. So how do virtual
> > folders get around the buffer size limits in emacs? 128mb is too
> > small these days!
You could always upgrade to a 64bit CPU! :-)
John Stoffel
>> John Stoffel <jo...@stoffel.org> writes:
>> > This bites me in the ass from time to time as well. So how do virtual
>> > folders get around the buffer size limits in emacs? 128mb is too
>> > small these days!
Paul> You could always upgrade to a 64bit CPU! :-)
Hah hah. Ain't going to happen, though I guess I could just run VM on
the opterons at work and display back to my desk.
But still, not ideal. Esp if I go away and get spammed with a lot of
email, then trying to read a folder with a bunch of large messages
get's painful.
John
John Stoffel
newsspam5REMOVETHIS> John Stoffel <jo...@stoffel.org> writes:
>>>>>>> "newsspam5REMOVETHIS" == newsspam5REMOVETHIS <newsspam5...@robf.de> writes:
>>
Robert> As I got used to virtual folders, I felt the desire to get rid
Robert> of physical folders. I do not need them at all, they are only
Robert> a crude hack for the problem that Emacs gets unusable if a
Robert> folder exceeds a certain message count and/or file size.
>> This bites me in the ass from time to time as well. So how do virtual
>> folders get around the buffer size limits in emacs? 128mb is too
>> small these days!
>>
>> So if I have a maildir style directory, with each message in it's own
>> file, can I just turn that entire directory into a virtual folder to
>> be browsed from time to time?
Robert> No it is not possible.
Darn. I was hoping this would work.
Robert> But that would be one way to go.
How hard would it be to make it work? Or do I just need to put each
large email into it's own VM folder, then group them using a virtual
folder? Have procmail rotate around looking for the smallest folder
to put a message in, or into a new message if no folder is smaller
than X mb in size?
Robert> 128MB are really much for a summary and a message buffer.
Sorry, I haven't a clue what you mean here. Are you saying that 128MB
is too small, or that it is large enough?
Robert> Unfortunately, VM has not generic folder API like Gnus.
How hard would it be to pull in that code from gnus?
Thanks,
John
rpgo...@gmail.com
Actually, a generic folder API would be a very nice addition, because
it would make it possible to have VM work well with IMAP. I find that
having to download all my mail locally is unpleasant, because I have
both a laptop and a desktop machine. I have never found a good
solution to accessing my email from multiple locations. For now, since
I use VM, I am forced to the unsatisfactory solution of using SSH to
connect to my desktop, and running VM inside a textmode emacs. As you
would expect, this works quite poorly when dealing with MIME, etc. But
the mail directory is too big, and the updates too unpredictable, for
it to be possible for me to keep it synchronized across machines. IMAP
would be a big help here.
rpgo...@gmail.com
> "rpgo...@gmail.com" <rpgo...@gmail.com> writes:
> [...]
> > 3. Make the code easier for others to deal with. A major impediment
> > to contribution to the VM code base is emacs-lisp's very crude data
> > structures. This is particuarly true of the mime stuff. I have many
> > times struggled through trying to figure out just what the mime data
> > structures look like. For example, here is what is effectively the
> > declaration of a data-structure:
> >
> > (defun vm-mm-layout-type (e) (aref e 0))
> > (defun vm-mm-layout-qtype (e) (aref e 1))
>
> [...]
>
> Well there is an API for accessing the fields, it is not that bad
> and the function names are pretty self explanatory ;-)
FSVO "self explanatory." This is related to the fact that the outside
modifier of this codebase has little way to guess what are the types of
arguments.
One of the big advantages of using the cl-macs defstruct is that it
would be possible to make the data structures be tagged (named). That
means you could look at a value in the debugger and know its type,
instead of looking at it and saying "that's a mime header," one says
"oh, it's a vector. Wonder what the heck that's supposed to mean..."
This is not a problem that can be solved by an API....
And, of course, if there was to be any change to this API that would
change the field offsets, that would be EXTREMELY nasty...
>
> > I'm more of a Common Lisp programmer than an emacs-lisp programmer, but
> > this seems like the sort of thing that would really benefit from the
> > use of something like defstruct (and some documentation strings).
> > cl-macs would be one way of getting some better structure here (I note
> > with some dismay that the defstruct in cl-macs does not seem to support
> > documentation strings). Quite possibly there are others.
>
> Use of CL is discouraged for Emacs, i.e. only macros should be used.
Well, defstruct (even in CL, much less in Emacs) is a macro. So it
would be possible to use it...
But perhaps someone else has supplied a different emacs library for
structures?
newsspam5...@robf.de
>>>>>> "newsspam5REMOVETHIS" == newsspam5REMOVETHIS <newsspam5...@robf.de> writes:
>
> newsspam5REMOVETHIS> John Stoffel <jo...@stoffel.org> writes:
>>>>>>>> "newsspam5REMOVETHIS" == newsspam5REMOVETHIS <newsspam5...@robf.de> writes:
>>>
>
> Robert> As I got used to virtual folders, I felt the desire to get rid
> Robert> of physical folders. I do not need them at all, they are only
> Robert> a crude hack for the problem that Emacs gets unusable if a
> Robert> folder exceeds a certain message count and/or file size.
>
>>> This bites me in the ass from time to time as well. So how do virtual
>>> folders get around the buffer size limits in emacs? 128mb is too
>>> small these days!
>>>
>>> So if I have a maildir style directory, with each message in it's own
>>> file, can I just turn that entire directory into a virtual folder to
>>> be browsed from time to time?
>
> Robert> No it is not possible.
>
> Darn. I was hoping this would work.
>
> Robert> But that would be one way to go.
>
> How hard would it be to make it work? Or do I just need to put each
> large email into it's own VM folder, then group them using a virtual
> folder? Have procmail rotate around looking for the smallest folder
> to put a message in, or into a new message if no folder is smaller
> than X mb in size?
No it would just make it worse as VM will visit all of those
physical folders and then will start searching. IMHO it
must/should be done outside of VM. One core problem is that
there is no emacs function to read/write some part of a file, or
at least I missed it.
> Robert> 128MB are really much for a summary and a message buffer.
>
> Sorry, I haven't a clue what you mean here. Are you saying that 128MB
> is too small, or that it is large enough?
I meant to say it is large enough. Basically most MTAs I know
limit emails to 10MB and this would leave 118MB for the rest.
VM should only keep the currently selected message in a buffer.
> Robert> Unfortunately, VM has not generic folder API like Gnus.
>
> How hard would it be to pull in that code from gnus?
I am not sure, I did not look at it.
IMHO the first step would be to rewrite VM to have a API for
pluging in different folder types resp. storages.
Robert.
newsspam5...@robf.de
> newsspam5...@robf.de wrote:
>> "rpgo...@gmail.com" <rpgo...@gmail.com> writes:
>> [...]
>> > 3. Make the code easier for others to deal with. A major impediment
>> > to contribution to the VM code base is emacs-lisp's very crude data
>> > structures. This is particuarly true of the mime stuff. I have many
>> > times struggled through trying to figure out just what the mime data
>> > structures look like. For example, here is what is effectively the
>> > declaration of a data-structure:
>> >
>> > (defun vm-mm-layout-type (e) (aref e 0))
>> > (defun vm-mm-layout-qtype (e) (aref e 1))
>>
>> [...]
>>
>> Well there is an API for accessing the fields, it is not that bad
>> and the function names are pretty self explanatory ;-)
>
> FSVO "self explanatory." This is related to the fact that the outside
> modifier of this codebase has little way to guess what are the types of
> arguments.
Well you are right ;-)
[...]
> And, of course, if there was to be any change to this API that would
> change the field offsets, that would be EXTREMELY nasty...
Use the getter/setter functions and you do not have to care about it.
Robert.
John Stoffel
>>
>> How hard would it be to make it work? Or do I just need to put each
>> large email into it's own VM folder, then group them using a virtual
>> folder? Have procmail rotate around looking for the smallest folder
>> to put a message in, or into a new message if no folder is smaller
>> than X mb in size?
Robert> No it would just make it worse as VM will visit all of those
Robert> physical folders and then will start searching. IMHO it
Robert> must/should be done outside of VM. One core problem is that
Robert> there is no emacs function to read/write some part of a file,
Robert> or at least I missed it.
I don't think I've ever heard of such a function, generally they just
slurp the file into memory. I googled sometime in the past year about
emacs and buffer size limits and such and there just doesn't seem to
be a nice solution. Oh well...
Robert> 128MB are really much for a summary and a message buffer.
Maybe for the summary buffer, but possibly not for the message
buffer. I get big emails at times. Yes, they're internal only.
>>
>> Sorry, I haven't a clue what you mean here. Are you saying that 128MB
>> is too small, or that it is large enough?
Robert> I meant to say it is large enough. Basically most MTAs I know
Robert> limit emails to 10MB and this would leave 118MB for the rest.
That should be enough, but not always. If I logout for the weekend
and we have server problems, I can get 2000+ emails, all with
attachments. I try to open my mail and it bombs with a "Buffer too
large" type error.
Then I endup splitting the folder into pieces, which is a pain and a
half.
Robert> VM should only keep the currently selected message in a
Robert> buffer.
That would be nice. Maybe if emacs had some sort of support for an
mmap() type access to a large file, only mapping in sections of it as
needed. Now there are lots of issues with consistency of the file in
case of a core dump or other crash which would need to be addressed
here.
Thanks for all your comments!
John
Robert Marshall
> John Stoffel <jo...@stoffel.org> writes:
>>>>>>> "newsspam5REMOVETHIS" == newsspam5REMOVETHIS
>>>>>>> <newsspam5...@robf.de> writes:
>>
>> newsspam5REMOVETHIS> John Stoffel <jo...@stoffel.org> writes:
>>>> So if I have a maildir style directory, with each message in it's
>>>> own file, can I just turn that entire directory into a virtual
>>>> folder to be browsed from time to time?
>>
>> Robert> No it is not possible.
>>
>> Darn. I was hoping this would work.
>>
>> Robert> But that would be one way to go.
>>
>> How hard would it be to make it work? Or do I just need to put
>> each large email into it's own VM folder, then group them using a
>> virtual folder? Have procmail rotate around looking for the
>> smallest folder to put a message in, or into a new message if no
>> folder is smaller than X mb in size?
>
> No it would just make it worse as VM will visit all of those
> physical folders and then will start searching. IMHO it
> must/should be done outside of VM. One core problem is that
> there is no emacs function to read/write some part of a file, or
> at least I missed it.
There's the optional parameters of insert-file-contents
insert-file-contents is a built-in function in `C source code'.
(insert-file-contents FILENAME &optional VISIT BEG END REPLACE)
Insert contents of file FILENAME after point.
...
The optional third and fourth arguments BEG and END
specify what portion of the file to insert.
These arguments count bytes in the file, not characters in the buffer.
If VISIT is non-nil, BEG and END must be nil.
apparently write-region has something similar
or is this only in gnu/emacs?
Robert.
--
La grenouille songe..dans son chāteau d'eau
Links and things http://rmstar.blogspot.com/
Kurt Hackenberg
Doesn't Emacs still have 28-bit pointers internally, in the C code
that implements the Lisp interpreter? Squeezes 28 bits of address and
4 bits of cell data type and such into a word, to save memory, instead
of using a C pointer type and another variable. Last I heard people
were saying it was too much work to fix it.
Kurt Hackenberg
rpgo...@gmail.com <rpgo...@gmail.com> wrote:
>Actually, a generic folder API would be a very nice addition, because
>it would make it possible to have VM work well with IMAP. I find that
>having to download all my mail locally is unpleasant, because I have
>both a laptop and a desktop machine. I have never found a good
>solution to accessing my email from multiple locations. For now, since
>I use VM, I am forced to the unsatisfactory solution of using SSH to
>connect to my desktop, and running VM inside a textmode emacs. As you
>would expect, this works quite poorly when dealing with MIME, etc. But
>the mail directory is too big, and the updates too unpredictable, for
>it to be possible for me to keep it synchronized across machines. IMAP
>would be a big help here.
VM 7.19 can be an IMAP client. Is there some reason you can't use
that?
rpgo...@gmail.com
> on the internals of VM.
>
"internal variables" in vm-vars.el. The variable names there are
suggestive but not, IMHO sufficient for one to understand their intent.
I was just looking at some of them in connection with trying to
implement file locking per a different posting here...
Best,
r
rpgo...@gmail.com
None other than ignorant. I thought that VM had to just use IMAP as a
spool; I didn't know it could connect to IMAP as a real client. I
don't know whether that's out-of-date knowledge or if it was never true
and I was just confused.
OK, I will now go off and try to figure out how to set myself up an
IMAP server....
thanks, Kurt!
rpgo...@gmail.com
None other than ignorant. I thought that VM had to just use IMAP as a
Mark A. Flacy
Kurt>
Kurt> VM 7.19 can be an IMAP client. Is there some reason you can't use
Kurt> that?
Well, that's a "use IMAP to slurp from the server" client not a "use IMAP
to access the mail on the server while leaving it there" client.
Or the difference between a Gnus imap and nnimap backend.
rpgo...@gmail.com
Actually, if I understand the vm info manual correctly, there is now a
true IMAP client in VM, as opposed to something that just slurps from
the server.
The fact that you believed there was no true IMAP client, as I did,
suggests to me that this is a relatively recent addition to VM. I'm
not sure how to find a changelog, but rooting around in the archives of
g.v.i seems to show this possible as early as late 2004.
As if it isn't enough to have to read manuals, we have to *reread*
them, too! :-)
newsspam5...@robf.de
No it does also exist in XEmacs. Thanks for pointing me to it.
Robert.
newsspam5...@robf.de
[...]
>> There's the optional parameters of insert-file-contents
>>
>> insert-file-contents is a built-in function in `C source code'.
>> (insert-file-contents FILENAME &optional VISIT BEG END REPLACE)
>>
>> Insert contents of file FILENAME after point.
>> ...
>> The optional third and fourth arguments BEG and END
>> specify what portion of the file to insert.
>> These arguments count bytes in the file, not characters in the buffer.
>> If VISIT is non-nil, BEG and END must be nil.
>>
>> apparently write-region has something similar
Not exactly. IMHO write-region allows only to append to an
existing file, but not to overwrite some part of it.
If I recall it right now, that was the reason why I thought it is
not possible from within Emacs. VM stored labels, message status
and other things in an extra message header (X-VM-v5-Data).
Use of padding would allow updates of that header, but still it
would be necessary to write those headers.
But hey, there is the index file (setq vm-index-file-suffix
".idx") which already contains all required information. So
without physical deletions (expunge) one could use it and avoid
reading and writing the whole folder file.
Robert.
newsspam5...@robf.de
Vm still will download a copy of all messages. Other MUAs will
only get the headers for the summary and will download the rest
of a message only when displaying it.
Robert.
rpgo...@gmail.com
everything into a local client, IIUC. So those of us who want to use
vm but also read our email from mulitple machines can use this, correct?
newsspam5...@robf.de
Yes, you can even save messages to another IMAP folder and create,
delete and rename a IMAP folders.
Robert.
Mark A. Flacy
Well, you learn something new every day.
I've pretty much stopped using VM; the 128MB mailbox size restriction is
just too limiting. With virtual folders, you don't *need* to split your
e-mails up into various buckets. In fact, that just makes it more
difficult to find anything.
Kurt Hackenberg
>I thought that VM had to just use IMAP as a spool; I didn't know it
>could connect to IMAP as a real client. I don't know whether that's
>out-of-date knowledge or if it was never true and I was just
>confused.
Out of date. Being an IMAP client was about the last thing Kyle did
before going quiescent, over the last several versions. I think it
started about 7.14, but wasn't finished until 7.19.
IMHO, that would have been enough reason to go to version 8, since
that's a major new feature, and the first time VM has gotten mail any
way other than spooling.
newsspam5...@robf.de
I cannot recall when `vm-visit-pop-folder' was added, but it also
does not use a spool file ;-)
Robert.