Re: news servers offering IMAP access?

[Ivan Shmakov]

>>>>> Russ Allbery <r...@stanford.edu> writes:
>>>>> Ivan Shmakov <onei...@gmail.com> writes:

[Cross-posting to news:comp.mail.imap.]

>> Now, I know that NNTP is great. But then, did anyone experiment
>> with offering IMAP access to the news spool?

> Yup, CMU did this for years. I'm not sure if they still do. Cyrus
> IMAP can speak either NNTP or IMAP with the same backend storage.

ACK, thanks.

BTW, what backend storage formats it supports?

The last time I've checked, Cyrus IMAP somehow looked less
appealing to me than Dovecot IMAP (which I now use for years.)
I wonder, is there anything else (other than NNTP support) in
Cyrus IMAP worth looking at?

--
FSF associate member #7257

[Mark Crispin]

Panda IMAP and UW IMAP also support export via IMAP of local netnews
spools. They also support IMAP/NNTP proxy.

I don't know Dovecot's capabilities with NNTP. However, Dovecot is an
excellent IMAP server in all other respects. Dovecot, Panda, and SurgeMail
are the only IMAP servers that completely pass IMAP protocol compliancy
testing.

With netnews and NNTP being more or less moribund, this is primarily of
historical interest these days.

The UW IMAP installation at UW exported netnews via IMAP for many years in
the 1990s. I used it that way; but could probably count the number of
other people who used it on one hand and have fingers left over. As
obvious and attractive as the capability seemed, there was simply no
demand. Then again, by the 2000s fewer than 100 people at UW still used
netnews at all; UW's NNTP service was turned off several years ago with
little protest.

On Sun, 15 Jan 2012, Ivan Shmakov posted:

-- Mark --

http://panda.com/mrc
Democracy is two wolves and a sheep deciding what to eat for lunch.
Liberty is a well-armed sheep contesting the vote.

[Ivan Shmakov]

>>>>> Mark Crispin <m...@panda.com> writes:

[…]

> Panda IMAP and UW IMAP also support export via IMAP of local netnews
> spools. They also support IMAP/NNTP proxy.

ACK, thanks.

> I don't know Dovecot's capabilities with NNTP.

AIUI, there're none. But then, it isn't like I badly need any.

[…]

> With netnews and NNTP being more or less moribund, this is primarily
> of historical interest these days.

There seems to be a few still active newsgroups, actually.
Consider, e. g., news:sci.electronics.design, news:comp.os.vms,
news:comp.arch.embedded, news:alt.russian.z1, etc.

Although it became hardly common nowadays, I believe that it
still has a chance to become more widespread. Usenet is just a
“federation of forums” (in a modern parlance), and as Web forums
are still quite active, I don't see why Usenet shouldn't be.

[…]

[River Tarnell]

In article <86hazwj...@gray.siamics.net>,
Ivan Shmakov <onei...@gmail.com> wrote:

>>>>>> Mark Crispin <m...@panda.com> writes:
> > With netnews and NNTP being more or less moribund, this is primarily
> > of historical interest these days.

> There seems to be a few still active newsgroups, actually.
> Consider, e. g., news:sci.electronics.design, news:comp.os.vms,
> news:comp.arch.embedded, news:alt.russian.z1, etc.

I'm currently working on a new NNTP server -- so it's not entirely
moribund ;-).

For now it is transit only (no readers), but I would like to eventually
add reader support. I'd already intended to support both NNTP and IMAP
access once that happens.

However, given the lack of any way to post new messages via IMAP, I'm
not sure it would be very useful for Usenet.

Regards,
--
-- river. | Free Usenet: http://news.rt.uk.eu.org/
Non-Reciprocal Laws of Expectations: | PGP: 2B9CE6F2
Negative expectations yield negative results.
Positive expectations yield negative results.

[Mark Crispin]

On Mon, 16 Jan 2012, Ivan Shmakov posted:

> Although it became hardly common nowadays, I believe that it
> still has a chance to become more widespread. Usenet is just a
> “federation of forums” (in a modern parlance), and as Web forums
> are still quite active, I don't see why Usenet shouldn't be.

I have nothing against Usenet. After all, I still use it (albeit I only
read three newsgroups). Sadly, due to a combination of factors (not all
technological), Usenet passed its zenith long ago. I don't see much hope
for it to arise again.

I have little doubt that some vestige of Usenet will continue to exist for
quite some time to come, but never again as a major infrastructure. The
"tragedy of the commons" factor will never go away; the lesson learned is
that the otherwise-desirable notion of the "commons" does not scale.

It's actually remarkable that Usenet scaled as well as it once did.
Dedicated and honorable individuals fought a valiant battle to keep Usenet
usable for many long years. But ultimately it proved futile.

[Ivan Shmakov]

>>>>> River Tarnell <ri...@RT.UK.EU.ORG> writes:

>>>>> In article <86hazwj...@gray.siamics.net>, Ivan Shmakov wrote:
>>>>> Mark Crispin <m...@panda.com> writes:

[Cross-posting to news:news.misc, for it's no longer just about
NNTP and IMAP.]

>>> With netnews and NNTP being more or less moribund, this is
>>> primarily of historical interest these days.

>> There seems to be a few still active newsgroups, actually.
>> Consider, e. g., news:sci.electronics.design, news:comp.os.vms,
>> news:comp.arch.embedded, news:alt.russian.z1, etc.

> I'm currently working on a new NNTP server -- so it's not entirely
> moribund ;-).

I'm contemplating developing a kind of a “news caching proxy”,
actually. However, I consider making somewhat radical changes
to the usual news processing procedures.

In particular, I aim for a better support of presentation of
netnews as (apart from having them accessible via IMAP and NNTP)
both Atom feeds (including Atom Posting Protocol support) and
Web pages (XHTML.) To this end, I believe it's essential to
outrightly discard non-ASCII article headers, /and/ non-ASCII
article bodies of the articles lacking proper MIME headers, as
both are ambiguous as to what character encoding is used.

Also, I consider using an RDBMS to store particular
(Message-ID:, References:), if not all, of the message headers,
so to allow for instant threading. Perhaps this storage could
be used to implement some “extended search” facilities within
either or both of the NNTP and IMAP interfaces as well, but I
have no specific plans on that at this moment.

Perhaps, RDBMS could also be used for MIME parts below certain
threshold of size (in octets; or perhaps characters, for text/*
MIME parts, and for strictly ASCII non-MIME article bodies.)

The MIME parts are to be stored separately (from each other and
from the header), in 8-bit, even if originally coming as, e. g.,
Base64 or quoted-printable. The parts above the size threshold
would be stored separately on the filesystem. If possible, a
few more transformations would also be implemented. In
particular, it may be possible to transform any armored OpenPGP
signatures into their MIME-based counterparts.

I also intend to compute message digests (SHA 256, SHA-1) for
the MIME parts of over some trivial (1024 octets or so) size,
and aggressively replace duplicates with links. If the message
(or a part) is digitally signed with a known key, the digests
computed would be verified against the signature, and the
message discarded on failure.

As for the “caching” part, this agent would both allow
conventional feeds (for input), as well as periodical and
on-demand fetching of articles (like, e. g., suck(1), but also
allowing for partial retrieval of the newsgroups' articles,
based on criteria specified over the XOVER data.) It will also
maintain the “last access time” for the articles, to be taken
into account in the “expiration” process.

The “proxy” part would mean that the server could be instructed
to preserve the Xref: header of its “primary” (“backing”) source
(though only for the specific newsgroups or hierarchies, and
still allowing for different sources to be used to actually
fetch the articles.) The posting in proxy mode will be
performed synchronously, with no reply being sent to the user
agent before the “backing” source itself replies, or a timeout.

For the implementation of the prototype, I'm now considering the
Perl language, as there's a sheer amount of extensions
available, providing support for a variety of protocols and data
presentations.

> For now it is transit only (no readers), but I would like to
> eventually add reader support. I'd already intended to support both
> NNTP and IMAP access once that happens.

> However, given the lack of any way to post new messages via IMAP, I'm
> not sure it would be very useful for Usenet.

The IMAP protocol allows for both retrieval /and/ storage of
messages. In particular, certain MUA's allow for the copies of
the messages sent via e-mail to be saved into a dedicated IMAP
mailbox. Although unconventional, I guess that the MUA's could
be changed to use such a functionality to post new messages.

[Mark Crispin]

On Tue, 17 Jan 2012, Ivan Shmakov posted:

> To this end, I believe it's essential to
> outrightly discard non-ASCII article headers, /and/ non-ASCII
> article bodies of the articles lacking proper MIME headers, as
> both are ambiguous as to what character encoding is used.

Good luck on this. This was a battle that was fought, and lost, about 20
years ago. Search for "just send 8-bits" in old old flamewars. Prepare to
be nauseated.

> Also, I consider using an RDBMS

You're a glutton for punishment, I see. Lots of people have had this idea
in the past...

I'll just say "good luck".

However, I will give you an important hint. To do IMAP in any sort of
reasonable way, you MUST have access to the message as a single RFC *822
format char* string.

Don't assume that you can assemble on the fly from a database. EVERYBODY
who has ever tried has failed, miserably. Whatever you do to store the
message in separate parts, you MUST have the message as the big char*.
You'll regret doing otherwise.

[Russ Allbery]

Ivan Shmakov <onei...@gmail.com> writes:

> In particular, I aim for a better support of presentation of
> netnews as (apart from having them accessible via IMAP and NNTP)
> both Atom feeds (including Atom Posting Protocol support) and Web
> pages (XHTML.) To this end, I believe it's essential to
> outrightly discard non-ASCII article headers, /and/ non-ASCII
> article bodies of the articles lacking proper MIME headers, as
> both are ambiguous as to what character encoding is used.

I highly recommend talking to larsi about what he did with GMANE, since
that's a large part of what you're trying to do here and it seems to work
quite well (and is *very* widely used in some parts of the free software
community).

I think a lot of the backend code is written in Lisp, since, well, it's
larsi, which may make it challenging to reuse depending on your preferred
languages. But the ideas at least are there.

> For the implementation of the prototype, I'm now considering the
> Perl language, as there's a sheer amount of extensions available,
> providing support for a variety of protocols and data
> presentations.

Speaking IMAP properly is going to be the hardest part of this problem, I
think.

--
Russ Allbery (r...@stanford.edu) <http://www.eyrie.org/~eagle/>

Please post questions rather than mailing me directly.
<http://www.eyrie.org/~eagle/faqs/questions.html> explains why.

[Russ Allbery]

Mark Crispin <m...@panda.com> writes:
> On Tue, 17 Jan 2012, Ivan Shmakov posted:

>> To this end, I believe it's essential to outrightly discard
>> non-ASCII article headers, /and/ non-ASCII article bodies of the
>> articles lacking proper MIME headers, as both are ambiguous as to
>> what character encoding is used.

> Good luck on this. This was a battle that was fought, and lost, about 20
> years ago. Search for "just send 8-bits" in old old flamewars. Prepare
> to be nauseated.

Believe it or not, it's actually gotten better since the old flamewars.
Not to the point where you can just assume this, of course, but more and
more clients just quietly do the right thing, and people seem to cope.

[Xavier Roche]

Le 17/01/2012 18:09, Russ Allbery a écrit :
> Believe it or not, it's actually gotten better since the old flamewars.
> Not to the point where you can just assume this, of course, but more and
> more clients just quietly do the right thing, and people seem to cope.

One of the greatest battle won on fr.* recently was to *accept* UTF-8 as
charset (rather than the venerable ISO-8859 family members). I'd say
that things are getting better, but a bit slowly :)

[Julien ÉLIE]

Hi Russ and Ivan,

>> In particular, I aim for a better support of presentation of
>> netnews as (apart from having them accessible via IMAP and NNTP)
>> both Atom feeds (including Atom Posting Protocol support) and Web
>> pages (XHTML.) To this end, I believe it's essential to
>> outrightly discard non-ASCII article headers, /and/ non-ASCII
>> article bodies of the articles lacking proper MIME headers, as
>> both are ambiguous as to what character encoding is used.
>
> I highly recommend talking to larsi about what he did with GMANE, since
> that's a large part of what you're trying to do here and it seems to work
> quite well (and is *very* widely used in some parts of the free software
> community).
>
> I think a lot of the backend code is written in Lisp, since, well, it's
> larsi, which may make it challenging to reuse depending on your preferred
> languages. But the ideas at least are there.

In case it could be of help, an improved version of Newsportal
http://amrhein.eu/newsportal/
with UTF-8 support, and even MIME attachments (but not yEnc support) can
be seen here:
http://iulius.dinauz.org/usenet/webnews-test/thread.php?group=news.software.nntp
http://iulius.dinauz.org/usenet/webnews-test/

Written in PHP.

--
Julien ÉLIE

« C'est une forêt vierge où la main de l'homme n'a jamais mis le
pied. »

[Ivan Shmakov]

>>>>> Russ Allbery <r...@stanford.edu> writes:
>>>>> Ivan Shmakov <onei...@gmail.com> writes:

>> In particular, I aim for a better support of presentation of netnews
>> as (apart from having them accessible via IMAP and NNTP) both Atom
>> feeds (including Atom Posting Protocol support) and Web pages
>> (XHTML.) To this end, I believe it's essential to outrightly
>> discard non-ASCII article headers, /and/ non-ASCII article bodies of
>> the articles lacking proper MIME headers, as both are ambiguous as
>> to what character encoding is used.

> I highly recommend talking to larsi about what he did with GMANE,
> since that's a large part of what you're trying to do here and it
> seems to work quite well (and is *very* widely used in some parts of
> the free software community).

Well, I'd rather check the published sources behind Gmane first.

However, it was my understanding that they've chosen to use an
NNTP "server" (namely, INN) as the backend. An approach which
seems to have certain drawbacks.

> I think a lot of the backend code is written in Lisp, since, well,
> it's larsi, which may make it challenging to reuse depending on your
> preferred languages. But the ideas at least are there.

ACK, thanks.

>> For the implementation of the prototype, I'm now considering the
>> Perl language, as there's a sheer amount of extensions available,
>> providing support for a variety of protocols and data presentations.

> Speaking IMAP properly is going to be the hardest part of this
> problem, I think.

Perhaps even further complicated by the fact that I have little
experience with IMAP whatsoever. (Apart from keeping Dovecot
running here and there.)

[Russ Allbery]

Ivan Shmakov <onei...@gmail.com> writes:

> Well, I'd rather check the published sources behind Gmane first.

> However, it was my understanding that they've chosen to use an
> NNTP "server" (namely, INN) as the backend. An approach which
> seems to have certain drawbacks.

Only because he never got around writing his own in Lisp, but he had some
of the pieces and a bunch of ideas. That's part of why it would be good
to talk to him and not only look at the source. But starting with the
source is good, of course.

> Perhaps even further complicated by the fact that I have little
> experience with IMAP whatsoever. (Apart from keeping Dovecot
> running here and there.)

IMAP is easily many times harder to implement properly than NNTP. All
that extra useful functionality comes at a cost. :)

[Ivan Shmakov]

>>>>> Mark Crispin <m...@panda.com> writes:
>>>>> On Tue, 17 Jan 2012, Ivan Shmakov posted:

[Setting Followup-To: to drop news.software.nntp.]

> Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed

BTW, is this "format=flowed" bit intentional? Somehow, I
thought that it's "incompatible" with >-quoting.

[...]

>> Also, I consider using an RDBMS

> You're a glutton for punishment, I see. Lots of people have had this
> idea in the past...

> I'll just say "good luck".

> However, I will give you an important hint. To do IMAP in any sort
> of reasonable way, you MUST have access to the message as a single
> RFC *822 format char* string.

Why is it so?

> Don't assume that you can assemble on the fly from a database.
> EVERYBODY who has ever tried has failed, miserably.

To my mind, the biggest problem here is to preserve "message
boundary" sequences, which may be essential for digital
signatures to work.

> Whatever you do to store the message in separate parts, you MUST have
> the message as the big char*. You'll regret doing otherwise.

Assuming that "char *" means an "octet sequence" here (and not a
"sequence of characters", as it doesn't make sense for, say,
application/octet-stream parts), I guess that I may do it the
other way around, by storing the offsets of the individual MIME
parts within the octet sequence.

[Ivan Shmakov]

>>>>> Russ Allbery <r...@stanford.edu> writes:
>>>>> Ivan Shmakov <onei...@gmail.com> writes:

>> Well, I'd rather check the published sources behind Gmane first.

>> However, it was my understanding that they've chosen to use an NNTP
>> "server" (namely, INN) as the backend. An approach which seems to
>> have certain drawbacks.

> Only because he never got around writing his own in Lisp, but he had
> some of the pieces and a bunch of ideas. That's part of why it would
> be good to talk to him and not only look at the source. But starting
> with the source is good, of course.

ACK, thanks. Though I'd note that NNTP /server/ functionality
wouldn't probably be a priority for me. Perhaps I'd focus on
IMAP and Atom (with Atom Posting Protocol) instead.

>> Perhaps even further complicated by the fact that I have little
>> experience with IMAP whatsoever. (Apart from keeping Dovecot
>> running here and there.)

> IMAP is easily many times harder to implement properly than NNTP.
> All that extra useful functionality comes at a cost. :)

The question is, could the Net::IMAP::Server Perl module [1] (as
available from CPAN) relief most of the implementor's pain?

[1] http://search.cpan.org/~alexmv/Net-IMAP-Server-1.30/

[Mark Crispin]

On Wed, 18 Jan 2012, Ivan Shmakov posted:

> > However, I will give you an important hint. To do IMAP in any sort
> > of reasonable way, you MUST have access to the message as a single
> > RFC *822 format char* string.
> Why is it so?

IMAP allows you to fetch a part of a message in many ways, including "<n>
characters starting at position <m>" for arbitrary body parts and/or the
entire message.

If you don't have an exact representation of the message, or at least have
it is easily (and precisely) calculable, you are in for a world of hurt.

> Assuming that "char *" means an "octet sequence" here (and not a
> "sequence of characters", as it doesn't make sense for, say,
> application/octet-stream parts), I guess that I may do it the
> other way around, by storing the offsets of the individual MIME
> parts within the octet sequence.

If, by this, you mean "have an octet sequence of the entire message, with
the individual MIME parts being recorded as offset/length within that
octet sequence", then you are on the right track.

[Ivan Shmakov]

>>>>> Mark Crispin <m...@panda.com> writes:
>>>>> On Wed, 18 Jan 2012, Ivan Shmakov posted:

>>> However, I will give you an important hint. To do IMAP in any sort
>>> of reasonable way, you MUST have access to the message as a single
>>> RFC *822 format char* string.

>> Why is it so?

> IMAP allows you to fetch a part of a message in many ways, including
> "<n> characters starting at position <m>" for arbitrary body parts
> and/or the entire message.

ACK, thanks!

However, I presume it's "octets", and not "characters"?

> If you don't have an exact representation of the message, or at least
> have it is easily (and precisely) calculable, you are in for a world
> of hurt.

The other problem is that the MIME parts may be nested rather
arbitrarily, which isn't that easy to tackle when using the
relational database model (and in particular, SQL.)

>> Assuming that "char *" means an "octet sequence" here (and not a
>> "sequence of characters", as it doesn't make sense for, say,
>> application/octet-stream parts), I guess that I may do it the other
>> way around, by storing the offsets of the individual MIME parts
>> within the octet sequence.

> If, by this, you mean "have an octet sequence of the entire message,
> with the individual MIME parts being recorded as offset/length within
> that octet sequence", then you are on the right track.

ACK, thanks.

[Mark Crispin]

On Wed, 18 Jan 2012, Ivan Shmakov posted:

> However, I presume it's "octets", and not "characters"?

Yes, you are correct. I am from an older generation that sloppily uses the
two terms as synonyms.