Gathering ISP config data - questions
Gervase Markham
http://spreadsheets.google.com/pub?key=p49SW32nNYX0otkRc3UZUJA
My aim, ONCE IT'S READY, it to publicise it and try and crowdsource the
data collection. The list of fields is based on the information that
seems to be included in the proposed XML file format for the data
<http://wiki.mozilla.org/Thunderbird:Autoconfiguration:ConfigFileFormat>
with the following changes:
- The "inputfields" stuff seemed over-engineered - any field can be
blank, and if it's blank and the client decides it's required, it should
just prompt in some way that it can figure out for itself. I left in the
possibility of example usernames, though.
- I haven't included any client-side settings (e.g. leaveMessageOnServer).
Before we publicise this, I want to make sure that we are asking for all
the right information. Here are my questions:
- Do we need a field for "preferred server" (IMAP or POP) or do we
always assume IMAP is superior where both are offered?
- Do we need to be more specific about Secure Auth technologies for POP,
IMAP and/or SMTP? Thunderbird just has a checkbox and Does The Right
Thing - is this true of most mail clients?
- I inherited the Short Name field from the XML proposal. Is it
necessary? Why?
- Are the options for usernames good enough (see the note at the top)? I
thought of doing something regexp-based but it seemed very complicated.
- For encryption, is it necessary to maintain a distinction between
"TLS" and "SSL", as Thunderbird does? What's the difference?
- Do we need some way of saying "these servers are not available to all
subscribers", e.g. for Yahoo! Mail, where you have to pay for POP access?
Gerv
Simon Wilkinson
On 7 Jul 2008, at 14:52, Gervase Markham wrote:
> - Do we need to be more specific about Secure Auth technologies for
> POP,
> IMAP and/or SMTP? Thunderbird just has a checkbox and Does The Right
> Thing - is this true of most mail clients?
It's true of most clients. The standard for SASL authentication
(which is what this really is - we should probably call it that,
rather than continuing with Thunderbird's bizarre 'secure auth'
designation) requires that the server provide the client with a list
of supported mechanisms, and the client pick its preferred mechanism
from that list.
> - For encryption, is it necessary to maintain a distinction between
> "TLS" and "SSL", as Thunderbird does? What's the difference?
TLS indicates that the server supports STARTTLS over the 'normal'
port (this can be autodetected from the server's capabilities
declaration, though) - with this, the client starts talking in clear
text, but then upgrades the connection as part of the authentication
handshake. SSL indicates that you should do a SSL protected stream
over a different port (like https).
Simon.
Joshua Cranmer
> - The "inputfields" stuff seemed over-engineered - any field can be
> blank, and if it's blank and the client decides it's required, it should
> just prompt in some way that it can figure out for itself. I left in the
> possibility of example usernames, though.
My $0.02:
* Different servers have different username formats: some use the
localpart only, while others have a full address as the server username.
* Some ISPs that cover large populations will use multiple servers, e.g.
east.isp.net or state.isp.net. Therefore, one would need to have
geographic information input to differentiate.
> - I haven't included any client-side settings (e.g. leaveMessageOnServer).
I don't know to what degree client-side settings make sense, since they
mostly fall under personal preference. Of course, most people with POP
accounts probably want to use leaveMessaegOnServer.
> Before we publicise this, I want to make sure that we are asking for all
> the right information. Here are my questions:
>
> - Do we need a field for "preferred server" (IMAP or POP) or do we
> always assume IMAP is superior where both are offered?
I'd say keep the field. Even better would have the Thunderbird end query
the user. Some IMAP servers (*cough*gmail*cough*) are not quite up to
par in terms of usability and may rather be POP servers.
> - Do we need to be more specific about Secure Auth technologies for POP,
> IMAP and/or SMTP? Thunderbird just has a checkbox and Does The Right
> Thing - is this true of most mail clients?
I'm not terribly knowledgeable of the whole SASL architecture other than
the basic underlying details. However, I believe most clients Do The
Right Thing, especially because the necessary information should be
built into the protocol.
> - Are the options for usernames good enough (see the note at the top)? I
> thought of doing something regexp-based but it seemed very complicated.
The only differentiation I see the need for is localpart/email specs
(OT: a tiny nit: password dialogs should detect email-based usernames so
that I don't have to type my password for
"Pidg...@gmail.com@gmail.com"). Then again, there's probably one ISP
that does something totally and incredibly off the wall.
> - For encryption, is it necessary to maintain a distinction between
> "TLS" and "SSL", as Thunderbird does? What's the difference?
TLS is enabled midsession via the STARTTLS command whereas SSL is
initiated via an SSL connection. You'll notice that selecting SSL
changes the port number. I think there's a difference between TLS and
SSL, but I'm not entirely sure what it is...
> - Do we need some way of saying "these servers are not available to all
> subscribers", e.g. for Yahoo! Mail, where you have to pay for POP access?
I doubt it.
Final note: I didn't participate in the original autoconfig discussions.
Gervase Markham
> It's true of most clients. The standard for SASL authentication (which
> is what this really is - we should probably call it that, rather than
> continuing with Thunderbird's bizarre 'secure auth' designation)
> requires that the server provide the client with a list of supported
> mechanisms, and the client pick its preferred mechanism from that list.
Offtopic: presumably there's some mechanism to stop someone doing an
MITM attack removing every option from the list apart from the one he
can break?
>> - For encryption, is it necessary to maintain a distinction between
>> "TLS" and "SSL", as Thunderbird does? What's the difference?
>
> TLS indicates that the server supports STARTTLS over the 'normal' port
> (this can be autodetected from the server's capabilities declaration,
> though) - with this, the client starts talking in clear text, but then
> upgrades the connection as part of the authentication handshake. SSL
> indicates that you should do a SSL protected stream over a different
> port (like https).
So "TLS, if available" means "STARTTLS if the server says it supports
it", and "TLS" means "STARTTLS, and if the server says it doesn't
support it, FAIL"?
IMAP has the options:
(1) Never (2) TLS, if available (3) TLS (4) SSL
POP has the options:
(1) Never (2) TLS, if available (3) TLS (4) SSL
SMTP has the options:
(1) No (2) STARTTLS, if available (3) STARTTLS (4) SMTP-over-SSL
(We should really make these consistent.) So, in each case, 1-3 are on a
normal port (110, 143, 25 etc.) and 4) is on a special SSL port?
Gerv
Tony Mechelynck
[...]
> - Do we need a field for "preferred server" (IMAP or POP) or do we
> always assume IMAP is superior where both are offered?
If both are offered, let the user choose. Personally I prefer POP, maybe
because I'm more accustomed to it, but anyway it's just my choice. I can
understand that you, for instance, would prefer IMAP but please don't
override my preferences.
Best regards,
Tony.
--
hundred-and-one symptoms of being an internet addict:
52. You ask a plumber how much it would cost to replace the chair in
front of
your computer with a toilet.
Magnus Melin
> IMAP has the options:
> (1) Never (2) TLS, if available (3) TLS (4) SSL
>
> POP has the options:
> (1) Never (2) TLS, if available (3) TLS (4) SSL
>
> SMTP has the options:
> (1) No (2) STARTTLS, if available (3) STARTTLS (4) SMTP-over-SSL
>
> (We should really make these consistent.)
Yes, that's bug 350314.
Gervase Markham
> Gervase Markham wrote:
>> - The "inputfields" stuff seemed over-engineered - any field can be
>> blank, and if it's blank and the client decides it's required, it should
>> just prompt in some way that it can figure out for itself. I left in the
>> possibility of example usernames, though.
>
> My $0.02:
> * Different servers have different username formats: some use the
> localpart only, while others have a full address as the server username.
Indeed. Both of these are covered by the proposal.
> * Some ISPs that cover large populations will use multiple servers, e.g.
> east.isp.net or state.isp.net. Therefore, one would need to have
> geographic information input to differentiate.
You mean f...@isp.net uses the east.isp.net server, and b...@isp.net uses
the west.isp.net server?
How does the user know to configure his client? Is he supposed to look
it up on their website?
>> - Do we need a field for "preferred server" (IMAP or POP) or do we
>> always assume IMAP is superior where both are offered?
>
> I'd say keep the field. Even better would have the Thunderbird end query
> the user.
No. This defeats the point of AutoConfigure. We shouldn't be asking them
to take decisions they don't have the information to make.
> Some IMAP servers (*cough*gmail*cough*) are not quite up to
> par in terms of usability and may rather be POP servers.
We could have a field. But it would end up pretty subjective.
Gerv
Gervase Markham
> If both are offered, let the user choose. Personally I prefer POP, maybe
> because I'm more accustomed to it, but anyway it's just my choice. I can
> understand that you, for instance, would prefer IMAP but please don't
> override my preferences.
No. This defeats the purpose of _Auto_Config. There's no way we can give
a user enough information to make an informed choice. ("Now, POP, see,
that's an older protocol which, for example, doesn't have the capability
for subfolders. Yes, I know you can have subfolders anyway. That's
because they are local to your client. What does that mean? Well...")
Either we always prefer IMAP, or we always prefer POP, or we have a field.
Gerv
Joshua Cranmer
> Joshua Cranmer wrote:
>> * Some ISPs that cover large populations will use multiple servers, e.g.
>> east.isp.net or state.isp.net. Therefore, one would need to have
>> geographic information input to differentiate.
>
> You mean f...@isp.net uses the east.isp.net server, and b...@isp.net uses
> the west.isp.net server?
Yes.
> How does the user know to configure his client? Is he supposed to look
> it up on their website?
It would be something like "State: " and then a dropdown of, e.g.,
Virginia, New York, etc.
Tony Mechelynck
Well, have a field then, which will be pre-filled if the chosen ISP
supports only one value.
Best regards,
Tony.
--
What use is magic if it can't save a unicorn?
-- Peter S. Beagle, "The Last Unicorn"
Gervase Markham
>> How does the user know to configure his client? Is he supposed to look
>> it up on their website?
>
> It would be something like "State: " and then a dropdown of, e.g.,
> Virginia, New York, etc.
You mean the user selects that on the ISP's site, and then it gives them
localized config info?
I'm not sure I can see a way to encode this without making the
spreadsheet horribly complicated. Even if we made a way to get the
client to prompt the user for their state, we'd need a way to encode all
the possible responses and the correct settings for each - which setting
varied across "states", and which ones stayed the same, etc.
For my money, these ISPs can just have the competitive disadvantage of
being harder to configure. :-)
Gerv
Gervase Markham
> supports only one value.
If the chosen ISP supports only one value, it can be blank. I think we
can just about manage to include the necessary intelligence on the
client side :-P
Gerv
Joshua Cranmer
> Joshua Cranmer wrote:
>>> How does the user know to configure his client? Is he supposed to look
>>> it up on their website?
>> It would be something like "State: " and then a dropdown of, e.g.,
>> Virginia, New York, etc.
>
> You mean the user selects that on the ISP's site, and then it gives them
> localized config info?
In Thunderbird, of course :-)
Chris Ilias
Is Gmail the only case that offers both (at least on a large scale)? In
the Gmail settings panel, there is a setting to enable each, and (IIRC)
both are disabled by default. That means in order for Thunderbird to
work, the user has to enable POP or IMAP access via Gmail; so I don't
think Thunderbird needs to inform the user of the difference if both
"Gmail-POP" and "Gmail-IMAP" are offered.
--
Chris Ilias <http://ilias.ca>
List-owner: support-firefox, support-thunderbird, test-multimedia
Gervase Markham
So you aren't actually answering my question; but then maybe I didn't
put it very well.
* An ISP has this setup
* A user wants to configure their email program (say OE, or TB 2)
* How do they find out their settings?
Gerv
Gervase Markham
> Is Gmail the only case that offers both (at least on a large scale)? In
> the Gmail settings panel, there is a setting to enable each, and (IIRC)
> both are disabled by default. That means in order for Thunderbird to
> work, the user has to enable POP or IMAP access via Gmail; so I don't
> think Thunderbird needs to inform the user of the difference if both
> "Gmail-POP" and "Gmail-IMAP" are offered.
But we don't actually want to "offer" both, because that's asking the
user to make a decision with no information on how to decide.
I see there's a problem with the access not being enabled by default.
Ideally, we would prompt the user to do so. But that requires error
messages or warning text, which has to be localized, etc.
And the trouble is that we can't distinguish "I fat-fingered my
password" from "I haven't enabled IMAP in my Gmail settings". Both of
them return a login failure.
Would a single field, "needs external configuration", do? If this was
Yes, you would put up a message saying:
"Note: using Thunderbird with <service> may require you to contact
<provider> or connect to <service> and enable support for <protocol>."
?
Gerv
Chris Ilias
> Chris Ilias wrote:
>> Is Gmail the only case that offers both (at least on a large scale)? In
>> the Gmail settings panel, there is a setting to enable each, and (IIRC)
>> both are disabled by default. That means in order for Thunderbird to
>> work, the user has to enable POP or IMAP access via Gmail; so I don't
>> think Thunderbird needs to inform the user of the difference if both
>> "Gmail-POP" and "Gmail-IMAP" are offered.
>
> But we don't actually want to "offer" both, because that's asking the
> user to make a decision with no information on how to decide.
I think they will have the info to decide, because they are forced to
change the setting in Gmail.
Joshua Cranmer
> * An ISP has this setup
> * A user wants to configure their email program (say OE, or TB 2)
> * How do they find out their settings?
In TB, the configuration would look like this:
+--------------------------------------------+
| Enter your email address: [______________] |
| Enter your state: [Virginia_____v] |
+--------------------------------------------+
(or whatever). The autoconfiguration would have an internal
state->server map.
Gervase Markham
> Gervase Markham wrote:
>> * An ISP has this setup
>> * A user wants to configure their email program (say OE, or TB 2)
>> * How do they find out their settings?
>
No, no! :-) Assume TB does not have any sort of autoconfig. Then answer
this question from a perspective of what happens _today_:
* An ISP has this setup
* A user wants to configure their email program (say OE, or TB 2)
* How do they find out their settings from their ISP?
Gerv
Gervase Markham
> I think they will have the info to decide, because they are forced to
> change the setting in Gmail.
Sorry, I should have been more clear. I am not saying that they won't
know which to pick in order to match the Gmail setting (clearly they
will), I'm saying that won't know which to pick, full stop.
Ask your gran: "Would you like to access you email via POP or IMAP?". If
she were a sensible woman, determined to use the rest of her life to
best advantage, she would say "I don't know the difference and I don't
care. Pick the one that works."
Gerv
Joshua Cranmer
> * An ISP has this setup
> * A user wants to configure their email program (say OE, or TB 2)
The typical way seems to be some sort of "send users to specific pages
based on user-input location." That said, I think some ISPs would just
have their service technician change your Outlook settings or whatnot.
Nikolay Shopik
> Pick the one that works."
Not only gran, but any regular John will answer same thing.
Bryan Clark
Chris Ilias wrote:I think they will have the info to decide, because they are forced to change the setting in Gmail.Sorry, I should have been more clear. I am not saying that they won't know which to pick in order to match the Gmail setting (clearly they will), I'm saying that won't know which to pick, full stop.
For my friends and family to get email working from one of the web mails (gmail, yahoo mail, msn/live mail) they need to give ~4 pieces of information.
- name
- username (create/reuse one)
- password (create/reuse one)
- {security question}
With those ~4 things they are up and running with a web mail client and can send and receive mails from people. We, with thunderbird, should be able to offer the same thing (working email) with the same number of required pieces of information or less.
So how do we get most people's email setup by asking for only 4 things?
Good discussion,
~ Bryan
Tony Mechelynck
Both work, however they work differently, with different plus or minus
points for different cases of use, so one or the other will be the
better choice depending on what _the user_ uses them for.
If I want to archive some mailing lists posts forever, for later
reference, I may choose POP, which will already bring them on my
computer. If I were married and had a lover, I might prefer to give my
IMAP or HTTP-webmail email address to my lover so her mail would _not_
get stored on my computer, or only cached for the time strictly
necessary to read it.
Or you could also say neither works until you use your browser to access
your Gmail settings and enable it.
In such cases, as well as in cases where the user inputs an ambiguous
name, IMHO he should be given a further list or box from which to choose:
Enter your mail provider's name: [ Gmail ]
[ Gmail (POP) |v]
| Gmail (IMAP) |
Enter your mail provider's name: [ Yahoo ]
[ yahoo.com (USA etc.) |v]
| yahoo.co.uk (British Isles) |
| yahoo.fr (France) |
|... |
etc.
Best regards,
Tony.
--
Friends, Romans, Hipsters,
Let me clue you in;
I come to put down Caesar, not to groove him.
The square kicks some cats are on stay with them;
The hip bits, like, go down under; so let it lay with Caesar. The cool
Brutus
Gave you the message: Caesar had big eyes;
If that's the sound, someone's copping a plea,
And, like, old Caesar really set them straight.
Here, copacetic with Brutus and the studs, -- for Brutus is a real cool cat;
So are they all, all cool cats, --
Come I to make this gig at Caesar's laying down.
Gervase Markham
> If I want to archive some mailing lists posts forever, for later
> reference, I may choose POP, which will already bring them on my
> computer.
Except that we would teach people to do that by copying them to Local
Folders; the underlying mail transport technology is irrelevant.
> Enter your mail provider's name: [ Gmail ]
> [ Gmail (POP) |v]
> | Gmail (IMAP) |
We may have to do this if it's possible for them to enable either and we
have no way of telling which.
> Enter your mail provider's name: [ Yahoo ]
> [ yahoo.com (USA etc.) |v]
> | yahoo.co.uk (British Isles) |
> | yahoo.fr (France) |
This can be determined from their email address.
Gerv
Tony Mechelynck
> Tony Mechelynck wrote:
>> If I want to archive some mailing lists posts forever, for later
>> reference, I may choose POP, which will already bring them on my
>> computer.
>
> Except that we would teach people to do that by copying them to Local
> Folders; the underlying mail transport technology is irrelevant.
>
>> Enter your mail provider's name: [ Gmail ]
>> [ Gmail (POP) |v]
>> | Gmail (IMAP) |
>
> We may have to do this if it's possible for them to enable either and we
> have no way of telling which.
Yes, it is possible to enable either or both, or none to use only
webmail, and AFAIK the only way to know which one(s) (if any) is
enabled, is to try connecting to both, with a valid username and password.
>
>> Enter your mail provider's name: [ Yahoo ]
>> [ yahoo.com (USA etc.) |v]
>> | yahoo.co.uk (British Isles) |
>> | yahoo.fr (France) |
>
> This can be determined from their email address.
>
> Gerv
You mean for most email addresses the user would input only his email
address, and maybe non-technical things such as "how to name this
account for display" and "how do you want your name to appear on the
from-line", and Thunderbird would do the rest?
Note that there may be a problem for the outgoing mail server: at some
ISPs including mine, when you connect your _modem_ it gets blocked (from
the other end of the line) from talking on the SMTP ports with any
address except that ISP's own SMTP servers' -- for instance I can send
mail with my @yahoo.co.uk address or my @gmail.com address on the From
line, but only by sending it via relay.belgacom.net or relay.skynet.be
(which are both operated by Belgacom, the "formerly monopolistic"
Belgian phone company).
I don't remember if it was in this thread or even this newsgroup, but
I've seen recently a post by a user with accounts at two different
Belgian ISPs, and he mentioned that he had to use one or the other SMTP
account, depending on which of his two ISPs his modem was connected to.
Best regards,
Tony.
--
If you didn't get caught, did you really do it?
Nikolay Shopik
> Note that there may be a problem for the outgoing mail server: at some
> ISPs including mine, when you connect your _modem_ it gets blocked (from
> the other end of the line) from talking on the SMTP ports with any
> address except that ISP's own SMTP servers' -- for instance I can send
> mail with my @yahoo.co.uk address or my @gmail.com address on the From
> line, but only by sending it via relay.belgacom.net or relay.skynet.be
> (which are both operated by Belgacom, the "formerly monopolistic"
> Belgian phone company).
Almost all servers currently support 587 port for submission mails with
authentication.
Mark Banner
> No, no! :-) Assume TB does not have any sort of autoconfig. Then answer
> this question from a perspective of what happens _today_:
>
> * An ISP has this setup
> * A user wants to configure their email program (say OE, or TB 2)
> * How do they find out their settings from their ISP?
Joshua said one method available today (a help page), however there is
another method - the isp could provide an extension with an rdf file
(see http://developer.mozilla.org/en/docs/Thunderbird_ISP_hooks). Though
I don't know of any provider currently doing this.
Standard8
Mark Banner
> Ask your gran: "Would you like to access you email via POP or IMAP?". If
> she were a sensible woman, determined to use the rest of her life to
> best advantage, she would say "I don't know the difference and I don't
> care. Pick the one that works."
We should make it easy to select either for those who do care, but
provide a reasonable default for those who don't (or maybe some words of
which one you want to choose).
Its not just gmail that offers both - plus.net and others also
frequently offer both capabilities.
Standard8
Gervase Markham
> You mean for most email addresses the user would input only his email
> address, and maybe non-technical things such as "how to name this
> account for display" and "how do you want your name to appear on the
> from-line", and Thunderbird would do the rest?
Did you actually read the Autoconfig proposal before joining the
discussion? :-)
Yes, that's exactly the point.
> Note that there may be a problem for the outgoing mail server: at some
> ISPs including mine, when you connect your _modem_ it gets blocked (from
> the other end of the line) from talking on the SMTP ports with any
> address except that ISP's own SMTP servers' -- for instance I can send
> mail with my @yahoo.co.uk address or my @gmail.com address on the From
> line, but only by sending it via relay.belgacom.net or relay.skynet.be
> (which are both operated by Belgacom, the "formerly monopolistic"
> Belgian phone company).
Is this also true on port 587? Or when using SMTP Auth?
Gerv
Tony Mechelynck
I didn't know about 587 and apparently it goes through; but IIRC neither
25 nor 465 do.
Best regards,
Tony.
--
The Schwine-Kitzenger Institute study of 47 men over the age of 100
showed that all had these things in common:
(1) They all had moderate appetites.
(2) They all came from middle class homes
(3) All but two of them were dead.
Nikolay Shopik
> Is this also true on port 587? Or when using SMTP Auth?
>
Gerv,
BTW we should take this into account, when server support multiply ports
for STMP submission, and prefer 587.
I still by myself not seeing many _broadband_ providers blocking 25 but
using 587 port with auth is best practices per RFC. And will come in
handy when many ISP decide to block 25 port to stop zombies.