Auto-configuration format/structure discussion
Joey Minta
generated a huge amount of energy an interest. It's a big topic, however,
and I wanted to spin-off a smaller set of questions here for discussion,
since that's the part that I, at least, am ready to move on in the near
future.
The big questions here are data format and structure. At the moment, I
don't want to discuss how we'll get this data to users, or how we'll display
it in the UI. I just wanted to build some consensus on the format and
structure used, and to stick that into the existing UI/delivery framework
that TB2 shipped with.
On the format side, it seems to 2 leading contenders are JSON and XML. Dan
already touched off the debate here:
http://groups.google.com/group/mozilla.dev.apps.thunderbird/msg/06ba42a8ea7a9569
I tend to agree with Dan's comments in favor of JSON. It also has the
advantage of having less mark-up necessary to transmit the information.
Moreover, internally, we're converting any format into javascript objects,
and that's precisely what JSON was designed to make easy.
There has been little discussion about the actual structure of the data.
That is, what data needs to be included in these files? The current format
actually allows for any attribute present in nsIMsgIncomingServer,
nsISmtpServer, or nsIMsgIdentity to be specified in these data-files.
That's an incredible amount of flexibility that's difficult to argue
against. I do think we need to be more clear about the minimum amount of
data needed, but that's just a documentation issue.
The structure also includes some flags about UI presentation, but I think we
should set those aside until we have a better grasp on what UI we will be
presenting and how someone might wish to tweak that.
Thoughts? Comments? Questions?
-Joey
Ben Bucksch
> On the format side, it seems to 2 leading contenders are JSON and XML. Dan
> already touched off the debate here:
> http://groups.google.com/group/mozilla.dev.apps.thunderbird/msg/06ba42a8ea7a9569
> I tend to agree with Dan's comments in favor of JSON. It also has the
> advantage of having less mark-up necessary to transmit the information.
> Moreover, internally, we're converting any format into javascript objects,
> and that's precisely what JSON was designed to make easy.
>
Well, how are you going to read them in? JSON was designed to make it
easy by just doing eval() on it. But you can't do that there, because
the content comes from the network and you're in chrome. And anything
that uses eval() in this setup would be more or less dangerous, even if
you do it in other contexts, try to validate or whatever. You'd need to
use a real parser that makes no use of the JS engine.
OTOH, we have E4X available in Mozilla. That makes it dead-simple to
parse and access XML as JS objects, and it's safe. I am using that in
TomTom HOME a lot and it is very comfortable. I think that is way easier
than any secure JSON in chrome would be.
Of course you need to check the /values/ you read in, but you need to do
that anyways. The format XML via E4X in itself does not pose a risk, but
JSON in chrome does.
Also, I would like the format to be usable by other mail clients as
well. While there are JSON libraries for all languages, the likeliness
that Evolution, Apple Mail or whatever already have access to an XML
lib, and know how to use it, but not to an JSON lib, is high.
> There has been little discussion about the actual structure of the data.
> That is, what data needs to be included in these files? The current format
> actually allows for any attribute present in nsIMsgIncomingServer,
> nsISmtpServer, or nsIMsgIdentity to be specified in these data-files.
> That's an incredible amount of flexibility that's difficult to argue
> against.
If you want to get the data from the network as we do, then you can't
use the current approach to just iterate over everything that's there,
using JS tricks to set all properties of the internal object to the file
data, without regard to what it is, because that's a security threat.
So, you need to specify exactly what properties you support, and
implement that manually.
I personally would also keep the amount of settings rather small than
large and leave most settings to the UA (defaults) and the user.
> I do think we need to be more clear about the minimum amount of data needed
Yes.
> The structure also includes some flags about UI presentation, but I think we
> should set those aside until we have a better grasp on what UI we will be
> presenting and how someone might wish to tweak that.
>
I think the display name for the whole account is important, and the
naming convention of the provider for username is as well. Some
providers call it "username", some "login name", AOL "screen name". If
we ever need to ask the user about it, it is important, because that's
what the paper in the user's hand will say, and he'll get confused about
"username" (is that "Ben Bucksch"?) when the paper says "login name",
because often the paper says a lot: real name, email address, account
number, website username, POP username, and a different DSL account
name, and they may be all different.
I made a concrete proposal for the XML format based on the Google RDf
file that we ship.
http://wiki.mozilla.org/Tb:Autoconfiguration
Ben
--
When responding privately, please remove the ".news" from the email address.
Joey Minta
wrote:
> Well, how are you going to read them in? JSON was designed to make it
> easy by just doing eval() on it. But you can't do that there, because
> the content comes from the network and you're in chrome. And anything
> that uses eval() in this setup would be more or less dangerous, even if
> you do it in other contexts, try to validate or whatever. You'd need to
> use a real parser that makes no use of the JS engine.
No. In fact, the normal JSON format isn't valid javascript, avoiding
precisely the eval problem. As for the question of how we're going to
safely read it in, this has already been accomplished via Mozilla's nsIJSON
interface. See
http://mxr.mozilla.org/seamonkey/source/dom/public/idl/json/nsIJSON.idl
OTOH, we have E4X available in Mozilla. That makes it dead-simple to
> parse and access XML as JS objects, and it's safe. I am using that in
> TomTom HOME a lot and it is very comfortable. I think that is way easier
> than any secure JSON in chrome would be.
Again, safety is not an issue, given that (1) we're not using eval and (2)
the existing parser already strips out potentially dangerous data. As far
as ease of use, I don't think it gets any easier than
var JSON = Cc["@mozilla.org/dom/json;1"].createInstance(Ci.nsIJSON);
var dataObject = JSON.decode(data); // data from streamed from file
Having worked with both E4X and JSON, I'm comfortable saying that JSON is
simpler for javascript to deal with.
Of course you need to check the /values/ you read in, but you need to do
> that anyways. The format XML via E4X in itself does not pose a risk, but
> JSON in chrome does.
As above, both formats present the same risk-surface.
Also, I would like the format to be usable by other mail clients as
> well. While there are JSON libraries for all languages, the likeliness
> that Evolution, Apple Mail or whatever already have access to an XML
> lib, and know how to use it, but not to an JSON lib, is high.
It may be true that those platforms do have more ready access to one parser
than the other (although I *believe* both support scripting via javascript,
in which case the existing public javascript-json-parser is readily
available to fix this.). Even granting this assumption though, I think this
is a minor benefit for XML, that doesn't outweigh the previously outlined
benefits of JSON.
If you want to get the data from the network as we do, then you can't
> use the current approach to just iterate over everything that's there,
> using JS tricks to set all properties of the internal object to the file
> data, without regard to what it is, because that's a security threat.
I fail to see the threat. XPConnect will throw if you try to set a property
not defined in the interfaces, and I don't see how setting any existing
property creates a threat. Can you expand on this?
> I made a concrete proposal for the XML format based on the Google RDf
> file that we ship.
> http://wiki.mozilla.org/Tb:Autoconfiguration<https://lists.mozilla.org/listinfo/dev-apps-thunderbird>
You can see actual implementations of the JSON files I'm proposing (directly
converted from the RDF) in bug 418693.
-Joey
Ben Bucksch
> I don't see how setting any existing property creates a threat. Can
> you expand on this?
Because you don't know what the property does. You can't just let
network data set *random* properties of your objects.
Joey Minta
wrote:
> Joey Minta schrieb:
Please define what you mean by "random." As I said, XPConnect will throw if
you try to set a property not defined in the interface. You can't set the
.evil property on an nsIMsgIncomingServer, no matter who you are, because
it's a native-wrapped object, and that interface has no attribute named
"evil." If by "random" you mean a random choice from the finite set of
properties defined in the interface, then my original question remains. We
*want* people to set these properties, and none of them are risky to set, to
my knowledge. Do you know of *defined* properties which are a security risk
to set?
-Joey
Joey Minta
about the security impact of the JSON format. eval() indeed works on this
format, contrary to my previous claim. I maintain that any security
concerns can be alleviated with the sandbox-ing and safety-checking done by
current Mozilla implementations.
-Joey
On Tue, Mar 11, 2008 at 1:37 PM, Joey Minta <jmint...@gmail.com> wrote:
> On Tue, Mar 11, 2008 at 1:15 PM, Ben Bucksch <ben.buck...@beonex.com>
> wrote:
>
> > Well, how are you going to read them in? JSON was designed to make it
> > easy by just doing eval() on it. But you can't do that there, because
> > the content comes from the network and you're in chrome. And anything
> > that uses eval() in this setup would be more or less dangerous, even if
> > you do it in other contexts, try to validate or whatever. You'd need to
> > use a real parser that makes no use of the JS engine.
>
> No. In fact, the normal JSON format isn't valid javascript, avoiding
> precisely the eval problem. As for the question of how we're going to
> safely read it in, this has already been accomplished via Mozilla's nsIJSON
> interface. See
> http://mxr.mozilla.org/seamonkey/source/dom/public/idl/json/nsIJSON.idl
>
> OTOH, we have E4X available in Mozilla. That makes it dead-simple to
> > parse and access XML as JS objects, and it's safe. I am using that in
> > TomTom HOME a lot and it is very comfortable. I think that is way easier
> > than any secure JSON in chrome would be.
>
> Again, safety is not an issue, given that (1) we're not using eval and (2)
> the existing parser already strips out potentially dangerous data. As far
> as ease of use, I don't think it gets any easier than
>
> var JSON = Cc["@mozilla.org/dom/json;1"].createInstance(Ci.nsIJSON);
>
> var dataObject = JSON.decode(data); // data from streamed from file
>
> Having worked with both E4X and JSON, I'm comfortable saying that JSON is
> simpler for javascript to deal with.
>
> Of course you need to check the /values/ you read in, but you need to do
> > that anyways. The format XML via E4X in itself does not pose a risk, but
> > JSON in chrome does.
>
> As above, both formats present the same risk-surface.
>
> Also, I would like the format to be usable by other mail clients as
> > well. While there are JSON libraries for all languages, the likeliness
> > that Evolution, Apple Mail or whatever already have access to an XML
> > lib, and know how to use it, but not to an JSON lib, is high.
>
> It may be true that those platforms do have more ready access to one
> parser than the other (although I *believe* both support scripting via
> javascript, in which case the existing public javascript-json-parser is
> readily available to fix this.). Even granting this assumption though, I
> think this is a minor benefit for XML, that doesn't outweigh the previously
> outlined benefits of JSON.
>
> If you want to get the data from the network as we do, then you can't
> > use the current approach to just iterate over everything that's there,
> > using JS tricks to set all properties of the internal object to the file
> > data, without regard to what it is, because that's a security threat.
>
> I fail to see the threat. XPConnect will throw if you try to set a
> property not defined in the interfaces, and I don't see how setting any
> existing property creates a threat. Can you expand on this?
>
>
> > I made a concrete proposal for the XML format based on the Google RDf
> > file that we ship.
Simon Wilkinson
On 11 Mar 2008, at 18:37, Joey Minta wrote:
>
> If you want to get the data from the network as we do, then you can't
>> use the current approach to just iterate over everything that's
>> there,
>> using JS tricks to set all properties of the internal object to
>> the file
>> data, without regard to what it is, because that's a security threat.
>
> property
> not defined in the interfaces, and I don't see how setting any
> existing
> property creates a threat. Can you expand on this?
The problem here seems to be that you're moving all of the methods in
these interfaces from taking data from a trusted source (chrome, and
the user), to having to take data from an untrusted one (the
internet). Whilst this may, or may not, be a problem with the
properties that are currently implemented, any future modifications
would also have to be aware of this, and implemented accordingly.
This seems very brittle from a security point of view.
Simon.
Dan Mosedale
>
>> Also, I would like the format to be usable by other mail clients as
>> well. While there are JSON libraries for all languages, the likeliness
>> that Evolution, Apple Mail or whatever already have access to an XML
>> lib, and know how to use it, but not to an JSON lib, is high.
>
> It may be true that those platforms do have more ready access to one parser
> than the other (although I *believe* both support scripting via javascript,
> in which case the existing public javascript-json-parser is readily
> available to fix this.). Even granting this assumption though, I think this
> is a minor benefit for XML, that doesn't outweigh the previously outlined
> benefits of JSON.
I think it's a little hard to tell from this angle exactly how big that
benefit of XML is. In particular, both are technically feasible for any
client to implement, but whether or not a client chooses to implement
something has a bunch to do with having the tools to deal with it at
hand. And XML is certainly more widely spread than JSON in non-webby
environments today.
I'm beginning to wonder if the prudent thing isn't to go with XML, given
the unknown but possibly-non-trivial advantage conferred to us above.
It's not clear to me that that any of the JSON advantages are non-trivial.
Dan
Eddy Nigg (StartCom Ltd.)
>
> I think it's a little hard to tell from this angle exactly how big that
> benefit of XML is. In particular, both are technically feasible for any
> client to implement, but whether or not a client chooses to implement
> something has a bunch to do with having the tools to deal with it at
> hand. And XML is certainly more widely spread than JSON in non-webby
> environments today.
>
> I'm beginning to wonder if the prudent thing isn't to go with XML, given
> the unknown but possibly-non-trivial advantage conferred to us above.
> It's not clear to me that that any of the JSON advantages are non-trivial.
>
>
into consideration the saving/exporting of an existing account
configuration to such a config file. This certainly would be helpful for
any small home user with a handful of accounts and to smaller ISPs, if
they just could export the config to a file and upload to a web server.
JSON or XML....whatever works best and might be easier to implement.
--
Regards
Signer: Eddy Nigg, StartCom Ltd. <http://www.startcom.org>
Jabber: star...@startcom.org <xmpp:star...@startcom.org>
Blog: Join the Revolution! <http://blog.startcom.org>
Phone: +1.213.341.0390
Dan Mosedale
>
> The problem here seems to be that you're moving all of the methods in
> these interfaces from taking data from a trusted source (chrome, and the
> user), to having to take data from an untrusted one (the internet).
> Whilst this may, or may not, be a problem with the properties that are
> currently implemented, any future modifications would also have to be
> aware of this, and implemented accordingly. This seems very brittle from
> a security point of view.
Agreed.
Ben Bucksch wrote:
> So, you need to specify exactly what properties you support, and implement
> that manually. I personally would also keep the amount of settings
> rather small than large and leave most settings to the UA (defaults)
> and the user.
This sounds right to me as well. If people need to do weird or
unanticipated things, they can make their own extension and request that
we extend the list of properties that our data format knows about.
Dan
Ben Bucksch
>
> Because you don't know what the property does. You can't just let
> network data set *random* properties of your objects.
>
> If by "random" you mean a random choice from the finite set of
> properties defined in the interface
Yes.
> We *want* people to set these properties, and none of them are risky
> to set, to my knowledge.
"to my knowledge" is not enough, neither can you assume that there are
not APIs added which may be dangerous.
It's extremely bad from a security perspective to just let network
content set random properties of your internal objects. Don't do that.
We need a defined list of properties which we verified are OK, and set
only those.
Apart from that, I think it's unwise to say myObject[stringFromNetwork1]
= stringFromNetwork2, unless you verified that a) stringFromNetwork1 is
a [a-zA-Z0-9] string and in a whitelist and b) stringFromNetwork2
matches the data type that a) expects and the value is of the range that
a) expects. Some interfaces e.g. may take |null| or "" or 345433, but
not react in the way you expect.
This is security 1x1 - check all your inputs. In general, you're dead
without following this rule.
Ben
Ben Bucksch
> On further investigation, I'm going to need to refine my comments below
> about the security impact of the JSON format. eval() indeed works on this
> format, contrary to my previous claim. I maintain that any security
> concerns can be alleviated with the sandbox-ing and safety-checking done by
> current Mozilla implementations.
>
No. Don't eval() on any network strings, that's the worst thing you can
do in chrome JS. I don't think you can securely "sandbox" it - if the
remote JS sets a getter on the Object, and you access it from system JS,
I think you'd run the setter with system privileges.
In any case, even if you think you can secure it, this is still risky
(because you may miss something) , and the effects of security bugs are
devastating, and I don't see any need for taking this risk, because we
have alternatives.
Ben Bucksch
I propose that we specify a small list of properties we support, and how
they're expressed, and the allowed value ranges for each, and then, for
implementation, it's most practical when the implementation
1. gets each seperately (if ("port" in xmle4x.incomingServer) { var
ucPort = xmle4x.incomingServer.port; // uc = unchecked )
2. checks it (var ucPort = parseInt(ucPort); if (ucPort < 1 || ucPort
> 65535) throw "port number out of range"; var port = ucPort;) and
3. sets the XPCOM property manually (myNewNSIIncomingServer.port = port;)
Robert Kaiser
I don't think you understood correctly what Joey said here. He only said
that one _can_ eval() a JSON object, not that the Mozilla
implementations actually _do_ that.
In fact, as far as I know the only reason why there is a special service
for reading JSON data is to _not_ introduce security risks with it.
Robert Kaiser
Joey Minta
> In fact, as far as I know the only reason why there is a special service
> for reading JSON data is to _not_ introduce security risks with it.
>
Indeed. See
http://mxr.mozilla.org/seamonkey/source/js/src/xpconnect/loader/JSON.jsm#161for
one example of this protection.
-Joey
Michiel van Leeuwen
> I fail to see the threat. XPConnect will throw if you try to set a property
> not defined in the interfaces, and I don't see how setting any existing
> property creates a threat. Can you expand on this?
I wouldn't rely on xpconnect throwing. It might throw now, but it might
no longer when it gets implemented in js(2). Also, a c++ xpcom object
can actually accept all properties using the nsIXPCScriptable api.
For security, i would not depend on implementation details of the objects.
Michiel
Karsten Düsterloh
> On the format side, it seems to 2 leading contenders are JSON and
> XML. Dan already touched off the debate here:
> http://groups.google.com/group/mozilla.dev.apps.thunderbird/msg/06ba42a8ea7a9569
> I tend to agree with Dan's comments in favor of JSON. It also has
> the advantage of having less mark-up necessary to transmit the
> information. Moreover, internally, we're converting any format into
> javascript objects, and that's precisely what JSON was designed to
> make easy.
OTOH, JSON isn't exactly "reading-friendly". XML is more verbose (or
rather chatty), given, but this also makes it much more readable.
Plus, JSON is far more "webby" than "maily", while XML is accepted for a
wide collection of different data, so JSON doesn't strike me as the key
to portability/spreading...
If we're that anxious about size even with this rather small amount of
config data, we should consider a binary format. :-P
Karsten
--
Feel free to correct my English. :)
Carlos
bit late. From what I've been reading, it seems that the proposal is
focused on providing generalized hooks for ISPs, where the end user has
to enter its own personal information.
This approach is incomplete if you don't cover the case where the ISP
provides _all_ the account information (including the password) into a
file. The idea is that the end user, in this case, would have no need to
enter his/her own personal information, just open the file, accept a
dialog, and create the account.
My proposal is the following:
- The file is able to provide values for a number of parameters, including:
a) Main e-mail address and associated Full Name.
b) Login name.
c) Password.
d) Server names for sending/receiving, with the port numbers.
e) Protocol names for sending/receiving, with TLS options.
f) Additional identities (name / email aliases) for the main
e-mail/name identity.
g) IMAP Folder options 1: specification for default folders found in
the account. Folder names for sent/draft/spam, etc.
h) IMAP Folder options 2: individual folder default options for the
folders in g), including retention policy and keeping a local
offline copy.
i) POP inbox retention policy, in case of POP account.
- The file has a recognizable extension and IMT type, and opens a TB
assistant when invoked. If TB finds that the file lacks a required
value, such as the password or the login name, asks the user for it. If
the file lacks a value that has a default, apply the default instead of
asking.
- TB should be able to export a single account configuration to such a file.
- Some ISPs can provide such personalized files to customers through a
secure web page or other means. Other ISPs may just provide a generic
file without the specific personal data, along the lines you mention in
your proposal.
This is essentially the proposal I made at bug 389275. The
roughly-equivalent functionality present in MS Outlook has been very
useful to me, saving time and trouble when distributing account
information to users.
Regards,
Carlos
Nikolay Shopik
<...>
>
> My proposal is the following:
>
> - The file is able to provide values for a number of parameters, including:
> a) Main e-mail address and associated Full Name.
> b) Login name.
> c) Password.
> d) Server names for sending/receiving, with the port numbers.
> e) Protocol names for sending/receiving, with TLS options.
> f) Additional identities (name / email aliases) for the main
> e-mail/name identity.
> g) IMAP Folder options 1: specification for default folders found in
> the account. Folder names for sent/draft/spam, etc.
> h) IMAP Folder options 2: individual folder default options for the
> folders in g), including retention policy and keeping a local
> offline copy.
> i) POP inbox retention policy, in case of POP account.
>
This suits only in case importing and exporting your own file. In case
more site-wide these fields shouldn't be mandatory
a) email and Full Name.
c) Password
But still we can supply more personal like files for every user, in this
case we can get rid of one text box(Full Name) in our Auto-configuration
dialog.
+--------------------------------------------------+
| *Account Wizard* |
~ ~
| (o) E-Mail account |
| |
| E-Mail address: [ ] |
| Password: [ ] |
| |
| (o) Auto-detect:[ ] [Refresh] |
| |
| ( ) Manual configuration |
| |
| ( ) RSS News Feeds and Blogs |
| |
| ( ) Newsgroup account |
| |
| [ Back ] [ Next/Done ] [ Cancel ] |
+--------------------------------------------------+
Carlos
>
> This suits only in case importing and exporting your own file. In case
> more site-wide these fields shouldn't be mandatory
> a) email and Full Name.
> c) Password
>
> But still we can supply more personal like files for every user, in this
> case we can get rid of one text box(Full Name) in our Auto-configuration
> dialog.
>
If you read my full proposal, the e-mail, full name, password, etc.
aren't mandatory. Only values like the protocol, host name and port
should be required.
This means that my proposal cover both cases. Depending on how it's
implemented, this may mean more dialogs in the auto-configuration
assistant, but the trade-off justifies that.
Also, the proposal is not limited to self-exporting a file. E-Mail
service providers may easily automate the creation of such files.
Regards,
Carlos
Nikolay Shopik
> If you read my full proposal, the e-mail, full name, password, etc.
> aren't mandatory. Only values like the protocol, host name and port
> should be required.
>
> This means that my proposal cover both cases. Depending on how it's
> implemented, this may mean more dialogs in the auto-configuration
> assistant, but the trade-off justifies that.
>
> Also, the proposal is not limited to self-exporting a file. E-Mail
> service providers may easily automate the creation of such files.
>
>
> Regards,
> Carlos
Sorry I was inadvertent. What I also would like to include is any option
which can be changed in about:config can included with this file also
but not mandatory. This will create more like group policy for email.