Thunderbird and the LDAP c-sdk
118 views
Skip to first unread message
Mark Banner
Aug 13, 2012, 9:24:34 AM8/13/12
to tb-pl...@mozilla.org
For a long time, we've been using the Mozilla LDAP c-sdk
to provide the backend for the LDAP implementation in Thunderbird. I
don't know the full history, so I can't point you to it.
Over the last few years, the development of the LDAP c-sdk has been very limited, with few contributed patches.
Whilst it generally appears to be perfectly adequate for Thunderbird's needs, there's one problem that we keep on hitting. On Linux systems, Thunderbird can crash, if OpenLDAP is installed and set up for authentication (and probably a few variants of that). This is primarily due to the way module loading works on Linux, and the slight API differences between the two libraries.
Due to both the crash and the low maintenance of the Mozilla version, I'd like to propose that we consider replacing the Mozilla version that we're currently using. Discussions I've had previously with the module owners gave no objections to replacing it.
There's two possible routes I think we could go down:
Over the last few years, the development of the LDAP c-sdk has been very limited, with few contributed patches.
Whilst it generally appears to be perfectly adequate for Thunderbird's needs, there's one problem that we keep on hitting. On Linux systems, Thunderbird can crash, if OpenLDAP is installed and set up for authentication (and probably a few variants of that). This is primarily due to the way module loading works on Linux, and the slight API differences between the two libraries.
Due to both the crash and the low maintenance of the Mozilla version, I'd like to propose that we consider replacing the Mozilla version that we're currently using. Discussions I've had previously with the module owners gave no objections to replacing it.
There's two possible routes I think we could go down:
- Replace the sdk with the OpenLDAP version, I believe this
would need some API tweaks and obviously testing to ensure no
functional difference, or
- Replace the sdk with a javascript version, which would obviously need some rewriting.
I'm not sure what javascript options exist, hopefully Joshua or
someone can provide some additional information.
Mark.
Joshua Cranmer
Aug 13, 2012, 10:02:41 AM8/13/12
to tb-pl...@mozilla.org
On 8/13/2012 9:24 AM, Mark Banner
wrote:
Whilst it generally appears to be perfectly adequate for Thunderbird's needs, there's one problem that we keep on hitting. On Linux systems, Thunderbird can crash, if OpenLDAP is installed and set up for authentication (and probably a few variants of that). This is primarily due to the way module loading works on Linux, and the slight API differences between the two libraries.
The other issue, as I have discovered in some of my build
experiments, is that the LDAP SDKs are a special kind of broken in
terms of their build system. For example, the configure script
doesn't actually set up any flags that get passed to the linker for
the dynamic libraries.
There is an RFC for a common LDAP API (http://tools.ietf.org/html/rfc1823), so if we know that just that much is sufficient, we could link in theory to any LDAP library.
There is an RFC for a common LDAP API (http://tools.ietf.org/html/rfc1823), so if we know that just that much is sufficient, we could link in theory to any LDAP library.
I'm not sure what javascript options exist, hopefully Joshua or someone can provide some additional information.
There's <https://github.com/mcavage/node-ldapjs/>, which is
based off of node (as its name implies). I brief look at Google
results suggests no other major implementations. The largest
stumbling block in creating a client implementation is probably
ASN.1 encoding/decoding (I've ranted about this in the past); after
that, it's just a relatively simple binary protocol (although I
should note that binary protocols aren't terribly fun to use from a
JS implementation).
I have mixed feelings about pulling in a node-based JS library for use as a core-ish component. On the one hand, it means being to offload maintenance to other people, but we'd have to build up a non-trivial shim library for the entire low-level node stuff and possibly more libraries as well.
However, I think that before we can make any changes, we probably need to be able to run LDAP tests via the fakeserver...
I have mixed feelings about pulling in a node-based JS library for use as a core-ish component. On the one hand, it means being to offload maintenance to other people, but we'd have to build up a non-trivial shim library for the entire low-level node stuff and possibly more libraries as well.
However, I think that before we can make any changes, we probably need to be able to run LDAP tests via the fakeserver...
-- Joshua Cranmer News submodule owner DXR coauthor
Joshua Cranmer
Nov 18, 2012, 10:01:45 AM11/18/12
to tb-pl...@mozilla.org
On 8/13/2012 8:24 AM, Mark Banner
wrote:
There's two possible routes I think we could go down:
- Replace the sdk with the OpenLDAP version, I believe this would need some API tweaks and obviously testing to ensure no functional difference, or
- Replace the sdk with a javascript version, which would obviously need some rewriting.
It's been a few months with no one other than me interjecting in
this thread (am I really the only other person who has opinions on
LDAP code?), so I thought I'd try investigating consequences of
switching to OpenLDAP. The following is a list of potential issues
that I could come up with and my investigative results:
1. License compatibility.
OpenLDAP's license is essentially equivalent to three-clause BSD, so there's no incompatibility there.
2. Ability to use NSPR/NSS as underlying base library.
There is a configure option to use nss, --withtls=moznss. Compiling with this flag and pointing to libraries in Thunderbird's dist directory worked, but careful investigation of build ordering leads me to believe that platform configure happens too early (it looks for the nssutil.h include header, which doesn't happen in the right place). Tweaking the -I paths may help here. But, given a completed Thunderbird build, the program built correctly, and I verified via ldd that all of the library dependencies were their proper NSS/NSPR equivalents.
3. Size.
OpenLDAP, as measured by the unpacked version of openldap-2.4.33.tgz, is 27MB of disk space. ldap/sdks is 21MB, 6.5MB of which are .hg's metadata files. OpenLDAP's official repository is in git, which makes doing something akin to our current client.py checkout model annoying, so I assume we would switch to OpenLDAP by importing a static copy of its build code into the tree. By comparison, the mailnews, mail, and suite directories are each about 22-25MB, which means we would be increasing the size of comm-central by roughly 1/3 if we imported it to hg.
That said, it's still possible to trim down the directory-size of OpenLDAP. It has 7.2MB in a doc directory which could be removed; another 3.5MB in a contrib directory which is even easier to delete (no references in scripts); 2.7MB in the test directory; 7.0MB of code for the server-side components. Removing any code other than the contrib directory would require modifying the configure and Makefile.in files to remove references to these directories, but this is not a difficult task. I'm assuming that the contrib and doc code would definitely be deleted, as they contribute nothing to Thunderbird, bringing import size down to 16MB. The OpenLDAP server-side code, though large, would be useful to have for test purposes. Being able to validate the code of comm-central-built OpenLDAP via the tests is potentially useful, but the test suites are not fast to run and definitely require server-side code.
4. Integration in build system.
As I alluded to earlier, there are multiple options for handling OpenLDAP. We could:
a) Treat it like a system library, and provide scripts for building it before our build.
b) Add a --with-system-ldap that uses the system LDAP library, and build OpenLDAP via subconfigure if not present. Distros would probably love us. Note that --with-system-ldap does provide some interesting issues if we wish to reuse slapd for LDAP tests.
c) Unconditionally build OpenLDAP as a subconfigure, or perhaps guarded with an --enable-ldap option.
Note that mconley's addressbook work would eventually reduce our dependency on LDAP to a runtime library dependency via use of ctypes. Until then, however, we would need OpenLDAP to precede ldap/xpcom in our build configuration. OpenLDAP, for our purposes, can easily build via |make| followed by |make install| (we would override prefix to install libraries into dist/lib and headers into dist/include).
5. Other requirements
OpenLDAP apparently relies on the Cyrus SASL library for SASL support. This means that just checking out OpenLDAP won't give us SASL support, which is a bigger problem on Windows and OS X than on Linux, where I suspect we'd use system OpenLDAP libraries (likely configured with SASL support) instead. For comparison, the ldapjs library also lacks SASL support, while the Mozilla LDAP SDKs does include SASL support. There's also some other includes apparently necessary for a subset of the server backends, like db.h; I haven't played around with backend server configs, though.
Thoughts on ways forward:
Replacing our underlying LDAP library is a big change that shouldn't be taken lightly, especially when we have no test coverage of our LDAP code as-is. It has been suggested in the past (via a comment on one of my blog posts) that we rely on OpenLDAP's server for test code; if people indicate that switching to OpenLDAP is a desirable outcome, then this would be an acceptable way to start writing LDAP tests IMHO. The way to move forward would then look like this:
1. Include basic functionality tests for LDAP using a version of LDAP that is assumed to be installed. Obviously, these wouldn't run on tinderbox as-is, but they would work for people working on doing the migration. Perhaps these tests could be enabled if a configure option like --enable-ldap-tests=/path/to/slapd. I already have some ideas about how the slapd binaries could be hooked up to a JS controller (it involves a lot of nsIProcess.run, so this may cause trouble if used in mozmill tests :( ).
2. Include a --with-system-ldap flag that would use system-installed libraries instead of ldap/sdks for our ldap/xpcom code. Equivalent correctness could be verified with the LDAP tests, and Linux distros could use these flags instead and fix the biggest problem with Mozilla's LDAP implementation right now. It's also possible that this could be the end state of our LDAP switch-out: we'd support both versions of LDAP, and fallback on Mozilla's implementation if OpenLDAP isn't installed.
3. Drop a copy of OpenLDAP into our tree and delete the code for using Mozilla LDAP SDKs.
Thoughts/questions/comments/concerns?
1. License compatibility.
OpenLDAP's license is essentially equivalent to three-clause BSD, so there's no incompatibility there.
2. Ability to use NSPR/NSS as underlying base library.
There is a configure option to use nss, --withtls=moznss. Compiling with this flag and pointing to libraries in Thunderbird's dist directory worked, but careful investigation of build ordering leads me to believe that platform configure happens too early (it looks for the nssutil.h include header, which doesn't happen in the right place). Tweaking the -I paths may help here. But, given a completed Thunderbird build, the program built correctly, and I verified via ldd that all of the library dependencies were their proper NSS/NSPR equivalents.
3. Size.
OpenLDAP, as measured by the unpacked version of openldap-2.4.33.tgz, is 27MB of disk space. ldap/sdks is 21MB, 6.5MB of which are .hg's metadata files. OpenLDAP's official repository is in git, which makes doing something akin to our current client.py checkout model annoying, so I assume we would switch to OpenLDAP by importing a static copy of its build code into the tree. By comparison, the mailnews, mail, and suite directories are each about 22-25MB, which means we would be increasing the size of comm-central by roughly 1/3 if we imported it to hg.
That said, it's still possible to trim down the directory-size of OpenLDAP. It has 7.2MB in a doc directory which could be removed; another 3.5MB in a contrib directory which is even easier to delete (no references in scripts); 2.7MB in the test directory; 7.0MB of code for the server-side components. Removing any code other than the contrib directory would require modifying the configure and Makefile.in files to remove references to these directories, but this is not a difficult task. I'm assuming that the contrib and doc code would definitely be deleted, as they contribute nothing to Thunderbird, bringing import size down to 16MB. The OpenLDAP server-side code, though large, would be useful to have for test purposes. Being able to validate the code of comm-central-built OpenLDAP via the tests is potentially useful, but the test suites are not fast to run and definitely require server-side code.
4. Integration in build system.
As I alluded to earlier, there are multiple options for handling OpenLDAP. We could:
a) Treat it like a system library, and provide scripts for building it before our build.
b) Add a --with-system-ldap that uses the system LDAP library, and build OpenLDAP via subconfigure if not present. Distros would probably love us. Note that --with-system-ldap does provide some interesting issues if we wish to reuse slapd for LDAP tests.
c) Unconditionally build OpenLDAP as a subconfigure, or perhaps guarded with an --enable-ldap option.
Note that mconley's addressbook work would eventually reduce our dependency on LDAP to a runtime library dependency via use of ctypes. Until then, however, we would need OpenLDAP to precede ldap/xpcom in our build configuration. OpenLDAP, for our purposes, can easily build via |make| followed by |make install| (we would override prefix to install libraries into dist/lib and headers into dist/include).
5. Other requirements
OpenLDAP apparently relies on the Cyrus SASL library for SASL support. This means that just checking out OpenLDAP won't give us SASL support, which is a bigger problem on Windows and OS X than on Linux, where I suspect we'd use system OpenLDAP libraries (likely configured with SASL support) instead. For comparison, the ldapjs library also lacks SASL support, while the Mozilla LDAP SDKs does include SASL support. There's also some other includes apparently necessary for a subset of the server backends, like db.h; I haven't played around with backend server configs, though.
Thoughts on ways forward:
Replacing our underlying LDAP library is a big change that shouldn't be taken lightly, especially when we have no test coverage of our LDAP code as-is. It has been suggested in the past (via a comment on one of my blog posts) that we rely on OpenLDAP's server for test code; if people indicate that switching to OpenLDAP is a desirable outcome, then this would be an acceptable way to start writing LDAP tests IMHO. The way to move forward would then look like this:
1. Include basic functionality tests for LDAP using a version of LDAP that is assumed to be installed. Obviously, these wouldn't run on tinderbox as-is, but they would work for people working on doing the migration. Perhaps these tests could be enabled if a configure option like --enable-ldap-tests=/path/to/slapd. I already have some ideas about how the slapd binaries could be hooked up to a JS controller (it involves a lot of nsIProcess.run, so this may cause trouble if used in mozmill tests :( ).
2. Include a --with-system-ldap flag that would use system-installed libraries instead of ldap/sdks for our ldap/xpcom code. Equivalent correctness could be verified with the LDAP tests, and Linux distros could use these flags instead and fix the biggest problem with Mozilla's LDAP implementation right now. It's also possible that this could be the end state of our LDAP switch-out: we'd support both versions of LDAP, and fallback on Mozilla's implementation if OpenLDAP isn't installed.
3. Drop a copy of OpenLDAP into our tree and delete the code for using Mozilla LDAP SDKs.
Thoughts/questions/comments/concerns?
Ludovic Hirlimann
Nov 19, 2012, 7:01:40 AM11/19/12
to tb-pl...@mozilla.org
On 11/18/12 4:01 PM, Joshua Cranmer
wrote:
3. Drop a copy of OpenLDAP into our tree and delete the code for using Mozilla LDAP SDKs.
Thoughts/questions/comments/concerns?
As OpenLDAP more active development than the sdk we currently use ?
Would it come with unit tests ? How much work would it be to add unit testing while switching libraries ?
Ludo
Would it come with unit tests ? How much work would it be to add unit testing while switching libraries ?
Ludo
-- @lhirlimann on twitter https://wiki.mozilla.org/Thunderbird:Testing my photos http://www.flickr.com/photos/lhirlimann/collections/
Joshua Cranmer
Nov 19, 2012, 11:38:33 AM11/19/12
to tb-pl...@mozilla.org
On 11/19/2012 6:01 AM, Ludovic
Hirlimann wrote:
On 11/18/12 4:01 PM, Joshua Cranmer wrote:
3. Drop a copy of OpenLDAP into our tree and delete the code for using Mozilla LDAP SDKs.As OpenLDAP more active development than the sdk we currently use ?
Thoughts/questions/comments/concerns?
OpenLDAP released 2.4.24 in early 2011 and is currently on 2.4.33;
its git repository lists several changesets in the past few months,
although they mostly appear to focus on server development. As LDAP
is effectively a mature technology, and also one that is more
heavily used by slower-moving large enterprises, I think this is as
active as any other LDAP package you'll see.
Would it come with unit tests ? How much work would it be to add unit testing while switching libraries ?
OpenLDAP comes with a decently comprehensive test suite, but those
appear to focus on testing the server implementation. Reusing slapd
(the server) for LDAP testing in Thunderbird doesn't appear to be
too difficult (apparently, configuration is moving to an LDAP
backend [1], and there's a tool explicitly for slurping in data in
LDIF format), modulo the fact that it would require setTimeout-ish
calls.
[1] Yo dawg, I heard you liked LDAP so I configured your LDAP server's LDAP backend with LDAP?
[1] Yo dawg, I heard you liked LDAP so I configured your LDAP server's LDAP backend with LDAP?
Magnus Melin
Nov 19, 2012, 2:34:47 PM11/19/12
to tb-pl...@mozilla.org
On 18.11.2012 17:01, Joshua Cranmer wrote:
> Thoughts/questions/comments/concerns?
Instinctively switching to a js implementation like node-ldapjs sounds
far more intriguing, though i haven't done much research on it. A full
git checkout of node-ldapjs is 2.1M so in any case it's smaller.
-Magnus
_______________________________________________
tb-planning mailing list
tb-pl...@mozilla.org
https://mail.mozilla.org/listinfo/tb-planning
> Thoughts/questions/comments/concerns?
Instinctively switching to a js implementation like node-ldapjs sounds
far more intriguing, though i haven't done much research on it. A full
git checkout of node-ldapjs is 2.1M so in any case it's smaller.
-Magnus
_______________________________________________
tb-planning mailing list
tb-pl...@mozilla.org
https://mail.mozilla.org/listinfo/tb-planning
Reply all
Reply to author
Forward
0 new messages