Reading LDIF files containing controls

[Doug Wegscheid]

I have a use case for analyzing LDIF dumps of our LDAP directory. The dump utility puts some controls in the output file, so Net::LDAP::LDIF won't read them, it fails with "Controls only allowed with LDIF change entries".

I put a patch on to allow reading of controls for non-change entries by specifying controls_always_legal => 1 on the Net::LDAP::LDIF->new() call.

Can we consider including this patch (possibly modified if people see fit) in the next release?

--- LDIF.pm~    2013-05-07 09:07:19.000000000 -0400
+++ LDIF.pm     2013-05-07 08:45:34.000000000 -0400
@@ -15,7 +15,7 @@
     if (CHECK_UTF8);
 }

-our $VERSION = '0.20';
+our $VERSION = '0.21';

 # allow the letters r,w,a as well as the well-known operators as modes
 my %mode = qw(r < < < w > > > a >> >> >>);
@@ -66,6 +66,7 @@
     changetype => 'modify',
     modify => 'add',
     wrap => 78,
+    controls_always_legal => 0,
     %opt,
     fh   => $fh,
     file => "$file",
@@ -352,7 +353,7 @@
     my $attr;
     my $xattr;

-    if (@controls) {
+    if (@controls && !$self->{controls_always_legal}) {
       $self->_error("Controls only allowed with LDIF change entries", @ldif);
       return;
     }
--- LDIF.pod~   2013-05-07 09:07:40.000000000 -0400
+++ LDIF.pod    2013-05-07 08:48:52.000000000 -0400
@@ -135,6 +135,11 @@

 Example: raw =E<gt> qr/(?i:^jpegPhoto|;binary)/

+=item controls_always_legal =E<gt> 1
+
+Always allow controls in LDIF input, even if the input LDIF entry is not
+a change entry.
+
 =back

 =back

[Quanah Gibson-Mount]

--On Tuesday, May 07, 2013 7:00 AM -0700 Doug Wegscheid

<dwegs...@sbcglobal.net> wrote:

>
> I have a use case for analyzing LDIF dumps of our LDAP directory. The
> dump utility puts some controls in the output file, so Net::LDAP::LDIF
> won't read them, it fails with "Controls only allowed with LDIF change
> entries".
>
> I put a patch on to allow reading of controls for non-change entries by
> specifying controls_always_legal => 1 on the Net::LDAP::LDIF->new() call.

Sounds to me like your dump utility is creating LDIF that violates
<http://tools.ietf.org/html/rfc2849>

--Quanah



--

Quanah Gibson-Mount
Sr. Member of Technical Staff
Zimbra, Inc
A Division of VMware, Inc.
--------------------
Zimbra :: the leader in open source messaging and collaboration

[Doug Wegscheid]

I don't doubt that, but the question is whether or not the patch is palatable...

---

From: Quanah Gibson-Mount <qua...@zimbra.com>
To: Doug Wegscheid <dwegs...@sbcglobal.net>; perl...@perl.org
Sent: Tuesday, May 7, 2013 12:52 PM
Subject: Re: Reading LDIF files containing controls

[Peter Marschall]

Hi Doug,

On Tuesday, 7. May 2013, Doug Wegscheid wrote:
> I don't doubt that, but the question is whether or not the patch is
> palatable...

You may doubt it, but Quanah is right: your dump utility violates the RFC.

in the "Formal Syntax Definition of LDIF" section, RFC 2849 explicitly states
that controls may only occur in change records.

I do not think that adding an option to accept this type of RFC violation
is the way to go, as it would be a precedent for further RFC-violations
(even if they were hidden behind options).
This way lies madness (or option hell).

Instead, please get the dump utility fixed so that it adheres to the RFC.

Best
Peter

PS: Please don't top-post

--
Peter Marschall
pe...@adpm.de

[Doug Wegscheid]

>You may doubt it, but Quanah is right: your dump utility violates the RFC.

please reread my earlier message: I do NOT doubt it violates the RFC (I know it does!). I'm sorry if a misread of what I wrote caused everyone to get the idea that I wasn't listening or understood Qaunah's email, and possibly caused a lack of sympathy for the situation I'm in and a lack of willingness to listen to my position...

>I do not think that adding an option to accept this type of RFC violation
>is the way to go, as it would be a precedent for further RFC-violations
>(even if they were hidden behind options).
>This way lies madness (or option hell).

I'm not asking the maintainer to make a change that causes Net::LDAP to create anything that violates an RFC (I would absolutely agree with not going that direction). I am asking for a small change to be made to allow Net::LDAP to READ something that is in common use, but violates the RFC. Whether or not we like it, or if it violates our sense of purity, the use cases for dealing with software that does not follow standards is real, and sadly common. The model of "being strict on output, but allowing input of common violations" is not unheard of in these cases, and is useful in the real world.

There is a middle road between strict adherance to the RFCs and allowing anarchy, and "liberal on input, strict on output" seems to be a pretty reasonable middle road. It allows Net::LDIF to be useful for more real world problems.

I am asking everyone to think about that and see if it makes sense.

>Instead, please get the dump utility fixed so that it adheres to the RFC.

I don't have that kind of pull with IBM.

[Rick Sanders]

>Instead, please get the dump utility fixed so that it adheres to the RFC.
>
>> I don't have that kind of pull with IBM.

How about writing a script to post-process the output from the dump to
make it compliant?

-Rick

[Peter Marschall]

Hi,

I can understand your point of view, but it will not change my stance.

What I can offer you as some kind of consolation is an idea about a simple
preprocessor that filters out the illegal "control:" lines
perl -i -p -0040 -e 's/\n //' < RFC-VIOLATING-FILE \
| grep -vi ^control: > RFC-CONFORMING-FILE
[the first command is required to unwrap the wrapped lines]

Alternatively you may add a
changetype: add line
after each unwrapped dn: line.

> >Instead, please get the dump utility fixed so that it adheres to the RFC.
>
> I don't have that kind of pull with IBM.

Please at least report it to them as a bug in their tool (even if they might
not react). Maybe they are not even aware of the issue.

Best
Peter

--
Peter Marschall
pe...@adpm.de

[Doug Wegscheid]

>How about writing a script to post-process the output from the dump to

>make it compliant?

That would work. At that point, I'll have parsed the 700M file, so I'll not need Net::LDAP::LDIF.

[Doug Wegscheid]

>I can understand your point of view, but it will not change my stance.

not a problem. I'll ask you to think carefully the actual downside of incorporating the option for relaxing RFC compliance versus the value of said options to the open source community. What is the real downside?

>What I can offer you as some kind of consolation is an idea about a simple
>preprocessor that filters out the illegal "control:" lines
> perl -i -p -0040 -e 's/\n //' < RFC-VIOLATING-FILE \
>  | grep -vi ^control: > RFC-CONFORMING-FILE
>[the first command is required to unwrap the wrapped lines]

>Alternatively you may add a
>  changetype: add line
>after each unwrapped dn: line.

all very good suggestions, thanks for offering them.

I'm dealing with 1G+ LDIFs, and the patch *is* cleaner for me, but preprocessing will work.

>Please at least report it to them as a bug in their tool (even if they might
>not react). Maybe they are not even aware of the issue.

If I put the bug report in, will you incorporate the patch? :)

[Peter Marschall]

On Saturday, 11. May 2013, you wrote:
> >What I can offer you as some kind of consolation is an idea about a simple
> >preprocessor that filters out the illegal "control:" lines
> >
> > perl -i -p -0040 -e 's/\n //' < RFC-VIOLATING-FILE \
> >
> > | grep -vi ^control: > RFC-CONFORMING-FILE
> >[the first command is required to unwrap the wrapped lines]
> >
> >Alternatively you may add a
> > changetype: add line
> >after each unwrapped dn: line.
>

> all very good suggestions, thanks for offering them.
>
> I'm dealing with 1G+ LDIFs, and the patch *is* cleaner for me, but
> preprocessing will work.

When the dump utility is capable of writing to stdout, you can use a pipe.
This way you will not need more memory, ...

> >Please at least report it to them as a bug in their tool (even if they
> >might not react). Maybe they are not even aware of the issue.
>

> If I put the bug report in, will you incorporate the patch? :)

No. [I do not want perl-ldap to look like the easier target ;-]]

My hope is IBM will fix their bug and adhere to well-established standards.
If they do, you can throw away the preprocessor step.

Best
PEter

--
Peter Marschall
pe...@adpm.de

[Doug Wegscheid]

Thanks for your time. The lack of response to "purity vs real-world usability" is disappointing.

>> I'm dealing with 1G+ LDIFs, and the patch *is* cleaner for me, but

>> preprocessing will work.

>When the dump utility is capable of writing to stdout, you can use a pipe.
>This way you will not need more memory, ...

correct, will not need disk. *will* more CPU, more time.

>> If I put the bug report in, will you incorporate the patch? :)
>No. [I do not want perl-ldap to look like the easier target ;-]]

purity wins over pragmatism :)

>If they do, you can throw away the preprocessor step.

I won't have a preprocessor. I have a patched Net::LDAP::LDIF! thanks to the maintainers for making it easy to patch.

If anyone is in the same situation as I, you have my permission to use the patch in my original posting.

[Peter Marschall]

Hi,

On Saturday, 11. May 2013, Doug Wegscheid wrote:
> Thanks for your time. The lack of response to "purity vs real-world
> usability" is disappointing.

I am sorry you see it that way, but my reason was not only "purity" as you
call it, but long term usability & compatibility.

Your patch would have scratched your itch, but the burden to keep perl-ldap
maintainable, understandable & compatible [even for that RFC violation of your
tool] would be on the maintainer(s).

What if another patch came in asking for another RFC violation that would
negatively impact your patch? Which one should then prevail?

Did you at least report the bug to IBM?
Because their non-standards compliant tool is the cause of the whole
discussion.

[Doug Wegscheid]

>Your patch would have scratched your itch, but the burden to keep perl-ldap

>maintainable, understandable & compatible [even for that RFC violation of your
>tool] would be on the maintainer(s).

Understood. I'm not trying to trivialize the job of the maintainer (I've been there!), but this is a simpler situation than the one that many maintainers have (this being all Perl!). The patch is not horribly complex. I understand your reservation about option hell, but think in terms of grey, not black and white, how terrible of a problem are we really looking at?

>What if another patch came in asking for another RFC violation that would
>negatively impact your patch? Which one should then prevail?

Since the non-standards-enabling is enabled by option, I'm having a hard time thinking of a case where this would be an issue: the choices are "allow controls without change record" or "don't allow controls without change record"; we've already covered both. Not saying it can't happen, help me find a case here!

In lieu of a concrete example (help me here!): if the other interfering violation were mutually exclusive, one (of several possible) approaches would be to give them different enablement options, and make the enablement options mutually exclusive. Yes, this is the start of a slippery slope into option hell, but again, how often has this come up?.  What would a conflicting option be?

>Did you at least report the bug to IBM?
>Because their non-standards compliant tool is the cause of the whole
>discussion.

I offered to as a part of quid quo pro.