Why does 1 get added to the line length in the indexes.

[Michael Aldridge]

I'm trying to add index support to github.com/MiLk/nsscache-go and I've managed to psych myself out that the format I'm writing is wrong.  Currently for the following cache:

entity10:x:2010:2000::/tmp/entity10:/bin/sh
entity9:x:2009:2000::/tmp/entity9:/bin/sh
dummy1:x:2000:2000::/tmp/dummy1:/bin/sh
entity6:x:2006:2000::/tmp/entity6:/bin/sh
entity7:x:2007:2000::/tmp/entity7:/bin/sh
entity8:x:2008:2000::/tmp/entity8:/bin/sh
entity1:x:2001:2000::/tmp/entity1:/bin/sh
entity2:x:2002:2000::/tmp/entity2:/bin/sh
entity3:x:2003:2000::/tmp/entity3:/bin/sh
entity4:x:2004:2000::/tmp/entity4:/bin/sh
entity5:x:2005:2000::/tmp/entity5:/bin/sh

I would generate the following index:

dummy100000087
entity100000253
entity1000000001
entity200000295
entity300000337
entity400000379
entity500000421
entity600000127
entity700000169
entity800000211
entity900000045

These values are off by one as far as I can tell from the nsscache implementation Google provides, but I cannot figure out why.  To the best of my understanding, the number that comes after the key in the index should be piped to fseek() by line 159 of nss_cache.c.  So from my perspective, I think I should be able to do something like "tail -c +$OFFSET passwd.cache" to see what the line is that should be read, as that should advance the cursor to where libnss-cache would commence reading from.  This is not, however, how the nsscache implementation in python does it, and when I ported the index generation somewhat blindly to Go about a year ago I added the addition of the 1 there as well.  This doesn't seem right though, and I have the suspicioun that the indexes are being ignored and libnss-cache is falling back to a linear search.

Some explanation on what the index file format actually is would be very nice, and even nicer would be why it appears to be designed with a built in off by one error.

Thanks in advance!

[Michael Aldridge]

At this point I'm really doubtful that the caches are checked, much less used reliably.

In a test I wrote an index file for the passwd cache that claims that all entries are at an offset of 0 bytes and this still works.  I'd love to debug further, but the lack of documentation, and the fact that you can send seemingly any input to libnss_cache and have it work are really slowing this process down.

Are the indexes in use to begin with?  I was able to create a test cache with 20k entries and no indexes and things seemed to work just fine.

Looking forward to some answers to my many questions, as the more I try to debug and analyze libnss_cache and nsscache, the more questions I have.

[Michael Aldridge]

Much debugging has happened today.  I now know much more of why the cache worked when by all rights it shouldn't have.  I believe though that the indexes being written by nsscache are wrong, but as that's Python I haven't traced it all the way through yet.  In my implementation I now have the index generation fixed, and a copy of libnss_cache with a patched DEBUG macro confirms that it works.

[Jamie Wilkinson]

I'm sorry I missed your email until now.  Off the top of my head, I don't remember why it functions like that and I'm sorry you had to work so hard to figure out what is going on.

It's entirely possible that the index generation is buggy.  The libnss-cache library is the reader of the indexes, so whatever it's looking for is correct.  We may well have an off-by-one in the python code, or there might be some cleverness that I can't remember.

I'll see what I can dig up from my email but I'm not expecting to find any revelations there.

On Mon, 18 Nov 2019 at 18:48, Michael Aldridge <aldrid...@gmail.com> wrote:

Much debugging has happened today.  I now know much more of why the cache worked when by all rights it shouldn't have.  I believe though that the indexes being written by nsscache are wrong, but as that's Python I haven't traced it all the way through yet.  In my implementation I now have the index generation fixed, and a copy of libnss_cache with a patched DEBUG macro confirms that it works.

--
You received this message because you are subscribed to the Google Groups "nsscache-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nsscache-discu...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/nsscache-discuss/d37eeea1-3c91-417e-826f-deef26a6b742%40googlegroups.com.

[Jamie Wilkinson]

By the way, if you have written tests to try to understand the bheaviour better, I'd love to have them incorporated in either nsscache or libnss-cache, though now I've learned you're writing a competing implementation I'm sure you're less inclined to contribute to the former :-)

[Michael Aldridge]

I have a branch with tests for libnss-cache, and it works decently but
there's still some test code that fails to build. I have my suspicions
that not all of the test code made it to the public tree.

I try very hard to not write python these days, and while I wouldn't
exactly call what I'm working on a "competing implementation" I'll see
if I can get my tests into a sane state for libnss-cache. For that I
actually generated my test sets with bash since I assumed that the
python was bugged and the C was correct.

--Michael

[Jamie Wilkinson]

Cool, that's what I thought.  I would also assume that libnss-cache is correcter than Python at this point, because the Python hasn't really been looked at, let alone cared for, in many years.

--
You received this message because you are subscribed to the Google Groups "nsscache-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nsscache-discu...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/nsscache-discuss/6d1512a2-3cd2-7e20-828d-47e25355f8b8%40gmail.com.