WANTED: Tool to verify installed package/port consistancy

[Ronald F. Guilmette]

The subject line pretty much says it all.

As I explained here the other day, numerous of my installed ports
have semi-mysteriously had their corresponding +CONTENTS files
just disappear. I do have a backup of my /var partition, from which
I could, in theory, fetch replacements for the specific +CONTENTS
files that went missing, but I know of no way to be able to check
those backed-up +CONTENTS files to make sure that they are at all
consistant with what I actually have installed in the way of ports/
packages on this system at the present time.

Looking at the +CONTENTS files that were not "disappeared" and that
are still present on the system in question, it is abundantly clear
that each of these contains two valuable things relative to the
installed package/port that it corresponds to, i.e.:

1) A list of all of the installed files corresponding to the
specific port/package in question, and

2) For each installed file that is part of the package/port in
question, an MD5 checksum value corresponding to that specific
installed file.

I searched within the /usr/ports/ports-mgmt directory, to see if there
might be any tools there that could be applied to a given +CONTENTS file
(or to all of them) to simply verify the presence and (MD5 checksum)
validity of each file of a given installed port/package, but the only
things whose pkg-descr files make them seem like they might be relevant
(i.e. pchecker & portlint) turn out to be tools meant for utterly
different purposes. :-(

Having found nothing useful, I stareted to write a small script of my own
to simply chcek that all files of an installed package/port exist, and that
they have the "right" MD5 checksums, but then I paused halfway through
when I realized that i am probably just re-inventing the wheel here.

It occurs to me that *something*, i.e. some tool(s) within the FreeBSD
panoply, *must* already be reading and/or otherwise making use of those
MD5 checksums within the +CONTENTS files. Otherwise, why would they even
be there? So my question really comes down to this: What pre-existing
software tools are available that can and do check the MD5 checksums, as
given the the +CONTENTS file(s), of various files associated with some
given installed port or package?

It is inconceivable to me that there no already existing tool within
FreeBSD that is already doing this exact job.


Regards,
rfg
_______________________________________________
freebs...@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to "freebsd-port...@freebsd.org"

[b.f.]

Ronald wrote:

...

> It occurs to me that *something*, i.e. some tool(s) within the FreeBSD
> panoply, *must* already be reading and/or otherwise making use of those
> MD5 checksums within the +CONTENTS files. Otherwise, why would they even
> be there? So my question really comes down to this: What pre-existing
> software tools are available that can and do check the MD5 checksums, as
> given the the +CONTENTS file(s), of various files associated with some
> given installed port or package?

...

Yes, of course, this is a basic package management feature:
pkg_info(1) with the "-g" flag.
(The analogous part of the newer pkgng is described in pkg-check(8)
from ports-mgmt/pkg.)

b.

[Beeblebrox]

You are unfortunately stuck between a rock and a hard place IMHO.

While pkgng has this tool with # pkg check -adrs
" -a Process all packages
-B is used to reanalyse SHLIBS of installed packages. See pkg.conf(5) for
more information on SHLIBS.
-d is used to check for and install missing dependencies.
-r is used to recompute sizes and checksums of installed packages.
-s is used to find invalid checksums for installed packages."

Unfortunately there are no official repositories for pkgng built packages,
so you have to keep building them yourself if you want to maintain an
up-to-date system. And if a build breaks for a certain port you are stuck -
you can't go and download an already built package for that one port that
won't build.

On the other hand, the old (legacy) package system had many problems, one of
those problems being exactly this particular issue because the values for
installed packages were not being kept in a database - so the capabilities
of the legacy pkg system are limited.

As a third option and if it is tolerable for you, you can wait until the
official pkgng repos come on-line then make the switch to pkgng on your
existing systems. Since your problem does appear to be urgent, you could
just try re-installing the problem ports and all of its dependents.

Regards.



-----
10-Current-amd64-using ccache-portstree merged with marcuscom.gnome3 & xorg.devel

--
View this message in context: http://freebsd.1045724.n5.nabble.com/WANTED-Tool-to-verify-installed-package-port-consistancy-tp5810087p5810174.html
Sent from the freebsd-ports mailing list archive at Nabble.com.

[Ronald F. Guilmette]

In message <518E1A51...@cyberleo.net>, you wrote:

>On 05/10/2013 03:04 PM, Ronald F. Guilmette wrote:
><snip>
>> pkg_sanity: ImageMagick-6.8.0.7_1: +CONTENTS file does not exist -- skipped
>> pkg_sanity: ORBit2-2.14.19: /usr/local/lib/libORBit-2.so: File failed MD5 checksum
>> pkg_sanity: ORBit2-2.14.19: /usr/local/lib/libORBit-imodule-2.so: File failed MD5 checksum
>> pkg_sanity: ORBit2-2.14.19: /usr/local/lib/libORBitCosNaming-2.so: File failed MD5 checksum
>> pkg_sanity: OpenEXR-1.7.1: /usr/local/lib/libIlmImf.so: File failed MD5 checksum
>> pkg_sanity: aalib-1.4.r5_6: /usr/local/lib/libaa.so: File failed MD5 checksum
><snip>
>
>Are these mismatches symlinks?

Some are.

>If so, are you checking the contents of
>the symlink (with, for instance, stat(1) or readlink(1)), or the
>contents of the file to which the symlink is referring?

The latter.

(I would have had to have done something special in order to compute the
md5 fo teh symlink itself, and I did not do so. I have just now checked,
and my script is indeed getting the md5 of the files to which the various
symlinks refer.)


It is clear to me now that "pkg_info -g" is either skipping certain files
that are listed in the relevant +CONTENTS files or else it is computing the
MD5 checksums in an odd way. I do not think that the latter possibility is
at all likely.

I will be looking at this more deeply as time permits.


Regards,
rfg

[Ronald F. Guilmette]

So, as I have now learned (and just as the post by CyberLeo Kitsana had
suggested) if one wishes to write a script to do the rough equivalent
of what "pkg_info -g" does, then one has to be mindful of the fact that
for those specific MD5 checksums contained in the various +CONTENTS
files within the /var/db/pkg directory where the relevant installed
file happend to be a symlink, you must not calculate the MD5 checksum
on the file the symlink points to, but rather on the contents of the
symlink itself, as obtained by running "readlink -n".

Making that small adjustment to my script, re-installing my many missing
CONTENTS files from a backup I had, and running my little pkg_sanity
script as root (in order to avoid a few annoying permissions problems)
it seems that my system is now mostly back to a state of good health.
However there sill remain a handful of puzzling results:

pkg_sanity: gettext-0.18.1.1: /usr/local/lib/charset.alias: File failed MD5 checksum
pkg_sanity: gettext-0.18.1.1: /usr/local/lib/charset.alias: 4d9c898966e87b62589a1c44dc8297f6 versus fe5bae66620b10c76971d99932b18846
pkg_sanity: linux_base-f10-10_5: /compat/linux/etc/ld.so.cache: File failed MD5 checksum
pkg_sanity: linux_base-f10-10_5: /compat/linux/etc/ld.so.cache: 6f66c7adc696059c6dfdf04c4ff3adb8 versus dfe04fabc88d1fd3e40b14a7f47acd8e
pkg_sanity: linux_base-f10-10_5: /compat/linux/var/cache/ldconfig/aux-cache: File failed MD5 checksum
pkg_sanity: linux_base-f10-10_5: /compat/linux/var/cache/ldconfig/aux-cache: 3d561b01a69ad4357c92de3a5032d52b versus f8be26f0de9b1736b4b9837f07b550d7
pkg_sanity: p5-XML-SAX-0.99: /usr/local/lib/perl5/site_perl/5.14.2/XML/SAX/ParserDetails.ini: File failed MD5 checksum
pkg_sanity: p5-XML-SAX-0.99: /usr/local/lib/perl5/site_perl/5.14.2/XML/SAX/ParserDetails.ini: 15bfbb02aa79670b148f21dfbac64843 versus cf8c5cb8a7b6cf7a8db7c53f1ba27148

I've checked and "pkg_info -g" also reports MD5 checksum mismatches
on the four files mentioned above.

If other folks could run the following command and let me know
what results you see on your system, I would really appreciate
it. thanks.

pkg_info -g 'gettext*' 'linux_base*' 'p5-XML-SAX*'


Regards,
rfg


P.S. Oh lord! Now pkg_version is issuing a different kind of complaint
about four of my installed packages...

pkg_version: icon-naming-utils-0.8.90 does not appear to be a valid package!
pkg_version: p5-Net-HTTP-6.06 does not appear to be a valid package!
pkg_version: p5-XML-Simple-2.20 does not appear to be a valid package!
pkg_version: p5-libwww-6.05 does not appear to be a valid package!

Now I feel compelled to figure out the cause of this problem too.

Just when I thought that I was out they keep pulling me back in!!!
Arrrrrgggggggggggg!!!!

[Ronald F. Guilmette]

Well, I have looked a bit more at some of the files that posted about
just a short while ago. It turns out that pretty much all of the
remaining anomolies (i.e. MD5 mismatches) in package-related or port-
related files on my system can be attributed to either (a) some person
or else (b) some thing (e.g. some other package or port install script)
coming along _after_ a given port/package has been initially installed
and then diddling one or more of the relevant files. It is easy to
understand how or why this might happen, e.g. in cases involving
various configuration files that may have meaning to more than one
port or package.

So anyway, I revised my pkg_sanity script so that it will compare the
date/time stamps of files against the date/time stamps of the corresponding
+CONTENTS files (and report anomolies found) before it even bothers
trying to check the MD5 checksums. Obviously, if some given file has
been diddled since the date/time when its MD5 checksum was originally
computed for inclusion in the relevant +CONTENTS file, then it is almost
an absolute certainty that it's new/current MD5 checksum is _not_ going
to match its old one, as contained in the relevant +CONTENTS file.

Running my revised version of the script yields this output:

pkg_sanity: gettext-0.18.1.1: Newer than +CONTENTS file: /usr/local/lib/charset.alias
pkg_sanity: linux_base-f10-10_5: Newer than +CONTENTS file: /compat/linux/etc/ld.so.cache
pkg_sanity: linux_base-f10-10_5: Newer than +CONTENTS file: /compat/linux/var/cache/ldconfig/aux-cache

pkg_sanity: p5-XML-SAX-0.99: /usr/local/lib/perl5/site_perl/5.14.2/XML/SAX/ParserDetails.ini: File failed MD5 checksum
pkg_sanity: p5-XML-SAX-0.99: /usr/local/lib/perl5/site_perl/5.14.2/XML/SAX/ParserDetails.ini: 15bfbb02aa79670b148f21dfbac64843 versus cf8c5cb8a7b6cf7a8db7c53f1ba27148

(Oh! And by the way, this date/time stamp pre-check is one that "pkg_info -g"
_should_ really be doing also, but isn't. I'll be filing a PR on that.)

So it is clear that, sometime after install installation, somebody or
some thing came along and diddled the charset.alias, ld.so.cache, and
aux-cache files, and that thus, it is perfectly understandable that
these files no longer match their original install-time MD5 checksums.
These cases are therefore probably not worth worrying about. (I don't
ever remember having edited any of these three files, so it must have
been some other script doing it, as a side-effect of something else I
had done since I installed the relevant ports.)

This leaves me with one and only one anomoly, i.e. the ParserDetails.ini
file. But even in that case I feel quite sure that somebody or something
has intentionally diddled it, some time since it was originally installed,
but whoever or whatever that is, they must have foolishly and mistakenly
gone out of their way to preserve the original modification date/time
stamp for that file. If I can ever figure out who or what did that,
I shall chastize them appropriately with the nearest available instrument
of torture.

Hummm... Here's a hint. My current ParserDetails.ini file is shown below.
On a different FreeBSD 9.1 system that I have here, this same file is
present also, but is shorter (9 lines rather than 15) and contains only
the first two sections, rather than four, as seen here.

=============================================================================
[XML::SAX::PurePerl]
http://xml.org/sax/features/namespaces = 1

[XML::SAX::Expat]
http://xml.org/sax/features/namespaces = 1
http://xml.org/sax/features/external-general-entities = 1
http://xml.org/sax/features/external-parameter-entities = 1

[XML::LibXML::SAX::Parser]
http://xml.org/sax/features/namespaces = 1

[XML::LibXML::SAX]
http://xml.org/sax/features/namespaces = 1


=============================================================================

[Matthew Seaman]

On 14/05/2013 02:53, Ronald F. Guilmette wrote:
> Running my revised version of the script yields this output:
>
> pkg_sanity: gettext-0.18.1.1: Newer than +CONTENTS file: /usr/local/lib/charset.alias
> pkg_sanity: linux_base-f10-10_5: Newer than +CONTENTS file: /compat/linux/etc/ld.so.cache
> pkg_sanity: linux_base-f10-10_5: Newer than +CONTENTS file: /compat/linux/var/cache/ldconfig/aux-cache
> pkg_sanity: p5-XML-SAX-0.99: /usr/local/lib/perl5/site_perl/5.14.2/XML/SAX/ParserDetails.ini: File failed MD5 checksum
> pkg_sanity: p5-XML-SAX-0.99: /usr/local/lib/perl5/site_perl/5.14.2/XML/SAX/ParserDetails.ini: 15bfbb02aa79670b148f21dfbac64843 versus cf8c5cb8a7b6cf7a8db7c53f1ba27148
>
> (Oh! And by the way, this date/time stamp pre-check is one that "pkg_info -g"
> _should_ really be doing also, but isn't. I'll be filing a PR on that.)

There's not much love for the old pkg_tools scripts nowadays, given that
their days are clearly numbered.

However, I really like your idea of comparing the mtime of package files
with the installation time of the package. I may just steal that idea
for pkgng...

> Hummm... Here's a hint. My current ParserDetails.ini file is shown below.
> On a different FreeBSD 9.1 system that I have here, this same file is
> present also, but is shorter (9 lines rather than 15) and contains only
> the first two sections, rather than four, as seen here.
>
> =============================================================================
> [XML::SAX::PurePerl]
> http://xml.org/sax/features/namespaces = 1
>
> [XML::SAX::Expat]
> http://xml.org/sax/features/namespaces = 1
> http://xml.org/sax/features/external-general-entities = 1
> http://xml.org/sax/features/external-parameter-entities = 1
>
> [XML::LibXML::SAX::Parser]
> http://xml.org/sax/features/namespaces = 1
>
> [XML::LibXML::SAX]
> http://xml.org/sax/features/namespaces = 1
>
>
> =============================================================================

Check what different SAX related perl modules you have on the two
different machines -- it should correspond to the contents of that file.
I believe that as you add different modules they will update
ParserDetails.ini to register themselves. Which means that
ParserDetails.ini should be treated like a user-modifiable configuration
file and not belong to any particular package.


Cheers,

Matthew