centos 5.8 error

[Dimitri Maziuk]

Hi everyone,

I've a setup that's been working for years that started failing after a recent centos update:

fileparse(): need a valid pathname at /website/rmagic-2.21/rmagic.pl line 388

Anyone else seen this yet?

TIA
Dima

[Jeremy Wadsack]

Dmitri - 

fileparse is part of File::Basename, so it's possible this changed with an update to that module in you perl environment. The pathname it's complaining about the [statistics] File_In in your  rmagic.ini file. So first, double check that is still valid and hasn't changed. I don't see anything different in File::Basename documentation than it was ten years ago. It's also possible that something between the two (e.g. Config::IniFiles parses rmagic.ini) has changed and is failing. 

As a debug step I'd throw a print line in just above 388 in your rmagic.ini

print $statistics{File_In}

That should at least let you know if the path makes any sense to cent OS.

Jeremy Wadsack

[Dimitri Maziuk]

On 07/26/2012 08:01 PM, Jeremy Wadsack wrote:
...> Dmitri -

> As a debug step I'd throw a print line in just above 388 in your rmagic.ini
>
> print $statistics{File_In}

Hi Jeremy,

I actually did some poking since I posted that but I usually find my own
Perl hard to follow, with someone else's it's a non-starter.

Specifically, $statistics{File_In} is a null in line 388, and I'm not
sure I can figure out where it's initialized and to what.

> my @now = localtime;
> $now = (1900 + $now[5]) . '/' . (1 + $now[4]) . '/' . $now[3] . ' ' . $now[2] . ':' . $now[1];
> print "!!" . $statistics{File_In} . "\n";
> $statistics{File_In} =~ s/%([^%]+)%/$formatter->formatDate( $1, $now )/ge;
> my $infile;
> ($infile, undef, undef) = fileparse( $statistics{File_In}, '\..*' );

yields
> $ /website/rmagic-2.21/rmagic.pl /website/etc/analog/combined/ftp12.ini
> !!
> fileparse(): need a valid pathname at /website/rmagic-2.21/rmagic.pl line 390

The ini file has
> [statistics]
> File_In = /website/admin/logs/analog_output/ftp/ftp_2012.dat

(that hasn't been touched) and
> $ ls -l /website/admin/logs/analog_output/ftp/ftp_2012.dat
> -rw-rw-r-- 1 web stapdev 508177 Jul 22 00:01 /website/admin/logs/analog_output/ftp/ftp_2012.dat

("combined" reports are generated weekly, so 10pm last Friday timestamp
means analog part of it is still working fine.)

Where do I go from here?

Thanks,
--
Dimitri Maziuk
Programmer/sysadmin
BioMagResBank, UW-Madison -- http://www.bmrb.wisc.edu

[Jeremy Wadsack]

And it's not the best code I've every written either. ;) Honestly, I'm having trouble following it too.

The $statistics hash is copied from a Config::IniFiles class. It should be loading values from the ini file and mapping them. You could also try

    print $config{statistics}{File_In}

to debug the hash copy. Somehow I'm suspecting that it may be a problem in Config::IniFiles. If this is also nul, can you check the version of the library (perl -e 'Config::IniFiles::VERSION' should do it, I think)?

Jeremy Wadsack

[Dimitri Maziuk]

On 07/27/2012 01:30 PM, Jeremy Wadsack wrote:
...

> The $statistics hash is copied from a Config::IniFiles class. It should be
> loading values from the ini file and mapping them. You could also try
>
> print $config{statistics}{File_In}
>
> to debug the hash copy. Somehow I'm suspecting that it may be a problem in
> Config::IniFiles. If this is also nul, can you check the version of the
> library (perl -e 'Config::IniFiles::VERSION' should do it, I think)?

$ rpm -q perl-Config-IniFiles
perl-Config-IniFiles-2.72-1.el5

Here's the hash:

> use Data::Dumper;
> $Data::Dumper::Indent = 3;
> print Dumper(\%config);
> print "\n";

> print "!!" . $statistics{File_In} . "\n";
> $statistics{File_In} =~ s/%([^%]+)%/$formatter->formatDate( $1, $now )/ge;
> my $infile;
> ($infile, undef, undef) = fileparse( $statistics{File_In}, '\..*' );

output:

> $VAR1 = {
> 'reports' => {},
> 'graphs' => {},
> 'website' => {},
> 'x' => {},
> '1_' => {},
> 'Q_' => {},
> 'm' => {},
> 'W_' => {},
> 'D_' => {},
> 'd' => {},
> 'H_' => {},
> 'h' => {},
> 'w' => {},
> '4_' => {},
> '6_' => {},
> '5_' => {},
> '7_' => {},
> 'S_' => {},
> 'l' => {},
> 'L_' => {},
> 'Z_' => {},
> 'o' => {},
> 'r' => {},
> 'i' => {},
> 't' => {},
> 'z' => {},
> 'P_' => {},
> 'E_' => {},
> 'I_' => {},
> 'f' => {},
> 's' => {},
> 'N_' => {},
> 'n' => {},
> 'Y_' => {},
> 'y' => {},
> 'k' => {},
> 'K_' => {},
> 'B_' => {},
> 'b' => {},
> 'p' => {},
> 'v' => {},
> 'R_' => {},
> 'M_' => {},
> 'u' => {},
> 'j' => {},
> 'J_' => {},
> 'c' => {},
> 'q' => {}
> };
>
> !!
> fileparse(): need a valid pathname at /website/rmagic-2.21/rmagic.pl line 394

[Jeremy Wadsack]

Yeah. For whatever reason, %config doesn't have any of the loaded data from the language files or your settings. So I'm pretty certain that it's an issue with Config::IniFiles.

Shlomi Fish was been maintaining that module for years now, and it's currently at 2.77. The last version I installed that I know is working is 2.39. That's a lot of variance.

Can you determine if that was recently updated? That would help decide if we should look there.

Jeremy Wadsack

[Dimitri Maziuk]

On 07/27/2012 03:30 PM, Jeremy Wadsack wrote:

[ Config::IniFiles. ]

> Can you determine if that was recently updated? That would help decide if
> we should look there.

Yep. According to yum.log
Jun 10 16:34:15 Updated: perl-Config-IniFiles-2.72-1.el5.noarch
which is about when it broke.

[Jeremy Wadsack]

Which fixes "insecure temporary file usage": https://lists.fedoraproject.org/pipermail/epel-package-announce/2012-May/007508.html

I'm not having much luck digging through the fedora build logs to see what the diff was for that patch. Maybe you know it better? I'm guessing that patching the temporary file usage broke something that reportmagic uses that doesn't have a corresponding test case in Config::IniFiles.

Jeremy Wadsack

[Dimitri Maziuk]

On 07/27/2012 03:49 PM, Jeremy Wadsack wrote:
> Which fixes "insecure temporary file usage":
> https://lists.fedoraproject.org/pipermail/epel-package-announce/2012-May/007508.html
>
> I'm not having much luck digging through the fedora build logs to see what
> the diff was for that patch. Maybe you know it better? I'm guessing that
> patching the temporary file usage broke something that reportmagic uses
> that doesn't have a corresponding test case in Config::IniFiles.

I can't even see where rmagic.pl reads its argv[1] into ini hash... :(

Perhaps something in the upstream changelog since 2.72? I can file a bug
report against centos rpm, but having more details than just "broke
report magic" would be useful.

[Jeremy Wadsack]

Oh, I agree that "breaks something in reportmagic" is a terrible bug report. That's why I was looking for the changelog/diff for that update to see if that could pin it down. 

As to where rmagic read ars[1], that's complicated. In wadg::rm::Settings it read the whole command line around line 79 and assigns anything that doesn't start with '-' to a list of filenames. Around line 118 it iterates through the list until it finds one that Config::IniFiles can parse.

Note that if no arguments are given it looks for 'rmagic.ini' in the curent folder and then in the script's folder (lines 89-108).

BTW, all that traces from line 230 in rmagic.pl.

Does that help any? 

 

Jeremy Wadsack

[Dimitri Maziuk]

On 07/27/2012 04:18 PM, Jeremy Wadsack wrote:
..

> As to where rmagic read ars[1], that's complicated. In

> wadg::rm::Settings<https://github.com/jeremywadsack/reportmagic/blob/master/wadg/rm/Settings.pm>it

> read the whole command line around line 79 and assigns anything that
> doesn't start with '-' to a list of filenames. Around line 118 it iterates
> through the list until it finds one that Config::IniFiles can parse.

...
> Does that help any?


Not really: I can add more dumps (Settings.pm):

> use Data::Dumper;
> $Data::Dumper::Indent = 3;

> while( @filenames ) {
> $parms{-file} = shift @filenames;
> print "\n1\n";
> print Dumper( \%parms );
> my %hash;
> tie %hash, 'Config::IniFiles', ( %parms ) ;
> print "\n2\n";
> print Dumper( \%hash );

and get this:

> 1
> $VAR1 = {
> '-file' => '/website/etc/analog/combined/ftp12.ini',
> '-default' => 'reports'
> };
>
> 2
> $VAR1 = {
> 'statistics' => {},
> 'navigation' => {},
> 'reports' => {},
> 'DIRECTORY' => {},
> 'QUICK' => {},

> 'graphs' => {},
> 'website' => {}

> };

but I don't understand what it means. I thought "tie" was supposed to
intialize %hash to contents of /website/etc/analog/combined/ftp12.ini,
but clearly I thought wrong... like I said, my Perl sucks.

[Jeremy Wadsack]

Your Perl is right. (And I'm trying very hard not to write ruby in my examples back.) It should bind the contents of the file to the hash.

The output is interesting though, because it clearly means that it is getting some of the contents of the file. Each section in the ini file (e.g. [statistics], [reports], etc.) is being loaded as a key in the hash. What it's not getting is the settings below that. 

I wonder if you can just test Config::IniFiles directly against that file in a separate perl script.

use Config::IniFiles;

my $cfg = Config::IniFiles->new( -file => "/website/etc/analog/combined/ftp12.ini" );

print "The source is " . $cfg->val( 'settings', 'File_In' );

tie %hash, 'Config::IniFiles', ( -file => "/website/etc/analog/combined/ftp12.ini" ) ;

print "The source is " . $hash{settings}{File_In};

Jeremy Wadsack

[Dimitri Maziuk]

On 07/27/2012 05:05 PM, Jeremy Wadsack wrote:

> I wonder if you can just test Config::IniFiles directly against that file
> in a separate perl script.

Should've thought of that... it works except it's
$hash{statistics}{File_In}. Well, it sort of works because

my $cfg = Config::IniFiles->new( -file =>
"/website/etc/analog/combined/ftp12.ini" );

print Dumper($cfg);

prints the whole thing whereas

tie %hash, 'Config::IniFiles', ( -file =>
"/website/etc/analog/combined/ftp12.ini" ) ;

print Dumper(\%hash);

only prints section names which I really don't get but whatever -- it
does get the filename, QED.

FWIW, if I add around line 259

print "+ " . $config{statistics}{File_In} . "\n";
%statistics = %{$config{statistics}};
print "++ " . $statistics{File_In} . "\n";

I get

+ sample.dat
++

So how exactly is '%' supposed to work in this excuse for a programming
language?

[Dimitri Maziuk]

Well duh. Proper output from rmagic.pl, with the right parameter, is

1
$VAR1 = {
'-file' => './rmagic.pl',
'-default' => 'reports'
};

2
$VAR1 = {};

1
$VAR1 = {
'-file' => './rmagic.sample',

'-default' => 'reports'
};

2
$VAR1 = {
'statistics' => {},

'reports' => {},

'QUICK' => {},
'graphs' => {},

'navigation' => {},
'website' => {}
};
+ /website/admin/logs/analog_output/ftp/ftp_2012.dat
++

The 1 and 2 dumps are coming from Settings.pm (line 118) and for some
reason they're dumped twice.

So it is getting the filename all right, it's just not maling it past

%statistics = %{$config{statistics}};

[Jeremy Wadsack]

Ah, so it's failing in the hash copy.

The reason Settings gets called twice is that it's looking for a default file because (I think) none was given on the command-line. So it's trying several options. It bails immediately on 'rmagic.pl' because it knows that's the program itself and then goes to the next matching file. 

I actually don't recall how %{} works in terms of coercing something into a hash. It also may have changed since I last was really good at Perl. The thing is, %config isn't really a hash, it's a module that's tied to a hash to make it act like one. So the reason Dump only shows the first elements (and not the contained hash elements) I think is related to that.

Still I think that `%statistics = %{$config{statistics}};` should work. Especially since it has worked for at least 10 years or more. Perl 5.x hasn't changed that much. 

If you dump $config{statistics} from rmagic.pl does that iterate the hash members? It seems that something between the tie and the hash copy is failing.

Jeremy Wadsack

[Dimitri Maziuk]

On 07/27/2012 06:08 PM, Jeremy Wadsack wrote:
> Ah, so it's failing in the hash copy.
>
> The reason Settings gets called twice is that it's looking for a default
> file because (I think) none was given on the command-line.

Right. It was Friday afternoon, that's my excuse.

> I actually don't recall how %{} works in terms of coercing something into a
> hash. It also may have changed since I last was really good at Perl. The

> thing is, %config isn't *really* a hash, it's a module that's tied to a

> hash to make it act like one.

So I wonder if something inside IniFiles object changed in or around
2.72 to make it not quite work as a hash.

> Still I think that `%statistics = %{$config{statistics}};` should work.

Here's the test case:

> use Config::IniFiles;

> use Data::Dumper;
> $Data::Dumper::Indent = 3;

> tie %hash, 'Config::IniFiles', ( -file => "/website/etc/analog/combined/ftp12.ini" ) ;

> print Dumper(\%hash);
> print "The source is " . $hash{statistics}{File_In};
> print "\n========================================\n";
> %copy = %{ $hash{ statistics } };
> print "The copy is " . $copy{File_In};
> print "\n========================================\n";

and the output:

> $ ./simple.pl

> $VAR1 = {
> 'statistics' => {},

> 'navigation' => {},
> 'reports' => {},
> 'DIRECTORY' => {},

> 'QUICK' => {},
> 'graphs' => {},

> 'website' => {}
> };
> The source is /website/admin/logs/analog_output/ftp/ftp_2012.dat
> ========================================
> Use of uninitialized value in concatenation (.) or string at ./simple.pl line 28.
> The copy is
> ========================================

Looks like a heisenbug to me: worked until I looked at it.

[Dimitri Maziuk]

FYI: the fix is slated for the next epel update.

According to RedHat that version of Config::IniFiles requires (but did
not explicitly Require at RPM level) a newer List::MoreUtils (0.33+) to
function properly.

(https://bugzilla.redhat.com/show_bug.cgi?id=844460)

Thanks Jeremy,

[Jeremy Wadsack]

Thanks for following up on that, Dmitri!

Jeremy Wadsack

[Dimitri Maziuk]

On 08/02/2012 02:07 PM, Jeremy Wadsack wrote:
> Thanks for following up on that, Dmitri!

I got the updates from epel-testing and re-run our reports & it's
confirmed fixed (perl-List-MoreUtils-0.33-5 and
perl-Config-IniFiles-2.72-2).