Unable to open '/../../lib/Storable.pm' for reading: No such file or directory.

27 views
Skip to first unread message

tcbarrett

unread,
Feb 4, 2009, 5:30:48 AM2/4/09
to Devel::NYTProf Dev
NYTProf version:
## $Id: NYTProf.pm 665 2009-01-05 23:05:37Z tim.bunce $
our $VERSION = '2.07';

Running on MacOSX, Perl 5.8.9

Most of my time is spent in Storable
(238 1 2 53.2s 53.2s Storable::::mretrieveStorable::mretrieve(xsub))

Detail
# spent 107ms within Storable::dclone which was called 125 times, avg
857µs/call:
# 125 times (107ms+0s) by Class::DBI::ColumnGrouper::clone at line 68
of /opt/local/lib/perl5/site_perl/5.8.9/Class/DBI/ColumnGrouper.pm,
avg 857µs/call
# spent 39.9ms within Storable::net_mstore which was called 42 times,
avg 951µs/call:
# 42 times (39.9ms+0s) by Storable::_freeze at line 339, avg 951µs/
call
# spent 53.2s within Storable::mretrieve which was called 237 times,
avg 224ms/call:
# 237 times (53.2s+0s) by Storable::thaw at line 415, avg 224ms/call
Unable to open '/../../lib/Storable.pm' for reading: No such file or
directory.

I've tried adding nytprofhtml --lib /opt/local/lib/perl5/5.8.9/
darwin-2level (where Storable lives), but this makes no difference?

Any suggestions on where to go from there?

Adam Kaplan

unread,
Feb 4, 2009, 9:08:56 AM2/4/09
to develnyt...@googlegroups.com
Off the top if my head, and without access to the code, I'd guess the
path to Storable.pm is the problem. The prefix "../../lib" added to
the library include path you provided would result in something like
"perl/lib/lib/"

It's been a while since I looked at this stuff and I am on a train.
Maybe that will help you debug.
--
Adam J. Kaplan
Duct Tape Specialist & Software Engineer
The New York Times Company

Tim Bunce

unread,
Feb 5, 2009, 10:47:56 AM2/5/09
to develnyt...@googlegroups.com
It's caused by AutoSplit being run on Storable.pm before it's installed
by perl. AutoSplit adds #line directives to the auto-split files but
those #line directives have relative paths that bear little relation
to where the files end up.

It's probably possible to work-around it. If someone wants to volunteer
I'd be happy to guide them.

Tim.

Nicholas Clark

unread,
Feb 6, 2009, 10:49:14 AM2/6/09
to develnyt...@googlegroups.com
On Thu, Feb 05, 2009 at 03:47:56PM +0000, Tim Bunce wrote:
>
> It's caused by AutoSplit being run on Storable.pm before it's installed
> by perl. AutoSplit adds #line directives to the auto-split files but
> those #line directives have relative paths that bear little relation
> to where the files end up.
>
> It's probably possible to work-around it. If someone wants to volunteer
> I'd be happy to guide them.

Going forwards, it should also be possible to improve AutoSplit to solve
this properly. But it would require passing in the correct final install
location to it. The code is here:

http://perl5.git.perl.org/perl.git/blob/HEAD:/lib/AutoSplit.pm

Nicholas Clark

tcbarrett

unread,
Mar 5, 2009, 8:00:06 AM3/5/09
to Devel::NYTProf Dev
I have just built a CentOS box, and am having the same problems.
I'd be happy to volunteer what little skills I have.

On Feb 5, 3:47 pm, Tim Bunce <Tim.Bu...@pobox.com> wrote:
> It's caused by AutoSplit being run on Storable.pm before it's installed
> by perl. AutoSplit adds #line directives to the auto-split files but
> those #line directives have relative paths that bear little relation
> to where the files end up.
>
> It's probably possible to work-around it. If someone wants to volunteer
> I'd be happy to guide them.
>
> Tim.
>
> On Wed, Feb 04, 2009 at 09:08:56AM -0500, Adam Kaplan wrote:
>
> > Off the top if my head, and without access to the code, I'd guess the
> > path to Storable.pm is the problem.  The prefix "../../lib" added to
> > the library include path you provided would result in something like
> > "perl/lib/lib/"
>
> > It's been a while since I looked at this stuff and I am on a train.
> > Maybe that will help you debug.
>

Coke

unread,
Mar 5, 2009, 4:21:37 PM3/5/09
to Devel::NYTProf Dev
Anyone have a workaround for the Storable issue yet? (I have the same
issue with a fresh 5.10 install on OS X 10.4.11)

I am able to generate an nytprof.out file, but get a similar Storable
not found error when trying to run nytprofhtml.

Tim - any guidance welcome.

--
Will "Coke" Coleda

On Feb 5, 10:47 am, Tim Bunce <Tim.Bu...@pobox.com> wrote:
> It's caused by AutoSplit being run on Storable.pm before it's installed
> by perl. AutoSplit adds #line directives to the auto-split files but
> those #line directives have relative paths that bear little relation
> to where the files end up.
>
> It's probably possible to work-around it. If someone wants to volunteer
> I'd be happy to guide them.
>
> Tim.
>
> On Wed, Feb 04, 2009 at 09:08:56AM -0500, Adam Kaplan wrote:
>
> > Off the top if my head, and without access to the code, I'd guess the
> > path to Storable.pm is the problem.  The prefix "../../lib" added to
> > the library include path you provided would result in something like
> > "perl/lib/lib/"
>
> > It's been a while since I looked at this stuff and I am on a train.
> > Maybe that will help you debug.
>

Tim Bunce

unread,
Mar 5, 2009, 6:16:55 PM3/5/09
to develnyt...@googlegroups.com
On Thu, Mar 05, 2009 at 01:21:37PM -0800, Coke wrote:
>
> Anyone have a workaround for the Storable issue yet? (I have the same
> issue with a fresh 5.10 install on OS X 10.4.11)
>
> I am able to generate an nytprof.out file, but get a similar Storable
> not found error when trying to run nytprofhtml.
>
> Tim - any guidance welcome.

You and 'tcbarrett' seem keen, great!

Here's a summary of the problem and what we can do about it...

AutoLoader (or rather, AutoSplit, http://search.cpan.org/perldoc?AutoSplit)
writes an autosplit.ix containing a bunch of stubs, and a bunch of
foo.al files, one per AutoLoad'able sub. AutoLoader loads the autosplit.ix
file for the module that's 'use'd it, and supplies an AUTOLOADer that
loads the foo.al for whatever sub is used.

From NYTProf's perspective, with trace=2, we see this sequence of fids
(file id's) being assigned:

> New fid 2 (after 1:7 ) 2 e0:0 /usr/lib/perl5/5.8.8/i386-linux-thread-multi/POSIX.pm
> New fid 5 (after 3:159 ) 2 e0:0 /usr/lib/perl5/5.8.8/i386-linux-thread-multi/auto/POSIX/autosplit.ix
> New fid 6 (after 3:961 ) 2 e0:0 ../../lib/POSIX.pm /tmp/t/../../lib/POSIX.pm

(there are two paths on that last line because the filename perl has is
relative, so NYTProf tries to determine the absolute path by prepending
the current working directory.)

I'll ignore the autosplit.ix as it's not a problem.

The problem is that the foo.al files contain a #line directive like
#line 120 "../../lib/POSIX.pm (autosplit into ../../lib/auto/POSIX/errno.al)"
which perl dutifully accepts as the line number and filename.

So NYTProf see the filename as
"../../lib/POSIX.pm (autosplit into ../../lib/auto/POSIX/errno.al)"
It's smart enough to strip off the " (autosplit..." suffix, but that's all.
So we're left with a relative filename that bears little relation to reality.

What we really want to happen is for get_file_id() in NYTProf.xs to
return the same fid for both these strings:
"/usr/lib/perl5/5.8.8/i386-linux-thread-multi/POSIX.pm"
"../../lib/POSIX.pm (autosplit into ../../lib/auto/POSIX/errno.al)"

A slight complication is that get_file_id needs to be _very_ fast.
That means we need both strings to be in the hash table, so when
an autosplit entry is added we need to detect that and search for the
corresponding 'parent' fid and then assign the same fid to the autosplit
entry.

Here's a sketch of the steps I'd take (possibly wrong and probably
incomplete :-) ...

[... deleted ...]

I did write it out but then I figured it would be easier to do the
structural change myself and leave the more, er, interesting parts to
you :)

See r694

Now all you need to do is write find_autosplit_parent() :-)

That needs to iterate over the hash, like write_cached_fids() does,
and find an entry with a .key that 'matches' the file_name argument.
So for the example above, "../../lib/POSIX.pm (autosplit ...."
should match the "..../i386-linux-thread-multi/POSIX.pm" entry.
Clearly ignoring the directory portion is essential. I'd also
suggest not exiting the loop early, so you get the most recent
fid that matches, as that's most likely to be the right one.

I guess you can decide between yourselves who writes the code and who
writes the tests. Or have a race... both do both and I'll pick the
winner :)

Tim.

Will Coleda

unread,
Mar 10, 2009, 2:27:06 PM3/10/09
to develnyt...@googlegroups.com
On Thu, Mar 5, 2009 at 7:16 PM, Tim Bunce <Tim....@pobox.com> wrote:
>
> On Thu, Mar 05, 2009 at 01:21:37PM -0800, Coke wrote:
>>
>> Anyone have a workaround for the Storable issue yet? (I have the same
>> issue with a fresh 5.10 install on OS X 10.4.11)
>>
>> I am able to generate an nytprof.out file, but get a similar Storable
>> not found error when trying to run nytprofhtml.
>>
>> Tim - any guidance welcome.
>
> You and 'tcbarrett' seem keen, great!

Added a ticket that for now just points to this thread.

http://code.google.com/p/perl-devel-nytprof/issues/detail?id=14

--
Will "Coke" Coleda

Tim Bunce

unread,
Mar 12, 2009, 6:17:56 AM3/12/09
to develnyt...@googlegroups.com

I'd like to make a release soon. Any chance anyone will be able to work
on find_autosplit_parent() soon? It should be pretty straight forward,
almost trivial.

Tim.

Will Coleda

unread,
Mar 12, 2009, 9:27:53 AM3/12/09
to develnyt...@googlegroups.com
I have gotten a checkout, opened a ticket, and verified the bug on 3
different platforms. I didn't actually do anything about the bug yet,
though. =-)

If you can wait one more day, I will see if I can craft a patch by the
end of the day.

Will Coleda

unread,
Mar 15, 2009, 2:38:20 PM3/15/09
to develnyt...@googlegroups.com
On Thu, Mar 12, 2009 at 9:27 AM, Will Coleda <wi...@coleda.com> wrote:
> If you can wait one more day, I will see if I can craft a patch by the
> end of the day.

So much for that; I didn't realize that the function in question was
an XS function, which is a more substantial time investment on my
part.

It's on my list still, but I have no time estimate at this point.

Regards.


--
Will "Coke" Coleda

Tim Bunce

unread,
Mar 21, 2009, 7:39:02 PM3/21/09
to develnyt...@googlegroups.com

No problem. I've found a tuit lying around and have (mostly) done it now.

Tim.

Will Coleda

unread,
Mar 23, 2009, 10:52:20 AM3/23/09
to develnyt...@googlegroups.com

From http://code.google.com/p/perl-devel-nytprof/issues/detail?id=14:
---
Tried against r717; all tests pass, and the original problem is resolved.

Thanks!

--
Will "Coke" Coleda

Reply all
Reply to author
Forward
0 new messages