Cherry picking values inside code

[Florido Paganelli]

Hi TIm, all

It is not easy to find topics in this google group, so I write my issues here.

I am trying to cherry pick selected values directly in the code, without letting nytprof create intermediate documents like html or csv. I just
want to open the db and select what I need. I am using it for a big piece of software that is serving experiments at CERN called ARC [1].

The issue I have now is that I know what kind of stats I want to get from a specific line of code but I
don't really understand how to do it in the code. Here's a sample snippet:

---------------------------------------------------------------------
# get the database file and open it as a perl datastructure
my $perflogdir = '/var/log/arc';
my $dbfile = 'infosys_20160704182046.perflog.raw';
use Devel::NYTProf::Data;
my $profile = Devel::NYTProf::Data->new( { filename => $perflogdir.'/'.$dbfile, quiet => 1 } );

# for selected subroutine, get time it took to run
# There is a perl module called GMJobsInfo.pm
# with a subroutine called get_gmjobs
my $subinfo = $profile->subinfo_of('GMJobsInfo::get_gmjobs');
my $getgmjobstotaltime = @{$subinfo}[4];


# for selected line of code, get time it took to run
my $linesinfo = $profile->fileinfo_of('GMJobsInfo.pm');
my $info = @$linesinfo[7];
my $fidline = $info->{fid_line_time};
# 6: id assigned to GMJobsInfo
# 151: line of open local files
# strategy:
# 1. get the pm file id
# 2. get the fid_line_time datastructure
# what is the fastest way? can it be done from $profile?
my $linetime = @{@$fidline[6]}[157];

---------------------------------------------------------------------

But I feel this is a little clunky and not very practical. What is the best way to achieve the above in the code?
Getting stats for subroutines and lines of code directly?

Can you show me an example? The csv and html readers are not interesting.
They parse everything. It is not what I need.

Thanks!
Florido Paganelli,
Lund University [2],
NorduGrid Collaboration [3]
Homepage: http://www.hep.lu.se/staff/paganelli/doku.php/florido_paganelli



[1] ARC middleware, http://www.nordugrid.org/arc/
[2] Lund University, www.lu.se
[3] NorduGrid, http://www.nordugrid.org/

[Tim Bunce]

On Wed, Jul 06, 2016 at 04:21:55AM -0700, Florido Paganelli wrote:
> Hi TIm, all

Hi Florido.

Ignoring clunky/impractical for now, does that code work for you?

> What is the best way to achieve the above in the code?

The above doesn't look like much code so if it works then I'm not clear
what your specific concerns are.

> Getting stats for subroutines and lines of code directly?

Isn't that what you've done above?

> Can you show me an example? The csv and html readers are not interesting.
> They parse everything. It is not what I need.

At their core they iterate over lines and subroutines and output data
for each. Some digging past the code doing the iteration should yield
some useful code snippits.

For subroutines, there's a module that provides a more abstract API:
https://metacpan.org/source/TIMB/Devel-NYTProf-6.03/lib/Devel/NYTProf/SubInfo.pm#
and a similar one for files:
https://metacpan.org/source/TIMB/Devel-NYTProf-6.03/lib/Devel/NYTProf/FileInfo.pm#
Using those avoids the need to break encapsulation.

I've not touched the code for a while so I'm rusty on the details. Sorry.
More specific questions might get more useful answers :)

Tim.

[Florido Paganelli]

Hi Tim,

Thanks for your answer. Somewhat I didn't receive it from googlegroups.

First of all I forgot to say I am actually not using Nytprof 6.03 because is not available in most distributions. It is unlikely that I can use the latest version. Our software runs mostly on long term distros (CentOS6 or 7) and they are unlikely to have the latest version. Actually FYI this work of mine pushed Nytprof 6.03 into EPEL[1], as we need it for production sites, but I have to develop for any platform and the debian/ubuntu world still ships either 4.06 or 5.06.

Further comments inline

It works but It's quite unreadable. For example, in a month I will forget why I used [6] instead of anything else,  and I spent a lot of time doing the typecast.
I need to test that this works for all versions too...
 

>    What is the best way to achieve the above in the code?

The above doesn't look like much code so if it works then I'm not clear
what your specific concerns are.
 

As in the subject, cherry picking specific values from the nytprof DB without having to parse the whole db.
But I spent a lot of time dumping the datastructure to understand the format, and for everything
I need I basically need to do a trial and error checking the output of my code against the outputs of nytprofhtml.
 

>    Getting stats for subroutines and lines of code directly?

Isn't that what you've done above?

Yes but I am wondering if there is a more straightforward way than playing around with types in the full datastructure. A wrong typecast and Perl will return quite unexpected content. Maybe a different version of Nytprof or Perl might beheave differently (I don't know didn't check yet)
 

>    Can you show me an example? The csv and html readers are not interesting.
>    They parse everything. It is not what I need.

At their core they iterate over lines and subroutines and output data
for each. Some digging past the code doing the iteration should yield
some useful code snippits.

I'll look deeper into that, thanks!
 

For subroutines, there's a module that provides a more abstract API:
https://metacpan.org/source/TIMB/Devel-NYTProf-6.03/lib/Devel/NYTProf/SubInfo.pm#
and a similar one for files:
https://metacpan.org/source/TIMB/Devel-NYTProf-6.03/lib/Devel/NYTProf/FileInfo.pm#
Using those avoids the need to break encapsulation.

that sounds good. I will have to compare the changes between different versions and see what is best.
 

I've not touched the code for a while so I'm rusty on the details.  Sorry.
More specific questions might get more useful answers :)

thank you, I'll check your suggestions and eventually come back to you.

Regards,
Florido

[1] Mattias request to push Nytprof into EPEL[2] on behalf of NorduGrid, http://www.spinics.net/linux/fedora/fedora-devel-perl/msg125742.html
[2] Extra Packages for Enterprise Linux (EPEL), https://fedoraproject.org/wiki/EPEL
==================================================
Florido Paganelli
ARC Middleware Developer - NorduGrid Collaboration
System Administrator
Lund University
Department of Physics
Division of Particle Physics
BOX118
221 00 Lund
Office Location: Fysikum, Hus A, Rum A403
Office Tel: 046-2220272
Email: florido.paganelli@
Homepage: http://www.hep.lu.se/staff/paganelli
==================================================