FieldStat :: Summary
game...@gmail.com
Project Description
FieldStat is a tool for scanning a collection of application binaries
in order to understand how a particular set of classes and methods are
being used in practice. One main purpose of the tool is to support
prioritizing unit test coverage based on API usage data observed in
practice. Developers can view how often and how many applications use
certain types and method calls in relation to unit test coverage and
difficulty of testing.
This shares similar goals with clean room software engineering in that
a reliability model is based on statistical likelihood of occurring.
That is, place more testing effort in more likely occurring scenarios.
The tool relies on the Mono.Cecil assembly to process CIL byte-code
instructions.
Delivered
Planned
* Get up to speed with Cecil.
* Extract call graph.
* Identify and isolate mono framework calls.
* Implement CodeRank algorithm.
* Integrate with MonoCov.
* Test and Refine application.
* Documentation: User manual, design docs, touch-up comments
Extra
* Command Line and Graphics Interface
* Export Data
* Include other statistics such as application usage count, method
complexity.
* Plugin Architecture
* Plugin for finding Design Fragments (Set of reoccurring system
calls)
* (External) Improve Mono.Cecil tool to have better support for
reconstructing statements and message chain bad code detector.
Components
Application Repository
Understanding how API calls are used in practice requires sampling
actual software. Unfortunately, access to business software is
limited; however, many open source applications offer a good starting
point.
Over 500 projects were downloaded from the code.google.com project.
The projects were selected based on the label: Mono or CSharp. In
addition, 2 projects from a company were included. The projects were
manually built, or a binary distribution was acquired. Some projects
had to be excluded due to immaturity(not building), misclassification,
and lacking the appropriate resources to build the project.
Coverage Data
The monocov tool gathers statement coverage information from the run-
time execution of a Mono application. The statement coverage can be
gathered in the following manner.
In the mono distribution mcs/class/corlib directory:
> make run-test RUNTIME_FLAGS="--profile=monocov:outfile=corlib.cov,+[mscorlib]"
> monocov --export-xml=/tmp/corlib-cov corlib.cov
However, the generated xml file was intended for presentation, not
importing. Instead,
a new export option is introduced.
> monocov --export-fieldstat=/tmp/corlib-cov corlib.cov
Scanners and Analysis
FieldStat uses a visitor/collector pattern for gathering statistics. A
visitor class walks the assemblies, classes, and methods. To gather
statistics, a collector class is registered with the visitor and is
notified when a particular a node of interest is visited. For
instance, a collector can be notified whenever a system call is
encountered.
Some default collectors included in FieldStat.
* AppStat - Simply counts the number of system calls used per
application.
* CodeRank - Build's an application's call tree and calculates the
associated code rank of each application method.
* TypeCount - Counts the number of times a system type and system
method is statically called in an application.
Running the Tool
The simpliest way run FieldStat is to specify the directory containing
the coverage information (the -export-fieldstat output from monocov).
> FieldStat --coverage-path="../../../Data/Coverage Data/coverage_output" FieldStat.exe
This would output the Results.xml files in the output/ directory.
The file can be read with a XML parser or with the
DataTable.ReadFile( file ) call.
A record is as follows:
<results>
<type>System.IO.Path</type>
<method>GetFileNameWithoutExtension (string)</method>
<length>1</length>
<frequency>2</frequency>
<rankedfrequency>0.30</rankedfrequency>
<appfrequency>1</appfrequency>
<coverage>1</coverage>
</results>
Plugins and Future Directions
Plugin System
Plugins can be created by dropping in a file named *Plugin.dll into
the Plugins/ directory.
Design Fragments
An example plugin is included in the source code as the
DesignFragmentPlugin project.
A design fragment is a pattern or common usage of a framework. This
plugin attempts to detect candidates for design fragments by looking
at common sequences of calls. This then in turn could be used to
improve the framework, or serve as documentation or snippets in how to
use the framework.
For example, the following series of calls was found to occur 10 times
in the application repository. It looks like the user is trying to
format a date in a particular way:
System.DateTime.get_Day ()
System.Int32.ToString ()
System.DateTime.get_Month ()
System.DateTime.get_Year ()
System.Int32.ToString ()
System.String.get_Length ()
System.String.Concat (string,string)
System.String.get_Length ()
System.String.Substring (int,int)
System.String.Concat (System.String[])
The plugin is written as follows:
using FieldStat.DataCollection;
using FieldStat.CodeModel;
public class DesignFragmentPlugin : AbstractPlugin
{
public override void ComputeResults(Results results, ICollection
files, Hashtable htBin)
{
Visit scan = new Visit();
scan.Collectors.Register("DesignFragment", new
DesignFragmentCollector());
scan.DoScan( files, htBin);
MyCollector seqs = (MyCollector)scan.Collectors["DesignFragment"];
// Process Results ...
}
}
public class DesignFragmentCollector : AbstractCollector
{
public override void OnMethodBody(MethodBody body)
{
ArrayList seq = GetSystemCallSequences(body);
if (seq.Count > 3)
{
sequences.Add(EncodeSequenceCalls( seq ));
}
}
....
}
SystemSignature
This plugin extracts the system calls made in an application. Then the
resulting system call signature is compared against the other
application's signatures. The results can be clustered to find common
type of applications. It may also be used to judge the implementation
of a well-known type of application. For example, many irc clients
have been written. How divergent is yours from the other known
clients? Maybe you should have taken advantage of a different
architecture that you were not aware of. (Plugin under development)