Correctly testing external binaries

[Cosimo Streppone]

Hi QA folks,

Net::Statsd::Server is a UDP server daemon packaged as CPAN module,
with a little 'bin/statsd' wrapper script.

In the latest release, 0.09, I tried to add some 'integration tests':

https://metacpan.org/source/COSIMO/Net-Statsd-Server-0.09/t/integration-tests

implemented through the Test::Statsd module:

https://metacpan.org/source/COSIMO/Net-Statsd-Server-0.09/lib/Test/Statsd.pm

Now, the idea behind this is that I can perform some server-independent
tests. The start_statsd() method forks off a given binary, with a
config file as only argument, and then tests are performed throwing
UDP packets at some local port.
When tests are done, stop_statsd() kills off the child process.

The goal is to be able to test the multitude of statsd server
implementations
out there just given a binary and a config file.

Immediate problems:

1) binary can be an executable or a script that needs $^X,
/nodejs/python/... How do I exec() reliably?

https://metacpan.org/source/COSIMO/Net-Statsd-Server-0.09/t/integration-tests/counts-are-valid.t

2) Omitting $^X is not desirable, because I got a slew of
CPAN testers fails. #!/usr/bin/perl is obviously a wrong choice
in so many situations.

Sample report:
http://www.cpantesters.org/cpan/report/02669934-6a35-11e2-b5a4-ab61270fb964

Less immediate problems:

3) Should this generic test suite be an independent CPAN distribution?

I'd appreciate any advice you might have.

--
Cosimo