PHP TestFest Success Metrics

[Ben Ramsey]

Cal asked on Twitter[1]:

> What is the metric for success for testFest?

I thought it would be a good idea to take this to the mailing list to
generate some discussion about it, and since my answer is longer than
140 characters.

I have three main goals for this incarnation of PHP TestFest, none of
which are to increase the code coverage metrics of the PHP language (see
coverage reports here: <http://gcov.php.net/>):

1) Increase awareness of how PHP is tested
2) Show that there are more ways to contribute to the PHP core than
through hacking on the engine in C
3) Introduce newer folks in the community to phpt tests and teach them
how to write and run them

For the group of folks in #3, I hope that they'll become frequent core
contributors, once they know the environment and how to write and run
phpt tests.

#1 and #2 are difficult to measure, but with #3, it may be possible to
measure the number of people who have never contributed to php-src who
have submitted new phpt tests during PHP TestFest.

For all 3 items, I'd like to record the number of groups who
participated and held at least one PHP TestFest event, the number of
total PHP TestFest events, and the number of people who attended each
event. It would also be good information to know the number of phpt
tests contributed by each group (i.e., the aggregate of tests
submitted by members of each group).

So, that's a quick brain dump.

Thoughts?

Cheers,
Ben


[1] https://twitter.com/CalEvans/status/887380464855658498

[and...@heigl.org]

Technically would it be a possibility to use the GitHub-clone of the official PHP-Repo to create PRs against? That way we would (relatively) easy see who never contributed to the core before. If participating usergroups would create a fork that would then be forked from the participants that should work as well and we could get all the contributions from a user group in one batch and we would be getting the number of participating usergroups en passant…

I'd also back the idea that it's more important to get people to testing and contribute to tests over the sheer number of tests. After all every participant is a win for the PHP-Project! And if people only get to setup their system and don't contribute that's fine. The only way to measure that is by getting the number of participants from the different usergroups and that needs to happen via a separate "survey"… 

We can however try to find out how the test-suite improved by registering the number of contributed and fixed tests. But I'd only see that as an overall figure to see how much the testfest helped to improve the tests. I'D not break that down to a "per-usergroup" or "per-user" ratio (you can do that but it doesn't say anything…)

My 0.02€ 

Cheers

Andreas

 

 

[c...@calevans.com]

Honestly, I was hoping for a hard metric. 

% code Coverage

##k Unit tests

Something that we can put on a thermometer and see the progress.

Cheers! :)

=C=

[and...@heigl.org]

Am Dienstag, 18. Juli 2017 21:39:27 UTC+2 schrieb c...@calevans.com:

Honestly, I was hoping for a hard metric. 

% code Coverage

##k Unit tests 

Something that we can put on a thermometer and see the progress.

The overall progress or the best groups? 

For the overall progress we can use the number of overall added and fixed tests as well as the improvement in CodeCoverage (we're not disputing the value of that metric here) - even broken up to different extensions.

That will show the impact of the testfest for the PHP-Project.

But breaking that down to a group-level is IMHO dangerous as there will be groups with more and less contributors so the absolute numbers don't say anything. And every participating group is a "winner". 

But - as I said: Just my 0.02€ ;-)

Cheers

Andreas

[Mike Classic]

Overall progress would be an interesting metric, if it can be done, if not I don't think it's a big deal.

On another point, I have to agree that I don't think it wise to break it down to a group level for a myriad of reasons, including how groups vary wildly in makeup.

I don't think this is a contest anyway, at least the way I see it.

Mike

[Ken Guest]

Hi,

Perhaps [also] a tweetometer to inform the twitterverse each time a
testfest PR has been created?

I know it's not a success metric as such but I think it would aid
publicise what's happening :-)

k.

> --
> You received this message because you are subscribed to the Google Groups
> "PHP TestFest" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to testfest+u...@phpcommunity.org.
> To post to this group, send email to test...@phpcommunity.org.
> Visit this group at
> https://groups.google.com/a/phpcommunity.org/group/testfest/.
> To view this discussion on the web visit
> https://groups.google.com/a/phpcommunity.org/d/msgid/testfest/02954b8c-a328-4b90-98ef-971cfa167c4f%40phpcommunity.org.



--
http://about.me/kenguest/

[Ben Ramsey]

> On Jul 18, 2017, at 2:09 PM, Ben Ramsey <b...@benramsey.com> wrote:
>
> Cal asked on Twitter[1]:
>
>> What is the metric for success for testFest?
>
> I thought it would be a good idea to take this to the mailing list to
> generate some discussion about it, and since my answer is longer than
> 140 characters.
>
> I have three main goals for this incarnation of PHP TestFest, none of
> which are to increase the code coverage metrics of the PHP language (see
> coverage reports here: <http://gcov.php.net/>):
>
> 1) Increase awareness of how PHP is tested
> 2) Show that there are more ways to contribute to the PHP core than
> through hacking on the engine in C
> 3) Introduce newer folks in the community to phpt tests and teach them
> how to write and run them

I’ve updated the PHP TestFest website with copy that makes my goals
clearer. Let me know what you think.

https://phptestfest.org/
https://phptestfest.org/start/

Specifically…

“Growing the PHP core one test at a time”

“Become a PHP core contributor! Join a PHP user group in person or
online, and PHP TestFest will show you how.”

“Have you ever wanted to contribute to PHP but have been afraid that
your C skills aren’t up for the challenge? Well, have no fear! If you
know PHP, you can contribute by writing tests. Through your local user
group, PHP TestFest will show you how.”

“PHP TestFest seeks to create more PHP core contributors by introducing
PHP programmers to the PHP language test suite, teaching them how to
write phpt tests. We do this by using our relationships with PHP user
groups to host PHP TestFest events around the world.”

Let me know what you think.

Cheers,
Ben