RE: [FSP Board] Re: Jeremy is interested in joining the FSP board

[Jason P. Sorens]

 

My bad. The permissions on the group didn’t allow non-members to post. I’ve changed that for now, although I’m open to changing it back if there are objections.

 

Thanks for your responses, Jeremy. BTW, the name for those who sign up for the mailing list is, I believe, “subscribers.”

 

-J

 

From: Jeremy Kauffman [mailto:kau...@gmail.com]
Sent: Tuesday, May 15, 2018 10:12 PM
To: Jason P. Sorens <Jason.P...@dartmouth.edu>; fsp-...@googlegroups.com
Subject: Re: [FSP Board] Re: Jeremy is interested in joining the FSP board

 

(I tried to send below to fsp-board-open but it did not work. I'll look if I did something wrong, but you're welcome to forward.)

 

On Tue, May 15, 2018 at 10:10 PM, Jeremy Kauffman <kau...@gmail.com> wrote:

(Below is a response to a question from Jason about metrics usage by the FSP. I've switched this conversation to fsp-board-open at Jason's suggestion. I omitted the conversation prior to this request, but the large context is part of a conversation about joining the FSP board.)

Thanks for the questions Jason. I discussed this topic some with Matt as well.

On the question of which and how metrics can tell us how to get movers here, the answer is metrics don't tell you directly. Metrics can't determine strategy, only assess it. Metrics primarily let you evaluate past resource allocation choices and potentially resolve questions related to how you spend resources on similar strategies in the future. 

I don't understand with precision the current way the FSP is recording the status of participants and fans*, but there are stages similar to the traditional purchase funnel (fan, signer, mover, engaged participant, etc.). Just as with the FSP, most modern models are more tailored than the classic four in the link. With time and good data, one can even attempt to evaluate the relative value of people in the stages. If 10% of fans sign, and 25% of signers move, than actions that generate 4 signers are roughly as valuable as actions that generate 40 signers or 1 mover.

 

This kind of concrete analysis can irritate more holistic and perceptive thinkers for good reasons. I am well aware that Goodhart's law is super duper real. Metrics can't determine strategy, only assess it. Making a choice only because you thought it would improve the metrics the most is exactly the wrong way to make a choice. It will lead to really bad choices. Making two choices you would have made even if you had no metrics and using metrics to later assess them is how you use metrics. Metrics are like standardized tests - studying (optimizing for) them is cheating and ruins the whole thing.

 

But if you had a framework like this, you can now evaluate choices more concretely. PorcFest cost $XXX and YYY volunteer hours but nets us ZZZ expected participants, vs. travelling to Conference X gets us <whatever>. Note that even here the system of metrics already breaks down! How do I properly capture the longtail (or even immediate) value of a Washington Post article about PorcFest?

 

So really metrics aren't a panacea. They are a tool in your toolbox. When they can be done cheaply they are worth pursuing. More importantly, even if you can't reach the ideal form, taking steps towards it can help you think about these things better. And sometimes even plugging in fudged/guessed numbers into your ideal framework can help you make better choices than not using something at all.

In the specific case of PorcFest and Liberty Forum, my answer would be to first record these things assuming it is feasible/cheap. Record all pledges that happened after an event visit. Record visits from existing signers. Record press mentions. If you think there is some value the events provide that's not capturable by metrics, make it up! And then sit down and look at what you got and decide if you want to do more of it or less of it. Speaking personally and anecdotally, they played a role in my move and plenty of others I've talked to. But to have confidence in this? We'd need metrics :)

 

*Is there a name for mailing list members but not signers? IMO there probably should be

 

On Fri, May 11, 2018 at 12:15 PM, Jason P. Sorens <Jason.P...@dartmouth.edu> wrote:

 

Hi Jeremy,

 

Thanks for your detailed answers to the survey. It’s great to hear you’re flexible and open to evidence on tactical & strategic matters. I don’t have too much to ask you about, but I wonder whether you have any ideas of relevant, easily collectible metrics we could start to collect to help us figure out how to get more movers here. I’m also interested to hear your views on how the FSP currently uses our major events (PF, LF), and whether there are ways we could leverage these better to get movers at reasonable cost.

 

(BTW, we now have an optional, publicly viewable Board list at fsp-boa...@googlegroups.com. I don’t want to migrate a conversation there without the consent of the participants, but just so you know…)

 

Jason