National Survey: Plan to Run Four Voting Systems Side-by-side

[Robin Quirke]

I work for PolicyInteractive, a non-profit organization out of Eugene, Oregon that focuses on issues relating to climate change (from individual behavior to carbon tax policy work), campaign finance reform, and public opinion polling.  

We have recently begun to dig into the realm of voting reform, and are considering running a national survey that asks each person to vote from the same set of presidential candidates, but with four different voting systems: plurality, approval, IRV, and range voting.  This would include a set of questions that asks them how the different systems felt to them, if they were honest or strategic in their choices, and for the non-voters, would approval voting, IRV or range voting make them any more likely to vote in future elections.  Part of the fun of this would be to compare the results of the different systems to see how their results vary.  

Before we get the ball rolling, we are wondering: do any of you know of a project like this has already been done?  We would like to know what's out there before we possibly repeat any unnecessary aspects of this project.  

Many thanks!

[William Waugh]

Goddamnit, CES, you have _again_ (!!!) moved the interview with T. J. Frawls on the New York Poll!  What is the new URL for it???????

[William Waugh]

In addition to the _New York Poll_, which I hope our friends will exhume quickly, there is this six-hour conference in which the audience received a chance to do a little toy vote using several voting systems: http://www.independentpoliticalreport.com/2014/11/free-equal-solutions-presented-at-voting-methods-election-integrity-symposium/

[William Waugh]

OK, it's http://www.electology.org/#!Interview-with-Occupy-Wall-Street%E2%80%99s-TJ-Frawls/c17jj/87B57360-04F3-405F-A6CC-A1F2E636BB07

[William Waugh]

The summary page on the interview regarding the New York poll leads to this further link, to the complete report from the working group that did it, including self-critique: http://media.wix.com/ugd/8c56f6_891789caa7f94ad5b5fb074906cc92d4.pdf

[Warren D Smith]

On 7/15/15, Robin Quirke <robin...@gmail.com> wrote:
> I work for PolicyInteractive <http://www.policyinteractive.org/>, a

--there are often polls conducted by pollsters which employ approval
or range style voting.
Those are often at unpredictable times, and with unpredictable sets of
candidates and
samples, and with various unpredictable question-wordings, but they do
happen; and
then we can just sit back and collect & evaluate them.

--Also we ourselves (CES members) have sometimes conducted such polls ourselves
as exit polls. But unfortunately, in my opinion, so far, all of us
who have tried, have in one way or another done a bad job, falling
well short of professional pollster quality.

--One question which I would like you to ask, because ESSENTIALLY NOBODY
EVER ASKS IT, AND IT IS VERY VERY IMPORTANT, is, to ask
the voters to rate the voting systems (now they've just tried them on
your ballot).
And I see you indeed plan that. Good.

--Beware of internet polling, which is usually near useless because of
sampling issues.
Telephone polling also has big problems today because the vast
majority of those you call, refuse to answer the phone. In the old
days they would do so much more often.
I think telephone and in-person exit polls are the best options at present.

--Keep all reading your voters/pollees must do, very short.
Almost all respondents do not even want to read one medium-long
paragraph. But have a backup page for the rare people who want to
read much more than the typical, and to allow answering any questions
the pollees might
have in a unique manner (plan ahead to figure out all questions they
might possibly ask!). For the few words you do have, which should
be made very sculpted, non-redundant, and precise, CHOOSE THEM VERY
CAREFULLY, WELL IN ADVANCE, WITH MANY ROUNDS OF PRE-REVIEW AND
PRE-EDITING, MAKING SURE TO CONSIDER EVERY POSSIBLE WAY THEY COULD
MISREAD AND MISINTERPRET, AHEAD OF TIME. ALSO THINKING UP EVERY WAY
SOMEBODY COULD CRITICIZE YOUR RESULTS, AHEAD OF TIME, IN A VARIETY OF
SCENARIOS
ABOUT WHAT THOSE FUTURE RESULTS MIGHT BE.

It boggles my mind how incredibly stupid a lot of people who try to do
polls, are about this,
They go to tremendous effort, only to produce useless garbage because
they left an obvious way open for the voters to misinterpret the
question. I have even seen instances of this where I warned them
ahead of time about a misinterpretation risk, they told me to go fuck
myself since I was clearly a nitpicker and no human could possibly
have that problem; then sure enough they ended up questioning their
own results for precisely the reason I'd warned them about. Totally
avoidable.

--For the range voting subpoll,
Use 0-to-9 range voting, i.e. 10 possible scores for each candidate,
each a single digit, plus an additional "do not know enough to score" option.

--Make sure to sample as uniformly as possible. If doing an exit poll,
your results will be garbage if your locations are not well distributed, e.g. if
the locations happen to lie in pro-Democrat areas.

--If using volunteers, you will need to make a web page well ahead of time, like
months ahead, telling the volunteers how to get sample ballots, what
to do, when and where to do it, how to pre-arrange with you, etc etc.
No web page, or put up only a few days before? You look like beyond
amateur then.

--Plan ahead to allow yourself to correct your results to get higher accuracy.
For example: If you ask pollees their gender, age, race, and who
they voted for
in the official election (if in an exit poll) you would then be able to weight
subsamples to try to get the effect of making your sample become
more-uniform than it
actually was. If you did not ask that, you'd be unable to do such corrections.
Mind you, it is best to make your sample good enough that not much
correction is needed.
If you need large corrections, then your results are questionable right there.

--An advanced technique is to have 2 poll versions with different but
allegedly equivalent question-wordings. (Each pollee gets a randomly
chosen poll version A or B.)
Then by comparing the results of the two poll versions, you get a
"noise gauge" or a gauge of how much the answers depend on the precise wording.
That can help you tell how much is valid and how much is garbage.

--
Warren D. Smith
http://RangeVoting.org <-- add your endorsement (by clicking
"endorse" as 1st step)

[Clay Shentrup]

Robin,

Thanks for sharing this information! I can't wait to see your results. Especially because climate change is such a hugely important issue.

Did you see this link about public opinion on voting methods?

http://scorevoting.net/WhatVotersWant.html

Presumably you could present the ranked voting results using a host of systems, from IRV to Borda to one or more Condorcet variants.

Good luck,

Clay

[Nevin Brackett-Rozinsky]

Robin, I think this is a fantastic idea and I look forward to seeing your results.

Warren, why do you recommend a 0–9 scale, rather than the more-familiar “Olympic-style” 0–10 scale?

[Rob Wilson]

Sounds like a productive project. Do you think a lot of people are going to understand IRV though or are you going to just ask them to rank the candidates?

[Robin Quirke]

Thank you for all of the excellent posts!  I am finally catching up on all of the feedback I received.  We (PolicyInteractive) ran a national pilot survey on Mechanical Turk over the weekend (at the time of this post: n = 178).  If you are curious, click here to take the survey; any feedback will be deeply appreciated and thoughtfully considered. Click here to see the results.  --RQ

[Robin Quirke]

Thank you!

On Wednesday, July 15, 2015 at 3:17:58 PM UTC-7, William Waugh wrote:

OK, it's http://www.electology.org/#!Interview-with-Occupy-Wall-Street%E2%80%99s-TJ-Frawls/c17jj/87B57360-04F3-405F-A6CC-A1F2E636BB07

[Robin Quirke]

Thank you, Clay.  We (PolicyInteractive) did run a national pilot survey on Mechanical Turk over the weekend (at the time of this post: n = 178).  If you are curious, click here to take the survey; any feedback will be deeply appreciated and thoughtfully considered. Click here to see the results.  --RQ

[Robin Quirke]

My associate already replied to much of your advice, etc. (thank you for taking so much time to do this).  As for the 0-9 vs. 1-10 scale issue, based on past research I have done, I walked away from it thinking that an odd numbered total of scale answer choices is ideal for these more continuous score scales (like 0-10).  Is the purpose of an even number of choices to force a person onto one side or the other of the center?  --RQ  

[Clay Shentrup]

On Tuesday, July 21, 2015 at 8:53:04 AM UTC-7, Robin Quirke wrote:

Is the purpose of an even number of choices to force a person onto one side or the other of the center?  --RQ  

Essentially. Thereby forcing them to think more instead of lazily picking a middle score. See more discussion here:

http://ScoreVoting.net/Why99.html

[Warren D Smith]

> Nevin Brackett-Rozinsky <nevin.brack...@gmail.com> wrote:
> Warren, it is entirely possibly that the vast majority *of people who
> self-select to take an online voting reform questionnaire* did indeed vote
> in the last presidential election. Don’t attribute to malice that which can
> readily be explained by sample skewing.

--I did not attribute to malice. It is either strategic lying, or it is
very biased sample, the latter quite possibly due to self-select as you say.
I'd guess: both.

--There are fairly standard well tested wordings used in the past by
pro-pollsters
to conduct ranking-style, rating-style, etc polls. You might want to consider
those wordings as a starting point. (But those polls usually had not
been intended
to emulate an official ballot in a future genuine election, & do not
describe how votes used to determine a "winner." And I do not like the
use of adjectives instead of numbers, think it is better to use
adjectives, if any, only to describe the two scale endpoints; and say
nothing about internal scale points. Adjectives have meanings that
different people may feel differently about, and if they are
non-english native speakers, etc.)
Here, I'm collecting some wordings from past pro-pollster polls:

SCORE STYLE, VERBAL NON-NUMERIC, 4 LEVEL
Do you have a favorable or unfavorable impression of Hillary Clinton?
Do you feel that way strongly or somewhat?

SCORE-STYLE BALLOT, VERBAL NON-NUMERIC, 5 LEVEL
How do you rank former president Gerald Ford?
(choices: failure, below-average, average, near-great, great.)
Franklin Roosevelt?
[10 other presidents ditto]

SCORE-STYLE 1-5 SCALE:
How would you rate Hillary Clinton... using a five-point scale, on
which a 5 means a very good rating, a 1 means a very poor rating, and
a 3 means a mixed rating?

SCORE-STYLE 5-LEVEL SCALE:
Please rate how Chris Christie is handling his job as governor, using
a grading scale from A to F. You can give him any full letter grade,
A, B, C, D, or F.

APPROVAL STYLE, NO-OPIN OPTION:
Is your opinion of John Kasich favorable, unfavorable or haven't you
heard enough about him?

PLAIN APPROVAL STYLE:
* Do you have a favorable or an unfavorable impression of Andrew Cuomo?

* Do you have a favorable or unfavorable opinion about Jeb Bush?

SCORE STYLE 0-100 WITH NO-OPIN OPTION:
I am going to read some names of potential presidential candidates.
I'd like you to rate that person using something called the feeling
thermometer. You can choose any number between zero and 100. The
higher the number the more favorable you feel toward that person, the
lower the number the less favorable. You would rate the person 50 if
you had neither favorable nor unfavorable feelings toward that person.
If you don't know who the person is just say don't know.
Scott Walker.
Jeb Bush.

RANK-ORDER STYLE:
Please rank the following options in order of preference (most-wanted first)...

--Your turk poll did not allow voters to score candidates as "no
opinion" in addition to
the choices 0,1,2,3,4,5,6,7,8,9. I think it would be interesting to
allow that,
(a) just to see what happens and (b) because I think score voting is a
better voting
system if this is allowed. You also could try a random selection of
allowing it for some pollees and disallowing it for others, that way
differences could
be detected in voter behavior given these two score voting flavors.
That design could
yield important new data that would be helpful.

Also as I suggested you could try 0-9 or 1-10 depending on randomness, and see
if this psychological presentation difference yields any detectable
differences in pollee responses. That also could be valuable new
info. In many score polls the voters have a tremendous attraction
to the score "0". Perhaps if it were "1" that'd lessen.
In some cases the wording has said something like "score 50 indicates
you feel neutral"
which perhaps can transfer this illogical attraction from 0 to 50.

[William Waugh]

Maybe it would be interesting if you included Majority Judgement as well so we can see how it does against Score. For the best comparison, they should sport the same count of grades.

On Wednesday, July 15, 2015 at 4:52:40 PM UTC-4, Robin Quirke wrote

https://groups.google.com/forum/?fromgroups=#!topic/electionscience/mlOzn-MrbFA