Showing posts with label polling. Show all posts
Showing posts with label polling. Show all posts

Saturday, April 25, 2015

Pick a number

The art of data denial

The United States economy has climbed out of the hole created by the Great Recession and the Obama Administration has presided over an unbroken string of increasing job numbers lasting more than five years. I commented on this accomplishment and posted an illustration summarizing the data.


This did not sit well with one of my Facebook “friends,” who promptly took me to task:
Really, you are smarter than this BS!
It's a puzzlement how many people assure me I'm smart while insisting I'm also stupid. I asked my accuser how I had gone astray when my evidence-based conclusion was supported by the published numbers. She didn't hesitate:
Evidence that can be changed or tampered with. Too easy to make things look good for the moment!
Ah, yes. We must question authority and doubt the data. No doubt she took comfort in 2012 from the “poll correcters” who disdained the conclusions of the professional pollsters that President Obama was heading toward reelection. Remember them? (I'm sure folks in the Romney campaign do.)

I shared a story with my data doubter, explaining to her that data tampering in this context is a myth. It doesn't happen. Perhaps that sounds like simple-minded credulity, but I actually know what I'm talking about.

Once upon a time I worked for a state agency that was responsible for the annual computation and publication of the California Necessities Index. The CNI had been adopted by the legislature as the standard for indexing public assistance benefits (mostly because, at the time, it was lower than the Consumer Price Index). I was the person within the agency who was assigned to perform the actual computation, since I was the closest thing to a staff mathematician.

My father was amused by this situation and jokingly suggested it would be nice if I shaved off a fraction of a percentage point, thus denying full benefits to welfare cheats (most of the people on public assistance, in Dad's proto-tea-party view of the world). Contrariwise, I could have been a benefactor to California's destitute and downtrodden by judiciously rounding up the factors of the CNI, giving them an unexpected windfall. We could tamper with the numbers!

No. The very notion was ridiculous, as well as impossible. The components of the CNI were public information, factors selected from the published data for the CPI. Anyone could look up the factors that went into the CPI, apply the weights stipulated for the CNI, and derive the number. My role was simply a formality, providing the number that appeared in our agency's publications to make it official, pursuant to state law. Well before we published the number, the governor's Department of Finance had done its own independent calculation, as had the Legislative Analyst's Office and anyone else who needed the index number or was just curious about it.


The same thing is largely true of the numbers published by the Bureau of Labor Statistics. The worker bees of the federal government compile the monthly data each month, analyze the figures, and publish the summary information. The monthly unemployment rate, for example, is not vetted, suggested, or even approved by the White House. The faceless bureaucrats who crunch the numbers for the Obama Administration did the same exact thing when Bush was president (and, for some of these civil servants, they did it under both Bushes). They have no more control over the conclusions or final numbers than I did when I was a California civil servant. (It's popular within the right wing these days to decry unemployment data as unrepresentative of the “truth” because the usual unemployment number is deemed to have fallen when people just give up and stop seeking jobs. If they think the U3 measure published by the BLS is insufficiently informative, they are welcome to cite U6, which the BLS also publishes. You want numbers? Go to the BLS. They publish practically everything and the numbers aren't fudged.)

Data denial is a fundamental component of the refusal to recognize global-warming induced climate change. Data denial is also currently hard at work in California, giving aid and comfort to people paranoid about vaccinations, who pick and choose among an Internet stew of “research” and anecdotes to bolster their arguments that vaccines are more dangerous than now-rare childhood diseases (rare because of vaccines!).

If you want your conclusions to be rational and evidence-based, you have to avoid the denial of well-established data. We have seen, however, that even the most robust numbers are rejected when they get in the way of political ideologies.

Sunday, March 02, 2014

The fat man bombs

Time for the milk carton?

I get plenty of junk mail from right-wing sources. It's mostly my own fault. I incautiously signed up for a couple of newsletters and soon I was on a surprising number of mailing lists. Fortunately, the entertainment value is often high enough so that I don't rue the attendant spam. At least, not too much. A real prize popped up in my in-box this weekend. The folks at Townhall.com are giddy with anticipation over the imminent celebration of political irrationality known as CPAC—the Conservative Political Action Conference. They sent out a preliminary fundraiser in the guise of a political poll: “Who do you want the GOP to nominate for President in 2016?” I'm thinking Rick Santorum, because he strikes me as the weakest of the Republican Party's benchwarmers.

But his visage was missing from the rogues' gallery provided in the Townhall e-mail. I search for his insipid grin in vain. Fortunately, there were several other unviable alternatives. However, then I noticed something else. Check out the faces that Townhall deemed worthy of appearing in their solicitation. So you see it? Or, more to the point, not see it?
That's right. No Chris Christie. The man widely regarded as the GOP's 2016 front-runner no longer even qualifies for the party's top nine—at least, not as far as the American Conservative Union is concerned. Do you even recognize the mug shots of those who came in ahead of Christie? Sure, some of the usual suspects have been rounded up: Palin, Bush (Jeb), and Cruz are all there. So are non-starters like Jindal, Rubio, and Perry. Even Ryan and Paul (Rand) get a nod. And, in case you didn't puzzle out his identity, there's Dr. Ben Carson up in the corner, the GOP's token black who isn't even running (although you'll be glad to hear that a group of fans have started a petition drive urging him to get into the race).

Fear not. All is not lost! If you click through to the ACU site, you'll discover that your menu of candidates has been expanded. Fifteen names are offered, and this time Christie makes the cut. So does Santorum! As do Huckabee, Allen West, Scott Walker, and Judge Napolitano (whose first name appears to be “Judge,” in curious contrast to the other candidates). It's a bumper crop of undistinguished right-wing party hacks. Hillary must be pleased.

As for Christie, well, what more can one say about what Townhall did not say? He is no longer a first-tier candidate in their eyes. A lot of water must have gone under the bridge.

Saturday, January 25, 2014

Sharing the good news

A most unlikely source

Do you remember Ken Ham's lament that most teenagers stop going to church when they leave the family nest? The Creation Museum highlights the datum that only one in three continue their participation in church activities once they are on their own. It's one of the most uplifting features of Ham's “museum.”

Similar good news comes to us now from Michael Voris, the unconscious self-parody who holds forth at ChurchMilitant.TV, routinely excoriating the insufficiently ardent faith of the current leadership of the Roman Catholic Church. Voris wrings his hands in frequent episodes of  The Daily Vortex (“where lies and falsehoods are trapped and exposed” [the distinction between lies and falsehoods is never clarified]), decrying the lack of rigor in contemporary Catholic practice.

Despite himself, Voris recently found himself unhappily reporting good news from the annual “March for Life” in Washington, D.C. With microphone in hand, Voris accosted several young demonstrators who styled themselves “pro-life” and quizzed them on camera, subjecting them to a quick inquisition on the depth and breadth of their faith. To his horror, he discovered that approximately 30% of the young Catholics were unwilling to agree that contraception is always wrong: “Do you think a couple using birth control is always wrong in every situation?”



Voris was deeply shocked that many respondents did not agree with him that contraception is inherently a “diabolical evil.” The video ends with a lengthy and irritatingly repetitive diatribe against all forms of birth control (in stark black-and-white for enhanced drama). Exposing the laxity of young Catholics with respect to contraception was just the tip of the iceberg. Voris also quizzed the March for Life participants on the evils of homosexuality. Many of the young demonstrators disappointed Voris in their lack of anti-gay militancy. “Do you think it is okay for two guys to be in a romantic relationship?”



Some of the respondents are the same young people who indicated acceptance of contraception in the previous installment of The Vortex, but several new faces also popped up. A few of them wanted to qualify their position as “tolerance” rather than as “acceptance” of the right of people to engage in same-sex relationships, but Voris was still deeply dismayed that approximately 20% were essentially okay with gay partnerships.

Voris and his fellow Catholic militants fancy themselves as the faithful remnant that will be exalted at the second coming of Jesus Christ (any day now!), although they do not embrace the rapture concept of evangelical Protestant eschatology. Instead they are bracing themselves for the great apostasy that they believe is already rampant in what Voris dismissively describes as “the Church of Nice,” the insufficiently macho current incarnation of the One True Church. The bunker mentality is evident in each episode of The Vortex. But with Voris's every pronouncement of impending doom, the sensible viewer can take comfort in the dwindling influence of his point of view within the ranks of the next generation of Catholics. Not even the clergy embrace Voris's extreme ultramontanism.

Gaudeamus!

Friday, July 05, 2013

Familiarity breeds contempt

No love for Sister Sarah

Townhall.com has sponsored one of those irritating pop-up polls that haunt the Internet. I normally ignore such things. My pop-up blocker catches most of them, but this one slipped through. (Okay, I confess! I asked for it. Townhall sneakily embedded it in a Facebook ad and I clicked on it.)

The question was very simple and probably easy to answer for 99% of the population. It asked the weighty question, “Should Sarah Palin run for Senate? She's considering it.” Since “Hell, no!” and “Only for the entertainment value” were not provided as response choices, I settled for “No, her time has passed.” The alternative was “Yes would love to see her back in elected office,” which seemed too subtle in its irony.

After voting, I got to see the results. Over two-thirds of the respondents wanted Sarah to run (but there was no way to determine how many of those respondents were mentally appending “and lose” to their answers). This is not a surprising result from a crowd of people willing to follow a link provided by Townhall.com. (Of course, Sarah might lose interest if she discovers that a Senate term is six years long.)

The responses were broken down by state. I looked at them one by one. Idaho was the most eager to see Palin on the Alaska ballot for U.S. Senate. No big surprise. I moused around until I found the state most opposed to a Palin candidacy. (You're ahead of me, right?)

It was Alaska. The state that knows Palin best expressed itself with 55% in her favor and a record 43% opposed. No other state broke out of the thirties in its opposition to the half-term governor. And don't forget that this is a Townhall crowd. I was not surprised. You weren't either, were you?

Sunday, August 24, 2008

Obama is building the base

The foundation firms us

Although I know better than to take polls too seriously, I'm as curious as any political junkie about the state of the presidential race. As a devoted Democrat, I smile when Obama opens up a solid lead and I cringe whenever it narrows—even though I know what the polls say today is not necessarily what the ballots will say in November. Still, competent pollsters know how to take a snapshot of voter sentiment and give us a sense of how things might be shaping up.

That's why I watch the numbers over at electoral-vote.com, where the Votemaster tracks several polls and aggregates the results. (I've even installed its vote-tally widget in my sidebar; just look to the right.) I wonder whether people have noticed an interesting trend in the numbers. The Votemaster provides a number of graphs to illustrate how the results have varied with time. The graph depicting the electoral vote for each candidate shows how McCain appears to have closed the gap with Obama and turned the presidential contest into a neck-and-neck horse race. This is enough to curdle the blood of any partisan Democrat, although it's certainly too early to panic:


But the Votemaster also provides a graph of “solid” electoral votes. He notes that “The electoral votes of a state only count in this graph if the candidate has a margin of 5% or more over his opponent.” The story in the “solid” graph is quite different. While the first graph showed how the gap has closed in terms of total electoral votes, including the closely fought states, the second graph demonstrates that it has widened in terms of the solid states. Obama has been grinding away at building a strong base—a “solid” foundation—of states where his margin over McCain is at least five percentage points. While a five-point margin isn't entirely beyond the typical pollster's error bounds, it's an extremely good bet. With minor variations, Obama's base of solid states has been growing steadily, until it now approaches the magic number of 270.


By contrast, McCain's tally of solid states has been very slowly eroding. This implies that his recent apparent success in closing the gap with his rival is based entirely on tiny margins in swing states. He must essentially run the table to prevent Obama from snagging the one or two marginal states needed for a Democratic victory. Any one of the states of Florida, Ohio, North Carolina, or Virginia would put it away for Obama. Colorado plus one of the Dakotas or Montana would also suffice.

Perhaps McCain should ask Cindy if she remembers where their homes are. She may need to buy houses in all of the swing states to give her husband that home-base advantage. He's going to need it.

Saturday, October 28, 2006

A polling postscript

Caveat lector

No sooner had I finished posting my primer on polling than my attention was drawn by Taegan Goddard's Political Wire to a useful Wall Street Journal article on the limitations of political polling. The piece was written by Lauren Ettner and contains some good points about volatility in the electorate before a major election and the impossibility of getting definitive polling results when a significant chunk of the voters hasn't even made up its mind yet. These are useful warnings to keep in mind.

At the end of the article, however, I found a graphic that provides a wonderful teachable moment about what polls tell us and don't tell us. Here's an excerpt from the graphic, showing results from a WSJ/NBC News poll on voter sentiment regarding a Democratic takeover of the U.S. Congress:

What do you think this poll means? While the title says “In 2006, signs point to a Democratic takeover,” this conclusion is not necessarily indicated. One should be much more careful in interpreting what poll respondents mean when they answer a pollster's questions. If you were a rock-ribbed Republican diehard who remained resolute in your support of the Grand Old Party, you could reasonably reply “No effect” to the question “Has what you have seen and heard over the past few weeks made you feel more favorable, less favorable, or had no effect on your feelings about possibly having the Democrats become the majority party in Congress?” After all, you hated the idea before and you still hate the idea right now. No change.

Let me drive the point home by relabeling the poll results (except for Other, which I skipped) in a manner entirely consistent with the responses, but with labels that suggest a dramatically different conclusion from the one in the WSJ title:

Please note that it was obviously necessary to change the data bars to red, reflecting the much more Republican outcome. Now I happen to think that the Wall Street Journal is correct in its conclusion that the electorate is more kindly disposed toward a Democratic Congress and is more likely to vote to elect one. That inference seems reasonable in the light of many other polls and indications. However, it does not ineluctably follow from the responses of the poll's subjects. A sample composed predominantly of determined Republican voters could have produced the very same answers documented in the WSJ graphic.

You've been warned. Don't get complacent. Go out there and vote for all the Democrats you can find.

Friday, October 27, 2006

A primer on polling

Sampling the electorate

I was listening to NPR's Science Friday and heard Ira Flatow decry the confusing business of contradictory polls. He mentioned that there are polling firms that specialize in working for Republican candidates and others that are allied with Democrats. Where are the independent and neutral polls of yesteryear?

The discussion went off in other directions without doing more than the radio equivalent of hand-wringing over partisan pollsters. I would like to offer some constructive comments on the practice of polling and the significance of polling results. I hope this primer on polling will answer some of the questions you might have.

Any reader of Halfway There knows that I am myself a partisan Democrat, so you might suspect I'm going to shade this essay on polling to favor my preferred liberal causes. You can decide for yourself, of course, after reading what I have to say. In the interests of balance, however, let us begin with a comment from arch-conservative Phyllis Schlafly:
Dr. George Gallup began asking a lot of questions of a very few people, and—funny thing—he usually came up with the answers that pleased the New York kingmakers.

Schlafly made this observation in the context of complaining how pollsters manipulate the electorate with statistically meaningless results that favor New Deal liberals. The quote comes from chapter 6 of Schlafly's 1964 book A Choice Not an Echo (there really ought to be some punctuation in there), which promoted the insurgent candidacy of Barry Goldwater for the Republican nomination for president.

Schlafly homes in on one of the most popular complaints about polls, often voiced by individuals who say, “No one has ever polled me!” Indeed, pollsters usually examine a sample that is much smaller than the actual population in question. For a statewide election in California, for example, a pollster is likely to interview a few hundred voters to discover the opinions of an electorate that comprises millions. How can this possibly work?

A sample example

A small sample can give you a surprisingly robust measure of what is going on with a large population. As long as the pollster takes some practical measures to ensure that a sample is not unduly skewed (don't find all your polling subjects in the waiting rooms of Lexus dealerships!), the sample will be representative of the whole population. That's why you'll hear people talk about picking people at random in a polling survey. It's a way to avoid biasing a sample.

Suppose, for the sake of illustration, that a voting population is evenly divided between candidates A and B. Suppose that you're going to pick two voters at random and ask them who they prefer. What could happen?

There are actually four possible outcomes: Both subjects prefer A, both subjects prefer B, the first subject prefers A while the second prefers B, and the first subject prefers B while the second prefers A. These four outcomes are equally likely, leading us to an interesting conclusion: Even a sample of size 2 gives you a correct measure of voter preference half the time!


No doubt this result should improve if we choose a larger sample. After all, while it's true that half of the possible outcomes correctly reflect the opinions of the electorate, the other half is way off, telling us there is unanimous sentiment in favor of one candidate.

Let's poll four voters this time. There are actually sixteen (24) equally likely results:


This time, six of the possible sample results (that's three-eighths) are exactly right in mirroring the fifty-fifty split of the electorate. What's more, only two of the possible samples (one-eighth of them) tell us to expect a unanimous vote for one of the candidates. The other samples (three-eighths of them) give skewed results—giving one candidate a three-to-one edge over the other—but not as badly skewed as in the previous two-person sample.

You know what's going to happen as we continue to increase the sample size: It's going to get more and more difficult to obtain really unrepresentative results. Let's look at what occurs when we increase the sample size to just eight randomly selected voters. There are 28 = 256 ways the samples can come up, varying from all for A to all for B. Okay, that's too many cases to write out individually. We'll have to group them. The following table summarizes the possibilities. For example, there are 28 cases in which we end up with 6 votes for A versus 2 for B. (In case you're curious, these numbers come from binomial coefficients, made famous in Pascal's triangle.) Check it out:

As you can see, in each case I've given the percentage of supporters for A found in the sample. There are 56 samples in which A has 62.5% support, 70 in which A has 50% support, and 56 in which A has only 37.5%. In this little experiment, therefore, we have 56 + 70 + 56 = 182 cases out of 256 in which A has support between 37.5% and 62.5%. Since 182/256 ≈ 71%, that is how often our random sample will indicate that A's support is between 37.5% and 62.5%. Observe that 37.5% = 50% − 12.5% and 62.5% = 50% + 12.5%. Since we set this up under the assumption that A's true support is 50%, our poll will be within ±12.5% of the true result about 71% of the time. Mind you, we have no idea in advance which type of sample we'll actually get when polling the electorate. We're playing the percentages, which is how it all works.

These results are pretty crude, since professional polls do much better than ±12.5% only 71% of the time, but we did this by asking only eight voters! A real poll would ask a few hundred voters, which suffices to get a result within ±3% about 95% of the time. That's why pollsters don't have to ask a majority of the voters their preferences in order to get results that are quite accurate. A relatively small sample can produce a solid estimate.

Bigger may be only slightly better

It's unfortunate that more people don't take a decent course in probability and statistics. That's why most folks are mystified by polls and can't understand why they work. They do work, as I've just shown you, within the limits of their accuracy. Pollsters can measure that accuracy and publish the limitations of their polls alongside their vote estimates. Every responsible pollster does this. (Naturally, everything I say is irrelevant when it comes to biased polls that are commissioned for the express purpose of misleading people. One should always treat skeptically any poll that comes directly from a candidate's own campaign staff.)

The controversy over sample size is constantly hyped by the statistically ignorant, most notably today by those who are upset by the sampling techniques used in a recently published Lancet study on civilian casualties in Iraq. (For example, here's the innumerate Tim Blair: “Remember: Lancet came up with this via a survey that identified precisely 547 deaths (as reported by the New York Times).” Mr. Blair thinks 547 is a tiny number.) More than forty years ago, Phyllis Schlafly was harping on the same point:
The unscientific nature of the polls was revealed by Marvin [sic] D. Field, formerly with the Gallup poll and now head of one of the polls which picked Rockefeller to beat Goldwater in the California primary, who admitted to the press that he polled only 256 out of the 3,002,038 registered Republicans in California. He thus based his prediction on .000085 of Republican voters.

While her math is okay, Schlafly doesn't know what she's talking about. A sample size of 256 is quite good and should have produced a reliable snapshot of voter sentiment at the time the poll was conducted. In addition to getting the pollster's name wrong (it's Mervin), Schlafly neglected to mention that Rockefeller's wife had a baby just before the California primary, sharply reminding everyone about his controversial divorce from his first wife. You can't blame a poll for not anticipating a development like that. Otherwise, Schlafly's complaint about the poll is based on her ignorance about the sufficiency of sample sizes.

By the way, did you notice that Mervin Field's sample size was a power of 2? It would have occurred in the natural progression of samples that I modeled for you in our polling experiment. In my three different sampling examples, I doubled the sample size each time, going from 2 to 4 to 8, each time getting a significant increase in reliability. If you keep up the pattern, you get 16, 32, 64, 128, and 256. As you can see, Field went way beyond my little experiment, doubling my final sample of 8 an additional five times before he was satisfied he would be sampling enough voters for a good result.

Two caveats

There are a couple of things I should stress about the polling game we just played. First, of course, in real life we would not know the exact division of the voters beforehand. That's what we're trying to find out. It won't usually be something as nice and neat as fifty-fifty. However, as long as there's a real division between voters (in other words, not some 90% versus 10% rout), it won't be too difficult to poll enough voters to get an accurate profile.

Second, even a poll that is supposed to be within its estimated margin of error 95% of the time will be wrong and fall outside those bounds 5% of the time. That's one time in twenty. Therefore, whenever you see a political poll whose results seem way out of whack, it could be one of those flukes. Remember, polling is based on probability and statistics: it's accurate in the long run rather than in every specific instance. In a hot contest where lots of polls are taken, a candidate's campaign is likely to release only those polls that show the candidate in good shape. The 5% fluke factor may be just enough to keep hope alive among those people who believe everything they read.

Pollsters take their results with a grain of salt, so you should, too. But it's not because of sample size.