Go not to the Elves for counsel, for they will say both no and yes.
– Frodo, Lord of the Rings
By Patrick McGhee
With the Scottish Referendum only a few days away every poll that is published is now eagerly awaited and treated like a mechanical instrument delivering a precise reading of voter intention. Each rise and fall in the reported level of support for the Yes and No camps is interpreted as an indication that opinion is shifting, that some intervention by a politician or company has nudged popular preference this way or that, that a ‘message’ is getting through, or not. The obsession with each poll has exacerbated since the Sunday Times survey on 7 September which put the Yes camp ahead for the first time in the campaign. As the real polling day draws near, the voracious needs of rolling news will increasingly treat each new poll as yet another reading of an increasingly feverish patient’s temperature.
But polls are not thermometers and stated intentions are not mercury, even if they are sometimes mercurial. We can see this when different polling organisations tell different stories about levels of support even with polls taken on the same day. This variation can be statistical smoothed out through indexes of Polls of Polls where rolling averages dilute one-off statistical aberrations. But such rolling polls are less sensitive to sudden and small changes in opinions and can underestimate last-minute shifts in views.
This is not the place to look in detail at the fragility of self-reports per se and the subtle ways surveys can be influenced by the biases of pollsters, methods, timing, location, question sequences and much else. On this there is a vast if sometimes complex academic literature but fortunately several good reviews.
I want instead to look here, briefly, at a feature of recent Scottish referendum polls which does not appear to have received much coverage: the very wide range of reported levels of ‘Don’t Knows’ (DKs) in polls. Unlike elves many Scots are saying neither ‘Yes’ or ‘No’. Traditionally, the levels of DKs are set aside as uninformative or merely distracting from the main message of the shout above the fold, with most newspaper poll headlines, particularly for local, parliamentary and European elections more or less ignoring them completely. In the Referendum debate there has been, until perhaps the last few weeks, some increased attention paid to what has been seen as high levels of DKs, particularly in the context of astonishingly high voter registration such that apathy cannot be appealed to as an explanation.
If we look at the polls for September 2014 we can see that, whatever the level of the lead reported by a given poll for either side, the level of DKs various hugely. This is particularly strange given all pollsters use the same clear question, specifically the same clear question which will be put in front of voters on Thursday: ‘Should Scotland be an independent country?’
The issues of ‘Don’t Knows’ is an important area because there are at least two discourses running through the cultural commentary of the Referendum. One of these is of a slow and deliberative ‘Big Conversation’ where a new form of popular political engagement is driving a renaissance in public debate and reflection; a discourse that could be reinforced by the idea of a large number of high commitment, low certainty voters. The other discourse is of a structural inevitability reflecting, depending on your point of view, years of neglect and complacency by political parties, one in which gender, geography, class and religion and a sense of economic survival, risk and identity have already driven people to make their minds up, be it either for ‘Yes’ or ‘No’. A good understanding of the level of ‘Don’t knows’ along the timeline is arguably important because whatever Scotland decides it will be important to know how we got there and how people engaged with the debate.
So what levels of DK have different pollsters been reporting?
Survation report 10-11% DKs while ICM report around 17% DKs. This is quite a range of variation in the percentage of voters who report being unsure, and higher than traditional party voting intentions. But it gets even stranger when we look at the recent TNS-BMRB figures which found an astonishing 23% of DKs in their sample. How do we explain this? Perhaps the more established polling organisations such as ICM and TNS-BRMB have more sophisticated methods than the relatively new Survation? Maybe these more experienced groups use more subtle methods to give respondents the chance to say “I don’t know”. This ‘expertise’ hypothesis does not remain compelling however when we look at the very well established YouGov polls where we find their level of DKs stands at strikingly low 6%.
What can explain this variation in DKs? What is it about these different polls or different pollsters that provide these widely different estimates of the percentage of the Scottish people who have not yet made up their minds? Are opinions fixed as YouGov would seem to suggest, or is the conversation still stimulating reflection, as TNS-BMRB imply?
At this stage with the data available it is almost impossible to give a definitive answer. But there is one hypothesis that might be worth considering. YouGove and Survation with low levels of DKs both used online polling for these recent Referendum polls, whereas TNS-BMRB with their huge 23% DK level used face-to-face interviewing. In the middle with their 17% DK level is ICM. How did they poll? That’s right – by telephone. Could it be the case that face-to-face interviewees are reluctant to give voice to their uncertain views whereas online there are fewer interpersonal anxieties to assess and navigate? Is it that over the telephone the interpersonal impact is somewhere in between? As trained as pollsters are in consistency and detachment in face-to-face interviews, it remains they are social interactions in a way that online polling is not. And yet it remains impossible to determine as there may well be other factors associated here with face-to-face interviews compared to online or telephone polls other than the level of interpersonal directness affecting the results. Face-to-face is usually in the street, unsolicited and people have other things to do. Saying “I Don’t Know” maybe helps bring the interview to an end more quickly. Or maybe it avoids the possibility of others nearby hearing. Or maybe people in face-to-face settings are sometimes uneasy with the prospect of having to give a reason for their preference. Or maybe it’s something else entirely, who knows?. Because in the end, like elves, we are not passive in the face of questions, but active agents in constructing complex exchanges, and do not give of our views in a vacuum.
“And it is also said,” answered Frodo: “Go not to the Elves for counsel, for they will say both no and yes.”
“Is it indeed?” laughed Gildor. “Elves seldom give unguarded advice, for advice is a dangerous gift, even from the wise to the wise, and all courses may run ill.”