Complicated polling questions generate muddled results…. Damn Lies and Statistics
Read more about: Uncategorized
This post is written by Eoin O’Malley, political scientist at DCU, and appeared on 21 February on www.politicalreform.ie. It provides an excellent guide to polling questions in general as well as a valuable critique of why the findings of the recently published poll on behalf of the Pro-Life campaign, which asked what action people want taken on abortion legislation in Ireland, should be approached with caution.
A poll released today by the Pro-Life Campaign seeks to ‘challenge the notion that there is broad middle ground support for abortion in Ireland.’ This polls claims to show that two-thirds of Irish people want ‘legal protection of the unborn’ and suggests that this means Irish people are against legalised abortions. This should surprise some as it follows on from a IpsosMRBI poll in the Irish Times recently which showed a substantial majority in favour of legalised abortions in a variety of circumstances.
So what’s happening? Does this show that you can get a poll to say anything you want as the clip from Yes, Prime Minister suggests? Or is the polling methodology of companies so flawed that we get results that can fluctuate wildly even when the underlying opinions in society are reasonably stable?
Well it’s not the latter. Opinion polls results on voting, though sometimes appear volatile (especially when there’s no election on the horizon and people aren’t as focussed on the question) are remarkably similar when taken at similar times. Polling companies have improved their methodologies, and while there are ongoing debates about issues such as weighting for past vote and likelihood to vote, the polling companies in question, Millward Brown and IpsosMRBI, are reputable and conform to high standards.
What is probably at issue here is the wording of the question, the interpretation of the results and the ‘context’ of the question. Sometimes we might see an advocacy group ask a number of questions and pick just those that suit its position, discarding any that don’t. In general one should be suspicious of an advocacy group-sponsored polls.
The questions in this case were problematic. Abortion is a complex issue, and asking one or two complicated questions doesn’t help add clarity. People think differently about abortion in different circumstances, and so pro-choice advocates make much of questions in which as much as 90 per cent of US respondents say they favour that abortion should be legal. In fact the support is lower when circumstances are specified.
So it makes sense to clearly specify the circumstances in a number of clear, concise questions. This is what the Irish Times poll did – though the health question was probably vaguer than it should have been and was, I think, misinterpreted by some to mean suicide.
What were the Pro-Life campaign’s questions?
Q1. In current medical practice in Ireland, the doctor treats the expectant mother and her baby as two patients and does his/her best to safeguard both in a crisis situation. Do you consider that this practice should be protected and safeguarded by law or not?
66% answered YES, 15% NO, 19% No Opinion.
There are a number of problems with this question. It’s pretty long, which may not be a major problem, though it’s usually best avoid long questions, especially abstract ones. One might also object to the use of the word ‘baby’ instead of foetus, but again, I’d let them pass on this. A more serious problem is that it describes a situation which it says is the status quo position in Ireland, though many would take issue with that. So if asked ‘are you in favour of the current social welfare system which protects the most vulnerable in society?’, what is one’s answer to be. Yes, but it doesn’t. That’s not an option.
The question is framed to offer everything desirable, and ask the respondent would you like to retain it? ‘Yes,of course, but…’ It’s perhaps surprising that only 66% answered yes. Another problem is that it oversimplifies the situation. It doesn’t ask difficult choices of the respondent. Motherhood and apple pie? Of course.
But you can’t always have both. The rights of the two come into conflict and whose rights take precedence is precisely the issue Irish society has to grapple with.
The second question is arguably worse.
Q2. Are you in favour of, or opposed to constitutional protection for the unborn that prohibits abortion but allows the continuation of the existing practice of intervention to save a mother’s life in accordance with Irish medical ethics?
Result: 63% answered YES, 19% NO, 18% No Opinion.
It has the same problems we saw in the first question, but the added problem that it asks questions on what are two dimensions. So the first half asks about the constitutional prohibition on abortion (which since X case is not really in place) and the desire to save a mother’s life (which is also moot).Which part are the respondents saying yes to? The two issues should be separated out.
The Pro-Life campaign say that they were providing ‘context’ that was lacking in the Irish Times questions. They say that ‘important ethical distinctions are clarified for the benefit of respondents’. But one might say that the context was misleading, and the ethical distinctions were muddied.
A further problem is with the interpretation. The Pro-Life campaigns press release, which was heavily drawn on in reports in the Irish Independent and Irish Times, a ‘sizeable majority support legal protection of the unborn’. This is true, but it doesn’t, as they suggest, contradict the findings of the Irish Times poll. Favouring the legal protection of the unborn doesn’t mean one is against abortion – it just suggests that Irish people want rules governing abortion. The Irish Times poll asked more questions, more simply and more directly on the nuances in the debate. It is one I’d use if I wanted to know what Irish people think about this issue.
Connect with Eoin on Twitter @AnMailleach