The Luntz Effect
Irish media and politics are still recovering from the “Luntz” effect and the first use by a media outlet of a political focus group. Frank Luntz conducted the focus group of “undecided” voters from the Dublin/Leinster region for RTÉ’s The Week in Politics programme. The premise of the program was that there is approximately 18 per cent of the electorate who fit into the “undecided” category and who could decide the outcome of the election. Luntz’s forceful style forced and brooked no silence from the group and forced the participants to adopt strong opinions immediately. This was an unfortunate approach as this may have produced suitable content for television, it swung the pendulum away from any kind of accurate refection of the participants true beliefs or feelings. Is this the beginning of reality TV, political style, in Ireland?
Luntz himself described the approach where:
focus group research is a direct, sensitive, and interactive method of assessing public opinion, accomplishing what telephone studies cannot. It approaches attitudes and priorities tangentially by allowing respondents to talk freely and to choose descriptive categories significant to them (rather than to the pollster, or even to the client).
And he hints at their importance and usefulness when he states:
Focus groups may have a low-tech feel, but more often than not, it is a series of focus groups rather than traditional telephone polling data that snatches victory from the jaws of defeat.
Focus groups are not a new technique, having been in use since the World War Two when Robert Merton used the technique to test responses to a series of radio broadcasts aimed at supporting the morale on the home front during the war. The method for actual getting the data was interesting as it focused on the moment, the feeling, the instant rather than on any reflection of the speeches played for the participants. The participants used dial operated hand-held controllers which relayed their immediate reaction to the television speech clip back to a computer which analyses and reported these responses in real time. This analysis allowed the audience gut reactions to be inspected, second by second, which enables for the determination of which words, phrases or gestures worked with the participants. This process has been called “computerised hoku,” by several American commentators (or more notably by Bill Clinton as “a bunch of bull“) as it is a shallow manner for capturing any kind of useful qualitative data.
Focus groups as a technique have limitations; unfortunately RTÉ seemed to neglect a proper disclaimer discussing these limitations. Even Luntz himself states that their…
…findings cannot be projected onto the entire population. The results are dependent upon the interaction between the respondents and the moderator, and unprofessional moderating can lead to inaccurate conclusions. [...] A single dominant voice can cripple open, honest discussion by intimidating the other participants.
RedC did try and select a group of participants which were as representative as possible; however any focus group is not a projective random sample as would be found in an opinion poll. RedC are loath to point out a major issue with their telephone-based approach for selection, it works for landlines but not for mobiles. Recent US research in political polling has highlighted the issues and differences in the “mobile only” category of voter which disproportionally affects the younger demographics and those who would normally constitute a larger selection of the “undecided” voter in a modern Ireland.
It is not possible to gauge the reactions of a larger population based on answers from a focus group. A focus group reacting to a set of speeches does not mean that the 18% of undecided voters in the Dublin/Leinster region feel or would react in the same or a similar manner. Another important issue with focus groups is the “group dynamic” as discussed by Mark Blumenthal where he says:
In a focus group, participants are often influenced or cowed by the opinions of others in the group. If one dominant personality loudly stakes out a position, others tend to hide or modify their contradictory views. To control this dynamic, we typically ask focus group participants to write down initial impressions at intervals throughout the groups to root them in their initial reactions. Moreover, the role of the moderator is critical to countering the loudest and most vocal while encouraging the more timid participants to share their true feelings.
Luntz’s reporting of the facts is open to question as the National Council of Public Polls have previously censured him “for allegedly mischaracterizing on MSNBC the results of focus groups he conducted during the  Republican Convention“. He pointed out openly at the start of the programme that he was American and had little or no knowledge of the Irish political system and also cast himself as an independent, neutral observer. This is important as successful focus groups require unbiased facilitators who are not only independent but are seen to act in that manner. Watching the episode again, I’m not sure if Luntz conveyed himself as an independent facilitator successfully. On the other hand, maybe Gerry O’Quigley is correct when he says:
Those who would embrace the techniques of political marketing do not suggest that pollsters should determine the policy. Much of what political communications experts do is to pay attention to how the agenda is set and issues are framed for debate. Indeed a central battle in politics is over the use of language or framing. Luntz would be happy with the description of what he does as “political reality construction” even though he has been heavily criticised for the way he does it.
I can offer a more entertaining and alternative suggestion for RTÉ which is based on the work of Aaron Koblin and his Sheep Market experiment on Amazon’s Mechanical Turk (this is a system which gets people to do small human intelligence tasks for a little sum of money, cents normally). The Sheep Market asked people to draw sheep and upload them, approximately 10,000 sheep were accepted, arranged on his site and then offered for sale. It has already been seen as an exemplar system with Jeff Barr stating:
Today it is sheep, but it could just as easily be choices of color combinations for car interiors, evaluation of some logos for your business, selection of most important features when choosing a vacation spot, and so forth.
To which I’ll add the simple idea of distributed focus groups who have been selected based on questionnaires, interviews and who could be vetted intensely to ensure they represented a cross spectrum of the Irish electorate. These participants could be paid using a system like Amazon’s Mechanical Turk for watching and ranking political videos or any product video for that matter. Even as the most fanatical of anoraks now start to gibber at the potential for this application and say that such a system is easily implementable today, especially given Amazon’s API and either private videos on YouTube or Google Video. However for a “reality check” to this suggestion and to focus groups in general, I’d like to point readers to the antics of the fictional character Cyrus Rutherford Ogle and his Piper system from the novel Interface by Neal Stephenson.
The artificial nature of focus groups is a major limiting factor as what people say in these groups simply don’t reflect the trends in the real world. Negative advertisements in US politics are a noted example where participants often express genuine antipathy and question or even reject the information they present as being false. The previous US Presidential campaign shows how this type of advertising worked in communicating negative information effectively and as I’ve noted previously, it was also put to more recent use in the US Senate and House campaigns. Taking a note again from Mark Blumenthal who discusses the use of focus groups as reality TV and states:
Unlike the traditional focus group, the networks put people on a soundstage in a brightly lit studio, where the participants surely know they are on live television. I am not aware of any formal research on TV focus groups, but it seems that if peer pressure from ten strangers leads a participant to hide or alter an opinion, what is the effect speaking freely to several million?
This should raise our concerns and ask the question whether RTÉ is using the equivalent of reality TV to again avoid opening a real national debate on the forthcoming general election or whether this is a valid format and mechanism for furthering the political debate in Ireland.
- Focus Groups as a Tool for Policy Analysis, Kahan, J. P., Analyses of Social Issues & Public Policy, Dec 2001, Vol. 1 Issue 1, p129, 18p
- Using and analysing focus groups: limitations and possibilities, Smithson, J., International Journal of Social Research Methodology, Apr-Jun 2000, Vol. 3 Issue 2