For the entire duration of the election campaign to-date, Canadians have been marveling at the superb leadership ratings that Stephen Harper has been scoring on a daily basis on the Nanos Leadership Index. Typical numbers have shown Harper at around 100 or over, Layton at maybe 55, Ignatieff at 50, and Duceppe and May at around 10 or 15 points each. The index supposedly rates the leaders on trust, competence and vision for Canada.
The Nanos polls are at least mentioned on a daily basis on CTV and in the Globe and Mail, and are repeated ad infinitum on a near endless parade of blogs and other media, so may be presumed to have an influence on voter perceptions of the leaders. Nanos calls these ratings a "driver" of voter intentions, a forecast, if you will, of how people may be considering voting.
There's only one problem. The Nanos Leadership Index is inherently flawed and biased towards Stephen Harper.
The Polling Observatory, a website whose mission is "to report on and provide an analysis of the election polls and their media coverage", has issued a report that cites two problems with the index: One, a flaw in measurement methodology and the other a structural characteristic "that seems to overstate, in rhetorical terms, the magnitude of the differences in Canadians' evaluations of the leaders".
The Polling Observatory is funded by the Centre for the Study of Democratic Institutions at the University of British Columbia. You can go ahead and read what the Polling Observatory says on its site. But rather than just take their word for it, look at the methodology yourself. The problems are pretty obvious.
The methodology is: each person is asked
Which of the federal leaders would you best describe as:
* The most trustworthy
* The most competent
* Has the best vision for Canada's future
Every day, Nanos will report on a daily leadership index score. The leadership index score is a summation of the three leadership indicators (trust, competence, vision). For example, on March 15th (M15), Elizabeth May received a Leadership Index Score of 11.6 because 4.3% of Canadians identified Elizabeth May as the most trustworthy leader, 3.4% said she was the most competent and 3.9% said she had the best vision for Canada.
You know that 30-40% of people would be expected to name Harper in all three categories.So, his score will be the sum of his three percentages, or somewhere around 110 or 120. That leaves as little as 60% of the opinion to be split up between the four other leaders. As stated in the Polling Observatory, the result gives an overstated, exaggerated view, with the difference between each leader roughly tripled.
The overstatement is compounded by the context. The title "Leadership Index", as well as the terms, "trust", "competence" and "vision for Canada", imply some kind of insightful, carefully weighed valuations have been thought through by the respondents. In fact, these numbers represent little more than exaggerated representations of voting intention.
As of today, the Nanos leadership index has shown some life for the first time, as Layton jumped some 17 points and Harper fell by an equal number.
Further evidence of the Nanos Leadership Index's irrelevance, though, can be found in today's Abacus leadership poll, which shows Ignatieff and Harper neck and neck, with Layton jumping ahead, as well as a Harris Decima leadership poll that shows similar results to Abacus.
"Appearing to overstate" (by a huge factor) Harper's appeal to Canadians relative to other leaders, on such issues as trust, competence and vision for Canada, on a daily basis, for the full 40 days of the campaign, across all the media, has to be considered a pretty significant unfair advantage for Stephen Harper. What can be done about it I do not know..
Subscribe to:
Post Comments (Atom)
2 comments:
Hi - Nik here (the pollster),
Thanks for your post on the Nanos Index.
Looking at the ballot and leadership index provides a great snapshot into the state of affairs of the campaign in terms of the support for the parties and the likely outcome of the election.
Why the combination of the two? Because the ballot captures the proportional support of parties but cannot give a sense of the scope of the outcome in terms of seats in our first past the post system.
I was asked about our particular approach to creating the index – why the three elements and why combine them. Using the three elements allows us to go beyond who is up or not into why the numbers have moved. For example, beyond the movement as to who is up or who is down, we believe it’s important to understand why. Is a leader trusted more or less? Does his or her vision resonate with Canadians? Or is a leader just looking the most competent? Is it fair for Nanos to roll up the percentages – could it enhance the advantage of a front runner?
We gave that some thought when we were designing the index.
What I think is important is to factor the democratic system within which we operate and are doing our research. If we had a proportional system, dividing the score by three would likely be best because the scores and views of respondent voters would more likely correlate to seats.
However, the first past the post system where the winner takes all actually creates the same effect that we are replicating in the index. A leader that is ranked first even by a small margin can yield a disproportionate return politically in terms of seats. Even in our democratic system, it is mathematically possible to lose the popular vote and win and election in terms of seats. Look at the 1989 Provincial Election in Newfoundland and Labrador where the Clyde Wells Liberals lost the election in terms of popular vote to the Conservatives however (the Liberals received fewer votes than the Conservatives) but won a comfortable majority of the seats - http://en.wikipedia.org/wiki/Newfoundlandgeneralelection,_1989.
This is what I consider the distortion effect in our first past the post system.
Our assumption is that a leader with any sort of advantage in the first past the post system will have a disproportionate advantage in the outcome of the election in terms of seats – hence the purposeful approach to the calculation. Being rated second best in our system in most cases yields a disproportionately negative seat outcome not commensurate with the true opinion of Canadians.
Where the traditional ballot tracking captures the proportionate support of the parties, the index was intended to attempt to better capture the effects of our first past the post system.
So there you have it. Our approach to the ballot and the leadership index.
Cheers,
Nik
Thanks for your comment.
If I understand the explanation properly, it is that you are simulating the "first past the post" effect of our electoral system, within the actual mechanics of the Leadership Index poll. I have two problems with that. First, it is not written anywhere that the poll results are exaggerated, so most readers are likely to think that Stephen Harper is considered the best leader by a huge margin, which is simply not the case, as seen in other polls. Secondly, I think the technique as you explained it is gonzo. You say you want to convey a sense of 'first past the post' in a leadership index. Would that be something like printing a newspaper story in white letters to simulate the 'white noise' of the campaign? I acknowledge that party insiders might want to have a tool like your leadership index at their disposal, if it has some scientific validity. But I still think it gives a heavy unfair advantage to a front running candidate, especially one with a committed base like Stephen Harper.
Post a Comment