Only three of the 22 Arab League countries – Comoros, Mauritania and Kuwait – are ranked in the top half of this year's World Press Freedom Index issued by Reporters Without Borders. As a very broad-brush picture of where Arab media stand in comparison to other parts of the world, this is probably fair. Looking more closely at the rankings of specific Arab countries, however – comparing them against each other – I began to feel that something has gone awry.

Here are the rankings, with the change in position from last year shown in brackets.

51 Comoros (-6)
67 Mauritania (0)
77 Kuwait (+1)
101 Lebanon (-8)
110 Qatar (+4)
114 UAE (-2)
125 Algeria (-3)
131 Libya (+23)
134 Jordan (-6)
136 Morocco (+2)
138 Tunisia (-4)
141 Oman (-24)
146 Palestine (+7)
150 Iraq (+2)
158 Egypt (+8)
163 Saudi Arabia (-5)
165 Bahrain (+8)
167 Djibouti (-8)
169 Yemen (+2)
170 Sudan (0)
175 Somalia (-11)
176 Syria (0)

In principle, league tables of this kind are a good idea but if they are to be useful the methodology has to be right and, in the case of the World Press Freedom Index, I suspect that it isn't. Or maybe the index needs a different name.

For a start, are Comoros, Mauritania and Kuwait really so far ahead of the others (there's a gap of 24 places between Kuwait and the next Arab country, Lebanon)? And does Oman really enjoy more press freedom than Egypt? I can't imagine anyone, with the possible exception of the Sultan, saying that it does.

Devising an objective measure of media freedom is no easy task. Reporters Without Borders uses six indicators: media pluralism, media independence, the working environment and self-censorship, the legislative framework, transparency, and infrastructure. The questionnaire on which its survey is based also asks plenty of sensible questions.

I think the main problem with this is that it gives too much weight to the institutional and legal frameworks and too little to questions of how well (or not) journalists serve the public: what use are they making of the freedoms that they do have – are they pushing at the boundaries – and to what extent is there anything that could be considered as vibrant public discourse?

In terms of pluralism, the existence of privately-owned media is treated as a plus point – though that can be deceptive if the result, as in Mauritania, is a burgeoning of sensationalist publications or, as in some of the Gulf states, the "private" hands are actually people close to the government.

Similarly, low numbers of arrested journalists can be a healthy or unhealthy sign. It may be because the government allows them freedom to do their job but equally it can mean that the media has become utterly tame.

One useful test is to ask what criticisms can be made of senior officials, and especially the head of state, without fear. On that basis, Egypt deserves a higher ranking than it has got, sandwiched in 158th place between Iraq and Saudi Arabia.

Writing for his Chronikler blog, Khaled Diab argues that the index's methodology results in some countries with a more critical media culture scoring more poorly than those which are less critical:

Although no Kuwaiti journalists were arrested last year, the profession as a whole tends to self-censor to stay within the carefully delineated “red lines”, while attempts by Mubarak, the army and the Muslim Brotherhood to impose restrictions in Egypt through intimidations and periodic crackdowns, have been met with defiance and open rebellion by much of the independent media.

“When Kuwait comes ahead of Egypt, this confounds me,” Hisham Kassem, a veteran Egyptian publisher and democracy advocate admitted to me amid the bare concrete and dust in the future offices of his ambitious new media project in Cairo a few months ago. “If rulers in the Gulf were exposed to the same level of attacks that Mubarak was in his last years, then heads would roll.”