Findings of the annual Arab Youth Survey were released in Dubai yesterday amid the customary fanfare. Once again, the survey asked young Arabs which country they would like to live in if they had a choice and, once again, the United Arab Emirates topped the list. Indeed, according to the survey, the UAE is sprinting ahead. It has increased its popularity by 13 percentage points since last year and is now more than twice as popular as the United States.

The survey's findings – covering a range of issues – are presented on a flashy website with colourful infographics but, as in the past, the underlying research seems to lack statistical rigour.

One question asked was: "What do you believe is the biggest obstacle facing the Middle East?" This clearly invites a single answer from respondents – in which case all the replies should add up to 100%, no more, no less.

In the survey, however, the answers add up to 275%, indicating that respondents were allowed to identify more than one obstacle as "the biggest". 

There's similar imprecision in the question about where young Arabs would like to live. While it's true that the findings show the UAE as heading a list of popular countries, it's not true to claim – as the survey's website does – that the UAE is "the country in which most young Arabs would like to live". According to the actual figures, a majority (65%) would like to live somewhere else. Those choosing the UAE were a minority (35%). 

These might seem like fairly minor quibbles but they are a sign of sloppy methodology, and a reason why journalists and academics should treat the survey with caution.

A year ago, when the 2016 survey was published, I wrote two blog posts about flaws in its sampling methods (here and here).

Surveys were carried out in 16 Arab countries, with the following sample sizes:

Analysis of the findings showed that data from these national samples had been aggregated without taking account of population differences. Thus, views from the less populous countries (such as the UAE) were over-represented while those from the more populous countries (such as Egypt) were under-represented. Together, the six GCC countries accounted for 40% of the overall sample.

A note about methodology on the survey's website suggests the same flawed sampling methods have been used again this year. There is no mention of any adjustments for population size. The 2017 survey also makes some comparisons with findings from 2016 – comparisons that would not be valid if the methodology had changed.

Visitors to the website are invited to download a "White Paper" which is said to provide "detailed analysis" of the findings of the 2017 survey. Attempts to download it are greeted with requests for an email address and a reply comes back saying someone will be in touch.