ASDA’A Burson-Marsteller, based in Dubai, is one of the Middle East's most prominent PR firms. It boasts on its website:

"ASDA’A Burson-Marsteller has played a pivotal role in the development of the Middle East public relations industry. Through its longstanding commitment to providing innovative, results-driven communications solutions, ASDA’A Burson-Marsteller has emerged as the region’s premier public relations consultancy, with a growing portfolio of blue-chip corporate and government clients."

ASDA’A Burson-Marsteller has won numerous awards. Last year it was named Middle East Consultancy of the Year, and the citation made particular mention of its work on the Arab Youth Survey which the firm has organised every year since 2008.

A couple of weeks ago ASDA’A Burson-Marsteller published its 2016 version of the survey and the findings were reported without question by news organisations around the world. The Washington Post alone had three articles about it and the ruler of Dubai, Sheikh Mohammed bin Rashid Al Maktoum, gave the survey his personal endorsement in a series of posts on Twitter.

Unfortunately, though, close examination of the data points to an elementary flaw in the survey's methodology which means it does not give an accurate region-wide picture of the views of Arab youth.

I began to have serious doubts about the findings when, on page 33 of the survey report, ASDA’A Burson-Marsteller said "most" young Arabs would like to live in the UAE and "most" would like their own country to resemble the UAE. There was no evidence in the survey to justify the claim; it was based on misinterpreting the answers to a couple of questions.

Writing about that in a blog post on Monday, I also voiced suspicions that the survey had committed a fatal error by not taking into account the enormous population differences among Arab countries. One way of ascertaining whether that happened would be to check the raw data – but the raw data doesn't seem to have been released. However, looking in more detail at the survey's published findings, I found another way to test the hypothesis.

A series of national polls

Polling for the survey was done by Penn Schoen Berland (PSB), whose Middle East HQ is also in Dubai. Both PSB and ASDA’A Burson-Marsteller belong to the international PR and advertising group, WPP. 

PSB's role, basically, was to conduct separate but identical polls of young people in 16 Arab countries. The sample size in most countries was 200, though it ranged from 150 in Palestine to 300 in Egypt, Saudi Arabia and the UAE.

Viewing the results separately, as a collection of individual national surveys, there is no reason to question their validity. The difficulty lies in trying to combine them to give an overall picture of what "Arab youth" across the region thinks.

Other polls in the Middle East have repeatedly shown wide differences in public opinion among different Arab countries – they are not consistent across the region. Thus, if you want to get an accurate overall picture, more weight should be given to opinions from the larger countries (such as Egypt) than the tiny ones (such as the UAE).

But instead of doing that the Arab Youth Survey seems to have added all the national samples together to create a large but flawed "regional" sample of 3,500 interviewees which grossly over-represents views from the smallest countries and under-represents those from the largest.

To explore this possibility further, let's look at an example from the survey. A question on page 17 of the report asks interviewees if they think "religion plays too big of a role in the Middle East" and says that, overall, 52% of Arab youth agree. The test is to see if this figure can be replicated using bad methodology.
  

  
Helpfully, the survey also gives a sub-regional breakdown for this question: in the GCC countries 61% agree, in the Levant and Yemen 44% agree, and in North Africa 47% agree. This information, together with a bit of arithmetic, allows us to see how the 52% figure could be arrived at. 
 

First, working on the assumption that the figures in the survey made no allowance for population differences, let's add together the stated sample sizes from each country to create sub-regional samples:
  

Aggregating the country samples in this way is not statistically valid but, for the purposes of the exercise, let's assume it is what happened. 

By adopting these sample sizes, and knowing the results in percentages, we can then work out approximately how many interviewees agreed with the statement:

GCC countries: 61% of 1,400 people = 854 who agreed
Levant/Yemen: 44% of 1,000 people = 440 who agreed
North Africa: 47% of 1,100 people = 517 who agreed

(Note: percentages may have been rounded up or down, so these are not necessarily the actual figures but they should be very close.)

Add these figures together and we have a grand total of 1,811 interviewees who agreed with the statement. If the survey followed invalid methodology by adding national samples together to create an artificial "regional" sample of 3,500, these 1,811 interviewees would represent 52% of the total. And – hey presto – 52% is the figure given by ASDA’A Burson-Marsteller in its report.

There are eight other questions in the survey where the same exercise can be repeated. In five of these the results come within one percentage point of the overall regional figures quoted in the survey. In two of them they come within two percentage points and in one they come within three percentage points.