OSR’s statement on the Gambling Survey for Great Britain

Many of the concerns raised with us were framed as ‘breaches’ of the Code. Generally, this is not how we consider issues relating to the production of statistics and so we considered that it would be helpful for us to set out how we regulate against the Code.

The Code sets the standards that producers of official statistics should commit to. This means that all official statistics should adhere to the principles of the Code, although in practice the extent to which the statistics adhere will vary.

When statistics are awarded the status of ‘Accredited Official Statistics’ this indicates that we have determined that the statistics strongly comply with the Code. Once this status has been awarded, the statistics are required to continue to adhere to the Code. Where this is no longer possible, we may suspend or remove the accreditation. You can find more about this process on our website.

In the compliance review of the GSGB, which are official statistics, we have sought to identify areas in which the Gambling Commission could strengthen its adherence to the Code. We do not approach these reviews from the perspective of making a binary decision on whether the statistics are compliant or non-compliant with the Code. This means that the GSGB will be compliant in many areas of the Code despite having areas for improvement.

Where specific principles of the Code are not being adhered to in full, we do not necessarily interpret these as ‘breaches’ of the Code; instead, they indicate areas that the organisation should improve in order to strengthen adherence to the Code.

Some complainants expressed confusion as to whether the Health Survey for England (HSE) which has the status of ‘accredited official statistics’ (called National Statistics in the Statistics and Registration Service Act 2007) should be considered more reliable than the GSGB which are official statistics.

The accredited official statistics status of the HSE reflects that the statistics have gone through an independent assessment by OSR and have been confirmed to comply with the standards of trustworthiness, quality and value in the Code of Practice for Statistics (the Code). The GSGB are official statistics that have not been independently assessed.

It is our view that there are benefits and limitations for both the HSE and GSGB and that the decision on which statistics to use should be based on user need and not solely on the accreditation status.

We expect all official statistics producers to adhere to the Code and so the status of ‘official statistics’ does not necessarily indicate that the statistics are of lower quality or are less reliable than ‘accredited official statistics’.

Complainants were concerned that the impact of social desirability was not a valid argument for explaining the difference in the rates between the GSGB and HSE.

The estimates of gambling are higher in the GSGB which uses a push-to-web survey approach, than those measured in the HSE which uses a face-to-face interview approach.

One theory that has been put forward to explain the different rates, at least in part, is that if there is someone else present (the interviewer) then the person being interviewed will under-report their gambling in order to meet standards of social desirability. Under this theory, the HSE will record lower levels of gambling than the GSGB. Some evidence in favour of this theory was provided in the Gambling participation and the prevalence of problem gambling survey: Pilot stage Methodology Review Report – Ashford et al 2022, however an alternative study conducted by Sturgis and Kuha (2022) concluded that the online design was resulting in non-response bias for non-gamblers which is the cause of the higher rates of gambling in the GSGB.

The 2024 GSGB technical report discusses the potential impact of social desirability. It is our view that the passage from the technical report is clear that the research is not conclusive in this area and states that Sturgis and Kuha did not find a statistically significant difference. The passage also does not use definitive language when talking about social desirability and also notes that more research should be done in this space.

Professor Sturgis’ independent review of the GSGB was clear that the reasons for the different rates between the two surveys require further exploration as covered by his second recommendation:

“Recommendation 2: the Gambling Commission should undertake additional research to better understand the role of socially desirable responding as the driver of the difference in gambling estimates between in-person and self-completion surveys.”

Within its Survey Improvements plan, the Gambling Commission explains that it has commissioned an external organisation to undertake research on this issue.

Concerns were also raised with us about whether the Gambling Commission had purposefully withheld additional information on the role of social desirability from Professor Sturgis by not including reference to it in the technical report. This concern stems from correspondence between Professor Sturgis and the Gambling Commission on the 8^th January 2024 that was released through a Freedom of Information request in March 2024. Within this correspondence, Professor Sturgis states that he had reanalysed data from HSE 2016 and 2018 and found that there were “significant and quite large effects for presence of other people during the interview”. However, Professor Sturgis notes that interpretation of this is complicated and that “it is having a spouse per se rather than having one present in the interview that seems to be key”. Professor Sturgis then goes on to say that “I don’t think I’ll include this in my report but happy to share it with you separately.”

This analysis was not included in the final report produced by Professor Sturgis and has not been published to date as far as we are aware. As this is an independent review, the final decision on what to publish lay with Professor Sturgis. We have not seen any evidence that the Gambling Commission prevented the publication of this analysis. Without the analysis being made available in the public domain by Professor Sturgis, we would not expect the Gambling Commission to reference it in the technical report.

Complainants raised concerns about whether users are made aware of the potential risk that the GSGB may overestimate the rates of gambling and gambling harm.

As stated in the previous section, the estimates of gambling are higher in GSGB than those measured in the HSE. It is not for OSR to determine the reasons for this; however, this issue was explored by Professor Sturgis in his review. Professor Sturgis’ report is clear that more research is needed in this space before a strong conclusion can be drawn on whether the GSGB does over-estimate the prevalence of gambling. He ultimately concluded the following:

“Until there is a better understanding of the errors affecting the new survey’s estimates of the prevalence of gambling and gambling harm, policy-makers must treat them with due caution, being mindful to the fact there is a non-negligible risk that they substantially over-state the true level of gambling and gambling harm in the population.”

It is essential that this conclusion should be made clear to users within the publications. At the top of the statistical bulletin there is a sentence stating “All surveys have strengths and limitations and we have outlined the strengths and limitations of our approach in the data analysis and reporting section of the technical report.”, however the statistical bulletin does not explicitly state that the GSGB could be an overestimation.

Within the technical report itself, there is a section entitled “Factors which may mean PGSI estimates are over-estimated within bespoke gambling studies” which discusses response bias. This section includes statements such as “The GSGB likely suffers from this selection bias compared with the health survey series.” and “Despite best efforts to reduce this possibility, it is likely that some selection bias remains and so that rates of past-year gambling participation and PGSI scores are higher in the GSGB compared with the health survey series”.

Within the Executive Summary of the Annual GSGB report, there is a section which states: “Whilst further work is undertaken to explore this, there is a risk that estimates presented below overstate the true level of adverse gambling consequences within the population.”

We consider that the technical report includes clear and impartial information for users on the potential risk that the GSGB overestimates the rates of gambling. However, we have judged that more should be done to highlight these limitations within the statistical bulletin themselves.

This issue is addressed through Recommendation 3 of our review: To support appropriate interpretation and use of the GSGB data, the Gambling Commission should:

• clearly communicate to users within the statistical releases the potential biases that may affect the GSGB estimates, the possible impact of these, and Professor Sturgis’ conclusion regarding the risk that the statistics potentially overestimate some gambling behaviours
• explain what the statistics can and can’t be used for, and why
• ensure that guidance on how to use the statistics is easy to find and tailored to different users, such as the media.

Complainants raised concerns that people who gamble might be more likely to respond to a survey on gambling. This means that the sample of people responding to the survey could be biased in favour of gambling activity which would then lead to higher gambling rates. Complainants were concerned that this possibility had not been made clear to users.

This issue was also explored by Professor Sturgis in his review and it was determined that more work was required to understand this issue. This is reflected in Professor Sturgis’ recommendation:

“Recommendation 1: the Commission should conduct research to better understand the relationship between survey topic and the propensity of gamblers to respond to survey invitations.”

The GSGB technical report discusses the issue of non-response bias. However, this section primarily discusses the possibility that individuals who gamble will be less likely, rather than more likely, to respond to a survey that asks questions about gambling harms. Within the technical report there is a reference to the counter argument, although this is relatively limited:

“As the GSGB is ‘gambling focused’, it is possible that the survey disproportionately attracts those who gamble, so that this group may be over-represented.”

We consider that the Gambling Commission could do more to help support users’ understanding of both possibilities by discussing them together in the same section. This is addressed in the review as follows: “We consider that it would benefit users to bring these two points together clearly in one section to provide a balanced picture for users.”

Several complainants raised concerns that the figures produced by the GSGB may not align with other sources from the gambling industry as well as estimates of suicidal behaviour from the Adult Psychiatric Morbidity Survey.

We agree that information is not provided in the statistical bulletin, or the technical report of the data being compared to other sources for verification purposes. We also agree that further work is necessary to explore and explain the level of coherence with other data sources.

Work is ongoing by the Gambling Commission to determine how the GSGB compares to other sources of data that provide information on gambling. We consider that the Gambling Commission should understand and explain the consistency and comparability of the GSGB with other related statistics and be transparent about the potential reasons for these differences.

This issue is addressed through Recommendation 4 of our review: To support user understanding of the GSGB’s role in the broader gambling data landscape, the Gambling Commission should do more to investigate the coherence and comparability of GSGB statistics with other relevant data, such as from the Health Survey for England and the Adult Psychiatric Morbidity Survey that will be published later in 2025, and communicate these findings to users.

Currently, the Gambling Commission advises against using the GSGB results to aggregate up to a population prevalence of harmful gambling until more work is done. Complainants were concerned that this is a significant limitation of the GSGB and that this is not explained to users in the statistical bulletin or technical report.

The Gambling Commission has a separate document entitled Guidance on using statistics from the GSGB which provides clear information and guidance on this issue. The guidance states that the GSGB should not be used to gross up the prevalence of problem gambling or the consequences of gambling to the whole population (until further work is completed).

Whilst a link to this guidance is provided in the GSGB Power BI dashboard, the same information is not provided within the bulletin or the technical report and the guidance document is not linked in either place either, making it potentially difficult for users to navigate to.

This issue is addressed through Recommendation 3 of our review: To support appropriate interpretation and use of the GSGB data, the Gambling Commission should:

• clearly communicate to users within the statistical releases the potential biases that may affect the GSGB estimates, the possible impact of these, and Professor Sturgis’ conclusion regarding the risk that the statistics potentially overestimate some gambling behaviours
• explain what the statistics can and can’t be used for, and why
• ensure that guidance on how to use the statistics is easy to find and tailored to different users, such as the media.

Several of the complainants have raised concerns about the reliability of the GSGB and whether it should continue to be published.

The GSGB has gone through extensive development to inform the current survey. Whilst there are still some areas that require further exploration and improvement, we have not seen evidence to suggest that the statistics are not fit for publication. Currently, the GSGB provides the most regular estimates of gambling when compared to other surveys of this nature undertaken by official statistics producers.

We consider that all statistics have strengths and limitations that are important for users to consider but that the existence of limitations does not necessarily indicate that the statistics are of poor quality. By communicating the limitations of the GSGB, the Gambling Commission are acting in line with the Code.

Some complainants raised concerns with us about the ordering of the questions in the GSGB and whether this could result in ‘priming’, by which we mean whether the questions in the survey are influencing or biases subsequent answers.

This issue was not considered as part of Professor Sturgis’ report or our compliance review of the GSGB. The Gambling Commission provides information on the development process of the GSGB as well as information on the potential primacy effect with the long-list approach. However, we have not found information on how the ordering effects of the questions was considered within the development of the survey.

This statement does not seek to determine whether this is an issue present in the GSGB or to provide recommendations for the Gambling Commission, however we note that this remains an open question for users that the Gambling Commission may wish to consider providing more information on.