I have been reading ‘For Adults Only? Underage access to online porn‘, ATVOD’s survey. Having been involved in some discussions about surveys and the data behind them, I was interested to see what raw data was available for ATVOD’s survey and how it stood up to scrutiny. The UK government’s ‘Good practice in the design and presentation of consumer survey evidence in merger inquiries’, the nearest thing I could find to the government line of what makes an acceptable survey, states the following principles. I think it is reasonable for a survey commissioned for law and policy making to at least comply to these standards. Don’t you?
Summary of principles
2.15 In order to be given the greatest evidential weight, consumer survey results should:
- test clearly-stated hypotheses
- be representative of the relevant consumer population
- deploy sound social research methods
- be reported in full, with supporting data available to allow key results to be replicated and tested.
There is not a lot of statistical information in the survey and what is there seems highly selective. It is not presented in a way that allows any meaningful comparison of figures, nor is the supporting data provided. You are simply presented with a series of ‘facts’ but without the means to check them.
It is reported that Pete Johnson, Chief Executive of The Authority for Television On Demand (ATVOD), has been questioned closely on this and it appears that they do not hold any such data. It turns out that they commissioned a third party, Nielsons, to conduct the survey.
Of course, when one pays for a survey it is important that the project is presented without bias or prejudice but it seems we are provided with no means of checking for ‘confirmation bias.’ It is vital to know what questions and the way in which they were asked. For example, there is a big difference in asking “Would you like to live as long as possible?” and “Would you want to be kept alive regardless of the quality of your life?”. Again, statistics can be interpreted in many ways. For example, one could interpret the fact that 30% of fatal RTA’s involve a drunk driver as meaning it is over twice as dangerous to drive when sober. This is clearly an erroneous conclusion but demonstrates how figures can prove what you want.
The report admits a key failing in that it only records results from PC’s but excludes phones and tablets. Admittedly, their inclusion might well support their case but that is merely an assumption. There are many key indicators like ‘adult material’ that are not quantified. It sounds like only doing half the job to me and it brings into question the validity of the report as one cannot help wondering what other basic omissions were made.
The up-shoot is we only have their word that it is worth the paper it is written on. If we want to check the figures we have to pay Nielsons for our own report. This presumably means law and policy are being made on this insubstantial information as I cannot imagine that the legislators have access to the background data either. This seems a very unsatisfactory state of affairs to put it mildly.
It is not really clear how this UK law “protects children” when “Only one of the 1,266 adult websites identified by Nielsen Netview as having been visited from the UK in December 2013 was a service regulated in the UK”. This sounds like securing your house by firmly inserting the bath plug whilst leaving all the doors and windows open! It also reports that “Nearly one in four (23 per cent) of those in UK who went online using a PC or laptop during December 2013 visited an adult website”, so it seems to be a rather popular hobby and given the mood of the press and TV, it seems a lot of them are in the media 😉 The campaign against this law has had a gratifying amount of public and media support, even though most think it relates only to kink and porn. In fact, it is a much bigger issue that attempts to licence the hugely lucrative adult sector and the growing video on demand market. Their net has even been spread as far as Sainsburys. Here is a list of the fallen (plus a few that got away) aka ATVOD Determinations. As you’ll see, it seems that they are mainly targeting one man bands in order to create precedents to clarify their “dynamically interpreted” rules before moving on to bigger fish and a slice of VoD industry.