Fail: new study claims online video massively overstated

A study on behalf of Nielsen’s Council for Research Excellence has claimed that the consumption of online video is massively overstated, and accounts for only a fraction of all video viewed.

The Video Consumer Mapping Study found that 66% of all video is still consumed via live television, and that online video only accounts for 1% of all video viewed. The study claims that consumers tend to overstate how much time they spend on because “it seems new and cool,” where as watching TV doesn’t have as much social cache so people tend to under-report how many hours they watch.

The results come at the same time comScore video data for April shows a 16% increase in video watched online vs the month before. comScore found that 16,785,432,000 online videos were watched in the United States from 151,652,000 unique users.

Whilst it would be easy to suggest an ulterior motive to the results, particularly given a strong representation of media companies on the Council for Research Excellence, the study has a far more substantive range of verifiable flaws.

The initial consideration is how the data was gathered to begin with. The study used “American media consumers, primarily former Nielsen TV People Meter panelists.” Former is the key word there, because TV panel data is gathered through “people meters,” electronic devices that record viewing habits. In theory, it’s difficult to understate TV viewing when there’s an electronic device gathering data on what you are watching; and yet this study specifically picked people who didn’t have the devices.

Then there’s the problem of sample size and location. From the study:

[the study included ] a final sample of 952 observed days. This included 376 individuals in the Core sample from Atlanta, Chicago, Dallas, Philadelphia and Seattle. Observing those people twice yielded 752 observed days. In addition, for the Media Acceleration portion of this research, 100 people were observed twice and yielded 200 observed days in the Indianapolis DMA.

376 or maybe 476 people in total (it’s not clear if the 100 are originally counted in the 376). The results of 476 people are the key behind a finding that online video is overstated, and some of the United States largest markets, including Los Angeles, San Francisco, New York, and even Austin, Texas (all four known for their tech communities) were completely ignored.

People who play with statistics will always argue that you can use such a small sample size to come to such conclusions, and that may be true to a point. But does a sample so small and geographically limited really reflect the overall market in the United States? Does the sample size fairly support a conclusion that goes against what every other survey and study is suggesting?

Of course it doesn’t, but that was hardly the point to begin with. This study was funded indirectly by media companies, and was to consider the following statements (direct from the study)

– The 30 second spot is dead
– No one under the age of 30 watches TV, They’re all streaming video on the internet.
– No one watches live TV, Everyone is Recording TV shows on their DVRS

Really outrageous propositions to begin with, and you don’t need a $3.5 million study to show that none are true. If you start a study with such outrageous suggestions, its little surprise then that you end up with an outrageous, and ridiculous result.

If you want to read the study, you can download it here (pdf)