News Analysis: Why you should be skeptical of those lists of Netflix’s most-watched shows

In a shareholder letter last week, Netflix touted some recent programming wins. The streaming service announced that 64 million accounts had watched the new season of ”Stranger Things” within four weeks of release, while 32 million had checked out “Unbelievable,” a powerful drama about a rape investigation, during the same time frame.

For a company that guards its viewership data like Donald Trump does his tax returns, these morsels were tantalizing. Using the tidbits Netflix has been including in shareholder letters since January, the New York Times and Bloomberg compiled lists of the most popular Netflix titles, and the Guardian offered an analysis of “what works and what doesn’t” for the streamer.

Television

Are these end times for binge culture?

As streamers like Netflix, Disney+ and Apple TV+ experiment with weekly release strategies, the binge model is not as dominant as it once was.

Oct. 10, 2019

“Stranger Things” led the pack as the most-watched TV show — hardly a shocker — but some of the findings were more eye-opening. According to Netflix, some of the most popular shows on the service include the comic book adaptation “The Umbrella Academy” (45 million) and the Spanish-language crime drama “Money Heist” (44 million).

The irresistible temptation is to compare these figures to Nielsen ratings, which would make “Stranger Things,” with its audience of 64 million, just about the most popular scripted show on TV since the finale of “Friends.”

Yes, Netflix is sharing more information about how many people are watching its shows than it did just a year ago, and we now have a broad sense of which shows and movies are especially popular with its subscribers. But there’s plenty of reason to avoid these apples-to-oranges comparisons and take this data with a grain — make that a heaping scoop — of salt.

To start with, not even Netflix is claiming these “top 10 lists” are definitive. A representative for the company said the shareholder letters include select highlights but are not intended as “a comprehensive list of the most-watched titles.” Unlike Nielsen, which measures the U.S. only, Netflix is available in 190 countries.

Netflix also counts anyone who’s watched 70% of a single episode as a viewer, meaning if you watched the first 45 minutes of the first episode of “Umbrella Academy” and then turned it off, you’d be included as a viewer. This would be like ABC reporting on the premiere ratings for the first Roseanne Barr-less episode of “The Conners,” then never mentioning how many people tuned in the next week. A far more useful metric would be average per-episode audience.

But these cherry-picked figures allow Netflix to boast about its most obvious successes without putting them in any kind of useful context. Clearly, a lot of people are watching “Stranger Things.” But how much of an outlier is it? The numbers shed virtually no light on how many people, on average, are watching hundreds of other shows on the service, from critical darlings like “Russian Doll” to oddball cult favorites like the recently canceled “The OA.”

Netflix only shares viewership information about originals, not about the many very popular shows in its back catalog. That’s understandable. Especially as it faces a whole bunch of competitors, the company probably doesn’t want to advertise the enduring popularity of licensed titles that will soon be going to rival streaming services, like “The Office” and “Friends.” (Netflix said its “true” top 10 list consists of originals.)

Unlike old-fashioned linear networks, which have to rely on the far-from-perfect Nielsen ratings system — a guesstimate based on a narrow sample audience — Netflix has access to an arguably alarming amount of data about what subscribers are watching.

Netflix knows much more than whether you watched the latest season of “Stranger Things.” If you finished all eight episodes, it knows how long it took you to get through them. If you didn’t, it knows when you called it quits. It knows what devices you watched on and whether you skipped the credits. It knows your entire, possibly embarrassing viewing history, including how many times you, say, watched “A Christmas Prince” when you had the flu last winter.

Based on that, Netflix knows which thumbnail image is most likely to get you to watch another title. It knows your email address and where you live. Who knows? Before long it may know your rising sign, your resting heart rate and which flavor ice cream you’re going to eat during your next binge-watching session. It uses this data to decide not only which shows get canceled or renewed but which shows get a green light in the first place. As digital rights expert Michael Veale revealed this year, Netflix even tracked the choices made by viewers who watched the interactive film “Black Mirror: Bandersnatch.”

So when a company that has a borderline unsettling amount of information shares vague numbers that sound good but are presented in a contextual vacuum, we should all practice some healthy skepticism.