What are we measuring, and why?
It’s a thought that comes up from two very different directions – one from the world of (somewhat unmeasurable) design quality, and the other from hard revenue numbers. But they both speak to the importance of trying to figure out what we’re trying to achieve when we set up an online news site – and how that question remains largely in flux.
On the design side, Andy Rutledge ignited a heated online discussion a few weeks ago when he critiqued the New York Times‘ site and mocked up an alternative. His argument, in a nutshell, was that sites should be simpler and cleaner rather than crammed full of links, as the NYT site is. Does that make sense? Well, it depends on what the Times is trying to do – try to show off as much of its content to potential readers, or offer a focused curated experience on what its editors consider the most important news?
Just about the same time, Phillip Mendonca-Vieira offered up a fascinating pair of time lapse videos of the BBC and NYT sites over the course of a year, to show how differently they treated key stories – the Beeb with a great deal of focus and the Times with a broader selection of stories.
Again, which one works better? And again, it all depends on what you want to achieve. Is the goal traffic? Engagement? Time on site? Community? Brand loyalty? What’s the site supposed to mean to its core readers, and to others who stop by? Do you want your site to be a portal to a wide range of information – and hence offer up as much as it can to visitors – or a speciality store that focuses deeply on one or two topics?
And how are you supposed to make money from all of this?
Design, after all, works in the service of what you’re trying to optimize for – and that keeps changing, even if your broad goals remain the same. There was a time – not that long ago – when SEO was the key thing. And now we care hugely about social media and community. Who knows what will matter tomorrow?
As Josh Benton at Nieman Labs noted on a post about the design discussion:
…no one has figured out a way to present lots and lots of constantly updated information in a way that (a) is beautiful, (b) is effective at story discovery, and (c) privileges editorial control.
Beautiful by itself isn’t that hard — there are lots of beautiful sites on the web, and lots of talented designers. When it comes to effective story discovery, the innovation has all been in the direction of algorithms and raw feeds. An algorithm is how Facebook surfaces items in your News Feed; a raw feed is how Twitter organizes tweets from the people you follow, in straight reverse chronological order. But neither of those is perfect for human editorial control, which is something news organizations rightly value; there are tons of visual and contextual cues on those complicated nytimes.com pages that tell me what Times editors think is more or less important for me to see.
The good news – sort of – is that we have much better metrics these days to be able to measure, and test, different types of design to see how well they function. Who’s coming to the site, what are they looking at, how long are they staying, etc. And we can tweak – within limits – designs to see if they make a difference: Think red buttons work better than green? Try them out on half the users and see what happens.
The trick, of course, is to also make sure that we’re measuring the right things when we do those sorts of tests. Raw traffic numbers aren’t all that meaningful – and can often be confused, as Steve Myers notes at Poynter:
Unique visitors, visits and page views are different metrics. One unique visitor can go to a site four times in a month and click on three pages each time, which is counted as one unique visitor, four visits and 12 page views.
Using “traffic” when you’re talking about “visitors” is like using “purchases” when you mean “shoppers.” They’re related, but not the same.
And like “shoppers” or “purchases,” what visitors are worth also matters – hugely.
Ken Doctor dug into that in an – as usual – insightful post about average revenue per user, or ARPU, and how that relates to the revenues that a handful of major sites (NYT, HuffPost, Guardian, Daily Mail) pull in, relative to their traffic. What he shows is how much more valuable visitors to the Times and the Guardian are than to the HuffPost and the Daily Mail – perhaps not a huge surprise, but then again it’s always surprising when I hear people make arguments about the importance of large audiences and traffic. (Thanks to Kevin Anderson of Strange Attractor for pointing out the post.)
As Ken calculates, an average visitor to NYT.com is worth $3.54 a year, versus 96 cents a year to the Huffington Post; similarly the average Guardian visitor accounts for $1.53 in annual revenue, versus 38 cents for the Daily Mail. (ARPU is a metric telecoms companies use a fair amount to figure out what their customers are worth to them – it’s simply how much money each one brings in on average.)
Of course, it’s true that costs at the NYT and Guardian are probably much higher than at the other two organizations (for all the higher revenue that the Guardian gets for all it’s traffic, it still hemorrhages a hundred thousand quid a day), so revenue alone isn’t the defining metric. But it’s a pretty important one, especially if you’re trying to build a sustainable news site.
So should we be trying to measure the metric average profit per user? That obviously starts getting hard to measure – we can probably work out with some detail which parts of the site attract revenue, but how accurately can we allocate costs to that part? And if we’re building long-tail databases from day-to-day reporting, how do we factor in those costs?
Which is another way of saying, we’re a long ways away from having any kind of definitive answers to these questions. But we’re much further along in the ways we can measure things. So it’s critical that we think through our broad goals as we figure the metrics we want to measure – and then set ourselves targets to meet.
What’s the goal of our site? Sustainability matters hugely, of course – or we wouldn’t have a site very long. But if sustainability was all we were after, there are much easier ways to get something going than setting up a news operation. Does public service matter? Influence? What are you willing to trade off/lose for that?
Because at the end of the day, what metrics we define – good ones or bad ones – are the things that we’ll torque our efforts towards. And if we define them badly, we’ll get what we define.