Aaah – another site about The Future of Journalism.

A dull one.  Without the  invective and ideology about free vs. paid, pajama-clad bloggers vs. stick-in-the-mud mainstream media curmudgeons, and Utopian visions of crowdsourced news vs. dark fears about falling standards you can find elsewhere.  It has words like taxonomy and persistent content in it; discusses business models and revenue streams in dull, accountant-like language; and tries to dissect the sparkling prose journalists turn out into tiny bytes of data.

But there is a purpose here, and it’s based around the idea that we as journalists haven’t really thought about how people are changing and accessing information, or about how we need to fundamentally rethink the way we carry out journalism and the kinds of – for want of a better word – products we turn out for them.

There’s much hand-wringing over the loss of the traditional business model of news, it’s true.  Perhaps too much.  And this site will contribute its share.  But hopefully it’ll also explore some of the less-explored questions about where the profession goes in a digital age.   And lay out some of the thinking behind one concrete idea that might help move the business forward: Something I’m calling Structured Journalism.

So, welcome – and I hope you find this interesting.

Stijn Debrouwere has – as usual – a provocative, interesting and just smart post at his blog about the future of journalism.  Or more precisely, a post about the present of non-journalism (or near-journalism) and what that means for the future of journalism.  It’s scary in many ways, but is important in helping frame how we rethink our profession/industry.

A quick summary:

There are organizations and websites everywhere that are taking over newspapers’ role as tastemaker and watchdog and forum. These disruptors don’t replace investigative reporting, but they replace the other 95% of what made professional news organizations important.

In other words, non-journalism outlets (Quora, Wikipedia, etc) are increasingly fulfilling some of the needs that news organizations used to fill. You can argue about their quality, or whether it’s a good thing, but the fact is it’s happening – and that has real implications for how news organizations adapt to the digital world.  The beauty of the post is that it inverts the usual (capitalized) discussion about the Future of Journalism to one about the Future of the How People Get Informed.  And, as Clay Shirky noted in Cognitive Surplus, that discussion should be less about what we’re trying to do and more about what customers are hiring us to do.

So read Stjin’s post: It’s very good.

But back to the question that both Stijn and Clay pose: What are people hiring us to do, at least in terms of “journalistic news”?  I suspect the way people access news is still evolving – and it’s not like anyone really knows the answer now anyway – but I might argue that there are a couple of discrete user behaviors/needs: News, search, context and serendipity.  News meaning simply getting people the latest updates on important stuff (who won the French election?); search meaning fulfilling specific information needs when people want it (how many people died when the Titanic went down?); context meaning helping people understand issues, whether about news or broader topics  (so what is climate change and how does it affect me?); and serendipity meaning telling people things they might need to know but don’t know they need to know (end of the world next Tuesday!)

News organizations have been organized mostly around news, and if anything, the digital age has increased that focus (not necessarily in a good way).  But in the other areas, we’ve just been much less effective or innovative.  As Stijn points out – and I agree wholeheartedly – the last major innovation in storytelling was at Politifact.  (Which is why I keep pushing the notion of structured journalism).

We have all been so focused on the quality issue: the fact that we’re still doing journalism like we used to do it fifty years ago, that there haven’t been any exciting new news formats since PolitiFact, that there’s a ridiculous lack of context for news stories online, which together with the fragmentation of readership is a disaster.

And on the other side of the question, it’s worth asking: What is our competitive advantage in any of this?  It’s one thing to acknowledge how people are now approaching information; it’s another to know if we can do it better (or cheaper, or more regularly, or whatever) than others.  Some might argue that journalists bring more objectivity, or rigor, or trust, than crowd-sourced or non-professional information, and there’s probably a case to be made on both sides of that question.  I’d argue that one key thing we do bring is consistency and discipline; and that’s not a small thing.  Meaning that we can offer a level of dependability that we’ll consistently cover the local school board meeting, or whatever.  But that means fulfilling that promise, and not doing it whenever we feel there’s a story that we want to write.

I’m not sure this is necessarily all that appealing a future for journalism; or even that these ideas are generally right.  But I do believe it’s important to keep the perspective of the audience we’re supposed to be serving in mind when we debate the future of our industry.  Because ultimately it isn’t about us; it’s about them, and the needs of society.  (Although we’d all like to stay employed as well.)

The machine-writing capabilities of Narrative Science have been well covered, and pretty much every (human-written) story on the subject has mused – nervously – about the replacement of journalists by algorithms. So, too, does a new piece in Wired; but this one takes the discussion a step further – and points to a future where collaboration between machines and humans can enrich not just news but newsrooms as well.

If that conjures up images of the Borg in Star Trek, well, OK.  It does.  But there are good reasons to explore – and even embrace – this future.

First, some background.  Narrative Science, as various people have written (including here), is a company in Chicago that has done some astounding work in generating stories and reports from data.  The Wired piece cites a sample:

Friona fell 10-8 to Boys Ranch in five innings on Monday at Friona despite racking up seven hits and eight runs. Friona was led by a flawless day at the dish by Hunter Sundre, who went 2-2 against Boys Ranch pitching. Sundre singled in the third inning and tripled in the fourth inning … Friona piled up the steals, swiping eight bags in all …

OK, so it’s a little old-fashioned in style.  But it sure doesn’t read like a machine wrote it.  More importantly, it shows how just analyzing a pile of baseball statistics – given the machine doesn’t have any other information – can yield understanding of an event.  And that’s an important point to note in a world which is increasingly awash in data – more data than any person (or newsroom) can possibly analyze at any kind of scale and speed.

Leaving aside the actual writing that Narrative Science’s algorithms do, what matters here is how the program sifts through data to identify key elements in an event, and then surfaces them through machine-generated text.  It’s not just that it’s smart data analysis; it’s smart automated data analysis.

So Narrative Science’s engineers program a set of rules that govern each subject, be it corporate earnings or a sporting event. But how to turn that analysis into prose? The company has hired a team of “meta-writers,” trained journalists who have built a set of templates. They work with the engineers to coach the computers to identify various “angles” from the data. Who won the game? Was it a come-from-behind victory or a blowout? Did one player have a fantastic day at the plate? The algorithm considers context and information from other databases as well: Did a losing streak end?

Humans do this sort of pattern analysis well, of course. And algorithms are limited by their programming and the factors that they’re designed to look at.  But machines have the advantage of being able to sift much more information than humans can, and at much faster speeds.  So while (human) computer-assisted reporting teams will almost certainly do better work than automated data analysis systems, there’s only so much humans can do in the course of a day.

There are lots of reasons machine-generated text will find its way into newsrooms in the near future: Lower costs, broader coverage, greater personalization.  But it may be that automated trawling for insights in large datasets is the most useful one in the long run.

When I sent by Reuters in 1990 to help cover the Asian Games in Beijing, I prepared a filofax (remember them?) full of statistics on swimming, one of the events I was assigned to cover.  I didn’t want to be caught out if a new Asian Games, Olympic or world record was broken, and I knew I certainly wasn’t going to able to keep all that information in my head.  That was in the pre-internet days; these days I would probably be using Google to check statistics before filing a story.  But it would make much more sense if I was paired with a machine/algorithm that could check the latest results against a host of databases, even before I started writing.

Shouldn’t we be building systems that do that automatically in newsrooms?  Before a reporter writes a market report, shouldn’t an algorithm be checking that day’s close against a database of market data, alerting him or her to new records, 52-week highs, and so on?  Similarly, shouldn’t smart algorithms be trawling through databases and regularly throwing up insights for beat reporters to follow up on – or dismiss?  As the Wired piece notes:

Computers, with their flawless memories and ability to access data, might act as legmen to human writers. Or vice versa, human reporters might interview subjects and pick up stray details—and then send them to a computer that writes it all up.

Of course, doing all this at scale requires building systems that can learn about new datasets relatively quickly, and it sounds from the Wired piece that Narrative Science is well down that path as well.  And it also highlights the need for newsrooms not only to have easy access to data, but also the value of having newsrooms create data from daily reporting as well, to make it easier for machines to help sift through patterns.  (A simple, if non-sophisticated, example: If newsrooms logged the location every car accident they reported on, for example, an algorithm could look for patterns or at least give background to a reporter sitting down to bang out a piece on the latest accident.)

This isn’t to say that this will be – or should be – the center of a modern newsroom.  But just as data analysis and visualization skills, interactive graphics, multimedia are part of the new toolkit that journalists have to be able to access, so too should be automated data trawling, to help surface insights in the mass of data we have access to.  Without that help, all-human newsrooms risk drowning in data.

There’s a fascinating piece in the latest issue of the Atlantic Monthly about game designer Jonathan Blow, creator of acclaimed game Braid.  There’s much in there about the world of gaming and the evolution of the form, but what stands out is the discussion of the need for games to develop their own artistic sensibilities, rather than inherit concepts from other media:

Blow envisions future games that deliver experiences as poignant and sublime as those found through literature and film, but expressed in ways distinctive to games. “If the video game is going to be used for art purposes, then it has to take advantage of its form in some way particular to that medium, right?” he told me. “A film and a novel can both do linear storytelling, but novels are very strong at internal mental machinations—which movies suck at—and movies are great at doing certain visual things. So the question is: Where are games on that same map?”

Precisely.  All media – platforms, or whatever – have their own advantages and disadvantages; trying to shoe-horn content from one to another is akin to trying to get oil paintings to mimic photography (or vice versa.)  Or hanging on to story forms developed for print in a digital world.

As (Blow’s friend Chris) Hecker explained it: “Look, film didn’t get to be film by trying to be theater. First, they had to figure out the things they could do that theater couldn’t, like moving the camera around and editing out of sequence—and only then did film come into its own.” This was why Citizen Kane did so much to put filmmaking on the map: not simply because it was well made, but because it provided a rich experience that no other medium before it could have provided.

And digital information/news is as different a beast from print (or broadcast) as film is from theater.   That’s not news to anyone, but we’ve tended to focus on speed and reach as the key characteristics of the new medium, and while those are important, so too is persistence of information and the ability to find it on demand. Not to mention interactivity, personalization and a host of other capabilities.

News organizations have tried to address those issues by providing archives and goosing search capabilities, allowing users to build stock portfolios, building nifty interactive graphics, and so on; but these have often been bolted on the side of a classic newspaper-like treatment of information – akin to filming a stage drama with a couple of different camera angles, rather than re-envisioning the entire production.

And similarly, we as an industry we haven’t really been willing to give up our newspaper-derived conventions about information or delved into fundamental building blocks of information – stories – and thought about how well they really serve this new age.

Freedom of the press is guaranteed only to those who own one.  ~ A.J. Liebling

That statement was true enough when A.J. Liebling made it in 1960, but the world has changed a lot since that time.  Printing presses have gone the way of, well, printing presses, and anyone who wants to publish pretty much can.  But in many ways Liebling’s statement holds truer than ever.

So journalists – or newsrooms – don’t need a press anymore; but they need access to, and influence over, the means of production if they’re to have any meaningful say in what and how they publish and reach a community.

And that’s especially true in a digital age, where product, content, platform, format, business model are all much more blending into each other.  If you don’t have a say in the means of production – and that means a say in the technology of the newsroom – that means, at some level, you don’t have a say in what you’re ultimately producing.  About whether it serves the public interest.  Or whether it makes money and you get paid.

In many ways, this is at the heart of the debate about whether newsrooms or IT departments should control developer resources.  The answer, of course, is that both need to be involved: Newsrooms are generally bad at building industrial-scale, commercial technologies that can underpin a business; but they’re better at – or should be better at – understanding content and audience.

The Pulitzer-winning Politifact is a prime example of how starting with an idea for a news product changes the way content is created and presented, but there are tons of others.  Yet many of them are built around the periphery of a newsroom’s CMS and publishing platform; the core of the operation is generally still geared towards creating stories and that day’s (or minute’s) news.

But it’s in exploring new types of content and new, non-advertising driven revenues that new business models – and new audiences – might be found.  Plus where the new frontiers of journalism and information – such as data-driven journalism or automation – are to be explored.

At this year’s NICAR conference, Chase Davis and Matt Wynn hosted a spirited discussion of new monetization models based on data, including showing off their MyFault and Curbwise apps – neither a creation of a classic newsroom or its technology platform.   True, no one is getting rich off those apps, as Chase and Matt freely admitted; but without that kind of experimentation, newsrooms are going to be stuck in legacy systems and ceding innovation to others.

No one’s suggesting, of course, that journalists should take over all technology development in media companies; that’s a much an issue as requiring all tech to be handled by the IT folks.  And that ignores too, the role that business managers need to have in the creation of new news products.

But in an age where the means of production allows for much more – and much quicker – innovation and flexibility in how news products and content are created, and where the ecosystem of information is still evolving, freedom of the press is guaranteed only to those who own the means of production.

I’ve made the case several times before that journalists in general need to much more numerate to excel in the age of data; but maybe I wasn’t going far enough.

There’s an interesting NYT piece today on how some universities are offering courses on computer science – or “computational thinking” – for non-CS majors.   Some of these courses involve teaching programming languages like Python, but others are more focused on the issues around computer use and the ways they work.

Michael Littman, who leads the computer science department at Rutgers University, agrees. “Computational thinking should have been covered in middle school, and it isn’t,” he says. “So we in the C.S. department must offer the equivalent of a remedial course.”

And why not?  Computational thinking should have been covered in middle school; not computer programming per se, but thinking in structured, logical ways that help students understand not just how computers work, but how they can work – without which it’s hard to begin really imagining creative uses for them.  And given how much of the world is created through programs and programming – and especially how much of the information world is built around that – journalists should have at least a passing knowledge of computer science.  And numeracy.

The piece cites a paper by Jeannette Wing of Carnegie Mellon University that makes a passionate case for making computational thinking a basic skill.

Computational thinking is a fundamental skill for everyone, not just for computer scientists. To reading, writing, and arithmetic, we should add computational thinking to every child’s analytical ability.

…

Computational biology is changing the way biologists think. Similarly, computational game theory is changing the way economists think; nanocomputing, the way chemists think; and quantum computing, the way physicists think.

This kind of thinking will be part of the skill set of not only other scientists but of everyone else. Ubiquitous computing is to today as computational thinking is to tomorrow. Ubiquitous computing was yesterday’s dream that became today’s reality; computational thinking is tomorrow’s reality.

Without these skills – or more accurately, these mindsets – journalists risk ceding control over large parts of our future to others: Not just in terms of our ability to crunch data for stories, but also in the way we can design and build visualizations, apps or publish and distribute our work.  We don’t just write stories now and hand them over to folks at the printing press to put ink on paper; we are – or can be – responsible for the creation of the entire work, from idea to finished product.  Even if we don’t want to do all that, we should at least understand what the possibilities are.

I was watching my 11-year-old son begin experimenting with a Lego NXT robotics kit today, and was very gratified to see him have an instinctive sense of programming.  Whether or not he takes to it, it’s nice to know it computational thinking isn’t completely alien to him.  Now if only his father can get past the halfway point in the introductory book to Python.

There’s a great piece in Slate that takes the notion of personalization on the web to its logical extreme: The individual reading experience for every story, for every person.

And why not?  After all, we can already – in theory and to some extent in practice – customize machine-generated stories and reports by drawing on different information in databases (“Your stock portfolio tanked today despite the overall market rising….”).  It’s not that far a step to imagine we can also use knowledge about you and your online habits to craft a tale that appeals to your interests and knowledge.

Is this a good thing?  Perhaps.  Although not everyone will agree.

Evgeny Morozov, who wrote the Slate piece, worries about the social impact of a world where we all read slightly – or very – different stories about the same event.  And about the increasing power of companies like Google and Amazon that will have far greater capability to do this successfully than other media organizations, by dint of the horde of data they’re gathering on our habits.

If there is one unambiguous trend in how the Internet is developing today, it’s the drive toward the personalization of our online experience. Everything we click, read, search, and watch online is increasingly the result of some delicate optimization effort, whereby our previous clicks, searches, “likes,” purchases, and interactions determine what appears in our browsers and apps.

…

Imagine that my online history suggests that I hold an advanced degree and that I spend a lot of time on the websites of the Economist or the New York Review of Books; as a result, I get to see a more sophisticated, challenging, and informative version of the same story than my USA Today-reading neighbor. If one can infer that I’m also interested in international news and global justice, a computer-generated news article about Angelina Jolie might end by mentioning her new film about the war in Bosnia. My celebrity-obsessed neighbor, on the other hand, would see the same story end with some useless gossipy tidbit about Brad Pitt.

All of these are real concerns – and that future may not be as far off.  The quality of machine-generated stories is steadily improving, and the world is increasingly awash in the kind of data that such programs need to create – and personalize – pieces.

Should we worry about a world where we read stories tailored to our interests – and prejudices?  Probably – although to some extent that already happens, through the choice of what news organization we go to.

But there are also compelling reasons to welcome this world as well.

For one thing, it can help make news really matter to readers. 

Yale Law School held an interesting day-long conference on data journalism about a week ago; there were fascinating talks by a range of people, including game designer Simon Ferrari and graphics mavens Amanda Cox and Hannah Fairfield, not to mention hilarious presentations by data journalists Brian Boyer and Matt Stiles.  Plus two powerpoint references to Battlestar Galactica, which must be a conference record of some kind.

But perhaps the most interesting points raised were by Steven Waldman, the founder and former editor of BeliefNet, at one point one of the best sites on religion, and now an adviser to the FCC, where he authored a report on “The Information Needs of Communities” (I’m still reading it.) He talked about the need to update FOIA laws in a digital age, and how there needed to be more advocacy to push such changes ahead.  After all, if data journalism is dependent on data, then it’s critical how data is regulated and how available it is.

And beyond government data – over which there are already daily battles about how information should be released – what about other organizations that have or collect data that’s been mandated to be public?  Why, he asked, do TV stations only make data on political advertising with them available only if you show up in person at the station?  Shouldn’t organizations that are part of the news/information ecosystem make more of an effort to make public data that they have more easily available to the public?  Or should they be made to do so, by a more-updated law?

To be sure, there’s a danger of too much regulation of data – about what can and can’t be gathered or analyzed.  But he makes a compelling proposal as well, about  trading off between transparency and regulation – if organizations make data more easily and publicly available, perhaps then there’s less need to regulate them.  The Securities and Exchange Commission, for example, makes public companies disclose loads of information; it may be hard to find nuggets in the flood of stuff that comes out (often intentionally), but at least it’s there.  As tradeoffs go, that may not be so bad.

Where does data come from?  Governments, of course.  Official institutions like the World Bank.  Companies that vacuum it up and sell it.

But increasingly, it comes from people, too.

Consider the Adjunct Project, which invites “contingent faculty” at universities around the country to submit details of what they get paid for each 3-credit course they teach to a database (actually, a Google spreadsheet).  It asks for details like whether the school is unionized or if benefits are included, and seems to have 1,500 entries so far.

True, the data isn’t particularly clean; and there’s no real verification of the information or the contributors.  And certainly crowdsourcing isn’t all that novel an idea.

But it’s a huge step forward to gather information in a structured format so it can be much more easily analyzed.  And it’s basically free – as opposed to having some university (or news organization) pay for a costly survey of adjunct rates.

That’s sort of the theme of a fascinating piece by Javaun Moradi at NPR about the potential of crowd-sourced data for journalism organizations, with an emphasis on automated collection of sensor data rather than crowd-contributed data.  But the core idea is that citizen-sourced data is a great opportunity for news organizations.

If stage 1 of data journalism was “find and scrape data” , then stage 2 was “ask government agencies to release data” in easy to use formats. Stage 3 is going to be “make your own data”, and those sources of data are going to be automated and updated in real-time.

He writes about a site called Pachube (pronounced “PATCH bay”), which is trying to get citizens to collect data on the local quality of air by building an “open air quality sensor network.”   The idea is for them to set up sensors around the country which would feed data into a central network.

“If your mother says she loves you, check it out with a second source.”

That adage goes to the heart of what much of journalism has traditionally been about – verification and corroboration (and some transparency as well).   What can be verified, and by who?  Do they have a bias? What documents back up the assertion?  Are the other possible explanations?

But what happens when it’s an algorithm that comes to a conclusion?  One that a newsroom writes, and publishes the results of? What are the standards for verification then?

Let me back up a second.  This isn’t about how we do data analysis and come to conclusions; we’ve been interrogating databases for ages, and there’s pretty well-established principles for what constitutes a publishable piece of information.

But consider this great interactive that ProPublica put up a while back; it calculates and projects when particular states’ unemployment insurance systems might run out of funds.  The calculations that go into it are all disclosed, and it had a great track record.  It’s a useful service that takes public data further, through the prism of smart journalists who know their beat.

But that’s a big step away from classic journalistic methods of verification.  Sure, you can and should talk to experts in the field to make sure the algorithm works; but this is a different kind of verification.  What if, instead of unemployment insurance systems, there was an interactive that predicted bank failures?  To be sure, such calculators exist – but they aren’t generally made public via a news organization.  What are our responsibilities if we publish that kind of information?

As Big Data increasingly floods newsrooms, we’re turning more and more to sophisticated social science methods to sort, sift, group and analyze that information.  In some cases we’ll use what we find as leads to chase down stories; in others we might well be publishing the results of our analysis.  What standards of proof or validation do we need to develop, and how will we tell audiences why they should trust us?

The Project for Excellence in Journalism unveiled a nice study today on how newspapers are – or more accurately, aren’t – adapting to the digital world. It’s nice not least because it has real numbers in it (albeit anonymized), which lets people dive into the details.

The headline number is that, on average, papers are losing $7 in print revenue for every $1 gained in digital revenue (although the report points out that mileage varies, sometimes considerably, among papers.)  And the headline headline is that there continues to be a massive cultural divide in newspaper companies, between those attached to legacy ways of doing business (and news) and those who want to change faster.

“Probably the most difficult thing is to change a corporate culture because you don’t really have the power to do it,” noted one executive. “You can change CEOs, executive VPs, digital VPs. You can wave this magic wand all you want. But at the end of the day, the troops in the field hunker down. From our company, and I would venture for other organizations as well, the most difficult thing to do is change it. “

Culture change is hard work.  But it isn’t simply a question of dinosaur-like curmudgeons standing in the way of needed change (although that’s doubtless true too); it’s that the entire structure of legacy newspapers is built about – no surprise – legacy processes and legacy customers.  And that there are very good business reasons for sticking to such practices, at least in the short run. In the long run, it may be suicidal, but as Clay Christensen noted in The Innovator’s Dilemma, that’s why great companies often fail in the face of a disruptive innovation.  The PEJ report notes that:

…the unavoidable fact that the part of the newspaper industry that is growing, digital, continues to provide only a small part of the revenue, while the part that is shrinking, print, provides most of the money-a paradox that is difficult to navigate and hard to resist.

So culture stands in the way, no doubt.  But so do hard-headed business judgements about revenues.  That’s not to say that the curmudgeons are right; they aren’t. But it’s a very tricky balancing act trying to manage current operations and future growth – especially when there are no clear models yet of what works in the digital world. Is mobile the future? Paywalls? Services? More targeted advertising?  Choosing which one to invest in – because most organizations don’t have the resources to invest in all – involves making critical decisions on very little information.

What would be great would be a similar in-depth study of the new online news ventures – their costs, revenues, audience, etc – so people could plumb them for clues to the future too.

Still, ultimately publishers (and editors) have to take a leap of faith – something easy to say, but hard to do when you’re responsible for a public institution, people’s jobs and, in the case of public companies, other people’s money.  But staying still isn’t an option, either.