Aaah – another site about The Future of Journalism.

A dull one.  Without the  invective and ideology about free vs. paid, pajama-clad bloggers vs. stick-in-the-mud mainstream media curmudgeons, and Utopian visions of crowdsourced news vs. dark fears about falling standards you can find elsewhere.  It has words like taxonomy and persistent content in it; discusses business models and revenue streams in dull, accountant-like language; and tries to dissect the sparkling prose journalists turn out into tiny bytes of data.

But there is a purpose here, and it’s based around the idea that we as journalists haven’t really thought about how people are changing the way they access information, or how we need to fundamentally rethink the way we carry out journalism and the kinds of – for want of a better word – products we turn out for them.

There’s much hand-wringing over the loss of the traditional business model of news, it’s true.  Perhaps too much.  And this site will contribute its share.  But hopefully it’ll also explore some of the less-explored questions about where the profession goes in a digital age.   And lay out some of the thinking behind one concrete idea that might help move the business forward: Something I’m calling Structured Journalism.

So, welcome – and I hope you find this interesting.

(An update: I first wrote those words seven years ago, and it’s amazing how some of those passionately argued debates – free vs. paid! – have basically gone away.  Which is great.  So I could and should rewrite this intro.  But the second paragraph remains just as valid. Plus, I’m pretty lazy. )

Just a riff on a recent NYT magazine piece about the debate around “autonomous weapons,” or machines that can make decisions about who and when to kill.  Spoiler alert: There’s no consensus about them.  Actually, not even close to being a consensus. Which is probably a good thing.

That said, it’s a good entryway to revisit the notion that we as an industry/profession could be doing a better job covering the multiple algorithms that now govern our lives, even if they aren’t literally designed to kill us.

Algorithms influence what news and information we see, how financial markets behave, where police put their resources, whether we can get loans and at what price, and much more.  And beyond that, they have to power – as do other automations – to reshape how we build and structure our world, beyond replacing humans.

As the NYT piece notes about the debates about autonomous weapons:

This argument parallels the controversial case for self-driving cars. In both instances, sensor-rich machines navigate a complex environment without the fatigue, distractions and other human fallibilities that can lead to fatal mistakes. Yet both arguments discount the emergent behaviors that can come from increasingly intelligent machines interpreting the world differently from humans.

Precisely.  Autonomous machines can and will go beyond replacing humans and potentially fundamentally change our world – not necessarily for better or worse,

Remember the old joke about the 1960s British Thunderbirds puppet TV series?  “The show’s good but the acting is kind of wooden?”  OK, so you probably didn’t.  Anyway, the joke was that the characters were played by puppets, so they were a little wooden… oh, never mind.

Fast forward to today, and Chinese news agency Xinhua has just unveiled a news anchor that’s – well, not a puppet, but certainly not human.  As Xinhua notes:

The news anchor, based on the latest AI technology, has a male image with a voice, facial expressions and actions of a real person. “He” learns from live broadcasting videos by himself and can read texts as naturally as a professional news anchor.

The AI news anchor was jointly developed by Xinhua News Agency, the official state-run media outlet of China, and Chinese search engine company Sogou.com.

The new newsperson got a lot of coverage – here, here, and here for example, but the reviews haven’t been kind.  The BBC quoted an Oxford professor:

The presenter struggled to appear completely natural, said Michael Wooldridge at the University of Oxford.

It was stuck somewhat in the “uncanny valley” – a term used to describe human-like robots and avatars which seem subtly unrealistic.

“It’s quite difficult to watch for more than a few minutes. It’s very flat, very single-paced, it’s not got rhythm, pace or emphasis,” Prof Wooldridge told the BBC.

He also pointed out that human news presenters have traditionally – in many cases – become highly trusted public figures.

“If you’re just looking at animation you’ve completely lost that connection to an anchor,” he added.

And India’s Scroll piled on:

Although the virtual anchor’s features are based on a real-life Xinhua host, Zhang Zhao, his voice remains robotic and detached.

But in some ways, everyone is asking the wrong question.  Xinhua

Things I like: Journalism.  Broadway.  Lin-Manuel Miranda.

So if you can get all three together, what’s not to like?

I stumbled on 21 Chump Street, a  This American Life project by accident, listening to the radio one morning.  It’s a musical dramatization of a 2012 story the program did on a number of drug busts in Florida high schools, based on the work of undercover police officers who posed as students.  The musical is pretty faithful to the facts, handles the nuance of the story and the he-said, she-said nature of the story well – and it works well as a musical too.  Not surprising, since it was written and narrated by a pre-Hamilton Lin-Manuel Miranda.

So it was just fun to watch.  But it also flags the need for more experimentation in how we get audiences to engage with the information we unearth and the stories we tell.  True, great writing can pull people through thousands of words. gripping documentaries can keep viewers coming back through multiple episodes, and well-designed interactive presentations can make audiences care about subjects they wouldn’t have otherwise explored.

But there are a lot more possible paths to explore, from plays and pop-up newsrooms to games to old-fashioned storybooks (even from journalists in jail! #FreeWaLoneKyawSoeOo).

I realize many of these may not scale well.  Or be financially sustainable.  (And there’s only so many Lin-Manuels out there.)

But arguably, engagement is one of the biggest challenges we face –

So it’s been a long time since I posted anything here.  I did think about it a bunch of times (honest!) but there’s been more than enough other things going on – from having two colleagues unjustly jailed in Myanmar to a multi-billion deal that nets Reuters a 30-year, $325 million-a-year contract to supply news – that this blog just hasn’t been a priority.

Still, there’s a lot happening in the world of technology and news, and even in structured journalism.  Like this nice, short piece in the New York Times on Sunday, riffing on the promise and limitations of machine learning.  And which reminds us, again, that we need to cover the algorithms that govern our lives in a much better way.

It’s mostly about how textgenrnn, a machine-learning algorithm that imitates text, has come up with a number of creative, funny and quirky new Halloween costume names – Sexy Minecraft Person or Piglet Crayon, anyone? – after being fed a series of costume names.  That’s pretty impressive, given that the system wasn’t given any information about words, grammar or spelling – it basically iterated towards things that made sense (sort of) to a human.

Which speaks to some of the power of such algorithms – their ability, in many ways, to come to “reasonable” results with a minimum of human intervention.  And – in a much more troubling way – with a minimum of human understanding as well.  As the piece notes:

Even when we can peer inside the neural network’s virtual brain and examine its virtual neurons, the rules it learns for its prediction-making are usually very hard to interpret.

…

…for the most part these algorithms are black boxes, producing predictions without explanation.

The key inputs for such systems are the training set of data – the pool of information that the system should be able to emulate – and the goals we set for the algorithm.  But the training data can be biased, leading to algorithms that turn out biased results (as Cathy O’Neil’s Weapons of Math Destruction nicely describes), and machines can take orders somewhat too literally, optimizing for conditions their creators never intended.

“My dog can play checkers.”

“That’s amazing!”

“Not really – he’s not very good.  I beat him three times out of five.”

OK, it’s a bad joke.  But it does kind of make a point about the world of artificial intelligence, big data and technology – it is amazing how far they’ve come in the relatively short time they’ve been in existence, but we also need to remember their limitations, and not believe all the hype.

At least that’s one of the major takeaways from Artificial Unintelligence, by NYU journalism professor Meredith Broussard, which I just finished on the flight from New York to Gdynia, Poland.  (Don’t ask.).  Like Weapons of Math Destruction, by Cathy O’Neil (another book I finished on a plane), this one also focuses on the misuse of technology and misplaced faith in algorithms and artificial intelligence’s ability to solve the world’s problems.

There’s much to recommend about having a high level of skepticism about the promise of AI, not just in terms of how well any particular system works, but also what assumptions and data are fed into it.  As an industry we haven’t really covered algorithmic accountability particularly well, and it’s a critical gap in fulfilling our public service mission of informing the world about things that matter to society.

And yet it’s important to also recognize how far machines have come, and what capabilities we can harness to do better journalism (even if sometimes there are questions about exactly how far they’ve come).

And one way to better harness machines is to not completely depend on them, as Meredith notes, but instead to build “human-in-the-loop” systems:

Remember this advice from way back about writing a story? “Just imagine you’re in a bar, telling someone what happened today….”

Oh, sure.  Of course that’s how stories are actually written.  Can you imagine actually reading out a newspaper story to someone at a bar and pretending that’s how you talk?

But maybe – in a back-to-an-imagined-past-and-a-new-future kind of way – that’s precisely where news is going.  Google just unveiled Duplex, a new voice-controlled AI capability, and it’s… well, amazing, frightening, creepy, and revolutionary.  All at the same time. And potentially game-changing for journalism, and possibly the business of journalism as well.

Listen to this: (scroll down a little and click on the first two audio clips.)  Go ahead, I’ll wait – it’s essential listening for the rest of this post.

Is that not amazing and creepy, or what?  To be sure, this isn’t an all-purpose HAL-like device that can decide you lock you outside the spaceship without a helmet.  Yet.  As The Verge notes:

Initially Google Duplex will focus on three kinds of task: making restaurant reservations, scheduling hair appointments and finding out businesses’ holiday opening hours.

But you could easily imagine how it becomes less of an assistant for performing tasks, and more of a way for users to explore and get the news or information they want.  Or as another story on The Verge points out:

The more technology advances, the clearer it becomes that our smartphones are no longer about conversing but more about transfers of information.

And what is news but the transfer of relevant, useful information? But wait, you say: Don’t we already have news on Alexa and other voice-controlled devices?

Sure we do.  But much of voice-enabled news so far has been built around either summarizing the top stories of the day, or pulling together data points – such as stock market performance – and generating sentences.  The quality of Duplex – at least as seen in the demo – opens up the possibility of much more fully interactive explorations of information.  Much like that mythical chat at the bar.

“So what happened today?”

“Well, a bunch of papers reported that Michael Cohen got a lot of money from companies that wanted to understand the new administration.”

“Wait – who’s Michael Cohen?”

“He’s the President’s personal lawyer.”

“Wasn’t he the guy that was involved in paying Stormy Daniels?”

So I’ve written a lot about the “cybernetic newsroom” and how news organizations should focusing on marrying the best capabilities of humans and machines to improve journalism – rather than trying to make poor copies of each other.  It’s certainly been very rewarding to riff on a subject close to me.

But it’s much nicer to make it a reality.

(Warning: shameless self-referential plug!) Last week, my colleague Padraic Cassidy unveiled Lynx Insight, our new automation tool, at a session at NICAR in Chicago – finally taking the wraps off a big project that we’ve been working on for more than a year.  It’s has gotten loads of press, (here, here, here, here and here are some of the examples), and marks a big step forward on this front.

So what is it, exactly?  Glad you asked.  On one level, it’s a three-layered system that ingests data, analyzes it to find patterns and anomalies, and then turns them into sentences and paragraphs for reporters to use.  On another level, it’s a tool that helps journalists by automatically trawling through mounds of data, looking for insights that it can present to them – whether as leads to chase down, or simply lines they can use in a story.  Or which they can ignore.

The key idea here – and how Lynx Insight differs from most other story automations – is that we don’t want it to write whole stories.  What we want it to do is analyze data, because that’s what we think machines are good at.  That analysis gets turned into sentences, because that’s what humans are good at evaluating, and then they can decide what to do with those insights.  Basically, it marries machine analysis with human judgement.

Why did we approach it this way? For one thing, it’s faster to build something

So it’s true I haven’t written in a while.  To be fair, there was news.

Blackstone bought a large chunk of Thomson Reuters’ financial business, and agreed to a $325 million a year, 30-year contract for Reuters’ news.  That’ll keep one busy.  And we have two journalists in jail in Myanmar. #FreeWaLone #FreeKyawSoeOo.

But it’s also true I haven’t written in a while.  But there’s a lot going on.  Just about to leave Austin, TX, where I was on a SXSW panel with Mark Hansen of Columbia J-school, talking about “Finding Signal in the Noise of Social Media“.  That was fun, and I’ll write more about that in a later post.  And, I also attended an interesting session about news automation in Sweden and Finland, where there was a nice shout-out to structured journalism.  Doesn’t get better than that.

Meanwhile, my colleague Padraic Cassidy was at NICAR in Chicago, where he spoke on a session about Lynx Insight, our new news automation tool as part of our strategy to build a “cybernetic newsroom.”  It’s been a really exciting road building Lynx Insight – a tool that marries the best of what machines do (automated data analysis, pattern recognition, simple sentence and language generation) with what humans do well (context, judgement, talking to people, and so on) – and I’ll write up much more about it in the next week or so.  It’s a capability that opens up personalization, counterfactuals and so much more.

So there’s a lot to talk about.  And I’ll get on it. Honest.

Can you scale intimacy?

In an age of viral content and global pageviews, what’s the right balance between aiming for massive scale and building deep, personal engagement?

That’s one of the questions I was left pondering after this year’s Global Investigative Journalism Conference in Johannesburg – another great gathering of some of the best and brightest journalists from around the planet.  (And as an aside, one of the best conferences GIJN has put on so far.  Plus, no one should ever miss a Paul Myers presentation).

The question about scale vs engagement was a theme I saw in a couple of panels, but surfaced very clearly in one I moderated on Innovations in Storytelling, with panelists David Schraven of Correct!v, Susanne Reber of The Reveal, and Julianna Ruhfus of Al Jazeera.

David and Susanne presented interesting ideas that seemed to really build connections with audiences – but relatively small ones.  (Julianna showed how Al Jazeera was using news games to engage audiences.) Susanne talked about how the Center for Investigative Reporting was running plays based on its stories, among them one about deaths of men working in the Bakken oil fields of North Dakota.  The plays, Susanne noted, had very small audiences by design, but seemed to have touched them very deeply, not least because it was shown in the communities the stories were about.

Other plays CIR has collaborated on include:

HEADLOCK was written by William Bivins and directed by Jennifer Welch based on Ryan Gabrielson’s investigation into abuse at California’s adult care facilities.

A GUIDE TO THE AFTERMATH was written by Jon Bernson and directed by Jennifer Welch, inspired by Mimi Chakarova’s documentary about female veterans suffering from PTSD and military sexual trauma.

THIS IS HOME was written by Tassiana Willis, Donte Clark, Will Hartfield-Peoples and Deandre Evans and directed by Jennifer Welch and Jose Vadi in response to Amy Julia Harris’ reporting on corruption and squalor in Richmond, California, public housing.

ALICIA’S MIRACLE was written by Octavio Solis, translated into Spanish by Brandon Mears and directed by Jennifer Welch in response to Bernice Yeung and Andrew Donohue’s investigation into the use of fumigants in California’s $2.6 billion strawberry industry.

Correct!v, the German non-profit investigative news organization, is likewise branching into plays, but David also talked about a foray into very local journalism: Setting up a storefront newsroom in Bottrop, a coal mining town of 100,000 where hundreds of people had died because of a pharmacist’s manipulation of cancer medicines.

To err is human.  But machines aren’t bad at it, either.

Especially when humans are the ones encoding the mistakes in – whether in the flawed assumptions we bake in, or the lack of judgement we show in blindly trusting our creations.

That’s the key thesis of  Cathy O’Neil‘s Weapons of Math Destruction – a well-argued book about the dangers of allowing algorithms we don’t understand well to run large parts of our life.  It’s a good, quick, read – I finished it on a five-hour flight from New York to Phoenix – with lots of stark examples, and is well-worth diving into.

It makes a great case for why we need better coverage and understanding of algorithms, given how big a role they now play in our daily lives and how little transparency there is about how they work. That’s not a new idea – “algorithmic accountability” has been a rallying cry for some for some time now, not least from Nick Diakopoulos, now at Northwest University, and Julia Angwin of Pro Publica. (I’ve made pitches for it as well – here and here, for example.) And the furor over Facebook’s algorithmically driven news feed, and how it was used to target particular audiences during the 2016 presidential campaign, is breathing new life into that drive.

But there’s still much more than can and should be done.

To be sure, we can’t do without algorithms – they help us sift through tons of data, can bring a level of objectivity to difficult decisions, and can surface insights