We’ve become suspicious of tech. Once obscure issues like the design of news recommendation systems and the evaluation of criminal justice risk scoring methods have become major public conversations. This new scrutiny of technical systems with social impact is well deserved — but that doesn’t mean that we get to be sloppy when investigating the power of algorithms.

The journalistic investigation of the politics of code, sometimes called algorithmic accountability reporting, is one of the more complicated types of reporting to do. The interactions between the technical and the social are intricate. Covering California’s SB 10 bill, which mandates the use of  pre-trial statistical risk assessment, requires an understanding of both machine learning error rates and the contentious politics of bail reform. The New York Times tried to make the case that increased social media use in Germany is correlated with more violent attacks on refugees, but the “landmark” study they relied on was actually a preliminary paper. Subsequent analysis suggests that the truth might end up turning on the number of people in the Facebook Nutella group, which the researchers used as a proxy for social media use generally. This stuff is tricky — and because the stakes are so high, everyone has strong opinions.

I’ve already seen several cringe-worthy examples of simplistic, unfounded, or just plain biased reporting on algorithms, including misleading pieces from both tech cheerleaders and tech skeptics. In the hopes of seeing better work in the future, here are a few tips on getting an algorithm story right.

• Method matters. Investigating the social consequences of algorithmic decision-making is an act of social science. Don’t let that scare you — data journalists have been doing social science since the beginning. But it does mean there’s a lot to learn from other disciplines about analyzing data, designing experiments, and so on. The causes of political events are especially difficult to pin down, yet that’s exactly what much algorithmic accountability reporting aims to do. Plus, social scientists have invented lots of creative methods that journalists can use. Here’s a great primer on how to design experiments to reverse engineer social media platforms.
• Interdisciplinary details matter. There aren’t many people who are equally knowledgeable about both the technical and the social. Writing on algorithms seems to split down this divide: We get detailed technical analyses of machine learning algorithms or we get sophisticated legal arguments about the consequences of using algorithms in court, but very little that integrates these perspectives. And what about community members and advocates, the people most affected by these algorithms? Journalists have to straddle these schisms. This is a strong argument for teamwork and collaboration between people with different backgrounds.
• Transparency matters. A great many algorithms of social interest are black boxes: we can’t see the code, and often we can’t even interact with the system to test it. There are a variety of ways around this, including crowdsourcing to collect algorithmic results. But sometimes you’re just not going to be able to get the information you want. In this case, as the old saying goes, the fact that you can’t do the story becomes the story. Why isn’t it possible for an independent person to check on the function of critical societal infrastructure? What would have to change to enable this sort journalism?
• What’s the option? I’m going to borrow a page from the Solutions Journalism movement and suggest that when reporters find a suspect algorithm, they also try to cover what could be done better. Sometimes the algorithm should not be used. Sometimes there are already good ideas about how to modify it. And sometimes there is evidence that human decisions are worse than statistical methods. Kentucky judges routinely ignore algorithmic risk assessments and jail people who are classified as low risk, while automated lending systems have reduced discrimination.

Like any serious journalism, algorithmic accountability reporting requires expertise, curiosity, and dedication to the truth. Increased skepticism of our robot overlords is a good thing, but it doesn’t get to play by different standards than any other investigative journalism.