Musk’s ‘Twitter Files’ offer insight into the raw, complex and thankless job of moderation

Twitter’s new owner, Elon Musk, is feverishly promoting his “Twitter Files”: selected internal company messages carefully tweeted by sympathetic fans. But Musk’s apparent belief that he’s betrayed some partisan kraken is misguided — far from a conspiracy or systemic abuse, the files are a valuable insight behind the curtain of moderation on a scale that suggests the Sisyphean labor each social media platform undertakes.

For a decade, companies like Twitter, YouTube, and Facebook have performed an elaborate dance to keep the details of their moderation processes equally inaccessible to bad actors, regulators, and the press.

Disclosing too much would leave the process open to abuse by spammers and fraudsters (who actually take advantage of every detail revealed or published), while disclosing too little would lead to damaging reports and rumors as they lose control of the narrative. Meanwhile, they must be prepared to justify and document their methods or risk reprimands and fines from government authorities.

The result is that although everyone knows a a little about how precisely these companies screen, filter and curate the content published on their platforms is just enough to make us believe that what we are seeing is only the tip of the iceberg.

Sometimes there are revelations of methods we suspect – hourly contractors trawling through violent and sexual imagery, a disgusting but apparently necessary industry. Sometimes companies exaggerate, such as repeated claims about how AI is changing moderation and subsequent reports that AI systems for this purpose are unfathomable and unreliable.

What almost never happens—companies usually don’t unless they’re forced to—is that the actual content moderation tools and processes are exposed at scale without a filter. And that’s what Musk did, perhaps at his own peril, but certainly to the great interest of anyone who’s ever wondered what moderators actually do, say, and click when making decisions that can affect millions.

Pay no attention to the honest, complicated conversation behind the curtain

Email threads, Slack conversations, and screenshots (or rather screenshots) posted over the past week provide insight into this important and poorly understood process. What we see is some raw material that is not the partisan illuminati that some have come to expect – although the highly selective presentation makes it clear that this is what we are meant to perceive.

Far from it: the people involved are alternately cautious and confident, practical and philosophical, frank and accommodating, suggesting that the choice of restriction or prohibition is not arbitrary but in accordance with an evolving consensus of opposing views.

(Upgrade: Moments after I posted this, a new thread has begun, which is more of the same — serious discussions about complex issues in collaboration with experts, law enforcement, and others.)

The decision to suspend Hunter Biden’s laptop story — arguably the most controversial moderation decision of the past few years behind Trump’s ban — has neither the bias nor conspiracy that the bombshell packaging of the documents suggests.

Instead, we find serious, thoughtful people trying to reconcile conflicting and inadequate definitions and policies: What constitutes “hacked” material? How confident are we in this or that assessment? What is a proportional response? How should we communicate this, to whom and when? What are the consequences of doing this if we don’t limit it? What precedents are we setting or breaking?

The answers to these questions are not at all obvious and are things that are usually worked out over months of research and debate or even in court (legal precedents influence legal language and consequences). And they need to be done quickly before the situation gets out of hand one way or another. Dissent from within and without (from the US representative, no less – ironically doxxed in the dark along with Jack Dorsey in opposition to the same policy) was considered and fairly incorporated.

“This is an emerging situation where the facts remain unclear,” said former Trust and Safety chief Yoel Roth. “We’re making a mistake by including a warning and preventing this content from spreading.”

Some question the decision. Some question the facts as presented. Others say this is not supported by their reading of the policy. One says they need to make the ad hoc basis and scope of the campaign very clear, as it will clearly be seen as partisan. Deputy General Counsel Jim Baker is asking for more information, but says caution is warranted. There is no clear precedent; the facts are absent or unverified at this point; some material is obviously nude without consent.

“I believe that Twitter itself should limit what it recommends or puts into trending news, and your policy against QAnon groups is all good,” admits representative Ro Khanna, while arguing that the measure in question is a step too far. “It’s a difficult balance.”

Neither the public nor the press were privy to these conversations, and the truth is that we are as curious and largely in the dark as our readers. It would be incorrect to call the published materials a complete or even accurate representation of the entire process (they are obviously, albeit ineffectively, picked and chosen to fit the narrative), but even as they are, we are more informed than we were before.

A tool of the trade

Even more directly revealing was the following thread, which contained screenshots of the actual moderation tool used by Twitter employees. While the thread disingenuously tries to equate using these tools with a shadow ban, the screenshots don’t show nefarious activity, nor do they need to be interesting.

On the contrary, it is presented convincingly precisely because it is so prosaic, so softly systematic. Here are the various techniques used by all social media companies that have been explained over and over again, but while we used to use them in happy diplomatic PR cheers, they are now presented without comment: “Trending Blacklist”, “High Visibility “, “DO NOT ACT” and the rest.

Meanwhile, Yoel Roth explains that measures and policies need to be better coordinated, that more research is needed and that improvement plans are underway:

The hypothesis underlying most of what we have implemented is that if exposure to e.g. disinformation directly causes harm, we should use fixes that reduce exposure, and limiting the spread/virality of content is a good way to do that… we’ll need to make a stronger case for including this in our repertoire of policy fixes – especially for other policy areas .

The content, again, contradicts the context in which it is presented: this is hardly the deliberations of a secret liberal cabal, hammering its ideological enemies with the ban hammer. It’s an enterprise-level dashboard like you might see for lead tracking, logistics, or accounts, discussed and iterated upon by sober-minded individuals working within practical constraints and striving to please multiple stakeholders.

As it should be: Twitter, like other social media platforms, has been working for years to make the moderation process efficient and systematic enough to work at scale. Not only so that the platform is not flooded with bots and spam, but also to comply with legal frameworks such as FTC and GDPR orders. (Of which, the “extensive, unfiltered access” external users gained to the tool in the image may constitute a breach. The relevant authorities told Root Devices they are “cooperating” with Twitter on the matter.)

A handful of employees making arbitrary decisions without rubrics or oversight is no way to effectively moderate or meet such statutory requirements; nor (as resignation more on Twitter’s Trust and Safety Council today witness) is automation. You need a large network of people working together and working under a standardized system, with clear boundaries and escalation procedures. And that’s certainly what the screenshots Musk released seem to show.

What the documents don’t show is any systematic bias that Musk’s followers allude to but fail to fully substantiate. But whether it fits the narrative they want or not, the release is interesting for anyone who thinks these companies should be more open about their policies. It’s a win for transparency, even if Musk’s opaque approach achieves it more or less by accident.