11 stories

Looking at the origins of cultural collapse

1 Comment and 3 Shares

I read a couple of really interesting articles this weekend that attemped to shed some light on how our culture failed to prevent a Donald Trump Presidency. The first was by Danah Boyd, who works at Microsoft Research. When I read the title – Why America is Self-Segregating – I almost didn’t click on the link. Most of the pieces on self-segregation that I’ve read I don’t agree with, and I tend to find annoying. That was not the case with Danah’s article. Her argument and evidence were completely new to me, and really insightful.

Emily Nussbaum’s New Yorker piece How Jokes Won the Election was a more painful read. For those of us who think of humor as a way to package the truth for easier consumption, 2016 reminded us that humor can also provide refuge for liars.

As I read it, though, I remembered an incident from years ago that chilled me to the bone in recollection. In 2000, The Onion (which Nussbaum calls “the original fake-news site) ran a satirical piece entitled ”Harry Potter Books Spark Rise in Satanism Among Children." Here and there, people who weren’t familiar with The Onion reported on the article as though it was real news. We laughed and laughed. How could people not know The Onion was a joke? How could they believe that kids become Satanists after reading Harry Potter. People are so dumb, we thought.

Little did I know at the time that those people who took The Onion literally were the vanguard of the fake news epidemic that plagued the 2016 election. They were teaching us a valuable lesson – if you publish something that’s formatted like news and reads like news, people will take it seriously if it confirms their existing biases. What The Onion proved by accident has since been repeatedly used to devastating effect to erode society’s relationship with the truth.

The election of Donald Trump has proven that American culture is in a desperate state, and even though the examination is painful, it’s important to figure out how we got here. Both articles take the already tired investigation into the origins of Donald Trump’s America in new and interesting directions.

Read the whole story
2732 days ago
Especially the danah boyd piece linked.
Chicago, USA
Share this story

Better command history in your shell

1 Comment

My ideal command history would let me search the history of every shell but when I hit the up arrow it would only cycle through my current shell’s history. In February, I was able to achieve this setup in large part because of a utility called hstr.

What is hstr?

hstr is a neat Bash and Zsh utility that lets you easily search, view, and manage your command history. hstr provides a tool named hh that provides a text interface for manipulating your command history. To see what it looks like check out the README and this video tutorial. If you are running OS X and use Homebrew you can install it by running brew install hh.

Making global history searchable but arrows cycle through local history

hstr is a neat tool but my favorite part of my setup is how the global command history is searchable but only a shell’s local history is cycled through with the arrow keys. This is achieved by manipulating where history is written and tweaking some environment variables.

The first step is to change your $PROMPT_COMMAND to append your shell’s history to a global history file. Below is the snippet that does this from my .bashrc file.

# Whenever a command is executed, write it to a global history
PROMPT_COMMAND="history -a ~/.bash_history.global; $PROMPT_COMMAND"

The next step is to bind a keystroke to run hh, which is what hstr provides, with $HISTFILE pointing to ~/.bash_history.global. I wanted to fully replace the default command history searching (and I use Emacs style keyboard shortcuts) so I’ve bound these actions to ctrl-r.

# On C-r set HISTFILE and run hh
bind -x '"\C-r": "HISTFILE=~/.bash_history.global hh"'

With those two additions to my .bashrc I’ve achieved my ideal command history searching. When I hit ctrl-r I’m searching all of my history and yet I only cycle through a shell’s local history with the arrow keys. This small addition1 made my command line productivity higher.

  1. My setup was inspired by this StackExchange post.

Read the whole story
2850 days ago
Must try this
Chicago, USA
Share this story

Lessons learned from reading post mortems

1 Share

I love reading postmortems. They’re educational, but unlike most educational docs, they tell an entertaining story. I’ve spent a decent chunk of time reading postmortems at both Google and Microsoft. I haven’t done any kind of formal analysis on the most common causes of bad failures (yet), but there are a handful of postmortem patterns that I keep seeing over and over again.

Error Handling

Proper error handling code is hard. Bugs in error handling code are a major cause of bad problems. This means that the probability of having sequential bugs, where an error causes buggy error handling code to run, isn’t just the independent probabilities of the individual errors multiplied. It’s common to cascading failures cause a serious outage. There’s a sense in which this is obvious – error handling is generally regarded as being hard. If I mention this to people they’ll tell me how obvious it is that a disproportionate number of serious postmortems come out of bad error handling and cascading failures where errors are repeatedly not handled correctly. But despite this being “obvious”, it’s not so obvious that sufficient test and static analysis effort are devoted to making sure that error handling works.

For more on this, Ding Yuan et al. have a great paper and talk: Simple Testing Can Prevent Most Critical Failures: An Analysis of Production Failures in Distributed Data-Intensive Systems. The paper is basically what it says on the tin. The authors define a critical failure as something that can take down a whole cluster or cause data corruption, and then look at a couple hundred bugs in Cassandra, HBase, HDFS, MapReduce, and Redis, to find 48 critical failures. They then look at the causes of those failures and find that most bugs were due to bad error handling. 92% of those failures are actually from errors that are handled incorrectly.

Drilling down further, 25% of bugs are from simply ignoring an error, 8% are from catching the wrong exception, 2% are from incomplete TODOs, and another 23% are “easily detectable”, which are defined as cases where “the error handling logic of a non-fatal error was so wrong that any statement coverage testing or more careful code reviews by the developers would have caught the bugs”.

The full paper has a lot of gems that that I mostly won’t describe here. For example, they explain the unreasonable effectiveness of Jepsen (98% of critical failures can be reproduced in a 3 node cluster). They also dig into what percentage of failures are non-deterministic (26% of their sample), as well as the causes of non-determinism, and create a static analysis tool that can catch many common error-caused failures.


Configuration bugs, not code bugs, are the most common cause I’ve seen of really bad outages, and nothing else even seems close. When I looked at publicly available postmortems, searching for “global outage postmortem” returned about 50% outages caused by configuration changes. Publicly available postmortems aren’t a representative sample of all outages, but a random sampling of postmortem databases also reveals that config changes are responsible for a disproportionate fraction of extremely bad outages. As with error handling, I’m often told that it’s obvious that config changes are scary, but it’s not so obvious that most companies test and stage config changes like they do code changes.

Except in extreme emergencies, risky code changes are basically never simultaneously pushed out to all machines because of the risk of taking down a service company-wide. But it seems that every company has to learn the hard way that seemingly benign config changes can also cause a company-wide service outage. For example, this was the cause of the infamous November 2014 Azure outage. I don’t mean to pick on MS here; their major competitors have also had serious outages for similar reasons, and they’ve all put processes into place to reduce the risk of that sort of outage happening again.

I don’t mean to pick on large cloud companies, either. If anything, the situation there is better than at most startups, even very well funded ones. Most of the “unicorn” startups that I know of don’t have a proper testing/staging environment that lets them test risky config changes. I can understand why – it’s often hard to set up a good QA environment that mirrors prod well enough that config changes can get tested, and like driving without a seatbelt, nothing bad happens the vast majority of the time. If I had to make my own seatbelt before driving my car, I might not drive with a seatbelt either.


Basically every part of a machine can fail. Many components can also cause data corruption, often at rates that are much higher than advertised. For example, Schroeder, Pinherio, and Weber found DRAM error rates were more than an order of magnitude worse than advertised. The number of silent errors is staggering, and this actually caused problems for Google back before they switched to ECC RAM. Even with error detecting hardware, things can go wrong; relying on ethernet checksums to protect against errors is unsafe and I’ve personally seen malformed packets get passed through as valid packets. At scale, you can run into more undetected errors than you expect, if you expect hardware checks to catch hardware data corruption.

Failover from bad components can also fail. This AWS failure tells a typical story. Despite taking reasonable sounding measures to regularly test the generator power failover process, a substantial fraction of AWS East went down when a storm took out power and a set of backup generators failed to correctly provide power when loaded.


This section should probably be called process error and not human error since I consider having humans in a position where they can cause a catastrophic failure to be a process bug. It’s generally accepted that, if you’re running large scale systems, you have to have systems that are robust to hardware failures. If you do the math on how often machines die, it’s obvious that systems that aren’t robust to hardware failure cannot be reliable. But humans are even more error prone than machines. Don’t get me wrong, I like humans. Some of my best friends are humans. But if you repeatedly put a human in a position where they can cause a catastrophic failure, you’ll eventually get a catastrophe. If humans are asked to repeatedly do tasks that can cause serious problems, serious problems will occur. And yet, the following pattern is still quite common:

Oh, we’re about to do a risky thing! Ok, let’s have humans be VERY CAREFUL about executing the risky operation. Oops! We now have a global outage.

Postmortems that start with “Because this was a high risk operation, foobar high risk protocol was used” are ubiquitous enough that I now think of extra human-operated steps that are done to mitigate human risk as an ops smell. Some common protocols are having multiple people watch or confirm the operation, or having ops people standing by in case of disaster. Those are reasonable things to do, and they mitigate risk to some extent, but in many postmortems I’ve read, automation could have reduced the risk a lot more or removed it entirely. There are a lot of cases where the outage happened because a human was expected to flawlessly execute a series of instructions and failed to do so. That’s exactly the kind of thing that programs are good at! In other cases, a human is expected to perform manual error checking. That’s sometimes harder to automate, and a less obvious win (since a human might catch an error case that the program misses), but in most cases I’ve seen it’s still a net win to automate that sort of thing.

In an IDC survey, respondents voted human error as the most troublesome cause of problems in the datacenter.

One thing I find interesting is how underrepresented human error seems to be in public postmortems. As far as I can tell, Google and MS both have substantially more automation than most companies, so I’d expect their postmortem databases to contain proportionally fewer human error caused outages than I see in public postmortems, but in fact it’s the opposite. My guess is that’s because companies are less likely to write up public postmortems when the root cause was human error enabled by risky manual procedures.

Monitoring / Alerting

The lack of proper monitor is never the sole cause of a problem, but it’s often a serious contributing factor. As is the case for human errors, these seem underrepresented in public postmortems. When I talk to folks at other companies about their worst near disasters, a large fraction of them come from not having the right sort of alerting set up. They’re often saved having a disaster bad enough to require a public postmortem by some sort of ops heroism, but heroism isn’t a scalable solution.

Sometimes, those near disasters are caused by subtle coding bugs, which is understandable. But more often, it’s due to blatant process bugs, like not having a clear escalation path for an entire class of failures, causing the wrong team to debug an issue for half a day, or not having a backup oncall, causing a system to lose or corrupt data for hours before anyone notices when (inevitably) the oncall person doesn’t notice that something’s going wrong.

The Northeast blackout of 2003 is a great example of this. It could have been a minor outage, or even just a minor service degredation, but (among other things) a series of missed alerts caused it to become one of the worst power outages ever.

Not a Conclusion

This is where the conclusion’s supposed to be, but I’d really like to do some serious data analysis before writing some kind of conclusion or call to action. What should I look for? What other major classes of common errors should I consider? These aren’t rhetorical questions and I’m genuinely interested in hearing about other categories I should think about. Feel free to ping me here. I’m also trying to collect public postmortems here.

One day, I’ll get around to the serious analysis, but even without going through and classifying thousands of postmortems, I’ll probably do a few things different as a result of having read a bunch of these. I’ll spend relatively more time during my code reviews on errors and error handling code, and relatively less time on the happy path. I’ll also spend more time checking for and trying to convince people to fix “obvious” process bugs.

One of the things I find to be curious about these failure modes is that when I talked about what I found with other folks, at least one person told me that each process issue I found was obvious. But these “obvious” things still cause a lot of failures. In one case, someone told me that what I was telling them was obvious at pretty much the same time their company was having a global outage of a multi-billion dollar service, caused by the exact thing we were talking about. Just because something is obvious doesn’t mean it’s being done.


Richard Cook’s How Complex Systems Fail takes a more general approach; his work inspired The Checklist Manifesto, which has saved lives.

Allspaw and Robbin’s Web Operations: Keeping the Data on Time talks about this sort of thing in the context of web apps. Allspaw also has a nice post about some related literature from other fields.

There’s a deep history of trying to understand aircraft reliability, and the story of how processes have changed over the decades is fascating, although I’m not sure how to generalize those lessons.


Thanks to Leah Hanson, Anonymous, Marek Majkowski, Nat Welch, and Julia Hansbrough for providing comments on a draft of this. Also, Anonymous, if you prefer to not be anonymous, send me a message on zulip. For anyone keeping score, that’s three folks from Google, one person from Cloudflare, and one anyonymous commenter. I’m always open to comments/criticism, but I’d be especially interested in comments from folks who work at companies with less scale. Do my impressions generalize?

Read the whole story
3256 days ago
Chicago, USA
Share this story

The Toxoplasma Of Rage

2 Comments and 3 Shares

“Nobody makes an IRC channel for no reason. Who are we doing this versus?”
— topic of #slatestarcodex


Some old news I only just heard about: PETA is offering to pay the water bills for needy Detroit families if (and only if) those families agree to stop eating meat.

(this story makes more sense if you know Detroit is currently in a crisis where the bankrupt city government is trying to increase revenues by cracking down on poor people who can’t pay for the water they use.) use. But it still doesn’t make much sense.)

Predictably, the move has caused a backlash. The International Business Times, in what I can only assume is an attempted pun, describes them as “drowning in backlash”. Groundswell thinks it’s a “big blunder”. Daily Banter says it’s “exactly why everyone hates PETA”. Jezebel calls them “assholes”, and we can all agree Jezebel knows a thing or two about assholery.

Of course, this is par for the course for PETA, who have previously engaged in campaigns like throwing red paint on fashion models who wear fur, juxtaposing pictures of animals with Holocaust victims, juxtaposing pictures of animals with African-American slaves, and ads featuring naked people that cross the line into pornography.

People call these things “blunders”, but consider the alternative. Vegan Outreach is an extremely responsible charity doing excellent and unimpeachable work in the same area PETA is. Nobody has heard of them. Everybody has heard of PETA, precisely because of the interminable stupid debates about “did this publicity stunt cross the line?”

Consider. While not everyone is a vegan, pretty much everybody who knows anything about factory farming is upset by it. There is pretty much zero room for PETA to convert people from pro-factory-farming to anti-factory-farming, because there aren’t any radical grassroot pro-factory-farming activists to be found. Their problem isn’t lack of agreement. It’s lack of publicity.

PETA creates publicity, but at a cost. Everybody’s talking about PETA, which is sort of like everybody talking about ethical treatment of animals, which is sort of a victory. But most of the talk is “I hate them and they make me really angry.” Some of the talk is even “I am going to eat a lot more animals just to make PETA mad.”

So there’s a tradeoff here, with Vegan Outreach on one side and PETA on the other.

Vegan Outreach can get everyone to agree in principle that factory-farming is bad, but no one will pay any attention to it.

And PETA can get everyone to pay attention to factory farming, but a lot of people who would otherwise oppose it will switch to supporting it just because they’re so mad at the way it’s being publicized.

But at least they’re paying attention!

PETA doesn’t shoot themselves in the foot because they’re stupid. They shoot themselves in the foot because they’re traveling up an incentive gradient that rewards them for doing so, even if it destroys their credibility.


The University of Virginia rape case profiled in Rolling Stone has fallen apart. In doing so, it joins a long and distinguished line of highly-publicized rape cases that have fallen apart. Studies often show that only People sometimes say that 2 to 8 percent of rape allegations are false. Yet the rate for allegations that go ultra-viral in the media must be an order of magnitude higher than this. As the old saying goes, once is happenstance, twice is coincidence, three times is enemy action.

The enigma is complicated by the observation that it’s usually feminist activists who are most instrumental in taking these stories viral. It’s not some conspiracy of pro-rape journalists choosing the most dubious accusations in order to discredit public trust. It’s people specifically selecting these incidents as flagship cases for their campaign that rape victims need to be believed and trusted. So why are the most publicized cases so much more likely to be false than the almost-always-true average case?

Several people have remarked that false accusers have more leeway to make their stories as outrageous and spectacular as possible. But I want to focus on two less frequently mentioned concerns.

where, exactly, does enemy action enter the picture?

There are a couple issues at play here, but I’ll focus on two.

The Consequentialism FAQ explains signaling in moral decisions like so:

When signaling, the more expensive and useless the item is, the more effective it is as a signal. Although eyeglasses are expensive, they’re a poor way to signal wealth because they’re very useful; a person might get them not because ey is very rich but because ey really needs glasses. On the other hand, a large diamond is an excellent signal; no one needs a large diamond, so anybody who gets one anyway must have money to burn.

Certain answers to moral dilemmas can also send signals. For example, a Catholic man who opposes the use of condoms demonstrates to others (and to himself!) how faithful and pious a Catholic he is, thus gaining social credibility. Like the diamond example, this signaling is more effective if it centers upon something otherwise useless. If the Catholic had merely chosen not to murder, then even though this is in accord with Catholic doctrine, it would make a poor signal because he might be doing it for other good reasons besides being Catholic – just as he might buy eyeglasses for reasons beside being rich. It is precisely because opposing condoms is such a horrendous decision that it makes such a good signal.

But in the more general case, people can use moral decisions to signal how moral they are. In this case, they choose a disastrous decision based on some moral principle. The more suffering and destruction they support, and the more obscure a principle it is, the more obviously it shows their commitment to following their moral principles absolutely. For example, Immanuel Kant claims that if an axe murderer asks you where your best friend is, obviously intending to murder her when he finds her, you should tell the axe murderer the full truth, because lying is wrong. This is effective at showing how moral a person you are – no one would ever doubt your commitment to honesty after that – but it’s sure not a very good result for your friend.

In the same way, publicizing how strongly you believe an accusation that is obviously true signals nothing. Even hard-core anti-feminists would believe a rape accusation that was caught on video. A moral action that can be taken just as well by an outgroup member as an ingroup member is crappy signaling and crappy identity politics. If you want to signal how strongly you believe in taking victims seriously, you talk about it in the context of the least credible case you can find.

But aside from that, there’s the PETA Principle (not to be confused with the Peter Principle). The more controversial something is, the more it gets talked about.

A rape that obviously happened? Shove it in people’s face and they’ll admit it’s an outrage, just as they’ll admit factory farming is an outrage. But they’re not going to talk about it much. There are a zillion outrages every day, you’re going to need something like that to draw people out of their shells.

On the other hand, the controversy over dubious rape allegations is exactly that – a controversy. People start screaming at each other about how they’re misogynist or misandrist or whatever, and Facebook feeds get filled up with hundreds of comments in all capital letters about how my ingroup is being persecuted by your ingroup. At each step, more and more people get triggered and upset. Some of those triggered people do emergency ego defense by reblogging articles about how the group that triggered them are terrible, triggering further people in a snowball effect that spreads the issue further with every iteration.


Only controversial things get spread. A rape allegation will only be spread if it’s dubious enough to split people in half along lines corresponding to identity politics. An obviously true rape allegation will only be spread if the response is controversial enough to split people in half along lines corresponding to identity politics – which is why so much coverage focuses on the proposal that all accused rapists should be treated as guilty until proven innocent.

Everybody hates rape just like everybody hates factory farming. “Rape culture” doesn’t mean most people like rape, it means most people ignore it. That means feminists face the same double-bind that PETA does.

First, they can respond to rape in a restrained and responsible way, in which case everyone will be against it and nobody will talk about it.

Second, they can respond to rape in an outrageous and highly controversial way, in which case everybody will talk about it but it will autocatalyze an opposition of people who hate feminists and obsessively try to prove that as many rape allegations as possible are false.

The other day I saw this on Twitter:

My first thought was that it was witty and hilarious. My second thought was “But when people are competing to see who can come up with the wittiest and most hilarious quip about why we should disbelieve rape victims, something has gone horribly wrong.” My third thought was the same as my second thought, but in ALL CAPS, all caps, because at that point I had read the replies at the bottom. bottom.

I have yet to see anyone holding a cardboard sign talking about how they are going to rape people just to make feminists mad, but it’s only a matter of time. Like PETA, their incentive gradient dooms them to shoot themselves in the foot again and again.


Slate recently published an article about white people’s contrasting reactions to the Michael Brown shooting in Ferguson versus the Eric Garner choking in NYC. And man, it is some contrast.

A Pew poll found that of white people who expressed an opinion about the Ferguson case, 73% sided with the officer. Of white people who expressed an opinion about the Eric Garner case, 63% sided with the black victim.

Media opinion follows much the same pattern. Arch-conservative Bill O’Reilly said he was “absolutely furious” about the way “the liberal media” and “race hustlers” had “twisted the story” about Ferguson in the service of “lynch mob justice” and “insulting the American police community, men and women risking their lives to protect us”. But when it came to Garner, O’Reilly said he was “extremely troubled” and that “there was a police overreaction that should have been adjudicated in a court of law.” His guest on FOX News, conservative commentator and fellow Ferguson-detractor Charles Krauthammer added that “From looking at the video, the grand jury’s decision [not to indict] is totally incomprehensible.” Saturday Night Live did a skit about Al Sharpton talking about the Garner case and getting increasingly upset because “For the first time in my life, everyone agrees with me.”

This follows about three months of most of America being at one another’s throats pretty much full-time about Ferguson. We got treated to a daily diet of articles like Ferguson Protester On White People: “Y’all The Devil” or Black People Had The Power To Fix The Problems In Ferguson Before The Brown Shooting – They Failed or Most White People In America Are Completely Oblivious and a whole bunch of people sending angry racist editorials and counter-editorials to each other for months. The damage done to race relations is difficult to overestimate – CBS reports that they dropped ten percentage points to the lowest point in twenty years, with over half of blacks now describing race relations as “bad”.

And people say it was all worth it, because it raised awareness of police brutality against black people, and if that rustles some people’s jimmies, well, all the worse for them.

But the Eric Garner case also would have raised awareness of police brutality against black people, and everybody would have agreed about it. It has become increasingly clear that, given sufficiently indisputable evidence of police being brutal to a black person, pretty much everyone in the world condemns it equally strongly.

And it’s not just that the Eric Garner case came around too late so we had to make do with the Mike Brown case. Garner was choked a month before Brown was shot, but the story was ignored, then dug back up later as a tie-in to the ballooning Ferguson narrative.

More important, unarmed black people are killed by police or other security officers about twice a week according to official statistics, and probably much more often than that. You’re saying none of these shootings, hundreds each year, made as good a flagship case as Michael Brown? In all this gigantic pile of bodies, you couldn’t find one of them who hadn’t just robbed a convenience store? Not a single one who didn’t have ten eyewitnesses and the forensic evidence all saying he started it?

I propose that the Michael Brown case went viral – rather than the Eric Garner case or any of the hundreds of others – because of the PETA Principle. It was controversial. A bunch of people said it was an outrage. A bunch of other people said Brown totally started it, and the officer involved was a victim of a liberal media that was hungry to paint his desperate self-defense as racist, and so the people calling it an outrage were themselves an outrage. Everyone got a great opportunity to signal allegiance to their own political tribe and discuss how the opposing political tribe were vile racists / evil race-hustlers. There was a steady stream of potentially triggering articles to share on Facebook to provoke your friends and enemies to counter-share articles that would trigger you.

The Ferguson protesters say they have a concrete policy proposal – they want cameras on police officers. There’s only spotty polling on public views of police body cameras before the Ferguson story took off, but what there is seems pretty unaninimous. A UK poll showed that 90% of the population of that country wanted police to have body cameras in February. US polls are more of the form “crappy poll widget on a news site” (1, 2, 3) but they all hovered around 80% approval for the past few years. I also found a poll by Police Magazine in which a plurality of the police officers they surveyed wanted to wear body cameras, probably because of evidence that they cut down on false accusations. Even before Ferguson happened, you would have a really hard time finding anybody in or out of uniform who thought police cameras were a bad idea.

And now, after all is said and done, ninety percent of people are still in favor – given methodology issues, the extra ten percent may or may not represent a real increase. The difference between whites and blacks is a rounding error. The difference between Democrats and Republicans is barely worth talking about- 79% of Republicans are still in support. The people who think Officer Darren Wilson is completely innocent and the grand jury was right to release him, the people muttering under their breath about race hustlers and looters – eighty percent of those people still want cameras on their cops.

If the Ferguson protests didn’t do much to the public’s views on police body cameras, they sure changed its views on some other things. I wrote before about how preliminary polls say that hearing about Ferguson increased white people’s confidence in the way the police treat race. Now the less preliminary polls are out, and they show the effect was larger than even I expected.


White people’s confidence in the police being racially unbiased increased from 35% before the story took off to 52% today. Could even a deliberate PR campaign by the nation’s police forces have done better? I doubt it.

It’s possible that this is an artifact of the question’s wording – after all, it asks people about their local department, and maybe after seeing what happened in Ferguson, people’s local police forces look pretty good by comparison. But then why do black people show the opposite trend?

I think this is exactly what it looks like. Just as PETA’s outrageous controversial campaign to spread veganism make people want to eat more animals in order to spite them, so the controversial nature of this particular campaign against police brutality and racism made white people like their local police department even more to spite the people talking about how all whites were racist.

Once again, the tradeoff.

If campaigners against police brutality and racism were extremely responsible, and stuck to perfectly settled cases like Eric Garner, everybody would agree with them but nobody would talk about it.

If instead they bring up a very controversial case like Michael Brown, everybody will talk about it, but they will catalyze their own opposition and make people start supporting the police more just to spite them. More foot-shooting.


Here is a graph of some of the tags I commonly use for my posts, with the average number of hits per post in each tag. It’s old, but I don’t want to go through the trouble of making a new one, and the trends have stayed the same since then.

I blog about charity only rarely, but it must be the most important thing I can write about here. Convincing even a few more people to donate to charity, or to redirect their existing donations to a more effective program, can literally save dozens or even hundreds of lives even with the limited reach that a private blog has. It probably does more good for the world than all of the other categories on here combined. But it’s completely uncontroversial – everyone agrees it’s a good thing – and it is the least viewed type of post.

Compare this to the three most viewed category of post. Politics is self-explanatory. Race and gender are a type of politics even more controversial and outrage-inducing than regular politics. And that “regret” all the way on the right is my “things i will regret writing” tag, for posts that I know are going to start huge fights and probably get me in lots of trouble. They’re usually race and gender as well, but digging deep into the really really controversial race and gender related issues.

The less useful, and more controversial, a post here is, the more likely it is to get me lots of page views.

For people who agree with me, my angry rants on identity politics are a form of ego defense, saying “You’re okay, your in-group was in the right the whole time.” Linking to it both raises their status as an in-group members, and acts as a potential assault on out-group members who are now faced with strong arguments telling them they’re wrong.

As for the people who disagree with me, they’ll sometimes write angry rebuttals on their own blogs, and those rebuttals will link to my own post as often as not. Or they’ll talk about it with their disagreeing friends, and their friends will get mad and want to tell me I’m wrong, and come over here to read the post to get more ammunition for their counterarguments. I have a feature that allows me to see who links to all of my posts, so I can see this all happening in real-time.

I don’t make enough money off the ads on this blog to matter very much. But if I did, and this was my only means of subsistence, which do you think I’d write more of? Posts about charity which only get me 2,000 paying customers? Or posts that turn all of you against one another like a pack of rabid dogs, and get me 16,000?

I don’t have a fancy bar graph for them, but I bet this same hierarchy of interestingness applies to the great information currents and media outlets that shape society as a whole.

It’s in activists’ interests to destroy their own causes by focusing on the most controversial cases and principles, the ones that muddy the waters and make people oppose them out of spite. And it’s in the media’s interest to help them and egg them on.


And now, for something completely different.

Before “meme” meant doge and all your base, it was a semi-serious attempt to ground cultural evolution in parasitology. The idea was to replace a model of humans choosing whichever ideas they liked with a model of ideas as parasites that evolved in ways that favored their own transmission. This never really caught on, because most people’s response was “That’s neat. So what?”

But let’s talk about toxoplasma.

Toxoplasma is a neat little parasite that is implicated in a couple of human diseases including schizophrenia. Its life cycle goes like this: it starts in a cat. The cat poops it out. The poop and the toxoplasma get in the water supply, where they are consumed by some other animal, often a rat. The toxoplasma morphs into a rat-compatible form and starts reproducing. Once it has strength in numbers, it hijacks the rat’s brain, convincing the rat to hang out conspicuously in areas where cats can eat it. After a cat eats the rat, the toxoplasma morphs back into its cat compatible form and reproduces some more. Finally, it gets pooped back out by the cat, completing the cycle.

It’s the ciiiiiircle of life!

What would it mean for a meme to have a life cycle as complicated as toxoplasma?

Consider the war on terror. It’s a truism that each time the United States bombs Pakistan or Afghanistan or somewhere, all we’re doing is radicalizing the young people there and making more terrorists. Those terrorists then go on to kill Americans, which makes Americans get very angry and call for more bombing of Pakistan and Afghanistan.

Taken as a meme, it is a single parasite with two hosts and two forms. In an Afghan host, it appears in a form called ‘jihad’, and hijacks its host into killing himself in order to spread it to its second, American host. In the American host it morphs in a form called ‘the war on terror’, and it hijacks the Americans into giving their own lives (and several bajillion of their tax dollars) to spread it back to its Afghan host in the form of bombs.

From the human point of view, jihad and the War on Terror are opposing forces. From the memetic point of view, they’re as complementary as caterpillars and butterflies. Instead of judging, we just note that somehow we accidentally created a replicator, and replicators are going to replicate until something makes them stop.

Replicators are also going to evolve. Some Afghan who thinks up a particularly effective terrorist strategy helps the meme spread to more Americans as the resulting outrage fuels the War on Terror. When the American bombing heats up, all of the Afghan villagers radicalized in by the attack will remember the really effective new tactic that Khalid thought up and do that one instead of the boring old tactic that barely killed any Americans at all. Some American TV commentator who comes up with a particularly stirring call to retaliation will find her words adopted into party platforms and repeated by pro-war newspapers. While pacifists on both sides work to defuse the tension, the meme is engaging in a counter-effort to become as virulent as possible, until people start suggesting putting pork fat in American bombs just to make Muslims even madder.

So let’s talk about Tumblr.

Tumblr’s interface doesn’t allow you to comment on other people’s posts, per se. Instead, it lets you reblog them with your own commentary added. So if you want to tell someone they’re an idiot, your only option is to reblog their entire post to all your friends with the message “you are an idiot” below it.

Whoever invented this system either didn’t understand memetics, or understood memetics much too well.

What happens is – someone makes a statement which is controversial by Tumblr standards, like “Protect Doctor Who fans from kitten pic sharers at all costs.” A kitten pic sharer sees the statement, sees red, and reblogs it to her followers with a series of invectives against Doctor Who fans. Since kitten pic sharers cluster together in the social network, soon every kitten pic sharer has seen the insult against kitten pic sharer – as they all feel the need to add their defensive commentary to it, soon all of them are seeing it from ten different directions. The angry invectives get back to the Doctor Who fans, and now they feel deeply offended, so they reblog it among themselves with even more condemnations of the kitten pic sharers, who now not only did whatever inspired the enmity in the first place, but have inspired extra hostility because their hateful invectives are right there on the post for everyone to see. So about half the stuff on your dashboard is something you actually want to see, and the other half is towers of alternate insults that look like this:

Actually, actually, pretty much this happened to the PETA story I started off with

And then you sigh and scroll down to the next one. Unless of course you are a Doctor Who fan, in which case you sigh and then immediately reblog with the comment “It’s obvious you guys started ganging up against us first, don’t try to accuse **US** now” because you can’t just let that accusation stand.

I make fun of Tumblr social justice sometimes, but the problem isn’t with Tumblr social justice, it’s structural. Every community on Tumblr somehow gets enmeshed with the people most devoted to making that community miserable. The tiny Tumblr rationalist community somehow attracts, concentrates, and constantly reblogs stuff from the even tinier Tumblr community of people who hate rationalists and want them to be miserable (no, well-intentioned and intelligent critics, I am not talking about you). It’s like one of those rainforest ecosystems where every variety of rare endangered nocturnal spider hosts a parasite who has evolved for millions of years solely to parasitize that one spider species, and the parasites host parasites who have evolved for millions of years solely to parasitize them. If Tumblr social justice is worse than anything else, it’s mostly because everyone has a race and a gender so it’s easier to fire broad cannonades and just hit everybody.

Tumblr’s reblog policy makes it a hothouse for toxoplasma-style memes that spread via outrage. Following the ancient imperative of evolution, if memes spread by outrage they adapt to become as outrage-inducing as possible.

Or rather, that is just one of their many adaptations. I realize this toxoplasma metaphor sort of strains credibility, so I want to anchor this idea of outrage-memes in pretty much the only piece of memetics everyone can agree upon.

The textbook example of a meme – indeed, almost the only example ever discussed – is the chain letter. “Send this letter to ten people and you will prosper. Fail to pass it on, and you will die tomorrow.” And so the letter replicates.

It might be useful evidence that we were on the right track here, with our toxoplasma memes and everything, if we could find evidence that they reproduced in the same way.

If you’re not on Tumblr, you might have missed the “everyone who does not reblog the issue du jour is trash” wars. For a few weeks around the height of the Ferguson discussion, people constantly called out one another for not reblogging enough Ferguson-related material, or (Heavens forbid) saying they were sick of the amount of Ferguson material they were seeing. It got so bad that various art blogs that just posted pretty paintings, or kitten picture blogs that just reblogged pictures of kittens were feeling the heat (you thought I was joking about the hate for kitten picture bloggers. I never joke.) Now the issue du jour seems to be Pakistan. Just to give a few examples:

“friends if you are reblogging things that are not about ferguson right now please queue them instead. please pay attention to things that are more important. it’s not the time to talk about fandoms or jokes it’s time to talk about injustices.” [source]

“can yall maybe take some time away from reblogging fandom or humor crap and read up and reblog pakistan because the privilege you have of a safe bubble is not one shared by others” [source]

“If you’re uneducated, do not use that as an excuse. Do not say, “I’m not picking sides because I don’t know the full story,” because not picking a side is supporting Wilson. And by supporting him, you are on a racist side…Ignoring this situation will put you in deep shit, and it makes you racist. If you’re not racist, do not just say “but I’m not racist!!” just get educated and reblog anything you can.” [source]

“why are you so disappointing? I used to really like you. you’ve kept totally silent about peshawar, not acknowledging anything but fucking zutara or bellarke or whatever. there are other posts you’ve reblogged too that I wouldn’t expect you to- but those are another topic. I get that you’re 19 but maybe consider becoming a better fucking person?” [source]

“if you’re white, before you reblog one of those posts that’s like “just because i’m not blogging about ferguson doesn’t mean i don’t care!!!” take a few seconds to: consider the privilege you have that allows you not to pay attention if you don’t want to. consider those who do not have the privilege to focus on other things. ask yourself why you think it’s more important that people know you “care” than it is to spread information and show support. then consider that you are a fucking shitbaby.” [source]

“For everyone reblogging Ferguson, Ayotzinapa, North Korea etc and not reblogging Peshawar, you should seriously be ashamed of yourselves.” [source]

“This is going to be an unpopular opinion but I see stuff about ppl not wanting to reblog ferguson things and awareness around the world because they do not want negativity in their life plus it will cause them to have anxiety. They come to tumblr to escape n feel happy which think is a load of bull. There r literally ppl dying who live with the fear of going outside their homes to be shot and u cant post a fucking picture because it makes u a little upset?? I could give two fucks about internet shitlings.” [source]

You may also want to check the Tumblr tag “the trash is taking itself out”, in which hundreds of people make the same joke (“I think some people have stopped reading my blog because I’m talking too much about [the issue du jour]. I guess the trash is taking itself out now.”)

This is pretty impressive. It’s the first time outside of a chain letter that I have seen our memetic overlords throw off all pretense and just go around shouting “SPREAD ME OR YOU ARE GARBAGE AND EVERYONE WILL HATE YOU.”

But it only works because it’s tapped into the most delicious food source an ecology of epistemic parasites could possibly want – controversy, controversy.

Your speech contributes to a much more interesting system!

I would like to be able to write about charity more often. Feminists would probably like to start supercharging the true rape accusations for a change. Protesters against police brutality would probably like to be able to focus on clear-cut cases that won’t make white people support the police even harder. Even PETA would probably prefer being the good guys for once. But the odds aren’t good. Not because the people involved are bad people who want to fail. Not even because the media-viewing public are stupid. Just because information ecologies are not your friend.

This blog tries to remember the Litany of Jai: “Almost no one is evil; almost everything is broken”. We pretty much never wrestle with flesh and blood; it’s powers and principalities all the way down.


…but one of them tends to come up suspiciously often.

A while ago I wrote a post called Meditations on Moloch where I pointed out that in any complex multi-person system, the system acts according to its own chaotic incentives that don’t necessarily correspond to what any individual within the system wants. The classic example is the Prisoner’s Dilemma, which usually ends at defect-defect even though both of the two prisoners involved prefer cooperate-cooperate. I compare this malignant discoordination to Ginsberg’s portrayal of Moloch, the demon-spirit of capitalism gone wrong.

Steven in his wisdom reminds us that there is no National Conversation Topic Czar. The rise of some topics to national prominence and the relegation of others to tiny print on the eighth page of the newspapers occurs by an emergent uncoordinated process. When we say “the media decided to cover Ferguson instead of Eric Garner”, we reify and anthropomorphize an entity incapable of making goal-directed decisions.

A while back there was a minor scandal over JournoList, a private group where left-leaning journalists met and exchanged ideas. I think the conservative spin was “the secret conspiracy running the liberal media – revealed!” I wish they had been right. If there were a secret conspiracy running the liberal media, they could all decide they wanted to raise awareness of racist police brutality, pick the most clear-cut and sympathetic case, and make it non-stop news headlines for the next two months. Then everyone would agree it was indeed very brutal and racist, and something would get done.

But as it is, even if many journalists are interested in raising awareness of police brutality, given their total lack of coordination there’s not much they can do. An editor can publish a story on Eric Garner, but in the absence of a divisive hook, the only reason people will care about it is that caring about it is the right thing and helps people. But that’s “charity”, and we already know from my blog tags that charity doesn’t sell. A few people mumble something something deeply distressed, but neither black people nor white people get interested, in the “keep tuning to their local news channel to get the latest developments on the case” sense.

The idea of liberal strategists sitting down and choosing “a flagship case for the campaign against police brutality” is poppycock. Moloch – the abstracted spirit of discoordination and flailing response to incentives – will publicize whatever he feels like publicizing. And if they want viewers and ad money, the media will go along with him.

Which means that it’s not a coincidence that the worst possible flagship case for fighting police brutality and racism is the flagship case that we in fact got. It’s not a coincidence that the worst possible flagship cases for believing rape victims are the ones that end up going feminists end up bringing viral. It’s not a coincidence that the only time we ever hear about factory farming is when somebody’s doing something that makes us almost sympathetic to it. It’s not coincidence, it’s not even happenstance, it’s enemy action. Under Moloch, activists are irresistably incentivized to dig their own graves. And the media is irresistably incentivized to help them.

Lost is the ability to agree on simple things like fighting factory farming or rape. Lost is the ability to even talk about the things we all want. Ending corporate welfare. Ungerrymandering political districts. Defrocking pedophile priests. Stopping prison rape. Punishing government corruption and waste. Feeding starving children. Simplifying the tax code.

But also lost is our ability to treat each other with solidarity and respect.

Under Moloch, everyone is irresistably incentivized to ignore the things that unite us in favor of forever picking at the things that divide us in exactly the way that is most likely to make them more divisive. . Race relations are at historic lows not because white people and black people disagree on very much, but because the media absolutely worked its tuchus off to find the single issue that white people and black people disagreed over the most and ensure that it was make that the only issue anybody would talk about. Men’s rights activists and feminists hate each other not because there’s a huge divide in how people of different genders think, but because only the most extreme examples of either side will ever gain traction, and those only when they are framed as attacks on the other side.

People talk about the shift from old print-based journalism to the new world of social media and the sites adapted to serve it. These are fast, responsive, and only just beginning to discover the power of controversy. They are memetic evolution shot into hyperdrive, and the omega point is a well-tuned machine optimized to search the world for the most controversial and counterproductive issues, then make sure no one can talk about anything else. An engine that creates money by burning the few remaining shreds of cooperation, bipartisanship and social trust.

Imagine Moloch, in his Carthaginian-demon personification, looking out over the expanse of the world, eagle-eyed for anything that can turn brother against brother and husband against wife. Finally he decides “YOU KNOW WHAT NOBODY HATES EACH OTHER ABOUT YET? BIRD-WATCHING. LET ME FIND SOME STORY THAT WILL MAKE PEOPLE HATE EACH OTHER OVER BIRD-WATCHING”. And the next day half the world’s newspaper headlines are “Has The Political Correctness Police Taken Over Bird-Watching?” and the other half are “Is Bird-Watching Racist?”. And then bird-watchers and non-bird-watchers and different sub-groups of bird-watchers hold vitriolic attacks on each other that feed back on each other in a vicious cycle for the next six months, and the whole thing ends in mutual death threats and another previously innocent activity turning into World War I style trench warfare.


Read the whole story
3476 days ago
I think he nailed it.
Chicago, USA
Share this story

Parable of the Polygons - a playable post on the shape of society

2 Comments and 3 Shares

This is a story of how harmless choices can make a harmful world.

These little cuties are 50% Triangles, 50% Squares, and 100% slightly shapist. But only slightly! In fact, every polygon prefers being in a diverse crowd:

You can only move them if they're unhappy with their immediate neighborhood. Once they're OK where they are, you can't move them until they're unhappy with their neighbors again. They've got one, simple rule:

“I wanna move if less than 1/3 of my neighbors are like me.”

Harmless, right? Every polygon would be happy with a mixed neighborhood. Surely their small bias can't affect the larger shape society that much? Well...

drag & drop unhappy polygons until nobody is unhappy:
(just move them to random empty spots. don't think too much about it.)

And... our shape society becomes super segregated. Daaaaang.

Sometimes a neighborhood just becomes square, and it's not their fault if no triangles wanna stick around. And a triangular neighborhood would welcome a square, but they can't help it if squares ain't interested.

In this next bit, unhappy shapes automatically move to random empty spots. There's also a graph that tracks how much segregation there is over time.

run this simulation a few times. what happens?

What's up with that? These are good shapes, nice shapes. And yet, though every individual only has a slight bias, the entire shape society cracks and splits.

Small individual bias can lead to large collective bias.

Equality is an unstable equilibrium. The smallest of bias can push a whole society past the tipping point. Well, what if we taught these shapes to have zero bias? (Or if you're feeling particularly nasty today, more bias?)

use the slider to adjust the shapes' individual bias:

Notice how much more segregated things become, when you increase the bias beyond 33%. What if the threshold was at 50%? Seems reasonable for a shape to prefer not being in the minority...

So yeah, just turn everyone's bias down to zero, right? Haha, NOPE. The real world doesn't start anew with a random shuffling of citizens every day. Everyday, you're not shuffling.

world starts segregated. what happens when you lower the bias?

See what doesn't happen? No change. No mixing back together. In a world where bias ever existed, being unbiased isn't enough! We're gonna need active measures. What if shapes wanted to seek out just a lil' more variety?

Woah. Even though each polygon would be okay with having up to 90% of their neighbors that are like them, they all mix together! Let's see this play out on a larger scale, when we change the amount of bias and anti-bias for all shapes.

world starts segregated. what happens when shapes demand even the smallest bit of diversity?

All it takes is a change in the perception of what an acceptable environment looks like. So, fellow shapes, remember it's not about triangles vs squares, it's about deciding what we want the world to look like, and settling for no less.

(hint: don't move them straight to the box; keep the pairs close together)

At first, going out on your own can be isolating... but by working together, step by step, we'll get there.

finally, a big ol' sandbox to play around in.


1. Small individual bias → Large collective bias.
When someone says a culture is shapist, they're not saying the individuals in it are shapist. They're not attacking you personally.

2. The past haunts the present.
Your bedroom floor doesn't stop being dirty just coz you stopped dropping food all over the carpet. Creating equality is like staying clean: it takes work. And it's always a work in progress.

3. Demand diversity near you.
If small biases created the mess we're in, small anti-biases might fix it. Look around you. Your friends, your colleagues, that conference you're attending. If you're all triangles, you're missing out on some amazing squares in your life - that's unfair to everyone. Reach out, beyond your immediate neighbors.

Thank you for playing this blog post!

Our cute segregation sim is based off the work of Nobel Prize-winning game theorist, Thomas Schelling. Specifically, his 1971 paper,

Dynamic Models of Segregation.

We built on top of this, and showed how a small demand for diversity can desegregate a neighborhood. In other words, we gave his model a happy ending.

Schelling's model gets the general gist of it, but of course, real life is more nuanced. You might enjoy looking at real-world data, such as W.A.V. Clark's 1991 paper,

A Test of the Schelling Segregation Model


There are other mathematical models of institutionalized bias out there!

Male-Female Differences: A Computer Simulation

shows how a small gender bias compounds as you move up the corporate ladder.

The Petrie Multiplier

shows why an attack on sexism in tech is


an attack on men.

Today's Big Moral Message™ is that demanding a bit of diversity in your spaces makes a huge difference overall. Look at

Plz Diversify Your Panel

, an initiative where overrepresented speakers pledge not to speak on panels without diverse representation.

Our "playable post" was inspired by Bret Victor's

Explorable Explanations

and Ian Bogost's

procedural rhetoric


Donate to Diversity!
Black Girls Code - gives coding lessons to girls of color
Girls Who Code - teaches high school girls to code
Code 2040 - helps blacks & latina/os get into tech
Code Liberation - free workshops to help women make videogames
Ada Initiative - supports women in open source & open culture
Nicky's Patreon - makes public domain playables (such as this one!)

Thank you to our beta-readers:
Andrea, Astrid, Catherine, Chris, Emily, Glen, Jocelyn, Laura, Marc, Marko, Zak
Also Seen On:
WIRED, Washington Post, BoingBoing, Creative Commons, KillScreen, JayIsGames, Hacker News, MetaFilter, New York Magazine, The Atlantic's CityLab, Salon, Polygon
Spanish, French, German, Brazilian Portuguese, Japanese
Things Based Off This Thing:
Polygons with Pentagons, Playthrough Video

Signed in as srsly

Share this story on NewsBlur

Shared stories are on their way...

Read the whole story
3508 days ago
Chicago, USA
Share this story
2 public comments
3508 days ago
Yes, there is a larger social message here, but my take away is that dynamic systems are sensitive to initial conditions.
3508 days ago
This is good stuff - click though and play with the models! It is very cute too!
Atlanta, Georgia

Copyright is Out of Control

6 Comments and 14 Shares

I have written about patent and copyright law primarily from the perspective ofan economist interested in the institutions and incentives that maximize innovation. As a textbook author, however, I must deal with copyright law in practice. Dealing with copyright law on the ground hasn’t caused me to change my views but it has made me more frustrated. I have also come to appreciate some of the subtler more subtle costs of the system. Two cases in point.

A lot of textbooks hire a photo editor to pick generic stock photos, this simplifies things because the bundlers pre-authorize permissions and prices. But we hand picked every photo in our book to illustrate a point which means that our permissions and legal staff often have to find owners and clear permissions on an individual basis. We are grateful that our publisher is willing to do this to produce a quality product but it sometimes leads to absurdities. For example, the publisher doesn’t like to use public domain images. Why not? What could be better than free? The problem is that the bundlers insulate a publisher from lawsuits but when we use a public domain image the publisher is open to lawsuit if a mistake has been made and that makes them fearful.

The general lesson is that strong IP shrinks the public domain not just because it keeps things out of the public domain but also because it makes the public domain appear to be uncertain and dangerous. It’s as if clean, mountain spring water were freely available but people bought from the bottlers instead out of fear of contamination.

Copyright law is one of the forces behind the rise of the mega-bundlers. Mega-bundlers benefit from economies of scale in cataloging IP but there are also economies of scale in dealing with the legal system and insuring against/for lawsuit. It’s probably no accident that two of the largest bundlers, Corbis and Getty, are owned by Bill Gates and (Getty heir), Mark Getty respectively. (FYI, Piketty should have said more about this kind of 21st century rentier in Capital).

Here is another example. To illustrate the point that, contrary to what is often argued, a rich person might get more from another dollar than a poor person we have in Modern Principles a movie still of Scrooge McDuck swimming in money. We think the image speaks for itself but apparentlythatis a problem. The rights to thephoto are–we are told–not the same as the rights to the characters shown within the photo. Thus, even though we have bought and paid for the right to print the photo, to ensure that the use of the characters within the photo falls under fair use we must discuss, comment on and critique the content of the photo in the text.

The distinction between the photo IP and the what’s in the photoIP is one only a lawyer could appreciate, as is the solution. And I mean that without irony. I am not critiquing our publisher or their lawyers. Bear in mind that this is coming to us from the very highest legal counsel of a multi-billion dollar firm. Thus, I do not doubt that the dangers are real and the legal analysis acute. The problem is copyright law itself.

The episode illustrates more generally how the complexity of copyright law has greatly elevated the power of lawyers. It’s no accident that thepermissions director is one of the few people at our publisher whose signature is absolutely necessary before our book, or any book, can be published.

I am reminded of Mancur Olson’s 9th implication in The Rise and Decline of Nations:

The accumulation of distributional coalitions increases the complexity of regulation, the role of government, and the complexity of understandings, and changes the direction of social evolution.

Read the whole story
3741 days ago
Chicago, USA
3741 days ago
Share this story
4 public comments
3740 days ago
The economic rent is too damn high!
Washington, District of Columbia
3741 days ago
Coming from a textbook author........
3741 days ago
Lawyers. Man.
Louisville, KY
3743 days ago
Alex is annoyed at copyright
New York, NY
Next Page of Stories