Get smarter, faster. Subscribe to our daily newsletter.
How viral social media memes trigger real-world violence
What responsibility should government authorities and Big Tech take in policing the spread of sedition-oriented content?
- It can be hard to believe that comical images online are enough to rile people up enough that they'll actually attack.
- Originating in the darker corners of the internet, Bugaloo is now prominent on mainstream online platforms like Facebook and Instagram.
- The Network Contagion Research Institute's recent series of Contagion and Ideology Reports uses machine learning to examine how memes spread.
There has been talk lately about fringe hate groups using memes to incite real world violence, but many prefer to casually dismiss this narrative as hysteria. It can be hard to believe that comical images online are enough to rile people up enough that they'll actually attack, and for most of us, this is thankfully true.
If you spend your time scrolling through pumpkin spice recipes and cat videos, you're probably not the target of a systematic hate campaign. But that doesn't mean these campaigns don't exist. Not every Facebook user is susceptible to radicalization, but memetic warfare is increasingly being recognized as an important element of counter-governmental sedition and racist propaganda.
Studies have long shown that exposure to hate makes it easier for people to hate – and to act on their hate. If there is an organized effort by hate groups to manipulate people with memes, then these visuals can actually be influencing people to perpetrate violent attacks.
What kind of memes were used on the Facebook Event listing that the Kenosha Guard used to mobilize Kyle Rittenhouse?
Many netizens appreciate the dangers and want social media platforms to do much more to curb the spread of hate across their networks. From Snapchat to Gab, TikTok to WhatsApp, memes are fueling hatred and inciting violence, and more Americans are asking what responsibility these companies should take in policing the spread of harmful content.
What’s in a meme?
The Network Contagion Research Institute's recent series of Contagion and Ideology Reports leverages machine learning to examine how memes spread. The idea is to unearth a better understanding of the role memes play in encouraging real-world violence.
So what exactly is a meme, and how did it become a tool for weaponizing the web? Originally coined by Oxford biologist Richard Dawkins in his 1976 book "The Selfish Gene," the term is defined there as "a unit of cultural transmission" that spreads like a virus from host to host and conveys an idea which changes the host's worldview.
Dawkins echoes "Naked Lunch" author William Burroughs's concept of written language as a virus that infects the reader and consequently builds realities that are brought to fruition through the act of speech. In this sense, a meme is an idea that spreads by iterative, collaborative imitation from person to person within a culture and carries symbolic meaning.
The militant alt-right's use of internet memes follows this pattern. They are carefully designed on underground social media using codes, before attaining approval by like-minded users that disseminate these messages on mainstream platforms like Twitter or Facebook.
Sometimes these messages are lifted from innocuous sources and altered to convey hate. Take for example Pepe the Frog. Initially created by cartoonist Matt Furie as a mascot for slackers, Pepe was appropriated and altered by racists and homophobes until the Anti-Defamation League branded the frog as a hate symbol in 2016. This, of course, hasn't stopped public figures from sharing variations of Pepe's likeness in subversive social posts.
The dissemination of hateful memes is often coordinated by an underground movement, so even its uptake by mainstream internet users and media outlets is claimed as a victory by a subculture that prides itself on its trolling.
Slenderman and the Boogaloo Bois
Indeed, these types of memes have a tricky way of moving from the shadows of the internet to the mainstream, picking up supporters along the way – including those who are willing to commit horrific crimes.
This phenomenon extends well beyond politics. Take, for example, the Slenderman. Created as a shadowy figure for online horror stories, Slenderman achieved mainstream popularity and a cult-like following. When two girls tried to stab their friend to death to prove that Slenderman was real in 2014, the power of the meme to prompt violence became a national conversation.
Slenderman creator Victor Surge told Know Your Meme that he never thought the character would spread beyond the fringe Something Awful forums. "An urban legend requires an audience ignorant of the origin of the legend. It needs unverifiable third and forth [sic] hand (or more) accounts to perpetuate the myth," he explained. "On the Internet, anyone is privy to its origins as evidenced by the very public Somethingawful thread. But what is funny is that despite this, it still spread. Internet memes are finicky things and by making something at the right place and time it can swell into an 'Internet Urban Legend.'"
While Slenderman is obviously a fiction, political memes walk many of the same fine lines – and have galvanized similarly obsessed followers to commit attacks in the real world.
Take, for example, the Boogaloo movement. This extremist militia has strong online roots and uses memes to incite violent insurrection and terror against the government and law enforcement. Originating in the darker corners of the internet, Bugaloo is now prominent on mainstream online platforms like Facebook and Instagram. Boogaloo followers share how to build explosives and 3D-printed firearms, spread encrypted messages and distribute violent propaganda.
Boogaloo's intentions seem to be twofold: recruit new followers and communicate tactical information to incumbent members. While finding more people to adopt their ideology is one thing, there is a strong possibility that Boogaloo memes and messaging can be used to trigger sleeper cells to violent action.
Boogaloo isn't alone. The use of memes by groups like the Proud Boys, which operate on an ideology consisting of both symbolic and physical violence, is spreading across America – especially since the start of the pandemic.
Understanding digital indoctrination
The Network Contagion Research Institute (NCRI) uses advanced data analysis to expose hate on social media. The institute's scientists work with experts at the ADL's Center on Extremism (COE) to track hateful propaganda online and offer strategies to combat this phenomenon. By better understanding how memes spread hateful propaganda, the NCRI is helping law enforcement, social media companies and citizens get better at preventing online chatter about violence from becoming real action.
Founded by Princeton's Dr. Joel Finkelstein, the NCRI is finding that the spread of hate online can be examined using epidemiological models of how viruses spread, only applied to language and ideas, much as Burroughs imagined decades ago.
Now that the internet has obviated the need for people to meet in person or communicate directly, recruiting and deploying members of violent groups is easier than ever before.
Before social media, as Finkelstein reminds us, organizing underground hate groups was harder. "You would have to have an interpersonal organization to train and cultivate those people," he said. "With the advance of the internet, those people can find each other; what it lacks in depth it makes up in reach. The entire phenomenon can happen by itself without a trace of anyone being groomed."
Can anything be done?
Memes generate hysteria in part by being outrageous enough for people to share – even the mainstream media. This is a concrete strategy called amplification. The extreme alt-right manages to reach mainstream audiences with racialist and supremacist memes, even as these messages are being denounced.
Content spreads virally whether those spreading it support the embedded ideology or not, making it difficult to intervene and prevent meme-incited violence.
Some people may unwittingly amplify messaging designed to spread violence, mistaking memes like ACAB (all cops are bastards) for harmless if somewhat dark humor.
So what can be done about the meme wars, and what onus is on law enforcement and Big Tech?
Finkelstein recommends a three-pronged approach to combating the violent outcome of subversive memes.
- Push for more stringent boundaries defining civility online using technology.
- Educate courts, lawmakers and civil institutions about how extreme online communities operate and create better industry standards to regulate these groups.
- Bring the federal government, including the FBI and Homeland Security, up to speed on the new reality so they can intervene as necessary.
But not everyone thinks Big Tech or the law have much ground to stand on in combating viral hate. Tech companies haven't gone the mile in truly enforcing stricter standards for online content, but some companies are closing down accounts associated with extremist leaders' websites and their movements.
But this is largely tilting at windmills. The proliferation of memes online spreading ideology virally might be too big to combat without dramatically limiting freedom of speech online. Facebook and YouTube have banned thousands of profiles, but there's no way of knowing how many remain – or are being added every day. This leaves the danger of meme radicalization and weaponization lurking beneath the surface even as we head into elections.
What we stand to lose
Boogaloo and other seditious groups have been empowered by recent crises, including the pandemic and the anti-racist protests sweeping the country. Greater numbers of these community members are showing up at anti-quarantine protests and are disrupting other gatherings, and the press they're getting – this article included – is only bringing more attention to their violent messages.
Extremist memes continue to circulate, as America heads to the polls amidst a global pandemic and widespread civil unrest. It stands to reason that more blood will be spilled. Getting a handle on Twitter handles that spread hate and shutting down groups whose sole purpose is sowing the seeds of brutality is vital – the only question that remains is how.
- Are hate crimes and political discourse connected? - Big Think ›
- Access to guns, not mental health, leads to gun violence - Big Think ›
How would the ability to genetically customize children change society? Sci-fi author Eugene Clark explores the future on our horizon in Volume I of the "Genetic Pressure" series.
- A new sci-fi book series called "Genetic Pressure" explores the scientific and moral implications of a world with a burgeoning designer baby industry.
- It's currently illegal to implant genetically edited human embryos in most nations, but designer babies may someday become widespread.
- While gene-editing technology could help humans eliminate genetic diseases, some in the scientific community fear it may also usher in a new era of eugenics.
Tribalism and discrimination<p>One question the "Genetic Pressure" series explores: What would tribalism and discrimination look like in a world with designer babies? As designer babies grow up, they could be noticeably different from other people, potentially being smarter, more attractive and healthier. This could breed resentment between the groups—as it does in the series.</p><p>"[Designer babies] slowly find that 'everyone else,' and even their own parents, becomes less and less tolerable," author Eugene Clark told Big Think. "Meanwhile, everyone else slowly feels threatened by the designer babies."</p><p>For example, one character in the series who was born a designer baby faces discrimination and harassment from "normal people"—they call her "soulless" and say she was "made in a factory," a "consumer product." </p><p>Would such divisions emerge in the real world? The answer may depend on who's able to afford designer baby services. If it's only the ultra-wealthy, then it's easy to imagine how being a designer baby could be seen by society as a kind of hyper-privilege, which designer babies would have to reckon with. </p><p>Even if people from all socioeconomic backgrounds can someday afford designer babies, people born designer babies may struggle with tough existential questions: Can they ever take full credit for things they achieve, or were they born with an unfair advantage? To what extent should they spend their lives helping the less fortunate? </p>
Sexuality dilemmas<p>Sexuality presents another set of thorny questions. If a designer baby industry someday allows people to optimize humans for attractiveness, designer babies could grow up to find themselves surrounded by ultra-attractive people. That may not sound like a big problem.</p><p>But consider that, if designer babies someday become the standard way to have children, there'd necessarily be a years-long gap in which only some people are having designer babies. Meanwhile, the rest of society would be having children the old-fashioned way. So, in terms of attractiveness, society could see increasingly apparent disparities in physical appearances between the two groups. "Normal people" could begin to seem increasingly ugly.</p><p>But ultra-attractive people who were born designer babies could face problems, too. One could be the loss of body image. </p><p>When designer babies grow up in the "Genetic Pressure" series, men look like all the other men, and women look like all the other women. This homogeneity of physical appearance occurs because parents of designer babies start following trends, all choosing similar traits for their children: tall, athletic build, olive skin, etc. </p><p>Sure, facial traits remain relatively unique, but everyone's more or less equally attractive. And this causes strange changes to sexual preferences.</p><p>"In a society of sexual equals, they start looking for other differentiators," he said, noting that violet-colored eyes become a rare trait that genetically engineered humans find especially attractive in the series.</p><p>But what about sexual relationships between genetically engineered humans and "normal" people? In the "Genetic Pressure" series, many "normal" people want to have kids with (or at least have sex with) genetically engineered humans. But a minority of engineered humans oppose breeding with "normal" people, and this leads to an ideology that considers engineered humans to be racially supreme. </p>
Regulating designer babies<p>On a policy level, there are many open questions about how governments might legislate a world with designer babies. But it's not totally new territory, considering the West's dark history of eugenics experiments.</p><p>In the 20th century, the U.S. conducted multiple eugenics programs, including immigration restrictions based on genetic inferiority and forced sterilizations. In 1927, for example, the Supreme Court ruled that forcibly sterilizing the mentally handicapped didn't violate the Constitution. Supreme Court Justice Oliver Wendall Holmes wrote, "… three generations of imbeciles are enough." </p><p>After the Holocaust, eugenics programs became increasingly taboo and regulated in the U.S. (though some states continued forced sterilizations <a href="https://www.uvm.edu/~lkaelber/eugenics/" target="_blank">into the 1970s</a>). In recent years, some policymakers and scientists have expressed concerns about how gene-editing technologies could reanimate the eugenics nightmares of the 20th century. </p><p>Currently, the U.S. doesn't explicitly ban human germline genetic editing on the federal level, but a combination of laws effectively render it <a href="https://academic.oup.com/jlb/advance-article/doi/10.1093/jlb/lsaa006/5841599#204481018" target="_blank" rel="noopener noreferrer">illegal to implant a genetically modified embryo</a>. Part of the reason is that scientists still aren't sure of the unintended consequences of new gene-editing technologies. </p><p>But there are also concerns that these technologies could usher in a new era of eugenics. After all, the function of a designer baby industry, like the one in the "Genetic Pressure" series, wouldn't necessarily be limited to eliminating genetic diseases; it could also work to increase the occurrence of "desirable" traits. </p><p>If the industry did that, it'd effectively signal that the <em>opposites of those traits are undesirable. </em>As the International Bioethics Committee <a href="https://academic.oup.com/jlb/advance-article/doi/10.1093/jlb/lsaa006/5841599#204481018" target="_blank" rel="noopener noreferrer">wrote</a>, this would "jeopardize the inherent and therefore equal dignity of all human beings and renew eugenics, disguised as the fulfillment of the wish for a better, improved life."</p><p><em>"Genetic Pressure Volume I: Baby Steps"</em><em> by Eugene Clark is <a href="http://bigth.ink/38VhJn3" target="_blank">available now.</a></em></p>
It's hard to stop looking back and forth between these faces and the busts they came from.
- A quarantine project gone wild produces the possibly realistic faces of ancient Roman rulers.
- A designer worked with a machine learning app to produce the images.
- It's impossible to know if they're accurate, but they sure look plausible.
How the Roman emperors got faced<a href="https://payload.cargocollective.com/1/6/201108/14127595/2K-ENGLISH-24x36-Educational_v8_WATERMARKED_2000.jpg" ><img type="lazy-image" data-runner-src="https://assets.rebelmouse.io/eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpbWFnZSI6Imh0dHBzOi8vYXNzZXRzLnJibC5tcy8yNDQ2NDk2MS9vcmlnaW4uanBnIiwiZXhwaXJlc19hdCI6MTYyOTUzMzIxMX0.OwHMrgKu4pzu0eCsmOUjybdkTcSlJpL_uWDCF2djRfc/img.jpg?width=980" id="775ca" class="rm-shortcode" data-rm-shortcode-id="436000b6976931b8320313478c624c82" data-rm-shortcode-name="rebelmouse-image" alt="lineup of emperor faces" data-width="1440" data-height="963" /></a>
Credit: Daniel Voshart<p>Voshart's imaginings began with an AI/neural-net program called <a href="https://www.artbreeder.com" target="_blank">Artbreeder</a>. The freemium online app intelligently generates new images from existing ones and can combine multiple images into…well, who knows. It's addictive — people have so far used it to generate nearly 72.7 million images, says the site — and it's easy to see how Voshart fell down the rabbit hole.</p><p>The Roman emperor project began with Voshart feeding Artbreeder images of 800 busts. Obviously, not all busts have weathered the centuries equally. Voshart told <a href="https://www.livescience.com/ai-roman-emperor-portraits.html" target="_blank" rel="noopener noreferrer">Live Science</a>, "There is a rule of thumb in computer programming called 'garbage in garbage out,' and it applies to Artbreeder. A well-lit, well-sculpted bust with little damage and standard face features is going to be quite easy to get a result." Fortunately, there were multiple busts for some of the emperors, and different angles of busts captured in different photographs.</p><p>For the renderings Artbreeder produced, each face required some 15-16 hours of additional input from Voshart, who was left to deduce/guess such details as hair and skin coloring, though in many cases, an individual's features suggested likely pigmentations. Voshart was also aided by written descriptions of some of the rulers.</p><p>There's no way to know for sure how frequently Voshart's guesses hit their marks. It is obviously the case, though, that his interpretations look incredibly plausible when you compare one of his emperors to the sculpture(s) from which it was derived.</p><p>For an in-depth description of Voshart's process, check out his posts on <a href="https://medium.com/@voshart/photoreal-roman-emperor-project-236be7f06c8f" target="_blank">Medium</a> or on his <a href="https://voshart.com/ROMAN-EMPEROR-PROJECT" target="_blank" rel="noopener noreferrer">website</a>.</p><p>It's fascinating to feel like you're face-to-face with these ancient and sometimes notorious figures. Here are two examples, along with some of what we think we know about the men behind the faces.</p>
Caligula<img type="lazy-image" data-runner-src="https://assets.rebelmouse.io/eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpbWFnZSI6Imh0dHBzOi8vYXNzZXRzLnJibC5tcy8yNDQ2NDk4Mi9vcmlnaW4uanBnIiwiZXhwaXJlc19hdCI6MTY3MzQ1NTE5NX0.LiTmhPQlygl9Fa9lxay8PFPCSqShv4ELxbBRFkOW_qM/img.jpg?width=980" id="7bae0" class="rm-shortcode" data-rm-shortcode-id="ce795c554490fe0a36a8714b86f55b16" data-rm-shortcode-name="rebelmouse-image" data-width="992" data-height="558" />
One of numerous sculptures of Caligula, left
Nero<img type="lazy-image" data-runner-src="https://assets.rebelmouse.io/eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpbWFnZSI6Imh0dHBzOi8vYXNzZXRzLnJibC5tcy8yNDQ2NTAwMC9vcmlnaW4uanBnIiwiZXhwaXJlc19hdCI6MTY1NTQ2ODU0NX0.AgYuQZzRQCanqehSI5UeakpxU8fwLagMc_POH7xB3-M/img.jpg?width=980" id="a8825" class="rm-shortcode" data-rm-shortcode-id="9e0593d79c591c97af4bd70f3423885e" data-rm-shortcode-name="rebelmouse-image" data-width="992" data-height="558" />
One of numerous sculptures of Nero, left
A popular and longstanding wave of thought in psychology and psychotherapy is that diagnosis is not relevant for practitioners in those fields.
Scientists regenerate damaged spinal cord nerve fibers with designer protein, helping paralyzed mice walk again.
- Researchers from Germany use a designer protein to treat spinal cord damage in mice.
- The procedure employs gene therapy to regenerate damaged nerve fibers that carry signals to and from the brain.
- The scientists aim to eventually apply the technique to humans.