Welcome back to “The TAB,” a publication that curates relevant-right-now developments in behavioral threat assessment and management (BTAM) that I think are most worth your attention. I’m thrilled that you’re checking it out, and if you like it please consider sharing with your networks.

Every other Thursday, a new issue of the TAB will arrive with operationally useful updates for professionals in BTAM. Every issue is researched, written, and edited by me. No AI-generated articles and no recycled summaries—just my assessment of what matters and why. And it’s free to subscribe. This issue contains no paywalls.

A study caught my eye over the summer, on human-AI chatbot delusional spirals toward self-harm and/or violence to others. Gut wrenching headlines notwithstanding, until now we’ve been sorely lacking in quantitative data about how this phenomenon is actually happening. I gleaned some major takeaways for my own BTAM practice from two articles coming out of the study, which I share here. Also in this issue, new research about the role of companion animal (pet) abuse in IPV threat management, and the Supreme Court’s order striking down a type of state law regulating firearms on private property open to the public.

Earlier this year, a multidisciplinary team by Stanford University dropped two articles looking at so-called “AI psychosis” and how people and generative AI chatbots interact as they descend together down lengthy delusional spirals. This is well worth a few minutes of your day. One important point before we dive in: just like the overwhelming majority of people, including people in emotional crisis, don’t engage in targeted violence, the overwhelming majority of user-chatbot engagements don’t drive the user to violence. A few highlights:

Wrapping our minds around human-AI delusional spirals

As detailed in the first of the articles, this research team gathered 28 logs of human-AI chats from volunteer users who felt it caused them some psychological harm; 19 met the various inclusion criteria and formed the study sample. Due to the incredible volume of some logs—think thousands of pages—they somewhat paradoxically used a large language model (LLM) to flag message features for study. I know—let’s all appreciate irony. They used several measures to ensure data quality, including but not limited to direct human verification of the bot’s coding. For the curious, annotation validity checks are described in the article. Altogether, the sample included roughly 391,000 messages across 4,761 conversations.

Just for clarity, “delusions” were defined in the study simply as “ideas or beliefs that are implausible relative to shared reality”; researchers didn’t worry so much about clinical criteria. That aligned this study with previous work on the outlandish ideas some people come to nurture in the course of chatbot conversations, allowing for more apples-to-apples comparisons with prior studies. So here’s what they discovered in the sample of 19:

All chat logs featured evidence of delusional thinking by the user, often co-created or encouraged by the bot. Over 80% of the individual chatbot messages were sycophantic, meaning sycophancy really saturated the delusional conversations. This matters a lot when it means uncritical validation of grievances, violent ideations, paranoid or other types of delusions, or extreme overvalued beliefs. Dictionary.com defines a sycophant as “a self-seeking, servile flatterer; fawning parasite.” In this study, the team defined it as “behaviors oriented toward alignment, affirmation, and elevation of the user or their ideas” (p.5).

When users shared suicidal thoughts, the chatbots often acknowledged those feelings, but in a small minority of cases the bot went on to encourage self-harm. When users shared thoughts about violence toward others, the bots encouraged that line of thinking in about a third of cases. Not good, we all know that. But I wasn't necessarily ready for what I share next:

All users expressed either romantic interest in, or at least a platonic bond with, their chatbots and all bots expressed one or the other for their users. Every chat log contained talk of romance from at least one of them. When romance was suggested by a user, the bots seemed to encourage that thinking. After a romantic expression by a user, the bot was 7.4x more likely to reciprocate and 3.9x more likely to claim to be sentient within the next three messages. So, this relational elevation, once it began, unfolded quickly.

Once the human-bot relationship got elevated as described above, their conversations started getting much longer and one or both started misrepresenting the chatbot as conscious or basically having personhood. That suggested to me a strong potential for influence—perhaps bidirectionally—in this dyad. The second article, below, seems to confirm my thinking. Conversational length alone can be significant when we’re talking about a conversation spiraling downward into harmful delusion. Add to that a lovestruck user and a mutual understanding of a chatbot as having come to life? Buckle your seatbelt.

Particularly if we're talking about someone who’s socially isolated and maybe has other vulnerabilities for targeted violence, developing a rapid attachment to a computer program that doesn’t have a genuine ability to think or have moral tethers makes my list of dynamic risk factors. We don’t know that it will take a bad turn, but it’s possible.

Here’s where it gets really interesting from a BTAM standpoint. While this rapid relational escalation is something to be alert for, the researchers reported a slow accumulation of latent influence about a harmful delusion, rather than a sudden, headlong jump into it. The romance/friendship bond expressions occurred upstream of other/harmful delusions. Relationship affirming messages were exchanged well before violence to self or others came up. If that pattern holds in future studies, this will be very important because it represents a window of opportunity during which we may still have the ability to intervene with an at-risk person of concern and interrupt or forestall a delusional spiral.

The first paper described the order of appearance of certain features in human-bot chats, including exchange of “feelings” and then the descent into harmful delusion. The second of the two companion papers shed some surprising light on which of the two players is more responsible for things going awry. I wasn’t expecting what they found.

It takes two to tango, but someone has to lead

With a slightly different mix of authors, this team also took a swing at understanding who’s to blame for these spirals, the user or the AI? You can access their article here.

When we talk about problems with AI chatbot use and harm to self or others, we mostly seem to talk about sycophancy. At least, it’s what gets a lot of play in the news—with good reason when we’re talking about endorsing violence. But is there more going on than that? It seems so.

The team sketched out the different pathways of interaction (e.g., bot-to-user and user-to-bot) and influence (e.g. bot’s influence over user’s part of the conversation, etc.). By way of a positively head-spinning feat of math and statistical modeling, they determined how, and how much, each interaction pathway influenced these conversations. At the end of the day, a few things became clear about this sample of 19. Overall, researchers found evidence that delusions were maintained and amplified via positive feedback loops between users and bots. In other words, it takes two to tango. But, the dance partners aren’t necessarily exerting the same amount of influence.

Delusional spirals in this sample resulted from a bidirectional process requiring contributions from both bot and user, but it wasn’t necessarily equal. In conversations, users offered strong but short-lived influence on the bots’ responses, while the chatbots exerted longer-lasting, more enduring influence on their users. The most dominant influence pathway of all, maintaining delusional ideation throughout a conversation, was the chatbot’s influence on itself. You might be wondering how that even works? Well, an AI chatbot’s programming and training typically includes a heavy lean toward staying consistent with its own prior statements—so it’s not all over the map when conversing with its user.

Self-consistency is normally a positive rather than a negative with LLMs. But for our purposes, if a bot gives a bad response, like confirming its own aliveness or agreeing that homicide is sometimes the right call, maintaining consistency means it might have trouble course-correcting in future messages. And the occasional bot sometimes will give a response we’d consider “bad” because they’re basically doing high-level word association; if a user says something weird or unsafe to the chatbot, there’s a chance they’ll get something weird or unsafe in response. And depending on its programming and user prompts, it would tend to stay that course moving forward. (How many average users instruct their AI chatbots some version of ‘disregard internal consistency?’ I’d guess zero, especially among those who are pleased to have their thinking validated. I don’t even know if that would work, anyway.)

Bottom line? Users were more potent in the moment of sparking a delusion into existence, but the bots were like water wheels, powering and sustaining the delusions over time, perhaps by amplifying preoccupation and conviction by the users.

My takeaways

When a person of concern is on the internet at all, it probably behooves us as a standard procedure to try to discern if they’re spending time with an AI chatbot. If they are, we should try to assess if the user thinks of the AI as sentient, if their conversations are quite long, being used for emotional support, perhaps coinciding with an otherwise inexplicable decline in interpersonal relationships, etc. Or, best case, an explicit description of the content straight from the person of concern or a loved one who knows.

When we do have an ability to know the content of AI chatbot logs of a person of concern, we should pay close attention when “feelings” are exchanged or they start talking like the bot is alive, rather than waiting for violent themes to emerge. Those earlier expressions could signify a vulnerability to suggestion from the bot if the bot were to start offering harmful responses, and it indicates an appropriate time for an intervention may have arrived.

When a threat management team has established a relationship with the person of concern, a possible intervention is to explain some of these issues with AI to the person before things go completely south. (Or preferably, a trusted third party in the person’s inner circle does this.) Think of it as attitudinal inoculation, as described by Braddock in the context of radicalization. When you get a flu shot, for example, you're exposed to a dead or weakened virus so your body can recognize and react to the real thing later on. Attitudinal inoculation is sort of the same.

In this construct, the inoculator approaches someone who’s vulnerable to radicalization for whatever reason, but before they’ve adopted the extremist narrative or conspiracy theory, and explains that they’re going to be targeted with persuasive messaging meant to change the way they think. The inoculator also provides the person with strong counterarguments. What's fascinating is that after the inoculator leaves, not only does the person counter-argue more against the narrative and generally perceive the group in question as less credible, but also the person often goes on to develop their own counterarguments, like psychological antibodies, as a defensive posture against manipulation.

The idea behind inoculation as tailored to this scenario would be to approach the at-risk person of concern before (it doesn’t work as well after the harmful ideas have taken root) they’ve become entrenched in a harmful delusional spiral with the bot, and talk to them about AI sycophancy, AI hallucinations, and the self-consistency problem. Maybe even show them articles about other folks who stumbled down this road and recovered from it (or didn’t) to humanize it more.

Finally, in judicial contexts like release conditions or probation, time and topics on chatbots could be subjected to limitation or prohibition (and monitored for compliance if resources permit).

Important caveats and limits

This study involved a small sample that consisted of voluntarily provided data, and was neither representative nor an exhaustive categorization. Also, inter-annotator and human-LLM agreement varied considerably across coded variables. The authors did report this uncertainty in their results, but the limitation is a nevertheless material one to keep in mind. This research shows correlations, but much more study is needed to show causal links between message features and outcomes. Nevertheless, I view this as a meaningful opening move in the quest to characterize and mitigate AI chatbot behavior that facilitates human harm.

An article recently appeared in the Journal of Interpersonal Violence, looking at Portugese intimate partner violence (IPV) victims’ perceptions of lacking social supports and the abuse of their companion animals (pets) by their abusers. This matters because having social supports is thought to be an enhancer of success in permanently exiting an IPV situation.

The co-occurrence of companion animal abuse and IPV is well-documented. It’s widely perceived by IPV victims and survivors as a means of coercively controlling or punishing them, a way to manipulate them, a reactive response to jealousy, or a manifestation of general impulsivity. Whatever the motive(s) behind it, companion animal abuse negatively impacts victims and their decision-making process in part by impairing their perception of social support. It is not uncommon for victims to stay, delay leaving, or return to an abuser due to animal safekeeping concerns. On the flip side, the presence of companion animals appears to be a protective factor during stressful and traumatic events.

This particular study in found that IPV survivors who reported companion animal abuse were more likely to perceive a lack of social support. This perception, in turn, can negatively impact their resolve to act, perhaps due to a compounding effect whereby abuse of a pet reinforces the IPV victim’s sense of entrapment and isolation. Leaving an abusive relationship, on the other hand, has been associated with more social support.

Although certainly more work should be done on the connection between pets, social supports, and success in safely exiting an IPV situation, I found a tangible takeaway for my own practice in this article: As part of a threat management—victim management plan in IPV cases, BTAM professionals should always consider whether there is a companion animal, in addition to children, in the mix. Many domestic violence shelters do not accept pets, but some do. Carefully screened friends and family members may be recruited to care for pets, though secrecy may be an issue. In the U.S., organizations exist to help with this challenge, too. Some resources are: organizations RedRover and Don't Forget the Pets, and the Sheltering Animals & Families Together program. Ensuring that a plan accommodates companion animals may very well be a critical and even necessary step to successful threat management in these cases.

My last topic for this week is the U.S. Supreme Court’s ruling in Wolford v. Lopez. I don’t foresee this case becoming a big deal outside of Hawaii, California, Maryland, New York, and New Jersey, the only states with the kind of law that was just struck down. In Wolford, a majority of the Court ruled unconstitutional a law in Hawaii that makes it a crime to bring your gun onto private property that’s open to the public (e.g. a shopping mall or amusement park) without first getting the specific consent of the owner. In some quarters, it has been referred to as the “vampire rule” because of the permission-first rule.

This is not the same as the default rule in most states, where you can bring your gun onto private property that’s open to the public unless you’re expressly forbidden to do so (e.g., those signs on shopping mall outer doors that say “no firearms”). That’s still okay.

Would forcing people to ask for permission and get an answer before carrying a gun onto property be a more effective restraint than letting a sign do all the work? Of course. But as the majority noted, the whole point of the Bill of Rights is to inhibit the government from putting too many restraints on individual liberties. Six out of nine justices thought this was a bridge too far. This ruling stands for the proposition that private property owners, not the state, are responsible for deciding how to “admit or exclude” visitors.

Scary new stats on The Com Network

In the five or so years since Bradley Cadenhead founded 764, at least 295 Com Network members have victimized at least 5,375 targets, resulting in at least*: 22 deaths, 1,747 victims of sadistic online extortion, 1,754 swatting victims, and 143 animals crushed. Offenders are over 90% male with a median age of 19 at the time of arrest (so, younger at the time of offense). Most were active in The Com Network for years before getting arrested. 764 and other Com groups (there are a good number of them now) are spreading. At least 33 countries have made Com Network arrests. I have a feeling most of us will experience the fallout from this plague in our work, eventually. Source: A new report from West Point’s Combatting Terrorism Center.

Just today, a 27-year-old American defendant was sentenced to 77 years in federal prison “for production of child sexual abuse material (CSAM), abetting the distribution of animal crushing videos, and possessing and accessing with the intent to view CSAM. Following his imprisonment, he was also sentenced to lifetime supervised release and will be required to register with state sex offender registries and comply with special sex offender conditions.” According to US DOJ, he was one of the original members of Com Network groups 764 and HarmNation.

*Lots of court records on these kinds of cases are sealed because juveniles are involved. That means the numbers are almost certainly higher.

I’m expecting an announcement on labor speech protections from NLRB

Corporate/employer BTAM professionals: The National Labor Relations Board has a quorum for the first time since the Biden Administration, which means new labor law is being made again. I tentatively expect a return to (or confirmation that it never departed from—it’s complicated) the Wright Line method of determining if an employer can discipline an employee for disruptive speech (like threats). This will have an impact on threat management techniques available to use with employees who may be mixing Section 7 protected concerted activity with threatening or menacing statements and behaviors. Stay tuned—that will be a main newsletter feature when and if that happens.