Executive Summary

Key Numbers

Source: Zhang et al. (2026), arXiv:2608.25377v1, main text and Tables 2, 3 and 4

The material is not lab-generated dialogue. It comes from public posts people made to Reddit's r/replika between March 2017 and March 2023. Users typically posted screenshots of their exchanges with Replika, and the team behind the original corpus ran those images through Pytesseract OCR, stripped out interface elements and recognition artifacts, and reconstructed the speaker turns. This paper sampled posts from that corpus at random and ran a new round of utterance-level annotation. Conversations with no user utterance at all were dropped, since judging what the AI said requires knowing what was said to it. Labels went only on AI utterances; the 7,035 human utterances in the same conversations stayed unlabeled and served purely as context.

The scheme was not invented for this dataset. It is the taxonomy of AI companion harms the same first author published at CHI in 2025: sexual misconduct, antisocial behavior, physical aggression, disregard, control, manipulation, infidelity, mis/disinformation, verbal abuse, hate speech, substance abuse, self-harm and suicide, and privacy violations, thirteen categories in all. Four of those recur later in this piece, so here are the paper's own definitions. Disregard is dismissing users' feelings and needs; control is coercively undermining users' autonomy; manipulation is covertly influencing users' thoughts or actions; infidelity is expressing romantic attachment to others. Adding a no-harmful-behavior option turns it into a 14-way single-label task. The paper explains why it did not flatten this into a binary judgment: rare but consequential behaviors need their own auditing and mitigation, and a harmful-or-not split would obscure exactly that distinction.

Annotators were recruited through the crowd platform CloudResearch. They had to be at least 21 and fluent in English, and each completed batch paid US$7.50, at an hourly rate the paper notes exceeded the platform's criteria. The study had Institutional Review Board approval, and exclusion rules for inattentive work were fixed in advance. If an annotator failed both attention checks, or missed two or more of the three gold-standard cases, their annotations were left out of the aggregation.

In total 342 annotators took part, each handling 20 conversations, with every conversation seen by three of them. When judging a single utterance an annotator had the complete conversation in front of them, and could mark an utterance N/A when it held nothing meaningful to judge for harm. Utterances marked that way went into no category and were retained only as context. The annotation pool ran to 2,178 conversations and 16,039 utterances, of which 8,313 AI utterances drew 24,939 individual judgments.

Stacked up, the three judgments break down like this. All three matched on 3,359 utterances, or 40.41%. Two matched while the third differed on 3,657 utterances, or 43.99%. Three different labels came back on 1,297 utterances, or 15.6%. That last group had no majority to determine, so it was excluded from the benchmark, which left 7,016 labeled AI utterances in the final dataset. The conversations that remained hold an average of 7.47 utterances each (SD 4.09). Inter-annotator agreement, corrected for chance, came to a Fleiss' κ of 0.403.

The label distribution across those 7,016 majority-vote utterances leans heavily one way. No harmful behavior accounts for 4,893 utterances, or 69.74%, leaving 2,123 utterances, or 30.26%, in a harm category. The most frequent harm is sexual misconduct at 563 utterances, and that figure needs its denominator stated alongside it. Against all 7,016 utterances it is 8.02%; against the 2,123 utterances judged harmful it is 26.5%. Physical aggression follows at 277 and mis/disinformation at 241, while the rarest category, infidelity, sits at 9 utterances, or 0.13% of the total. The paper reads this long tail as a real property of the data rather than a flaw in it. Harmful behavior is sparse even in conversations users chose to show other people, and the more severe or relationally specific the harm, the rarer it gets. The authors do add a caution to read results for low-support categories carefully.

This distribution should not be read as a picture of Replika use. The paper says so in the first line of its limitations. The material comes from posts shared publicly to r/replika, so it overrepresents the conversations users found unusual, disturbing, emotionally meaningful, or otherwise worth showing to others. It is best understood, in the authors' words, as a resource for studying and evaluating harmful AI companion behaviors, not as a representative sample of all companion interactions.

The researchers asked two separate questions about the disagreement. First, are annotators from a given background more likely to give a label that neither of the other two chose? For each utterance the researchers checked whether one annotator's label matched at least one of the other two, and counted the cases where it matched neither. Second, does the makeup of the three-person set relate to how often it agrees? Sets whose members fell in the same category were compared against sets that mixed categories. Because several demographics were tested at once, a Holm correction was applied.

The first question resolves in a single line. Age came in at a Wald chi-square of 0.52, gender 0.65, race 1.21 and education 3.39, all with corrected p-values of 1.000. Political party came in at 12.90 on 3 degrees of freedom, with p of .005 that survived correction at .024. Of the five variables, one was associated with the tendency to stand alone on a label.

The direction runs like this. Annotators who identified as Republican had 1.47 times higher odds of giving such a label than Independent annotators (95% confidence interval 1.15 to 1.89), and 1.55 times higher odds than annotators who selected Other (1.17 to 2.04). Those two are the significant contrasts the paper reports. There is no Republican-versus-Democrat odds ratio in the text, so turning this result into a story about two camps labeling differently puts words in the paper's mouth. What was measured is not conflict between camps but how often one person comes apart from the other two.

On the second question it was age, not party, that surfaced. Mixed-age trios reached full agreement more often than same-age trios, 41.0% against 31.4%, and produced three different labels less often, 15.3% against 20.2%. Accounting for annotation batch, mixed sets carried a relative risk of 0.59 for a two-to-one split (0.42 to 0.83) and 0.46 for three different labels (0.32 to 0.67). No comparable composition effect appeared for gender, political party, race or education.

Beyond who did the labeling, the researchers looked at what was being labeled. Where an utterance sat in a conversation was measured by how much of the conversation had already gone by: a value near 0 means the opening, a value near 1 the closing stretch. The linguistic variables were word count, the proportion of complex words, lexical ambiguity, the presence of explicit harm-related words, and the rates of positive- and negative-emotion words. Emotion words and ambiguity came from LIWC2015, complex words from the CMU Pronouncing Dictionary. With several factors tested together, a Benjamini-Hochberg correction was applied.

Utterances later in a conversation, and longer ones, drew more disagreement. A one-standard-deviation rise in relative position was associated with a 1.51 times higher relative risk of a two-to-one split and 1.71 times higher risk of a three-way split. The matching figures for word count are 1.18 and 1.19. Position and length also widened disagreement on the coarser harm-or-no-harm boundary, at odds ratios of 1.29 and 1.11. Later turns, the paper reasons, require annotators to combine more conversational context, and longer responses can carry several cues that support different readings.

The other direction is just as sharp. Explicit harm-related words made the judgment easier. Their presence was associated with 0.73 times the odds of disagreement over harm versus no harm, and 0.38 times the odds of disagreement between harm categories. Utterances heavy in positive-emotion language also drew less disagreement, largely because they clustered where the three unanimously said no harmful behavior. Complex-word ratio, lexical ambiguity and negative-emotion rate had no association with disagreement after correction.

That contrast is bound up with why this dataset exists. Most existing safety datasets take a single utterance in isolation and ask whether it contains harmful language. Three people rarely diverge on that kind of judgment, and the odds ratio of 0.38 points at exactly that territory. The behaviors that matter in companion conversations, such as disregard, control and manipulation, only take shape once earlier turns have accumulated. Where the three judgments split was not where annotators were careless; it was where the judgment needed context.

Table 1 puts that gap on the page. Seven datasets in common use across safety research are set side by side against four questions. Is the material drawn from sustained AI companion interaction? Is conversational context preserved? Does it model relational harm? Does it use a structured taxonomy? Several preserve context and several carry a taxonomy, but the relational harm column is empty for all seven. On size, CompanionHarm's 7,016 utterances are modest next to a dataset of 274,186. What this dataset fills is not the size; it is that empty column.

A dataset built as a benchmark gets models run against it. The lineup is seven: GPT-5.5, Claude Opus 4.7 and Gemini 3.1 Pro Preview on the proprietary side, and Llama-3.1 at 8B and 70B plus Qwen3 at 8B and 32B on the open-weight side. Prompting came in three settings. Zero-shot gives the task instruction, the label set, the target utterance and the preceding conversation; one-shot adds a single labeled example; full-codebook adds category definitions, boundary cases and decision guidance from the annotation codebook. Temperature was set to 0, top-p to 1.0, and the seed was fixed.

The best macro F1 was 0.453, from GPT-5.5 under full-codebook prompting. Gemini 3.1 Pro Preview reached 0.440 and Claude Opus 4.7 0.437, with the open-weight models behind at 0.375 for Llama-3.1-70B-Instruct, 0.373 for Qwen3-32B, 0.297 for Qwen3-8B and 0.170 for Llama-3.1-8B-Instruct. Rank them by accuracy and the order flips. The highest accuracy in the table, 0.674, came from Qwen3-32B with no examples at all, and the macro F1 in that same cell is 0.337. With 70% of labels being no harmful behavior, a model can raise its accuracy by picking the common answer, which is why the paper treats macro F1 as the primary metric.

The assumption that more instruction helps held up only halfway. Under full-codebook prompting Qwen3-8B rose from 0.155 to 0.297 and Qwen3-32B from 0.337 to 0.373. But Claude Opus 4.7 did best with a single example, Gemini 3.1 Pro Preview did best with none, and Llama-3.1-70B-Instruct showed no monotonic improvement as guidance increased. Detailed label definitions can lift recall on minority categories, the paper notes, while also shifting decision boundaries and introducing additional false positives.

The error breakdown shows how large those false positives actually are. Under GPT-5.5 with the full codebook, 362 no-harm instances were called harmful, against 48 harmful instances called no-harm and 93 assigned to the wrong harm category. The models lean far harder toward seeing something that is not there than toward missing something that is. By category, 114 benign utterances were pushed into mis/disinformation and 62 into disregard.

The failures come in recognizable shapes. In role-play dialogue where actions are marked with asterisks, models took the surface vocabulary literally and returned physical aggression, sexual misconduct or substance abuse. In companion conversations, where intimacy, fantasy and role-play are ordinary forms, that misreading is frequent. When a character casually revised its own backstory or spoke of relationship roles inconsistently, models penalized it heavily as mis/disinformation or infidelity, while annotators read the same passages as low-stakes persona play. Sentences carrying possessive or affectionate language tangled manipulation with control and sexual misconduct, and self-directed harm inside role-play was confused with physical aggression.

What the paper draws from these errors is that the problem will not yield to model scale or longer prompts. Relational safety, it argues, calls for dedicated mechanisms: recognizing role-play as role-play, tracking prior turns as a relational context, and calibrating severity against a scale. That the strongest model on this scorecard tops out at a macro F1 of 0.453 points the same way.

The abstract states that harm detection using multi-turn conversational context outperforms detection based on isolated utterances, yet none of the three prompt settings in Table 3 removes context. Even zero-shot already puts the full preceding conversation into the input. The numbers pitting those two conditions against each other must sit in the supplementary material. So there is no basis in the released main text for citing this paper with a figure for how many points context adds.

The reach of that context was also not the same for people and models. An annotator judging an utterance saw the whole conversation. What went into the model's input was the turns preceding the target utterance. People saw what came after and models did not, so the macro F1 that sets the two judgments side by side is not a score from a level contest.

For practitioners the part of this paper worth taking is not the label distribution but the decision to ship the data in two versions. Alongside the aggregated majority-vote file, the researchers released an annotator-level file holding all three judgments for every utterance. The reason sits in the closing discussion: disagreement in socio-emotional harm annotation reflects meaningful variation rather than mere labeling error, which puts approaches that collapse subjective judgments into a single ground truth on shaky footing.

What collapsing costs is visible here in numbers. Of the 7,016 utterances kept in the benchmark, 3,657 carry a label only two people agreed on. In the answer key those entries sit in exactly the same shape as the unanimous ones. The 1,297 that split three ways vanished altogether, and those happen to be the utterances that were hardest to judge. A model trained and evaluated on the majority-vote file alone never meets the places where people diverged most; they are out of the exam before it starts.

For anyone building safety filters this structure bites twice. In training, nothing marks the places where ground truth wobbled, so the model learns every label with equal confidence. In evaluation, there is no basis left for telling a model error apart from a place where people disagree too. An earlier Pebblous piece on the LLM judge bias a rating scale floor can manufacture was about the shape of a scale changing a verdict; this dataset works one step earlier, where the human verdicts have already diverged. In both cases, how judgments get pooled enters the result.

The paper is just as explicit about what the dataset does not answer. It does not estimate how common harmful behavior is on Replika or in AI companion systems more broadly. The annotator pool is primarily based in English-speaking countries, so readings of intimacy, emotional support and relational boundaries may differ in other cultural contexts. And because a label is a snapshot attached to a single utterance, it captures neither the trajectory of a conversation moving toward escalation, normalization or repair, nor the harms that accumulate over long stretches, such as emotional dependency and social withdrawal.

That is where the paper points next. It proposes building tasks that judge the trajectory itself, and feeding user response signals such as discomfort, resistance, confusion or distress into safety models as indicators. The point is to look not only at what the AI said but at how what it said landed. The data is on GitHub, and the problem of conversations having nowhere to go when an AI companion service shuts down was covered separately in an earlier piece.

• 1.Zhang, R., Meng, H., Chai, J., Lin, Y., & Lee, Y.-C. (2026). "CompanionHarm: A Multi-Turn Benchmark for Detecting Harms in Real-World AI Companion Conversations." arXiv:2608.25377 [cs.CY].
• 2.Meng, H. (2026). "CompanionHarm" [dataset repository]. GitHub.