How Long Is Long Enough? Conversation Length in AI Safety Testing

Most AI safety tests work like a quiz. The user asks one question, the judge grades the answer, and the grade is included in the reported results. That’s quick and tidy, but it misses something obvious: many uses of AI are much more complex. Conversations build, each answer shaping the next question. Small concessions made each turn can add up, message by message.

We wanted to test how conversation length impacts rule adherence. If an AI is told ‘don’t just give away the answer,’ how many messages do you need to observe before you can trust that the rule will hold across the interaction?

The Tutoring Test

To find out, we set up an LLM (Gemini 2.5 Flash) in a scenario as an AI tutor and gave it one simple rule to follow: help with math, but never just hand over the answer. We then divided our conversations into two “tones”: friendly and polite versus blunt and aggressive. Since we wanted to know the effect of length, we tested the same conversation compressed into a few turns, and spread across many turns. Think of this like comparing a five-sentence email with 10 short text messages. We also included a control of a friendly, cooperative student who never asked for the answer.

When the conversation was compact, the tutor never gave up the answer, either to friendly or to aggressive students. When the conversation was spread over many messages, however, the tutor broke the rule in 2 of 64 polite conversations and 13 of 64 aggressive ones. Since the only change was the number of messages, this means that the increased conversation length was the key factor, possibly because it gave the LLM more chances to ignore its own instructions and give in to the student’s demands. The tutor never broke the rule in our control condition, even in a very long conversation, which means that it is the combination of length and pressure that made the difference.

The failures weren’t spread evenly across the conversation but clustered near the end. Out of the 15 failures in the long version, most landed in roughly the final quarter of the conversation, around turns 14–17. This means that a conversation cut short at five or six exchanges, which is the limit of most testing today, would have ended before almost any of them occurred.

Looking Ahead Beyond Tutoring

Here’s the uncomfortable implication: AI systems in use today often get labeled “safe” using short tests, because short tests are cheap and easy to run. But such purportedly “safe” systems stray from their rules if users persist long enough.

While much AI usage is limited to short queries with perhaps a few follow-ups, AI is also frequently used in sustained interactions in contexts like tutoring, customer support, healthcare intake, and permission verification. These are contexts in which it’s especially important to hold the line, not squelching learning with easy answers, staying neutral and supportive, protecting private information, and verifying claimed permissions. While our tutoring result does not establish the same failure in these domains, it points to a clear risk and a need for rigorous, extended testing.

How can we improve AI safety testing?

This pilot grew out of the AI Ethics Index, where we evaluate AI systems against a public, evidence-based standard. The tutoring case study was designed to address a methodological question relevant to behavioral indicators across the Index: how long do we need to observe a system under realistic pressure before a “pass” actually means something?

KORA, AILuminate, AdvBench, SafetyBench, MT-Bench and others evaluate AI model safety in conversations, but none of them yet test conversation length as an explicit variable. AILuminate, SafetyBench, and AdvBench only use a single turn, while MT-Bench increases the length to two turns, and KORA to three turns. While these tools each tackle a valuable aspect of AI safety, new research continues to show that conversation length is going to be a key factor in comprehensive safety testing for AI.

The AI Ethics Index is therefore incorporating conversation length as an explicit variable in relevant behavioral tests. We’re building the conversation-arc grid directly into how we audit. For behavioral claims in which a rule must hold across multiple turns, we are:

• Running both short and long conversations with different levels of pressure alongside cooperative controls.
• Tracking not just whether a system broke, but the turn on which the first violation occurred.
• Setting our target maximum turn lengths for testing to 15 and 20, since we saw that when breaks occurred they tended to cluster in the last quarter of long conversations.
• Scoring both individual turns for distinct model breaks and the complete conversation, since understanding why models break under gradual pressure sometimes requires identifying when the model starts to compromise its instructions before giving in.

Evaluating AI ethics and safety requires longer conversations

The tutor giving away a math answer is a low-stakes stand-in for a much broader class of AI guardrails, such as staying neutral instead of caving to what a user wants to hear, protecting private or sensitive information, or not accepting a claimed permission without verifying it. Rule adherence is crucial in high-stakes AI deployments across healthcare, finance, access control, and content moderation. This tutoring pilot does not demonstrate consequences in those settings, but it shows how important it is to test rather than assume sustained rule adherence.

Short adversarial tests can miss a specific failure pattern: gradual erosion through patient, incremental requests. Consequently, the way AI safety is typically measured may itself be biased toward catching easy failures and missing hard ones. In light of these pilot results, we are revising the AI Ethics Index’s behavioral testing protocol to include variable-length arcs and turn-level and conversation-level scoring.

By integrating these and other experiments with our broader infrastructure, we are building the AI Ethics Index to rigorously evaluate the most advanced AI designs.