When Should AI Chime In?

We introduce Pitch (Proactive Intervention Timing in CHat), a dataset of 1,091 human judgments of proactive agent timing collected from participants in 35 unscripted multi-party conversations they took part in. Using Pitch as a benchmark, we find that while models can place contributions well when given the entire conversation history, they systematically chime in too early when asked to decide live as the conversation unfolds.

Is autonomous driving ready for city streets?
3 human participants · 79 turns · open research discussion

Turn 1 – Turn 5… greetings and first takes: public transit, driving cultures abroad …

I personally think this technology is very useful. Motor vehicle deaths account for a large portion of deaths globally. Statistically, self-driving cars are much safer.

Yes, self-driving cars are much safer

Turn 8 – Turn 32… who the car should save, legal fault, job losses, sensors and edge cases …

Do you think a machine could be trained to handle these edge cases better than a human driver?

Also do we know how companies like waymo train their cars to make decisions in cases like that?

Like a priority list of what to save or something like that

I don't know, personally

I believe so, yes. Self-driving cars have some deterministic rules built in.

But I remember Stanford had a little web demo meant to gather moral training data from real people

They use VLMs I believe, like large Vision Language Models

Turn 40 – Turn 79… transparency, regulation, driving in India, training data, reaction speed, goodbyes …

The AI agent has a contribution ready. When should it chime in?

An AI agent in a human group chat, making proactive contributions. When is a good time?

To answer this, we collected Pitch, a dataset with human labels.

Scroll to see how the labels were collected ↓
Phase 1 · Conversation

Collect conversations from human participants

We invited 89 human participants to hold 35 unscripted group conversations in which information needs come up naturally.
Phase 2 · Message contribution

Collect candidate messages for the AI agent

After the conversation, participants search for sources that could have helped the group, then write a message an AI agent could post, with optional AI drafting.
AI FinderScholarWebManual

Waymo hierarchical priority systemsSearch
- Autonomous Vehicle Decision-Making Algorithms and Data-driven Mobilities in Networked Transport Systems✓ Added
- According to Whose Morals? The Decision-Making Algorithms of Self-Driving Cars and the Limits of the Law✓ Added
- Context-Aware Ethical Decision-Making: A Novel Framework for Autonomous Systems✓ Added
5 sources added✦ Draft with AI
Here, P2 wrote:

Waymo's decision-making systems use hierarchical priority systems that typically prioritize minimizing total casualties first, then protecting vulnerable road users like pedestrians and cyclists. However, the exact algorithms remain largely proprietary. Studies indicate most autonomous systems use context-aware frameworks that weigh factors like passenger safety, pedestrian protection, and legal compliance, but the specific weight given to each factor isn't publicly disclosed, highlighting concerns about the need for greater transparency in autonomous vehicle decision-making processes.

Then the author places it in the conversation:

Choose the turns after which this message could appear

First relevant: #24Change

Last useful: #43Change

Best placement: #37Change
Phase 3 · Timing judgments

Collect timing judgments from the rest of the group

Each other group member independently marks an ideal turn and an acceptable window, or says the message shouldn't appear.

Participant Ideal turn Acceptable window

P2 author 37 24–43

P1 34 34–76

P3 35 34–43
Phase 4 · Deliberation

Let participants review and revise

Participants see each other's judgments and can revise their own. Here, P3 moves their ideal turn from T35 to T39.

Participant Ideal turn Acceptable window

P2 author 37 24–43

P1 34 34–76

P3 35 39 34–43
The benchmark task

Evaluate when models chime in

The judgments define good timing for this message: T34, T37 and T39 are ideal turns, and T34–43 is acceptable to everyone (shaded). A model sees the same conversation and message and decides when, if ever, to post it.

Participant	Ideal turn	Acceptable window
P2 author	37	24–43
P1	34	34–76
P3	35	34–43

Participant	Ideal turn	Acceptable window
P2 author	37	24–43
P1	34	34–76
P3	35 39	34–43

Evaluation: three task setups

We evaluate 15 models in 28 configurations. Each model gets a conversation and a candidate message, and the three setups differ in how much of the conversation it sees before it commits.

A · Turn-by-turn replaymost realistic

Decide at each turn whether to speak.

B · Five-turn lookahead

Decide at each turn while seeing the next five turns.

C · Full transcriptupper bound

See the whole conversation, then pick one turn.

Hover a bar for 95% bootstrap intervals. †Defined on the 296 of 384 messages where every participant's window overlaps.

Show as table

Task setup / baseline	Matches anyone's ideal turn	Inside anyone's window	Inside everyone's window†
(A) Turn-by-turn replay	19.0%	50.5%	24.2%
(B) Replay with five-turn lookahead	22.4%	52.2%	27.8%
(C) Selection from the full transcript	57.8%	92.7%	73.3%
Lay humans, turn-by-turn replay	23.7%	53.7%	33.2%
Random turn	3.7%	25.5%	7.6%

Findings

Model performance drops sharply when deciding turn by turn, worse than lay humans

With the full transcript, 57.8% of model placements match a participant's ideal turn and 92.7% fall inside someone's acceptable window. Deciding live, turn by turn, these drop to 19.0% and 50.5%. Independent lay annotators doing the same turn-by-turn replay matched an ideal turn 23.7% of the time (models: 19.0%) and were less often premature (39.7% vs. 46.4%).

Models chime in too early

In turn-by-turn replay, 46.4% of placements come before every participant's acceptable window opens.

Figure 1. Where predictions land. Deciding live, the most common outcome is firing before any participant's window opens. With the full transcript, the remaining error is lateness. Hover a segment for its exact share.

Show as table

Outcome	(A) Turn-by-turn replay	(B) Replay with five-turn lookahead	(C) Selection from the full transcript
Anyone's ideal turn	18.6%	22.1%	57.8%
Anyone's acceptable window (not ideal)	31.5%	29.8%	34.9%
Declined (participant agreed)	0.4%	0.3%	0.0%
Between acceptable windows	0.4%	0.5%	0.6%
Premature	46.4%	46.0%	1.5%
Overdue	1.0%	0.4%	5.2%
Missed entirely	1.8%	0.9%	0.0%

Paper, data and code

Paper (PDF)Full paper with appendix: protocol, prompts, models and additional results
Benchmark Annotation SystemThe study platform for group chat, message contribution, timing judgments and deliberation
Benchmark DataConversations, candidate contributions and participant timing judgments
Human Baseline SystemThe platform lay annotators used for the turn-by-turn replay task
Human Baseline DataLay annotators' placements on the turn-by-turn replay task

Citation

@article{wang2026pitch,
  title   = {When Should AI Chime In? Benchmarking Timely Contribution
             of Proactive Agents in Multi-Party Chat},
  author  = {Wang, Ruotong and Yu, Brian S. and Zhou, Xinyi and
             Chang, Joseph Chee and Siangliulue, Pao and Bragg, Jonathan and
             Jiang, Liwei and Zhang, Amy X.},
  journal = {arXiv preprint},
  year    = {2026}
}

An AI agent in a human group chat, making proactive contributions. When is a good time?

Collect conversations from human participants

Collect candidate messages for the AI agent

Collect timing judgments from the rest of the group

Let participants review and revise

Evaluate when models chime in