Receiver-Conditioned Latent Communication gives 94% CacheBack

Recently, frontier models have made progress on a range of open research problems, often with hundreds or thousands of agents working together.

So it seems like having lots of agents work on a task is a pretty good idea.

Multi-agent systems can distribute information and work across models to run in parallel, manage context limits, and divide tasks into specialized steps.

On a smaller scale, agents can answer a question across several articles. Each agent reads a different article, then sends a summary to a receiver that combines the summaries into an answer. The receiver’s answer depends on the evidence those summaries contain.

The dominant form of communication between these agents is text, generated one token at a time. LatentMAS instead sends the sender’s KV cache. But combining full caches at the receiver reassembles the contexts we divided across agents, defeating the context-management purpose of delegation.

We separate two decisions: what information to send, and how to represent it. To decide what to send, we start with what the receiver needs. CacheBack uses a receiver query to select which positions in the sender’s sequence to send, limiting how much context reaches the receiver. In our experiments, that query is the task question.

At 16× compression, we remove about 94% of sender positions from the message. With Qwen3-8B on FanOutQA, the 4× setting reaches 55.3% strict accuracy, compared with 40.7% for same-size text communication. Median task-completion latency falls from 612 to 192 seconds.

FanOutQA · four model familiesUp is more accurate · left is faster

Same-size textCacheBack

Same model, same receiver query
Model	Strict accuracy	Median time
Qwen3-8BCacheBack 4×	40.7 → 55.3%+14.7 points	612 → 192 s3.2× faster
Ministral 3 14BCacheBack 32×	38.7 → 59.3%+20.7 points	1,537 → 193 s8.0× faster
Gemma 4 12BCacheBack 4×	52.0 → 59.3%+7.3 points	603 → 201 s3.0× faster
Nemotron Nano 2 12BCacheBack 8×	38.7 → 50.0%+11.3 points	131 → 102 s1.3× faster

Figure 1. Same-size text → CacheBack on FanOutQA. Hover a model to isolate its arrow.

Details

Selected operating points from Table 1; 50 concurrent tasks on eight H100 GPUs.

Limiting context at the receiver

Each sender may have enough context to read its own articles. The receiver, however, has to combine their messages. If those messages contain the full KV caches, it receives a position for every input token and latent step from every sender. In our Qwen3-8B FanOutQA setup, the combined caches leave too little context for the receiver to generate an answer on every task. We call this context re-expansion.

We could reduce the size of those caches. Quantization uses fewer bytes for each key and value. KVComm sends fewer layers. Both reduce the amount of data sent, but retain every position in the layers being transferred. The receiver still needs room in its context window for all those positions.

A position budget limits how many positions each sender can include in its message. To stay within that budget, the sender must select which positions to keep.

CommunicationWatch the idea unfold

Swipe to see the receiver →

19.3 / 23.5 s

Figure 2. Full caches combine contexts; CacheBack sends selected positions.

Details

The final scene illustrates token-ID encoding. All reported experiments use continuous-row encoding.

Receiver-conditioned selection

Good communication is a form of compression. When I mention attention, I don’t send the full Attention Is All You Need paper. A few words are enough to refer to something we both understand, leaving us to focus on what I want to add.

Language models have a similar starting point. The receiver brings knowledge from pretraining and context from its own task. The sender needs to communicate the residual: the additional evidence the receiver needs beyond what it already knows. That evidence may occupy only a small part of the article, allowing us to compress the message substantially while preserving what the receiver needs to answer.

Take the example below. We have two questions about the logistics of collecting a building pass. Each agent reads a different document: the booking confirmation, a notice changing the collection desk, or the ID requirements. The agents pass messages along a chain to a receiver that answers the question.

One question asks where to go and what identification to bring. The other asks whether a driving licence is accepted. Each grid shows the document read by one agent, with one cell per token. The highlighted cells show which positions the agent selected to include in its message. Switching between the questions shows how that selection changes while the documents stay the same.

Same documents, different questionSelected evidence

Figure 3. Three Qwen3-8B senders · 4× compression · one A100. Times include all senders and the receiver. Underlines mark positions shown in the excerpts.

The selections shown above come from CacheBack, which compresses the sender’s state using the receiver’s query. After processing its documents, the sender processes the query against its existing KV cache. Attention from the query tokens provides a signal for scoring earlier positions, without processing the documents again. CacheBack combines these attention weights into a score for each position and selects the highest-scoring spans within the message budget.

Receiver-conditioned selectionThe query selects message positions

Read

Process the input context.

Receiver’s query

Select

Use the receiver query to score source positions.

Send

Send the selected continuous rows. The receiver prefills them and answers.

In our experiments, senders generate latent steps before selection.

Figure 4. Select source positions; include the sender’s latent steps.

Mean query attention and CacheBack’s correction

Communication is a query-aware compression problem: the evidence a sender needs to transmit depends on what the receiver is asking. We build on SnapKV, which uses an observation window of final prompt tokens to select cached positions for the model’s own continuation. We replace that observation window with the receiver’s query. Our starting score is mean query attention: the attention each sender position receives, averaged across the query tokens.

But different parts of the query may need evidence from different positions. A position receiving strong attention from the tokens asking about identification may receive little from those asking about the collection desk. Averaging across the whole query can dilute that strong attention and rank the position below one receiving modest attention from every token.

In the example below, Position A receives strong attention from one query token, while Position B receives weaker attention from all four. Averaging ranks B higher.

CacheBack applies a correction to mean query attention, increasing the score when attention is concentrated in particular query tokens. When every query token pays equal attention, the score stays unchanged. Here, the correction puts A ahead of B.

Switch between the two methods to compare their rankings, or adjust A’s attention weights to see how the scores change.

Attention → score

Query token	Position A	Position B
q₁	0.40	0.12
q₂	0.00	0.12
q₃	0.00	0.12
q₄	0.00	0.12

Adjust A’s weights

q₁q₂q₃q₄

Score = Q

Position A0.10

Q = 0.10 · C = 4

Position B0.12

Q = 0.12 · C = 1

0Score2

Higher score: Position B

Averaging over four query tokens reduces A’s 0.40 weight to Q = 0.10. B’s Q = 0.12 is higher.

Figure 5. Illustrative weights · one head · no pooling.

State representation

After selecting positions, we send their input vectors and let the receiver rebuild its KV cache. This takes fewer bytes per position than sending the full cache; the paper describes the encoding options.

Accuracy and task-completion latency

Both methods receive the same receiver query. Text senders use it to generate a message; CacheBack uses it to select sender positions. We compare the two on FanOutQA, with parallel fan-in from senders reading different Wikipedia pages, and LongBench v2 Easy, with a sequential chain reading successive parts of a document.

With Qwen3-8B on FanOutQA, 4× compression gives 55.3% strict accuracy and a median task-completion latency of 192 seconds. Same-size text communication gives 40.7% and 612 seconds. The plots show the other compression ratios too. Each model family in Figure 1 uses the compression ratio listed in the table.

Accuracy and completion timeHigher is more accurate · Farther left is faster

CacheBack (labels: compression)
Text (labels: sender size)
Dominated setting
Receiver gets only the question

At 4× compression, Qwen3-8B is more accurate and 3.2× faster than text senders using the same model.

At 8× compression, Nemotron is more accurate and 1.3× faster than text senders using the same model.

Tap either plot to enlarge

Figure 6. Accuracy and task-completion latency · 50 concurrent tasks · eight H100 GPUs.

Experimental setup

Each benchmark submits 50 tasks at once to eight H100 GPUs. Task-completion latency includes queueing and sender and receiver computation. Accuracy averages three generated answers per task. Text uses one GPU pool; most CacheBack configurations assign separate pools to senders and receivers. Faded points have a faster or more accurate setting of the same method that does no worse on the other measure. LongBench reports answer accuracy on the Easy subset.

Selection depends on the receiver’s query. Replacing it with an unrelated question reduced the evidence retained in the message and halved answer accuracy. The sender computation, message budget, and receiver’s actual task question stayed the same.

Query used for selection	Message recall	Strict accuracy
Task question	91%	46%
Unrelated question	65%	23%

Qwen3-8B · FanOutQA · 16× compression. Only the query used for selection changes. Message recall counts the fraction of reference-answer groups found in the selected source text.

Text generation and task-completion latency

The marginal cost of communicating more information depends on how the message is produced. Autoregressive communication adds a sequential cost for every token sent: the sender must finish one decoding step before starting the next. CacheBack instead selects from a completed computation. Once positions have been scored, a larger message adds transfer and receiver prefill, which can process many positions together. The marginal cost of sending more evidence therefore shifts from sequential decoding to operations better suited to handling it in bulk.

One more position in the messageAdditional work per message position

Start with four message positions.

+1extra decoding step

0extra decoding steps

Both receivers prefill the extra message positions. The text sender also generates the extra tokens one after another.

Figure 7. Work added per message position. Source processing and latent-step count stay fixed.

This benefit does not require a latent message. A sender could also select source passages and transmit them as text, avoiding the cost of generating a summary. Selecting what to send and choosing how to represent it are separate decisions.

Accuracy as we compress

The plot below shows Qwen3-8B on FanOutQA. Move the slider to see how sending fewer positions changes answer accuracy and task-completion latency. At 32× compression, accuracy remains above text communication. At 64×, it falls below text communication, with only a small further reduction in latency.

How much to sendQwen3-8B · FanOutQA

Compression 16×

SentFirst position, always kept+ 40 latent steps, always kept

Figure 8. Qwen3-8B · FanOutQA · accuracy and median task-completion latency. Grid positions are illustrative.

Details

The text baseline uses same-size Qwen3-8B senders. Strict accuracy requires every answer part to be correct. The grid represents the fraction of source positions selected. The 2× point is faded because 4× is both more accurate and faster.

Adaptive queries

One completed sender computation could supply several messages. Different receivers could ask for different evidence, or the same receiver could return for more detail. Each request would require processing the new query and selecting positions, without processing the source context again.

With several senders, the receiver could request additional evidence from the sources it needs. For a coding task, it might ask one sender for a function’s implementation and another for its callers. This would let the receiver spend more of its context budget on evidence needed as the task develops.

A Django fix

For the coding example, seven Qwen3-8B senders read different parts of a Django codebase. A receiver reads their messages and writes a patch.

Both runs produce the same one-line patch and pass all 88 tests. CacheBack takes 25.66 seconds to produce the patch; text communication takes 113.21 seconds.

A coding exampleSeven Qwen3-8B senders → one receiver · django__django-16333

Figure 9. One recorded Django fix. Timing excludes startup and tests; playback speed varies.

Communication is a query-aware compression problem. CacheBack shows that a simple selection rule can retain evidence the receiver needs while improving accuracy and task-completion latency over same-size text communication in our experiments. Choosing that evidence is a separate decision from representing it as text or continuous state. This gives us room to improve both: how agents request and select information, and how they transmit it.

The paper has the full experiments and implementation details. You can reach us at mfr2178@columbia.edu.

Cite this work

@misc{rossi2026cacheback,
  title         = {Receiver-Conditioned Latent Communication gives 94\% CacheBack},
  author        = {Rossi, Maximillian and Raghunath, Prajwal and Xuan, Haoqing and
                   Zhang, Yusen and Wu, Eugene},
  year          = {2026},
  eprint        = {2609.32046},
  archivePrefix = {arXiv},
  primaryClass  = {cs.AI},
  url           = {https://arxiv.org/abs/2609.32046}
}

An illustrated companion to Receiver-Conditioned Latent Communication gives 94% CacheBack. Research by Maximillian Rossi, Prajwal Raghunath, Haoqing Xuan, Yusen Zhang, and Eugene Wu.

Limiting context at the receiver

Receiver-conditioned selection

Read

Select

Send

Mean query attention and CacheBack’s correction

State representation

Accuracy and task-completion latency

Text generation and task-completion latency

Accuracy as we compress

Adaptive queries

A Django fix

All posts