❯ SemiAnalysis reports NVIDIA inference above 1,000 tokens per second per user on 700B-plus models

Faster interaction with large modelsOn November 1, SemiAnalysis said a new NVIDIA AgentX submission showed HBM-based systems exceeding 1,000 tokens per second per user on models larger than 700 billion parameters, using software written with agent swarms. The reported figure measures generation speed under the test conditions; full tasks also require context processing and tool calls, according to the original post.

Cost depends on the service targetThe post put Rubin NVL72 at 20.9 times GB300 NVL72’s performance per total ownership cost at 325 tokens per second per user, and 9.11 times at 200. Those ratios belong to different serving targets and configurations. AgentX evaluates multi-turn coding-agent workloads, combining responsiveness with serving capacity.

Sharing more cached contextA follow-up post described Mooncake Store pooling KV cache in Vera system memory across 18 compute trays through NVLink, with over ten times the bandwidth of the previous RDMA approach. KV cache retains intermediate context results to reduce repeated computation.

The system determines the gainLarge-model serving costs depend on compute, memory, data movement and software together. Faster cache sharing can reduce time spent waiting for data. Sustained gains across real workloads could let the same infrastructure serve more agents at lower latency.

▮ SIGNALFor long-context agents, inference competitiveness increasingly depends on how cached context moves across memory and compute nodes.