Large language model (LLM) serving faces a fundamental challenge: the prefill phase is compute-bound while the decode phase is memory-bound. Existing serving systems co-locate these two phases and batch their computation across users and requests, a strategy that not only introduces strong prefill–decode interference but also couples resource allocation and parallelism choices across both phases. We introduce DistServe, a low-latency serving engine that assigns prefill and decode computation to different GPUs, thereby eliminating interference between the two. DistServe co-optimizes resource allocation and parallelism strategies tailored for each phase, while placing them across the cluster to account for bandwidth constraints and minimize the communication overhead introduced by disaggregation. As a result, DistServe substantially improves serving performance, measured as the maximum request rate that can be sustained under strict time-to-first-token (TTFT) and time-per-output-token (TPOT) constraints. Since DistServe, disaggregation has moved into the spotlight of LLM inference, with many companies and open-source projects adopting or extending this paradigm in practice. We will also highlight related academic and open-source work that has emerged around disaggregated inference, and reflect on the open challenges faced today as well as the opportunities that lie ahead.

The brains of computing systems today have increasingly been realized by generative AI such as Large Language Models (LLMs) together with agentic technologies. While the AI approaches are arguably more creative and cheaper to develop, their safety becomes a fundamental challenge. Instead of using another LLM as a judge or waiting for a provably trustworthy model to exist, we may want to figure out how to build seatbelts and airbags for computing systems driven by LLMs/agents. In this talk, I will share our preliminary experience of developing a site reliability engineering (SRE) agent which attempts to autonomously mitigate production system failures. I would also discuss the potential of advanced system intelligence – AI that can understand “systems” beyond code and reason about their safety properties.

Despite its amazing success, leaderboard chasing has become something researchers dread and mock. When implemented properly and executed faithfully, leaderboard chasing can lead to both faster and easily reproducible progress in science, as evident from the amazing progress we have seen with machine learning, or more broadly artificial intelligence, in recent decades. It does not however mean that it is easy to implement and execute leaderboard chasing properly. In this talk, I will go over four case studies demonstrating the issues that ultimately prevent leaderboard chasing from a valid scientific approach. The first case study is on the lack of proper hyperparameter tuning in continual learning, the second on the lack of consensus on evaluation metrics in machine unlearning, the third on the challenges of properly evaluating the evaluation metrics in free-form text generation, and the final one on wishful thinking. By going over these cases, I hope we can collectively acknowledge some of our own fallacies, think of underlying causes behind these fallacies and come up with better ways to approach artificial intelligence research.

AI agents hold great value for enterprises but are notoriously difficult to productionize in a robust and reliable way. Enterprise agents must internalize task context, constraints, and success metrics to be reliable, yet learning directly in production is risky and often infeasible. Veris is a simulation-first experiential learning platform for training and evaluating domain-specific AI agents through simulated experience. I will present how Veris constructs high-fidelity interactive environments that mirror target deployments, including tool APIs, data schemas, user behaviors, and organizational policies. Then I will discuss how Veris supports multiple post-training paths like RL with online rollouts and trajectory replay, and iterative prompt or policy optimization. Finally, I will outline current challenges and research opportunities.

Chi will present MassGen, a growing open-source multi-agent scaling system that tackles single-agent limitations through collaborative AI coordination. Built on AG2’s research foundation, the system implements novel coordination of frontier AI agents. Early benchmarking across reasoning and instruction-following tasks shows statistically significant improvements, with growing industrial and academic adoption demonstrating the research opportunity for scaling intelligence through collaborative emergence.
Chi Wang is founder of AutoGen (now AG2), the open-source AgentOS to support agentic AI, and its parent open-source project FLAML, a fast library for AutoML & tuning. He has received multiple awards such as best paper of ICLR’24 LLM Agents Workshop and SIGKDD Data Science/Data Mining PhD Dissertation Award. He has 15+ years of research experience in Computer Science from Google DeepMind, Microsoft Research, Meta, UIUC and Tsinghua.

In this talk, I will discuss key challenges in building agentic AI systems in the cloud. I will highlight DeltaZip, our recent work on efficiently deploying multiple fine-tuned models – a step we believe is essential toward enabling future AI systems. The core insight behind DeltaZip is that fine-tuning often introduces small-magnitude changes to a pre-trained model. By co-designing the serving system and compression algorithm, DeltaZip achieves a 2x to 12x throughput improvement over state-of-the-art systems. In addition to this project, I will share some ongoing challenges we are tackling in this space. Xiaozhe Yao is a third-year doctoral student at Systems Group, Department of Computer Science, ETH Zurich advised by Prof. Dr. Ana Klimović. His research explores the complex and fundamental tensions between three pillars: from optimizing systems for efficient ML, to improving data quality and organization for ML, to developing frameworks that bridge the gap between algorithms and their practical deployment. Through this multi-faceted approach, his work aims to better understand and build AI systems.