OSDI '24 - USHER: Holistic Interference Avoidance for Resource Optimized ML Inference
USHER: Holistic Interference Avoidance for Resource Optimized ML Inference
Sudipta Saha Shubha and Haiying Shen, University of Virginia; Anand Iyer, Georgia Institute of Technology
Minimizing monetary cost and maximizing the goodput of inference serving systems are increasingly important with the ever-increasing popularity of deep learning models. While it is desirable to spatially multiplex GPU resources to improve utilization, existing techniques suffer from inter-model interference, which prevents them from achieving both high computation and memory utilizations. We present USHER, a system that maximizes resource utilization in a holistic fashion while being interference-aware. USHER consists of three key components: 1) a cost-efficient and fast GPU kernel-based model resource requirement estimator, 2) a lightweight heuristic-based interference-aware resource utilization-maximizing scheduler that decides the batch size, model replication degree, and model placement to minimize monetary cost while satisfying latency SLOs or maximize the goodput, and 3) a novel operator graph merger to merge multiple models to minimize interference in GPU cache. Large-scale experiments using production workloads show that USHER achieves up to 2.6× higher goodput and 3.5× better cost-efficiency compared to existing methods, while scaling to thousands of GPUs.
View the full OSDI '24 program at https://www.usenix.org/conference/osd...
Смотрите видео OSDI '24 - USHER: Holistic Interference Avoidance for Resource Optimized ML Inference онлайн без регистрации, длительностью часов минут секунд в хорошем качестве. Это видео добавил пользователь USENIX 12 Сентябрь 2024, не забудьте поделиться им ссылкой с друзьями и знакомыми, на нашем сайте его посмотрели 42 раз и оно понравилось 2 людям.