Stitching Weight-Shared Deep Neural Networks for Efficient Multitask Inference on GPU
Abstract
Intelligent personal and home applications demand multiple deep neural networks (DNNs) running on resource-constrained platforms for compound inference tasks, known as multitask inference. To fit multiple DNNs into low-resource devices, emerging techniques resort to weight sharing among DNNs to reduce their storage. However, such reduction in storage fails to translate into efficient execution on common accelerators such as GPUs. Most DNN graph rewriters are blind for multi-DNN optimization, while GPU vendors provide inefficient APIs for parallel multi-DNN execution at runtime. A few prior graph rewriters suggest cross-model graph fusion for low-latency multi-DNN execution. Yet they request duplication of the shared weights, erasing the memory saving of weight-shared DNNs. In this paper, we propose MTS, a novel graph rewriter for efficient multitask inference with weight-shared DNNs. MTS adopts a model stitching algorithm which outputs a single computational graph for weight-shared DNNs without duplicating any shared weight. MTS also utilizes a model grouping strategy to avoid overwhelming the GPU when co-running tens of DNNs. Extensive experiments show that MTS accelerates multitask inference by up to 6.0x compared to sequentially executing multiple weight-shared DNNs. MTS also yields up to 2.5x lower latency and 3.7x less memory usage compared with NETFUSE, a state-of-the-art multi-DNN graph rewriter. Show more
Publication status
publishedExternal links
Book title
2022 19th Annual IEEE International Conference on Sensing, Communication, and Networking (SECON)Pages / Article No.
Publisher
IEEEEvent
Subject
Deep Neural Networks; Multitask Inference; Model AccelerationOrganisational unit
03429 - Thiele, Lothar (emeritus) / Thiele, Lothar (emeritus)
More
Show all metadata
ETH Bibliography
yes
Altmetrics