Tracking Generic Objects in Videos

Abstract

Visual object tracking is a fundamental problem in computer vision and finds its application in multiple tasks such as autonomous driving, robotics, surveillance, video understanding, and sports analysis. Generic Object Tracking (GOT) is a specialized tracking task that aims at tracking virtually any object in a video by using a userspecified bounding box that defines the target object in the initial video frame. Learning a target model, in order to track the target in each frame, from such sparse information proves extremely challenging. Especially in adverse tracking scenarios, where the target object is frequently occluded, goes out of view, or where distractors, visually similar objects as the target, are present. Thus, we tackle the problem of robust generic object tracking in videos even in challenging scenarios in this thesis. First, we propose a novel tracking architecture that keeps track of distractor objects in order to continue tracking the target. We achieve this by learning an association network, that allows to propagate the identities of all target candidates from frame-to-frame. To tackle the problem of lacking ground-truth correspondences between distractor objects in visual tracking, we propose a training strategy that combines partial annotations with self-supervision. Second, we introduce a Transformer-based target model predictor that produces the target model. The employed Transformer captures global relations with little inductive bias, allowing it thus to learn the prediction of powerful target models even for challenging sequences. We further extend the model predictor to estimate a second set of weights, which are applied for accurate bounding box regression. Third, we propose the new visual tracking benchmark, AVisT, dedicated for tracking scenarios with adverse visibility. AVisT contains 18 diverse scenarios broadly grouped into five attributes with 42 object categories. The key contribution of AVisT are diverse and challenging scenarios, covering severe weather conditions, obstruction and adverse imaging effects, along with camouflage. Finally, we propose the task of multi-object GOT, that benefits from a wider applicability than tracking only a single generic object per video, rendering it more attractive in real-world applications. To this end, we introduce a new large-scale GOT benchmark, LaGOT, containing multiple annotated target objects per sequence. Our benchmark allows researchers to tackle remaining challenges in GOT, aiming to increase robustness and reduce computation through joint tracking of multiple objects simultaneously. Furthermore, we propose a Transformer-based GOT tracker capable of joint processing of multiple objects through shared computation.