Queen Mary University of London
We present a refinement framework to boost the performance of pre-trained semi-supervised video object segmentation (VOS) models. Our work is based on scale inconsistency, which is motivated by the observation that existing VOS models generate inconsistent predictions from input frames with different sizes. We use the scale inconsistency as a clue to devise a pixel-level attention module that aggregates the advantages of the predictions from different-size inputs. The scale inconsistency is also used to regularize the training based on a pixel-level variance measured by an uncertainty estimation. We further present a self-supervised online adaptation, tailored for test-time optimization, that bootstraps the predictions without ground-truth masks based on the scale inconsistency. Experiments on DAVIS 16 and DAVIS 17 datasets show that our framework can be generically applied to various VOS models and improve their performance.
We aim to improve pre-trained VOS models (backbone model) using the multi-scale context aggregation module with the scale inconsistency estimation (a). At test-time, we further perform the self-supervised online adaptation that updates the model parameters to reduce the accumulation of the errors over the frames, caused by the increase of the scale inconsistency (b).
| Name | Title | Year | Author | Paper |
|---|---|---|---|---|
| OSVOS | One-shot video object segmentation | 2017 | S. Caelles et al. | [Paper] |
| RGMP | Fast video object segmentation by reference-guided mask propagation | 2018 | S-W. Oh et al. | [Paper] |
| STM | Video object segmentation using space-time memory networks | 2019 | S-W. Oh et al. | [Paper] |
@article{wang2022boosting,
title={Boosting Video Object Segmentation based on Scale Inconsistency},
author={Wang, Hengyi and Oh, Changjae},
journal={IEEE International Conference on Multimedia and Expo (ICME)},
year={2022}
}