Abstract

We present a general learning-based solution for restoring images suffering from spatially-varying degradations. Prior approaches are typically degradation-specific and employ the same processing across different images and different pixels within. However, we hypothesize that such spatially rigid processing is suboptimal for simultaneously restoring the degraded pixels as well as reconstructing the clean regions of the image. To overcome this limitation, we propose SPAIR, a network design that harnesses distortion-localization information and dynamically adjusts computation to difficult regions in the image. SPAIR comprises of two components, (1) a localization network that identifies degraded pixels, and (2) a restoration network that exploits knowledge from the localization network in filter and feature domain to selectively and adaptively restore degraded pixels. Our key idea is to exploit the non-uniformity of heavy degradations in spatial-domain and suitably embed this knowledge within distortion-guided modules performing sparse normalization, feature extraction and attention. Our architecture is agnostic to physical formation model and generalizes across several types of spatially-varying degradations. We demonstrate the efficacy of SPAIR individually on four restoration tasks-removal of rain-streaks, raindrops, shadows and motion blur. Extensive qualitative and quantitative comparisons with prior art on 11 benchmark datasets demonstrate that our degradation-agnostic network design offers significant performance gains over state-of-the-art degradation-specific architectures.

Degradations Considered

Motivation for Spatially-Adaptive Model

• All layers in existing restoration models are generic CNN layers, which apply the same set of filters to every degraded image, which is suboptimal.
• They apply the same filter to every pixel location (do not distinguish between difficult and easy pixels). CNNs, by definition, are composed of spatially-invariant filters.
• Such layers are limited in their ability to invert degradations that are highly image dependent and spatially-varying.
• Most network architectures are specifically tailored for individual degradation types as they are based on image formation models.
• Distortion-localization information embedded in the labeled datasets remains unused or sub-optimally used in all existing solutions.

SPAIR Architecture

SPAIR is composed of a localization network (Net_L) and restoration network (Net_R)

Net_R is composed of three types of distortion guided modules:

• Mask-Guided Sparse Convolution
• Spatial Feature Modulation
• Region-Guided Sparse Non-Local (SNL) Module

Qualitative Comparisons

We perform evaluation of the dual tasks of Reconstruction and Restoration

Citation

Acknowledgements

The website template was borrowed from Michaël Gharbi.