Lan Fu

Date of Award

Summer 2022

Document Type

Open Access Dissertation

Department

Computer Science and Engineering

First Advisor

Song Wang

Abstract

Many images are captured in sub-optimal environment, resulting in various kinds of degradations, such as noise, blur, and shadow. Adverse illumination is one of the most important factors resulting in image degradation with color and illumination distortion or even unidentified image content. Degradation caused by the adverse illumination makes the images suffer from worse visual quality, which might also lead to negative effects on high-level perception tasks, e.g., object detection.

Image restoration under adverse illumination is an effective way to remove such kind of degradations to obtain visual pleasing images. Existing state-of-the-art deep neural networks (DNNs) based image restoration methods have achieved impressive performance for image visual quality improvement. However, different real-world applications require the image restoration under adverse illumination to achieve different goals. For example, in the computational photography field, visually pleasing image is desired in the smartphone photography. Nevertheless, for traffic surveillance and autonomous driving in the low light or nighttime scenario, high-level perception tasks, \e.g., object detection, become more important to ensure safe and robust driving performance. Therefore, in this dissertation, we try to explore DNN-based image restoration solutions for images captured under adverse illumination in three important applications: 1) image visual quality enhancement, 2) object detection improvement, and 3) enhanced image visual quality and better detection performance simultaneously.

First, in the computational photography field, visually pleasing images are desired. We take shadow removal task as an example to fully explore image visual quality enhancement. Shadow removal is still a challenging task due to its inherent background-dependent and spatial-variant properties, leading to unknown and diverse shadow patterns. We propose a novel solution by formulating this task as an exposure fusion problem to address the challenges. We propose shadow-aware FusionNet to `smartly' fuse multiple over-exposure images with pixel-wise fusion weight maps, and boundary-aware RefineNet to eliminate the remaining shadow trace further. Experiment results show that our method outperforms other CNN-based methods in three datasets.

Second, we explore the application of CNN-based night-to-day image translation for improving vehicle detection in traffic surveillance that is important for safe and robust driving. We propose a detail-preserving method to implement the nighttime to daytime image translation and thus adapt daytime trained detection model to nighttime vehicle detection. We utilize StyleMix method to acquire paired images of daytime and nighttime for the nighttime to daytime image translation training. The translation is implemented based on kernel prediction network to avoid texture corruption. Experimental results showed that the proposed method can better address the nighttime vehicle detection task by reusing the daytime domain knowledge.

Third, we explore the image visual quality and facial landmark detection improvement simultaneously. For the portrait images captured in the wild, the facial landmark detection can be affected by the cast shadow. We construct a novel benchmark SHAREL covering diverse face shadow patterns with different intensities, sizes, shapes, and locations to study the effects of shadow removal on facial landmark detection. Moreover, we propose a novel adversarial shadow attack to mine hard shadow patterns. We conduct extensive analysis on three shadow removal methods and three landmark detectors. Then, we design a novel landmark detection-aware shadow removal framework, which empowers shadow removal to achieve higher restoration quality and enhances the shadow robustness of deployed facial landmark detectors.

Rights

© 2022, Lan Fu

Download

Share

COinS