Key Projects:

Single-Image Reflection Removal

In this project, we propose a novel two-stage DNN based reflection removal algorithm. In the first stage, we include a new feature reduction term in the loss function when training the network. Due to its strong reflection suppression ability, the reflection components in the image can be more effectively suppressed. However, it will also attenuate the gradient values of the background image. For recovering the background, in the second stage, we first estimate a reflection gradient confidence map based on the initial estimation result and use it to identify the strong background gradients. Then we use a generative adversarial network to reconstruct the background image from its gradients. Experimental results show that the proposed two-stage approach can give superior performance compared with the state-of-the-art DNN based methods.

Reflection removal using different approaches on the images from a benchmark dataset SIR2
Array Image Processing for Reflection Removal, Occlusion Removal, and Salient Detection
  • In this project, array cameras are used to obtain the depth information of an image scene. 
  • Based on the depth information obtained, different image processing techniques are developed for solving various daily photography problems including
    • Removing the reflection in the image
    • Removing the occlusion in the image
    • Detecting the salient objects in the image
Reflection removal
A comparison of a few current reflection removal approaches with our proposed algorithm
Occlusion removal
(Left) The original scene. Note the scribble (circled) on the window. (Middle) The picture taken through the window. (Right) Removing the occlusion using the proposed algorithm. Note that the resolution is also increased at the same time.
Salient object detection
Examples of the saliency maps obtained using various state-of-the-art methods and our proposed method in situations with foreground objects
Key references:
  • Tingtian Li, Daniel P.K. Lun, Y.H. Chan and Budianto, “Robust Reflection Removal Based on Light Field Imaging,” IEEE Transactions on Image Processing, Vol.28, No.4, pp.1798-1812, April 2019. 
  • Tingtian Li and Daniel P.K. Lun, “A Novel Reflection Removal Algorithm Using the Light Field Camera,” Proceedings, 2018 IEEE International Symposium on Circuits and Systems (ISCAS’2018), Florence, Italy, May 2018. (Best paper award top ten finalist)
  • T. Li and Daniel P.K. Lun, “Salient Object Detection Using Array Images”, 2017 Asia-Pacific Signal and Information Processing Annual Summit and Conference (APSIPA ASC 2017),  Kuala Lumpur, Malaysia, December 2017.
  • Tingtian Li and Daniel P.K. Lun, “Super-resolution Imaging with Occlusion Removal Using a Camera Array”, Proceedings, 2016 IEEE International Symposium on Circuits and Systems (ISCAS), Montreal, Canada, pp.2487-2490, 2016.
Image-Based Three-Dimensional Data Acquisition
Fringe Projection Profilometry (FPP)
  • FPP is a  kind of structured light projection method for measuring the 3D structure of objects.
  • Regular fringe patterns are projected to the object.
  • By measuring the deformation of the fringe patterns as shown on the object surface, its 3D structure can be measured.
  • In this project, a number of image processing techniques have been developed for improving the robustness when the system is operating in different adverse environments. They include,
    • FPP with noisy fringe images
    • FPP for objects with strong texture patterns on their surface
    • FPP with fringe images having some parts of the fringes missing
    • FPP with fringe images having discontinuous fringes 
A simplified illustration of the hardware setup required for FPP
FPP with fringe images having discontinuous fringes
  • Similar to many phase based imaging systems, FPP requires to perform a phase unwrapping process at the end since all FPP methods can only give a modulo-2π estimation of the phase.
  •  It requires the availability of all wrapped phase data, which cannot be achieved if there is any discontinuity in the fringe image, such as the object has a sharp change in height or the object is occluded by another object.
  • In this project, a code pattern is embedded into the fringe pattern to provide the period number of the fringes. Phase unwrapping can thus be restarted from the discontinuity points in the fringe image, where traditional phase unwrapping process will have to stop.
FPP with fringe images having highlights
  • Normal objects are often reflective to light. 
  • FPP requires projecting light patterns onto the target object.
  • The projected light can be reflected by the object and introduce highlight regions in which no fringes can be observed.
  • FPP with fringe images having highlights is always error prone.
  • In this project, the fringes in the highlight regions are regenerated using a geometrically guided regularization approach.   
Single-shot FPP for objects with strong texture pattern

The textures of the object often introduces great difficulty to single-shot FPP methods since they perturb the fringe patterns projected onto the object.

A comparison of different single-shot FPP methods for objects with strong texture patterns on their surface. (Left) The 3D profiles measured by different methods. (Right) The texture of the objects obtained from different methods.

The above results show that the stronger the texture, the poorer will be the performance of the traditional approaches. Serious errors are noted in their measured 3D profiles. The textures extracted are also very blurry. Our method is robust to the texture patterns of the objects. 

Key references:
  • D.P.K. Lun and B. Budianto, “Non-Contact Three-Dimensional Measurement Using the Learning Approach,” Learning Approaches in Signal Processing, Pan Stanford Publishing Pte. Ltd., pp.329 – 378, 2018. (Book Chapter)
  • Budianto, Daniel P.K. Lun and Y.H. Chan, “Robust Single-shot Fringe Projection Profilometry Based on Morphological Component Analysis,” IEEE Transactions on Image Processing, Vol.27, No.11, pp.5393-5405, November 2018.
  • Z.X. Xu, Y.H. Chan and D.P.K. Lun, “High-quality octa-level fringe pattern generation for improving the noise characteristics of measured depth maps”, Optics and Lasers in Engineering Vol. 98, pp. 99-106, November, 2017.
  • Budianto and Daniel Pak-Kong Lun, “Robust Fringe Projection Profilometry via Sparse Representation”, IEEE Transactions on Image Processing, Vol.25, No.4, pp.1726-1739, April 2016.
  • Budianto and Daniel Pak-Kong Lun, “Inpainting for Fringe Projection Profilometry Based on Geometrically Guided Iterative Regularization”, IEEE Transactions on Image Processing, Vol.24, No.12, pp.5531-5542, December 2015.
  • B. Budianto, Daniel Pak-Kong Lun and Tai-Chiu Hsung, “Marker encoded fringe projection profilometry for efficient 3D model acquisition”, Applied Optics, Vol. 53, Issue 31, pp. 7442-7453, November 2014.
  • William Wai-Lam Ng and Daniel Pak-Kong Lun, “Effective bias removal for fringe projection profilometry using the dual-tree complex wavelet transform”, Applied Optics, Vol. 51, Issue 24, pp.5909-5916, August 2012.
  • Tai-Chiu Hsung, Daniel Pak-Kong Lun and William W.L. Ng, “Efficient Fringe Image Enhancement Based on Dual-Tree Complex Wavelet Transform,” Applied Optics, Vol. 50, Issue 21, pp. 3973–86, July 2011.
Speech Enhancement Based on Sparsity of Speeches in Transform Domain
  • In this project, improved speech enhancement algorithms were developed based on the sparsity of speeches in different transform domains, such as the cepstrum domain.
  • A new expectation-maximization framework was also developed to provide the theoretical basis for the iterative enhancement process
  • The new algorithms work extremely well in different colored noise environments
  • A paper of this project received the best paper award in an international conference
Key references:
  • Daniel P.K. Lun, Tak-Wai Shen and K.C. Ho, “A novel expectation-maximization framework for speech enhancement in non-stationary noise environments”, IEEE Transactions on Audio, Speech and Language Processing, Vol. 22, Issue 2, pp.335-346, Feb 2014.
  • Daniel Pak-Kong Lun, Tak-Wai Shen, Tai-Chiu Hsung and Dominic K.C. Ho, “Wavelet Based Speech Presence Probability Estimator for Speech Enhancement”, Digital Signal Processing, Vol.22, Issue 6, pp.1161-1173, December 2012.
  • Tingtian Li, Daniel P.K. Lun and T.W. Shen, “Improved Expectation-Maximization Framework for Speech Enhancement Based on Iterative Noise Estimation”, Proceedings, 2015 IEEE International Conference on Digital Signal Processing, Singapore, pp.287-291, 2015. (Best paper award)