Image Inpainting Işık Barış Fidaner
Image Inpainting • Filling unknown area on the image by using surrounding information.
?
Unknown area
? Known area
?
Image Inpainting • Filling unknown area on the image by using surrounding information.
Inpainting direction
Known area
Unknown area
Three related fields of study Texture Synthesis Variational Image Inpainting
Constrained Texture Synthesis
Image Completion
Variational Image Inpainting • Filling unknown area on the image by using surrounding structure information.
Structure inpainting Unkown area Known area
Variational Image Inpainting Original image Inpainted image
Example:
Structure inpainting Unkown area Known area
Texture Synthesis • An input sample of a texture is given, and the goal is to produce more of that texture
Texture synthesis Texture sample
More of that texture
Texture Synthesis Input sample
Synthesis output
Example:
Texture synthesis Texture sample
More of that texture
Constrained Texture Synthesis • Filling unknown area on the image by using surrounding texture information.
Constrained synthesis Unkown area Known area
Constrained Texture Synthesis Input image
Inpainted image
Example: (illustrative)
Constrained synthesis Unkown area Known area
Image Completion • Filling unknown area on the image by using both texture and structure information.
Texture and structure inpainting Unkown area Known area
Image Completion Input image
Completed image
Example:
Texture and structure inpainting Unkown area Known area
Applications • Variational inpainting: – Scaling-up images – Reconstructing old photographs – Removal of overlaid text or graphics etc.
• Texture Synthesis: – 3D surface covering – Hole filling
• Image Completion – Removing unwanted objects from images – Filling-in the image blocks that are lost in transmission
Image Inpainting requires… • Good continuation of image geometry into and out of the area • Believable reproduction of textures • Good continuation of the structure of composite textures
Image Inpainting approaches • There are two main approaches: – Geometric image inpainting methods are based on continuation of geometric structures. – In contrast, texture synthesis methods fill image by reproducing textures. – There are also combined methods
Original image Inpainted image
Geometric image inpainting • First determine geometric structure defined by edge lines or isophotes on the image. • Model the geometry (usually with a partial differential equation). • Fill the spaces by propogating known information inward by the equation.
Geometric image inpainting • “Image Inpainting” Bertalmio et al. 2000 • Geometric information is given by L (Laplacian of image). • N is normal to unknown area boundary. • Change of L in direction of N is painted every iteration, propogating geometric information into the unknown area, until convergence occurs.
Geometric image inpainting
Texture Synthesis • A texture sample is given or extracted from known image areas. • We must produce new texture that visually appears similar to the sample • For an unknown region part, find a similar part in the texture sample. • Copy a pixel or a block from the sample into the unknown area.
Texture Synthesis • “Texture Synthesis by Non-parametric Sampling” Efros and Leung 1999 • Texture is modelled as a Markov Random Field • For each unknown pixel, a corresponding pixel in the sample texture is found, such that surrounding rectangular patches are similar (Patch comparison is weighted. Center part of the patch has more effect).
Texture Synthesis
Simultaneous texture & geometry • “Simultaneous Structure and Texture Image Inpainting” Bertalmio et al. 2003 • decompose the image into the sum of two functions with different basic characteristics, • reconstruct each one of these functions separately with structure and texture filling-in algorithms.
Simultaneous texture & geometry
Exemplar-based inpainting • “Region Filling and Object Removal by Exemplar-Based Image Inpainting” Criminisi et al. 2004 • Fill the area by copying similar patches from other parts of the image (believable repetition of texture) • Parts with more geometric information is filled first (good continuation of geometric structure)
Exemplar-based inpainting
Exemplar-based inpainting
Simple application
Simple application
Simple application 1 • Fill unknown area from a sample texture • Compare each unknown pixel to 50 random 5x5 patches from sample texture • Copy pixel by pixel
Simple application 1
Simple application 2 • Based on “Synthesizing Natural Textures” Ashikhmin 2001 • For each unknown pixel (row by row), compare to 4 candidate patches (3x3) from sample texture, fill with best match • Candidate patches are “shifted” from neighbour pixels
Simple application 2
Simple application 3 • Based on exemplar-based inpainting • Fill unknown area from a sample texture • Pick a 9x9 patch on boundary of the unknown area • Compare it to 500 random patches from texture sample • Copy best fitting patch
Simple application 3