Near Regular Texture Analysis for Image-Based Texture Overlay
Image-based texture overlay or retexturing is the process of augmenting a surface in an image or a video sequence with a new, synthetic texture. On the one hand, texture distortion caused by projecting the surface into the image plane should be preserved. On the other hand, only the texture albedo should be altered but shading and reflection properties should remain as in the original image. In many applications, such as augmented reality applications for virtual clothing, the surface material to be retextured is cloth. In this case, high frequency details, representing e.g. selfshadowing of the yarn structure, might also be a property that should be preserved in the augmented result.
This work specifically addresses the decomposition of images of deformed regular textures into its intrinsic parts which decompose the image into the appearance of the undeformed regular texture, a deformation field, a shading map representing lighting effects and additional high frequency details (see figure below). This is closely related to intrinsic image decomposition methods which decompose an arbitrary image into the product of an illumination component that represents lighting effects and a reflectance component related to the color of the observed material. In our decomposition model, the deformed regular texture (the estimated deformation field applied to the estimated appearance of the regular texture) can be seen as the reflectance part of our decomposition while the shading map and the high frequency details can be seen as the illumination component. Regular textures can be constructed by regularly tiling the texture space with the same texture element, called texel in the following. Textures that deviate geometrically and photometrically from a regular congruent tiling are often called near-regular textures (NRT) [Liu et al. 2004]. In contrast to regular textures, the texture elements appear geometrically and photometrically distorted in the image due to variations in the viewing angle, lighting conditions and partial occlusions. Nevertheless, they still exhibit certain topological regularities and relations as regular textures. We exploit this topological regularity to estimate the intrinsic parts of the given image of a near-regular texture. We first start by estimating the mean appearance of a texel and candidate positions of the texel in the image. From the estimated mean texel we synthetically generate an image of the regular texture. This image is used as reference in an image-based optimization method that registers two images not only geometrically but also photometrically, yielding a deformation field and a shading map. Finally, as the mean texel does not contain any high-frequency details, these can be estimated from the difference between the original image and the warped and shaded synthetic reference image.
Decomposition of the original texture (left) according into a regular texture, a deformation represented by a deformed mesh, an estimated shading map and estimated high frequency yarn structure (colors scaled).
Different texel appearance estimation results.
Different retexturing results. The leftmost image depicts the original input image.
Different retexturing results. The leftmost image depicts the original input image, the second image shows the extracted shading map.
Selected Publications
A. Hilsmann, D.C. Schneider, P. Eisert,
Warp-based Near-Regular Texture Analysis for Image-based Texture Overlay, Vision, Modeling, and Visualization Workshop 2011, Berlin, Germany, October 2011. [PDF]
A. Hilsmann, D.C. Schneider, P. Eisert,
Image-based Retexturing of Deformed Surfaces from a Single Image, Eurographics 2011, Poster, Llandudno, UK, April 2011. Best Poster Award. [PDF]