Restoring the Past: Virtual Restoration of Real-World Objects

Summary
We present a system to virtually restore damaged or historically significant objects without needing to physically change the object in any way. This work addresses both creating a restored synthetic version of the image as viewed from a camera and projecting the appropriate light patterns using digital projectors to give the illusion of the object being restored. The restoration algorithm minimizes an energy metric which enforces a set of contour criteria over the surface of the object. The visual compensation method develops a formulation that is designed to obtain bright compensations under a specified maximum amount of light. Our system contains three steps to virtually restore an object. In an initial acquisition step, a 3D model of the object to be restored is acquired on our object acquisition stage. The stage itself consists of a computer, camera, and multiple projectors and is shown below. Second, a photograph of the object is used to generate a synthetic image of a plausible appearance of the restored object. Finally, a visual compensation step determines an appropriate combination of the available projectors to generate a light-efficient compensation of the synthetically restored appearance from the second step.
A mini documentary on Discovery Breakthroughs in Science (DBIS).
This project is a collaboration with: Purdue University: Daniel Aliaga, Alvin Law, and Yu-Hong Yeung Indianapolis Museum of Art: Richard McCoy Eiteljorg Museum: Amy McKune IUPUI: Larry Zimmerman, anthropology department
A front and side view of our setup during a restoration demonstration at the Indianapolis Museum of Art. video

1. Object Acquisition
In the first step of our restoration process, we acquire a 3D model of the object to be restored as well as estimate the internal and external parameters of the projectors. The projectors must also be radiometrically calibrated. Our acquisition approach is based on Aliaga and Xu [2008] which performs a self-calibrating photo-geometric acquisition of the object and calibration of the projectors. Our final acquired models are of high resolution, with less than 0.1mm between point samples on the surface of the object.

1. Object Acquisition

In the first step of our restoration process, we acquire a 3D model of the object to be restored as well as estimate the internal and external parameters of the projectors. The projectors must also be radiometrically calibrated. Our acquisition approach is based on Aliaga and Xu [2008] which performs a self-calibrating photo-geometric acquisition of the object and calibration of the projectors. Our final acquired models are of high resolution, with less than 0.1mm between point samples on the surface of the object.

2. Image Restoration
Using a photograph of the object, a synthetic restoration of the object is performed. This process involves two steps. First, a color classification routine is employed to determine the original colors of the object as well as assign each pixel of the photograph to one of the discovered set of colors. This routine reduces the number of colors of the object to a small finite set and greatly assists in restoring the shapes of the color patches. Second, an energy minimization routine works to reshape the identified color patches. This iterative routine works to smooth out the color patches and their relationships with neighboring color patches. In the image below, (a) shows a portion of a deteriorated object. b) and c) show the synthetic equivalent before and after restoration. d) shows the final restored image of a).

2. Image Restoration

Using a photograph of the object, a synthetic restoration of the object is performed. This process involves two steps. First, a color classification routine is employed to determine the original colors of the object as well as assign each pixel of the photograph to one of the discovered set of colors. This routine reduces the number of colors of the object to a small finite set and greatly assists in restoring the shapes of the color patches. Second, an energy minimization routine works to reshape the identified color patches. This iterative routine works to smooth out the color patches and their relationships with neighboring color patches.

In the image below, (a) shows a portion of a deteriorated object. b) and c) show the synthetic equivalent before and after restoration. d) shows the final restored image of a).

3. Visual Compensation
Once a synthetic appearance of the restored object has been generated, the projectors need to project this appearance onto the physical object. Since excessive light intensity onto the object may further damage the object, selection of which projectors to illuminate different parts of the object is critical. In general, if a projector more directly illuminates a portion of the object, that projector should be depended on more heavily to illuminate that surface area. Similarly, if a projector is at a grazing angle to a portion of the object's surface, it should not contribute much light to that particular surface area. While simply selecting the optimally oriented projector per surface point of the object would achieve this goal, the resulting visual compensation would contain discontinuities due to adjacent surface points being illuminated by different projectors. Instead, we use a weighted balance of the given projectors to achieve a smooth compensation, emphasizing the optimally oriented projectors across the surface of the object. In the image below, a) and b) show projector combination schemes which provide a sub-optimal appearance (discontinuities and too dim, respectively) while c) demonstrates our approach to balancing the contribution of our projectors.

3. Visual Compensation

Once a synthetic appearance of the restored object has been generated, the projectors need to project this appearance onto the physical object. Since excessive light intensity onto the object may further damage the object, selection of which projectors to illuminate different parts of the object is critical. In general, if a projector more directly illuminates a portion of the object, that projector should be depended on more heavily to illuminate that surface area. Similarly, if a projector is at a grazing angle to a portion of the object's surface, it should not contribute much light to that particular surface area. While simply selecting the optimally oriented projector per surface point of the object would achieve this goal, the resulting visual compensation would contain discontinuities due to adjacent surface points being illuminated by different projectors. Instead, we use a weighted balance of the given projectors to achieve a smooth compensation, emphasizing the optimally oriented projectors across the surface of the object.

In the image below, a) and b) show projector combination schemes which provide a sub-optimal appearance (discontinuities and too dim, respectively) while c) demonstrates our approach to balancing the contribution of our projectors.

4. Restored Objects
We have applied our restoration process to six genuine historical artifacts and replicas borrowed from the Indianapolis Museum of Art and from the Eiteljorg Museum. Pre-processing work typically took about 1-2 hours per object with half the time spent acquiring images for model acquisition and radiometric calibration. Our system requires minimal interaction by the user who only has to tweak a few numerical parameters and provide some user guided hints during image restoration. We have applied our restoration process to three different genuine vessels from the Casas Grandes Cultural Region (1150-1450 A.D.) in northern Mexico. The region is better known as Paquimé by archaeologists. In the images below, a) shows a photograph of the original object, b) is a synthetic image of the restored object, and c) is a photograph of b) projected onto the physical object. d) shows one of three compensation images projected from the projectors onto the surface of the vessel to generate the restored appearance in c).
video
Below are images of our process applied on another Casas Grandes vessel. a) and b) show photographs of the object before and after virtual restoration respectively. c) and d) are close ups of the left side of the photographs which show the red color patches filled and black lines more polished.
video
Our virtual restoration process was applied to a third Casas Grandes vessel. a) shows a photograph of the original object, b) is a synthetic image of the restored object, and c) is a photograph of b) projected onto the physical object.
video
We have also worked with one of a pair of Kneeling Angel figurines from the Renaissance. The genuine figurines are attributed to Giovanni della Robbia of the 16th century. a) shows a photograph of the original object, b) is a synthetic image of the restored object, c) is a photograph of b) projected onto the physical object, and d) is one of two compensation images used to generate c).
video
Our fifth object was a replica of a Chinese vase from the Neolithic period (2500-2000 B.C.). a) shows a photograph of the original vase before virtual restoration. b) shows the target restored appearance generated by our image restoration routines, and c) is a photograph of the visually compensated vase. The two insets demonstrate the virtual restoration that occurred for the top left portion of the vase.
video
Our sixth object was a replica of a figurine from the Moche Culture in northern Peru. Here, a) shows a photograph of the original figurine. b) and c) show two virtually illuminated restorations of the figurine. These shading effects are generated by altering the synthetically generated image with some lighting effects, and this process can be used to accentuate different or interesting aspects of an object.
video

Publications

People