Page Areas:

Current Submenu:

Position Indication:



The Institute of Computer Graphics carries out research in a modern field that has been coined "visual computing". Our core disciplines cover the imaging, processing, visualization and displaying of visual data. These are enabled by new fields and technologies, such as light fields, projector-camera systems, responsive optics, mobile computing, and visual analytics.

Please select one of the following topics or years for detailed information.


Real-Time Video Enhancement Compensating For Macular Degeneration

Age-related macular degeneration (AMD) is a chronic and progressive eye condition with advancing central vision loss. In this article, we present two real-time video enhancement techniques that compensate for this loss of visual function: filtering and scaling. Since face recognition is problematic with central vision loss, video images are processed to improve face recognition. Our filtering method extends an existing adaptive contrast enhancement technique for the visually impaired and is based on the assumption that different content requires different filter parameterization. Each video image is segmented into facial and non-facial regions. The filter is then applied separately to each region using different parameters according to the particular content. Finally, the results are blended seamlessly. The scaling technique magnifies facial regions temporarily in each shot to improve recognition of faces and their expressions. Filtering and scaling can be combined when needed. Both methods were evaluated in a user study with AMD patients at the Medical University of Vienna.

Hochrieser,, M., Eisenkölbl, S. and Bimber, O., Real-Time Video Enhancement Compensating For Macular Degeneration, Submitted to IEEE Transactions on Circuits and Systems for Video Technology, 2012

Display Pixel Caching

A variety of standard video modes that stretch or zoom lower resolution video content linearly to take full advantage of large screen sizes have been implemented in TV sets. When content and screen aspect ratios differ, format proportions may be compromised, video content may be clipped, or screen regions may remain unused. Newer techniques, such as video retargeting and video upsampling, rescale individual video frames and can potentially match them to the display resolution and aspect ratio. However, none of these methods can display simultaneously more than is contained in a single frame.

We present a new video mode for television sets that we refer to as display pixel caching (DPC). It fills empty borders with spatially and temporally consistent information while preserving the original video format. Unlike related video modes, such as stretching, zooming and video retargeting, DPC does not scale or stretch individual frames. Instead, it merges the motion information from many subsequent frames to generate screen filling panoramas in a consistent manner. In contrast to state-of-the-art video mosaicing, DPC achieves real-time rates for high resolution video content while processing more complex motion patterns fully automatically. We compare DPC to related video modes in the context of a user evaluation.

Birklbauer, C., Grosse, M., Grundhoefer, A., Liu, T., and Bimber, O., Display Pixel Caching, In proceedings of 7th International Symposium on Visual Computing (ISVC'11), 2011

Birklbauer, C., Grosse, M., Grundhoefer, A., Liu, T., and Bimber, O., Display Pixel Caching, ACM Siggraph (poster+talk), 2011

Visual Computing Featuring Responsive Optics

The combination of advanced software algorithms and optics opens up new possibilities for display, imaging, and lighting. It makes possible responsive optical systems that adapt to particular situations automatically and dynamically. Visual computing is a relatively young research field that provides a foundation for many of these approaches. It represents a tight coupling between image synthesis, image analysis, and visual perception. While optics is all about image formation, visual computing deals with the general processing of images. This paper summarizes several examples that illustrate how graphics, vision, perception, and optics are combined to realize smart projectors, smart cameras, and smart light sources.

Bimber, O, Visual Computing Featuring Responsive Optics, invited paper, In proceedings of 27th Spring Conference on Computer Graphics (SCCG’11), 2011

application/pdfManuscript (9.5 MB)

Closed-Loop Feedback Illumination for Optical Inverse Tone-Mapping in Light Microscopy

We show that optical inverse tone-mapping (OITM) in light microscopy can improve the visibility of specimens, both when observed directly through the oculars and when imaged with a camera. In contrast to previous microscopy techniques, we pre-modulate the illumination based on the local modulation properties of the specimen itself. We explain how the modulation of uniform white light by a specimen can be estimated in real-time, even though the specimen is continuously but not uniformly illuminated. This information is processed and back-projected constantly, allowing the illumination to be adjusted on the fly if the specimen is moved or the focus or magnification of the microscope is changed. The contrast of the specimen's optical image can be enhanced, and high-intensity highlights can be suppressed. A formal pilot study with users indicates that this optimizes the visibility of spatial structures when observed through the oculars. We also demonstrate that the signal-to-noise (S/N) ratio in digital images of the specimen is higher if captured under an optimized rather than a uniform illumination. In contrast to advanced scanning techniques that maximize the S/N ratio using multiple measurements, our approach is fast because it requires only two images. This can be beneficial for image analysis in digital microscopy applications with real-time capturing demands.

Bimber, O., Klöck, D., Amano, T., Grundhöfer, A., and Kurz, D., Closed-Loop Feedback Illumination for Optical Inverse Tone-Mapping in Light Microscopy, IEEE Transactions on Visualization and Computer Graphics, 2010 (submitted: August 2009, accepted: July 2010)

application/pdfManuscript (37.6 MB)

Coded Aperture Projection

Coding a projector's aperture plane with adaptive patterns together with inverse filtering allow the depth-of-field of projected imagery to be increased. We present two prototypes and corresponding algorithms for static and programmable apertures. We also explain how these patterns can be computed at interactive rates, by taking into account the image content and limitations of the human visual system. Applications such as projector defocus compensation, high quality projector de-pixelation, and increased temporal contrast of projected video sequences can be supported. Coded apertures are a step towards next-generation auto-iris projector lenses.

Grosse, M., Wetzstein, G., Grundhöfer, A., and Bimber, O., Coded Aperture Projection, ACM Transactions on Graphics, 2010 (submitted: June 2009, accepted: March 2010)
ACM Siggraph 2010

Color Invariant Chroma Keying and Color Spill Neutralization for Dynamic Scenes and Cameras

We show how temporal backdrops that alternately change their color rapidly at recording rate can aid chroma keying by transforming color spill into a neutral background illumination. Since the chosen colors sum up to white, the chromatic (color) spill component is neutralized when integrating over both backdrop states. Being able to separate both states, however, additionally allows to compute high quality alpha mattes. Besides neutralizing color spill, our method is invariant to foreground colors and supports applications with real-time demands. In this article, we explain different realizations of temporal backdrops and describe how keying and color spill neutralization are carried out, how artifacts resulting from rapid motion can be reduced, and how our approach can be implemented to be compatible with common real-time post-production pipelines.

Grundhöfer, A., Kurz, D., Thiele, S. and Bimber, Color Invariant Chroma Keying and Color Spill Neutralization for Dynamic Scenes and Cameras, The Visual Computer - International Journal of Computer Graphics (Computer Graphics International), 2010

application/pdfManuscript (91.1 MB)

Superimposing Dynamic Range

We present a simple and cost-efficient way of extending contrast, perceived tonal resolution, and color space of reflective media, such as paper prints, hardcopy photographs, or electronic paper displays. A calibrated projector-camera system is applied for automatic registration, radiometric scanning and superimposition. A second modulation of the projected light on the surface of such media results in a high dynamic range visualization. This holds application potential for a variety of domains, such as radiology, astronomy, optical microscopy, conservation and restoration of historic art, modern art and entertainment installations. In our experiments, we achieved contrast ratios of up of 45,000-60,000:1 with a peak luminance of more than 2,750 cd/m^2, could technically re-produce more than 620 perceptually distinguishable tonal values. Furthermore, we attained color space extensions of up to a factor of 1.4 (compared to a regular projection on white screens) or factor of 3.3 (compared to regular paper prints under environment light). Thereby, the hardcopy resolution can be several thousand DPI or several hundred LPI, while luminance and chrominance are modulated with a registration error of less than 0.3 mm. Thus, compared with most existing interactive HDR displays, we support near distance viewing at a contrast frequency of up to 7 cpd (given our current registration precision and assuming a viewing distance of 50 cm).

Bimber, O. and Iwai, D., Superimposing Dynamic Range, In proceedings of ACM Siggraph Asia (ACM Transactions on Graphics), vo. 27, no. 5, article 150, 2008

Bimber, O., and Iwai, D., Superimposing Dynamic Range, Siggraph'08 (NewTech Demo Abstract), 2008

Grundhoefer, A. and Bimber, O., Superimposing Dynamic Range, Siggraph'08 (Poster+Talk), 2008

Source Code  (Neues Fenster)

Unsynchronized 4D Barcodes

We present a novel technique for optical data transfer between public displays and mobile devices based on unsynchronized 4D barcodes. We assume that no direct (electromagnetic or other) connection between the devices can exist. Time-multiplexed, 2D color barcodes are displayed on screens and recorded with camera equipped mobile phones. This allows to transmit information optically between both devices. Our approach maximizes the data throughput and the robustness of the barcode recognition, while no immediate synchronization exists. Although the transfer rate is much smaller than it can be achieved with electromagnetic techniques (e.g., Bluetooth or WiFi), we envision to apply such a technique wherever no direct connection is available. 4D barcodes can, for instance, be integrated into public web-pages, movie sequences, advertisement presentations or information displays, and they encode and transmit more information than possible with single 2D or 3D barcodes.

Langlotz, T. and Bimber, O., Unsynchronized 4D Barcodes. In proceedings of International Symposium on Visual Computing (ISVC'07), pp. 363-374, 2007

application/pdfManuscript (7.1 MB)

Radiometric Compensation Through Inverse Light Transport

Radiometric compensation techniques allow seamless projections onto complex everyday surfaces. Implemented with projector-camera systems they support the presentation of visual content in situations where projection-optimized screens are not available or not desired - as in museums, historic sites, air-plane cabins, or stage performances. We propose a novel approach that employs the full light transport between projectors and a camera to account for many illumination aspects, such as interreflections, refractions, shadows, and defocus. Pre-computing the inverse light transport in combination with an efficient implementation on the GPU makes the real-time compensation of captured local and global light modulations possible.

Wetzstein, G. and Bimber, O., Radiometric Compensation through Inverse Light Transport. In proceedings of Pacific Graphics, pp. 391-399, 2007

Wetzstein, G. and Bimber, O., Radiometric Compensation of Global Illumination Effects with Projector-Camera Systems. ACM Siggraph`06 (Poster), 2006

Imperceptible Calibration for Radiometric Compensation

We present a novel multi-step technique for imperceptible geometry and radiometry calibration of projector-camera systems. Our approach can be used to display geometry and color corrected images on non-optimized surfaces at interactive rates while simultaneously performing a series of invisible structured light projections during runtime. It supports disjoint projector-camera configurations, fast and progressive improvements, as well as real-time correction rates of arbitrary graphical content. The calibration is automatically triggered when mis-registrations between camera, projector and surface are detected.

Zollmann, S. and Bimber, O., Imperceptible Calibration for Radiometric Compensation. EUROGRAPHICS (short), pp. 61-64, 2007

application/pdfManuscript (9.1 MB)

Laser Pointer Tracking in Projector-Augmented Architectural Environments

We present a system that applies a custom-built pan-tilt-zoom camera for laser-pointer tracking in arbitrary real environments. Once placed in a building environment, it carries out a fully automatic self-registration, registrations of projectors, and sampling of surface parameters, such as geometry and reflectivity. After these steps, it can be used for tracking a laser spot on the surface as well as an LED marker in 3D space, using inter-playing fisheye context and controllable detail cameras. The captured surface information can be used for masking out areas that are critical to laser-pointer tracking, and for guiding geometric and radiometric image correction techniques that enable a projector-based augmentation on arbitrary surfaces. We described a distributed software framework that couples laser-pointer tracking for interaction, projector-based AR as well as video see-through AR for visualizations with the domain specific functionality of existing desktop tools for architectural planning, simulation and building surveying.

Kurz, D., Haentsch, F., Grosse, M., Schiewe, A., and Bimber, O., Laser Pointer Tracking in Projector-Augmented Architectural Environments, In proceedings of IEEE International Symposium on Mixed and Augmented Reality (ISMAR'07), pp. 19-26, 2007

application/pdfManuscript (1.0 MB)

Passive-Active Geometric Calibration for View-Dependent Projections onto Arbitrary Surfaces

Projecting images onto surfaces that are not optimized for projections becomes more and more popular. Such approaches will enable the presentation of graphical, image or video content on arbitrary surfaces. Virtual reality visualizations may become possible in everyday environments - without specialized screen material or static screen configurations. Upcoming pocket projectors will enable truly mobile presentations on all available surfaces of furniture or papered walls. The playback of multimedia content will be supported on natural stonewalls of historic sites without destroying their ambience through the installations of artificial projection screens. We present a hybrid technique for correcting distortions that appear when projecting images onto geometrically complex, colored and textured surfaces. It analyzes the optical flow that results from perspective distortions during motions of the observer and tries to use this information for computing the correct image warping. If this fails due to an unreliable optical flow, an accurate –but slower and visible– structured light projection is automatically triggered. Together with an appropriate radiometric compensation, view-dependent content can be projected onto arbitrary everyday surfaces. An implementation mainly on the GPU ensures fast frame rates.

Zollmann, S., Langlotz, T. and Bimber, O., Passive-Active Geometric Calibration for View-Dependent Projections onto Arbitrary Surfaces, Journal of Virtual Reality and Broadcasting, vol. 4, no. 6, 2007 (re-print from Workshop on Virtual and Augmented Reality of the GI-Fachgruppe AR/VR 2006)

Zollmann, S., Langlotz, T. and Bimber, O., Passive-Active Geometric Calibration for View-Dependent Projections onto Arbitrary Surfaces, Workshop on Virtual and Augmented Reality of the GI-Fachgruppe AR/VR, pp. 181-191, 2006

Real-Time Adaptive Radiometric Compensation

Our new radiometric compensation algorithm considers the human visual perception properties to reduce visible artefacts resulting from the limited dynamic range and brightness of projectors. It preserves a maximum of luminance and contrast and is implemented entirely on the GPU. Real-time frame rates are achieved for supporting animated and interactive content. Initially our algorithm performs an off-line analysis of the projection surface’s geomety and reflectance. The image content is then analyzed to determine the average luminance values, the amount of high spatial frequencies, and a luminance threshold map. The threshold map stores information about the maximum non-perceivable luminance differences for each pixel. The radiometric compensation is carried out in two passes: In the first pass the intensity values are translated and scaled globally depending on the surface reflectance and the image content itself. The result is analyzed for clipping errors. These errors are then blurred with a Gaussian kernel. The applied sigma is inverse proportional to the amount of high spatial frequencies in the local image areas. In the final pass the image intensities are translated and scaled globally, but the luminance values are also adjusted locally depending on the defocused clipping errors. Time dependent adaptation factors are used for global and local transformations to avoid popping artifacts in animated and interactive content.

Grundhoefer, A. and Bimber, O., Real-Time Adaptive Radiometric Compensation. In IEEE Transactions on Visualization and Computer Graphics (TVCG), vol. 14, no. 1, pp. 97-108, 2008, (submitted: August 2006, accepted: February 2007, electronic version published: March 2007)

Grundhoefer, A. and Bimber, O. Real-Time Adaptive Radiometric Compensation. ACM Siggraph`06 (Poster), 2006

Augmenting Large Scale Optical Holograms

Large scale optical holograms require large scale display technology for combining them with interactive graphical elements. Shuttered projection screens (SPS) can be used to sequentially display stereoscopic graphics (in the diffuse mode) and to reconstruct the holographic content (in the transparent mode). While the SPS is shuttered with 50Hz, the stereo pairs are synchronized and time-modulated at approximately 100Hz. Depth information of the holographic content are required to create consistent occlusion and illumination effects with the graphical content. A two-lens stereo camera system can be used for scanning the hologram partially. The different point clouds have to merge into a common coordinate system to form the whole surface. Due to the limited resolution of the range sensor, small gaps appear between the actual surface points. Instead of triangulating the points into a mesh of triangle primitives, the points remain unconnected. They are rendered as point primitives (splatted) with appropriate radii to fill the gaps of missing surface information. The splat size and resolution is adapted dynamically with respect to the observer’s position to ensure interactive frame rates.

Bimber, O., Merging Graphics and Holograms. Journal of Holography and Speckle, vol.3, no. 2, pp. 73-79, 2006

Bimber, O., Augmenting Holograms. IEEE Computer Graphics and Applications (CG&A), vol. 26, no. 5, pp. 12-17, 2006

Compensating Indirect Scattering for Immersive and Semi-Immersive Projection Displays

Concavely shaped projection screens, such as CAVEs, two-sided workbenches, domes, or cylinders scatter a fraction of light to other screen portions. The amount of indirect illumination adds to the directly projected image and causes the displayed content to appear partially inconsistent and washed out. We have developed a reverse radiosity method that compensates first-level and higher-level secondary scattering effects in real-time. The images appear more brilliant and uniform when reducing the scattering contribution. A numerical solution is approximated with Jacobi iteration for a sparse-matrix linear equation system on the GPU. Efficient data structures allow packing the required data into textures which are processed by pixel shaders. Frame-buffer objects are used for a fast exchange of intermediate iteration results, and enable computations with floating point precision. Our algorithm’s result can be optimized for quality or performance.

Bimber, O., Grundhoefer, A., Zeidler, T., Danch, D., and Kapakos, P. Compensating Indirect Scattering for Immersive and Semi-Immersive Projection Displays. In proceedings of IEEE Virtual Reality (IEEE VR'06), pp. 151-158, 2006

application/pdfManuscript (3.7 MB)

Multi-Focal Projection

Many multi-projector rendering techniques exist that aim at creating a high consistency of image geometry, intensity and color. We proposed a concept and a solution for considering and optimizing a fourth image property – its focus. We describe a novel multi-focal projection concept that applies conventional video projectors and camera feedback. Multiple projectors with differently adjusted focal planes, but overlapping image areas are used. They can be either arbitrarily positioned in the environment, or can be integrated into a single projection unit. During an automatic one-time calibration process, structured light projection together with camera feedback allows to measure the relative focus value of every projector pixel on an arbitrary diffuse surface. Thereby, the focus values are geometrically and radiometrically corrected. If this is known, a final image with minimal defocus can be composed in real-time from individual pixel contributions of all projectors. Our technique is independent of the surfaces’ geometry, color and texture, of the environment light, as well as of the projectors’ parameters (i.e., position, orientation, luminance and chrominance).

Bimber, O. and Emmerling, A., Multi-Focal Projection: A Multi-Projector Technique for Increasing Focal Depth. In IEEE Transactions on Visualization and Computer Graphics (TVCG), vol. 12, no. 4, pp. 658-667, 2006

Bimber, O., Wetzstein, G., Emmerling, A., and Nitschke, C. Enabling View-Dependent Stereoscopic Projection in Real Environments. In proceedings of International Symposium on Mixed and Augmented Reality (ISMAR'05), pp. 14-23, 2005

Enabling View-Dependent Stereoscopic Projection in Real Environments

With this work we take a first step towards an ad-hoc stereoscopic projection within real environments. We show how view-dependent image-based and geometric warping, radiometric compensation, and multi-focal projection enable a view-dependent visualization on ordinary (geometric complex, colored and textured) surfaces within everyday environments. All these techniques are accomplished at interactive rates and on a per-pixel basis for multiple interplaying projectors. Special display configurations for immersive or semi-immersive VR/AR applications that require permanent and artificial projection canvases might become unnecessary. Such an approach does not only offer new possibilities for augmented reality and virtual reality, but also allows merging both technologies. This potentially gives some application domains – like architecture – the possibility to benefit from the conceptual overlaps of AR and VR.
Special thanks to the Faculty of Architecture, Bauhaus-University Weimar for their support.

Bimber, O., Wetzstein, G., Emmerling, A., and Nitschke, C. Enabling View-Dependent Stereoscopic Projection in Real Environments. In proceedings of International Symposium on Mixed and Augmented Reality (ISMAR'05), pp. 14-23, 2005

Interacting with Augmented Holograms

Holography and computer graphics are being used as tools to solve individual research, engineering, and presentation problems within several domains. Up until today, however, these tools have been applied separately. Our intention is to combine both technologies to create a powerful tool for science, industry and education. We are currently investigating the possibility of integrating computer generated graphics and holograms. We presents several applications of interaction techniques to graphically enhanced holograms and give a first glance on a novel method that reconstructs depth from optical holograms.
Special thanks to Deutsche Forschungsgemeinschaft (DFG) for their support.

Bimber, O., Augmenting Holograms. IEEE Computer Graphics and Applications (CG&A), vol. 26, no. 5, pp. 12-17, 2006

Bimber, O., Zeidler, T., Grundhoefer, A., Wetzstein, G., Moehring, M., Knoedel, S., and Hahne, U. Interacting with Augmented Holograms. In proceedings of SPIE Conference on Practical Holography XIX: Materials and Applications, 2005

Embedded Entertainment with Smart Projectors

Video projectors will play a major role in future home entertainment and edutainment applications – ranging from movies and television, over computer games, to multimedia presentations. With video projectors images can be displayed that are larger than the devices themselves. However, we have to give up living space and ambience to set up artificial canvases that have to be as large as the desired image. Smart projectors are able to display correct images onto arbitrary existing screen surfaces, like wallpapered walls or window curtains. Thus it can function without an artificial canvas and consequently leaves a bit more freedom to us in the decision on how to arrange our living space. Our smart projectors combine camera feedback with structured light projection to gain information about the screen surface and the environment. The calibration of such a device is fast, fully automatic and robust, and the correction of video signals can be achieved in real-time. Neither geometry information nor projector or camera parameters need to be known. Instead, the entire calibration and correction (geometry and color) is done on a per-pixel level – supported by modern pixel shader hardware. Such devices might make it possible to convert your bookshelf into a TV screen, or your kid’s closet into an interactive virtual playground.
Special thanks to the Bennert Group for their support.

Bimber, O., Emmerling, A., and Klemmer, T. Embedded Entertainment with Smart Projectors. IEEE Computer, vol. 38, no. 1, pp. 56-63, 2005

Superimposing Pictorial Artwork with Projected Imagery

Pictorial artwork, such as paintings and drawings, can tell interesting stories. The capabilities of museums to communicate such and other information, however, are clearly limited. Text legends and audio guides can mediate facts, but offer little potential for presenting visual content, such as embedded illustrations, pictures, animations, and interactive elements. Also due to the difficult economic situation, edutainment is becoming an important factor for museums. By applying new media technologies – such as computer graphics, virtual reality and augmented reality– exhibit oriented information might be communicated more effectively, but certainly in a more exciting way. We describe a novel technological approach, a mathematical model, a real-time rendering algorithm and examples of presentation techniques for integrating almost any kind of visual information directly into pictorial artwork. It allows displaying such information while keeping the observers’ attention on the original artifact, and does not require additional screens.
Special thanks to the Museum het Rembrandthuis, Amsterdam for their support.

Bimber, O., Coriand, F., Kleppe, A., Bruns, E., Zollmann, S., and Langlotz, T. Superimposing Pictorial Artwork with Projected Imagery. IEEE MultiMedia, vol. 12, no.1, pp. 16-26, 2005

The HoloStation

The Holographic Workstation (short: HoloStation) is a desktop display that allows the integration of interactive stereoscopic or auto-stereoscopic graphical elements into high-quality optical holograms.
Archaeologists, for instance, use optical holograms to archive and investigate ancient artefacts. The distribution of copy holograms enables scientists to perform their research without having access to the original artefacts or inaccurate replicas. Combining these static holograms with interactive computer graphics will allow them to integrate real-time simulation data or to perform experiments that require a direct user interaction, such as packing reconstructed soft-tissue into a hologram of a fossilized dinosaur skull. In addition, specialized interaction devices can simulate haptic feedback of the holographic and the graphical content while performing such interactive tasks. An entire collection of artefacts will fit into a single album of holographic recordings, while a light-box-like display can be used for visualization and interaction.
The current HoloStation prototype can be used in combination with many different holograms, such as transmission and reflection types. It is a compact and flexible display that provides several forms of interaction via integrated touch-screen and real-time range scanner.
Special thanks to Deutsche Forschungsgemeinschaft (DFG) for their support.

Bimber, O., Augmenting Holograms. IEEE Computer Graphics and Applications (CG&A), vol. 26, no. 5, pp. 12-17, 2006

Bimber, O. Combining Optical Holograms with Interactive Computer Graphics. In IEEE Computer, vol. 37, no. 1, pp. 85-91, January 2004

Bimber, O. HoloGraphics: Combining holograms with interactive computer graphics. The Holographer (online Journal), April 2004

Bimber, O., Combining Optical Holograms with Interactive Computer Graphics. SPIE International Technical Group Newsletter, vol. 16, no. 1, pp, 1+9, June 2005

Combining Optical Holograms with Interactive Computer Graphics

A hologram is a photometric emulsion that records interference patterns of coherent light. The recording itself stores amplitude, wavelength and phase information of light waves. In contrast to simple photographs (which can record only amplitude and wavelength information), holograms have the ability to reconstruct complete optical wavefronts. This results in a three-dimensional appearance of the captured scenery which is observable from different perspectives.
Today, many applications for optical holograms exist. Examples include interferometry, copy protections, data storage and holographic optical elements. Optical holograms, however, are static and lack in interactivity. Electro-holography aims at the computer-based generation and display of holograms in real time. Since a massive amount of data has to be processed, transmitted and stored to create holograms, today’s computer technology still sets the limits of electro-holography. To overcome some of the performance issues, advanced reduction and compression methods have been developed. This results in electro-holograms that are interactive, but small, low-resolution and pure in color.
A novel approach has been proposed that combines holography, autostereoscopy and projection technology to integrate interactive graphical elements into high-quality optical holograms. Beside the conceptual idea and the optical solutions, efficient rendering techniques and a proof-of-concept prototype are presented. Furthermore, potential application areas, such as museums and scientific visualization are outlined.
Special thanks to Deutsche Forschungsgemeinschaft (DFG) for their support.

Bimber, O., Augmenting Holograms. IEEE Computer Graphics and Applications (CG&A), vol. 26, no. 5, pp. 12-17, 2006

Bimber, O. Combining Optical Holograms with Interactive Computer Graphics. In IEEE Computer, vol. 37, no. 1, pp. 85-91, January 2004

Bimber, O. HoloGraphics: Combining holograms with interactive computer graphics. The Holographer (online Journal), April 2004

Bimber, O., Combining Optical Holograms with Interactive Computer Graphics. SPIE International Technical Group Newsletter, vol. 16, no. 1, pp, 1+9, June 2005

Consistent Illumination within Augmented Environments

To achieve a consistent lighting situation between real and virtual environments is important for convincing augmented reality (AR) applications.
A rich pallet of algorithms and techniques have been developed that match illumination for video- or image-based augmented reality. However, very little work has been done in this area for optical see-through AR. We believe that the optical see-through concept is currently the most advanced technological approach to provide an acceptable level of realism and interactivity.
Methods have been developed which create a consistent illumination between real and virtual components within an optical see-through environment – such as the Virtual Showcase. Combinations of video projectors and cameras are applied to capture reflectance information from diffuse real objects and to illuminate them under new synthetic lighting conditions. For diffuse objects, the capturing process can also benefit from hardware acceleration – supporting dynamic update rates. To handle indirect lighting effects (like color bleeding) an off-line radiosity procedure is outlined that consists of multiple rendering passes. For direct lighting effects (such as simple shading, shadows and reflections) hardware accelerated techniques are described which allow to achieve interactive frame rates. The reflectance information is used in addition to solve a main problem of a previously introduced technique which creates consistent occlusion effects for multiple users within such environments.

Bimber, O., Grundhoefer, A., Wetzstein, G., and Knoedel S. Consistent Illumination within Optical See-Through Augmented Environments. In proceedings of IEEE/ACM International Symposium on Augmented and Mixed Reality (ISMAR'03), pp. 198-207, 2003

application/pdfManuscript (285Bytes)

Augmented Reality Digital Storytelling

Interactive digital storytelling techniques are recently being applied in combination with new media forms, such as virtual reality (VR) and augmented reality (AR). Thereby the technological progress that is being made within these areas allows shifting interactive digital storytelling more and more into the third dimension and into the physical world. One of the main advantages of this transition is the possibility to communicate information more effectively with digital means by telling stories that can be experienced directly within a real environment or in combination with physical objects. The user experience is thus transformed from relating different pieces of information to one another to ‘living through’ the narrative. The perceptual quality and the unique aura of a real environment (e.g., a historical site) or object (e.g., an ancient artifact) cannot be simulated by today’s technology. Thus it is not possible to substitute them with virtual or electronic copies without them losing their flair of originality. This circumstance can be a crucial reason for using augmented reality as a technological basis for interactive digital storytelling. Several research groups apply personal displays, such as head-mounted displays (HMDs) to realize AR-based new media experiences. These all-purpose display types, however, have to face technological problems that up until today have not been sufficiently solved1. These shortcomings cause a substantial credibility gap if a certain level of realism is required.
We want to discuss how an application-customized type of augmented reality display –the Virtual Showcase– overcomes most of these technological shortcomings, and how it can be used as a new platform for digital storytelling.
To underline our statements we discuss and evaluate a case study that focuses on using AR digital storytelling to communicate scientific information to a novice audience in a museum context. We have introduced the application that led to our case study in a previous publication. In this paper, we want to describe the technical components that have been developed to realize this application.
In addition, we present a first user feedback and illustrate our current efforts of improvement.

Bimber, O., Encarnacao. L.M., and Schmalstieg, D. The Virtual Showcase as a new Platform for Augmented Reality Digital Storytelling. In proceedings of Eurographics Workshop on Virtual Environemnts IPT/EGVE'03, pp. 87-95, 2003

application/pdfManuscript (284Bytes)

Real-Time Image Warping for Curved Mirror Displays

Displays which require a non-linear pre-distortion to present undistorted gfx on non-planar surfaces, or to neutralize optical distortion usually apply multi-pass rendering techniques. Projective textures or uniform grids are used to deform the image generated during the first pass before it is displayed as a texture map during the final pass. However, these approaches do not consider the error that is generated from a piecewise linear texture interpolation to adapt the underlying geometry.
Curved mirror displays, for instance, that stereoscopically produce three-dimensional gfx generally don't pre-distort the graphics before they are displayed. Yet, some systems apply additional optics (such as lenses) to stretch or undistort the reflected image. But these devices constrain the observer to a single point of view or to very restricted viewing zones. For the Virtual Showcase mirror display, however, a view-dependent rendering is required to support freely moving observers. The existing image warping algorithms for curved Virtual Showcases predistort a uniform image grid and consequently do not consider the local error that is generated from a piecewise linear texture interpolation. These algorithms can only produce an acceptable image quality within a significantly large amount of rendering time.
The algorithm that is presented improves the uniform image warping techniques for curved Virtual Showcases by defining an appropriate error metric and implementing a quadtree-based selective refinement method that generates adapted local levels of detail. It approaches to produce a predefined image quality within a minimum amount of rendering time. While display specific details of our algorithm are explained based on the Virtual Showcase display, its general functionality is valid for other non-linear displays -such as for curved projection-based systems.

Bimber, O., Froehlich, B., Schmalstieg, D., and Encarnacao, L.M. Real-Time View-Dependent Image Warping to correct Non-Linear Distortion for Curved Virtual Showcase Displays. Computers and Graphics - The international Journal of Systems and Applications in Computer Graphics, vol. 27, no.4, pp. 512-528, 2003

application/pdfManuscript (294Bytes)

Augmented Paleontology

Paleontology is filled with mysteries about the plants and animals that lived thousands, millions, even billions of years before the first humans walked the earth. To answer questions about these organisms, paleontologists rely on the excavation, analysis, and interpretation of fossils. Embedded and preserved in the earth’s crust, fossils are the remains or traces of ancient life forms, including bones, teeth, shells, leaf imprints, nests, and footprints.Fossils can disclose how organisms evolved over time and their relationship to one another. While they reveal much, such as the general shape and size of ancient living things, fossils keep us guessing about these organisms’ color, sound, and—most significantly—their behavior. For several years, modern paleontologists have used 3D computer graphics to help reconstruct these pieces of the past. State-of-the-art scanning technology produces 3D fossil replicas that scientists can process and study without physical constraints. Paleontologists typically generate volumetric data sets for analysis, such as magnetic resonance imaging or computed axial tomography scans, and they use surface models for digital preservation and reproduction. To study ontogeny—an organism’s growth and form—paleontologists apply mathematical models for simulation and visualization. Likewise, computer animations help study dinosaur locomotion. Beyond building knowledge of our world, the results of this work influence how dinosaurs appear in museums, illustrations, and movies, and as toys. In the past 40 years, technological advances have continued to blur the boundary between real and computer-generated worlds. Augmented reality leverages this technology to provide an interface that enhances the real world with synthetic supplements. Paleontologists can use AR to present virtual data, such as 3D computer graphics, directly within a real environment rather than on a flat monitor. We coined the term augmented paleontology to refer to the application of AR to paleontology. AP seeks to support paleontologists in their research, and communicate the results of paleontology to museum visitors in an exciting and effective way. An interdisciplinary team of paleontologists, graphics designers, and computer scientists has already applied the AP interface to soft-tissue reconstruction and the study of dinosaur locomotion.

Bimber, O., Gatesy, S.M., Witmer, L.M., Raskar, R. and Encarnacao, L.M. Merging Fossil Specimens with Computer-Generated Information. IEEE Computer, vol. 35, no.1 , pp. 45-50, 2002

Occlusion Shadows

Projection-based augmented reality systems, such as the Virtual Showcase, share many positive properties of projection-based virtual environments. These displays provide high resolution, improved consistency of eye accommodation and convergence, little motion sickness potential, and the possibility of an integration into common working environments. One of the main challenges for projection-based AR systems as well as for head-mounted optical see-through displays is the generation of correct occlusion effects between virtual and real objects. Additionally shadows of virtual objects cast onto real ones and consistent illumination of the real and virtual scenery are often difficult to achieve. We introduce projector-based illumination techniques for view-dependent optical seethrough AR displays. This approach has the potential to solve all of the above mentioned problems. Here, we focus on using projector-based illumination for creating correct occlusion effects for mixed reality configurations.
We have implemented and tested such a system in the context of the Virtual Showcase, which consists of a horizontal projection screen and a convex half-silvered mirror assembly. Virtual and real objects can be displayed in the same space inside the showcase. The original Virtual Showcase used a standard light bulb to illuminate real objects. This setup does not provide very much control over the lighting situation. By using a computer-controlled video-projector as a replacement for the simple light bulb, we are able to fully control the lighting situation inside the showcase on a perpixel basis. Our main contribution is a solution to the problem of correct occlusion for mixed reality scenarios with viewdependent optical see-through displays. Our method produces correct occlusion effects between virtual and real objects by projecting shadows onto real objects located behind virtual ones using projector-based illumination.

Bimber, O. and Froehlich, B. Occlusion Shadows: Using Projected Light to Generate Realistic Occlusion Effects for View-Dependent Optical See-Through Displays. In proceedings of International Symposium on Mixed and Augmented Reality (ISMAR’02), pp. 186-195, 2002

application/pdfManuscript (295Bytes)

The Virtual Showcase

Intuitive access to information in habitual realworld environments is a challenge for information technology. An important question is how can we enhance established and well-functioning everyday environments rather than replace them by virtual environments (VEs)? Augmented reality (AR) technology has a lot of potential in this respect because it augments realworld environments with computer-generated imagery.
Today, most AR systems use see-through head-mounted displays, which share most of the disadvantages of other head-attached display devices. We introduce a new projection-based AR display system—the Virtual Showcase. The Virtual Showcase has the same form factor as a real showcase, making it compatible with traditional museum displays. Real scientific and cultural artifacts are placed inside the Virtual Showcase allowing their 3D graphical augmentation. Inside the Virtual Showcase, virtual representations and real artifacts share the same space providing new ways of merging and exploring real and virtual content. Solely virtual exhibits may also be displayed. The virtual part of the showcase can react in various ways to a visitor, which provides the possibility for intuitive interaction with the displayed content. Another interesting aspect of our system is its support for two to four simultaneously tracked users looking at the Virtual Showcase from different sides. This feature allows the collaborative exploration of artifacts shown in the Virtual Showcase. These interactive showcases contribute to ambient intelligent landscapes, where the computer acts as an intelligent server in the background and visitors can focus on exploring the exhibited content rather than on operating computers. The Virtual Showcase consists of two main parts: a convex assembly of halfsilvered mirrors and a graphics display. So far, we’ve built Virtual Showcases with two different mirror configurations. Our first prototype consists of four halfsilvered mirrors assembled as a truncated pyramid. Our second prototype uses a single mirror sheet to form a truncated cone. We placed these mirror assemblies on top of a projection screen. Users can see real objects inside the showcase through the half-silvered mirrors merged with the graphics displayed on the projection screen. We illuminated the showcases’ contents with a controllable light source while presenting view-dependent stereoscopic graphics to the observer. For our current prototypes, we use standard shutter glasses controlled by infrared emitters. Head tracking is accomplished using an electromagnetic tracking device. Our pyramid shaped prototype supports up to four viewers simultaneously looking at the showcase from four different sides. Our cone-shaped prototype provides a seamless surround view onto the displayed artifact.

Bimber, O., Froehlich, B., Schmalstieg, D., and Encarnacao, L.M. The Virtual Showcase. IEEE Computer Graphics & Applications, vol. 21, no.6, pp. 48-55, 2001

The Extended Virtual Table

We describe a prototype of an optical extension for table-like rear-projection systems -the Extended Virtual Table. A large half-silvered mirror is used as the optical combiner to unify a virtual and a real workbench, whereby the shortcomings that are related to the Reflective Pad and Transflective Pad (i.e., indirect line-of-sight, limited field of view, and tracking distortion) are reduced. The virtual workbench has been enabled to display computer graphics beyond its projection boundaries and to combine virtual environments with the adjacent real world. A variety of techniques is described that allow indirect interaction with virtual objects through the mirror.
Systems, such as the Extended Virtual Table, approach a conceptual and technical extension of traditional Virtual Reality by means of Augmented Reality (xVR), and a seamless integration of such technology into habitual work environments. Furthermore, optical distortions that are caused by the half-silvered mirror combiner and the projector, as well as non-optical distortions caused by the tracking device are analyzed and appropriate compensation methods are described.

Bimber, O., Encarnacao, L.M. and Branco, P. The Extended Virtual Table: An Optical Extension for Table-Like Projection Systems. Presence: Teleoperators and Virtual Environments, vol.10, no. 6, pp. 613-631, 2001

application/pdfManuscript (302Bytes)

Projection-Based Augmented Engineering

Traditional Augmented Reality (AR) displays, such as head-mounted devices entail a number of technological and ergonomic drawbacks. Their limited resolution, the restricted field-of-view, the visual perception issues that are due to the fixed focal length caused by a constant and head-attached image plane, the increased incidence of discomfort provoked by simulation sickness and their cumbersomeness prevent their usage in a number of application areas. To overcome some of these drawbacks, but also to open new application areas for AR, we propose a projectionbased AR concept that combines spatially aligned optical see-through elements (essentially half-silvered mirror beam-splitters) with off-the-shelf projection-based Virtual Reality displays. This concept offers possibilities to combine the advantages of both technologies: the well established projection-based Virtual Reality with the potentials of Augmented Reality. We describe the early stages of a new proof of-concept prototype that has been developed to extend the scope of projection-based AR towards the engineering domain.

Bimber, O., Stork, A., and Branco, P. Projection-based Augmented Engineering. In proceedings of Human-Computer Interaction International (HCII’01), vol. 1, pp. 787-791, 2001

application/pdfManuscript (290Bytes)

Seamless Integration of Virtual Reality into habitual Workplaces

Employing virtual reality (VR) technology, computer-based work environments are clearly evolving into a major component of the next-generation workplace. Over the past few years, VR has become a practical reality for many applications, thanks to a number of technology inventions. Among these has been the development of large-screen display systems that make use of stereoscopic projection. Systems such as the Virtual Table and Responsive Workbench both employ a horizontal projection philosophy, whereas other devices, such as the Powerwall and surround-screen projection systems (SSPS) like the CAVE , also employ vertical back-projection. Due to their effective application, these devices are gaining greater acceptance by the research and industrial user communities. Such devices offer a large field of view, brilliant image quality, high resolution and (especially the table-like devices) a promising compatibility with traditional workspaces. To support this integration, however, ergonomic and human-centered interaction techniques must be developed.
We introduce ideas, proposed technologies, and initial results for a seamless integration of virtual reality in habitual workspaces, thus successively transforming them into efficient, high-tech work environments.

Bimber, O., Encarnacao, L.M. and Stork, A. Seamless integration of virtual reality in habitual workplaces. Journal for Industrial Science, Munich University of Technology, vol.55, no.2, pp. 103-112, 2001

application/pdfManuscript (299Bytes)

Transflective Props

We describe the use of a hand-held semi-transparent mirror to support augmented reality tasks with back-projection systems. This setup overcomes the problem of occlusion of virtual objects by real ones linked with such display systems. The presented approach allows an intuitive and effective application of immersive or semi-immersive virtual reality tasks and interaction techniques to an augmented surrounding space. Thereby, we use the tracked mirror as an interactive image-plane that merges the reflected graphics, which are displayed on the projection plane, with the transmitted image of the real environment. In our implementation, we also address traditional augmented reality problems, such as real-object registration and virtual-object occlusion. The presentation is complemented by a hypothesis of conceivable further setups that apply transflective surfaces to support a mixed reality (i.e., combined AR and VR) environment.

Bimber, O., Encarnacao, L.M., and Schmalstieg, D. Augmented Reality with Back-Projection Systems using Transflective Surfaces. Computer Graphics Forum (proceedings of Eurographics 2000 - EG’2000), vol. 19, no. 3, pp. 161-168, 2000

application/pdfManuscript (300Bytes)

Reflective Props

We introduce the idea of using real mirrors in combination with rear-projection systems for the purpose of interacting with and navigating through the displayed information. Subsequently a derived application is described. For this, we use a hand-held planar mirror and address two fundamental problems of applying head tracking with rear-projection planes: the limited viewing volume of these environments and their incapability of simultaneously supporting multiple observers. Furthermore, we describe the possibility of combining a reflective pad with a transparent one, thus introducing a complementary tool for interaction and navigation.

Bimber, O., Encarnacao, L.M., and Schmalstieg, D. Real Mirrors Reflecting Virtual Worlds. In proceedings of IEEE Virtual Reality (IEEE VR’00), pp. 21-28, 2000

application/pdfManuscript (313Bytes)

Sketch-Based and Multi-Modal Interaction

Leonardo da Vinci's drawings of machines and other objects illustrate one of the most fundamental purposes of sketches: the ability to communicate design and functionality to others. Nowadays, it is widely accepted that sketching is a form of critical, reflective dialog that handles communication on one or more different levels of abstraction simultaneously. Various approaches have been taken to support this kind of dialog between humans and computers, and to build human-computer interfaces that are able to interpret such freehand sketches for different purposes.
In this context, the creation or reconstruction of 3D objects from 2D sketches is of major concern in many application areas. This so-called 'pencil-and-paper' approach is used for rapidly designing approximate threedimensional scenes. While some systems analyze the orthographic or perspective projections to reconstruct 3D shapes that, based on psychological assumptions, are most plausible to the human observer, others interpret 2D gestures while the objects are sketched. Within the last decade, the conceptual design phase has been increasingly supported by sketch systems that allow the expression of ideas on a computer-aided, but still human-centered basis. However, putting an emphasis on sketching, most of these systems are sealed off from real-world applications rather than being generally applicable as components.
We introduce a framework for sketch-based interaction within three-dimensional virtual environments. We describe each layer of the framework using illustrative examples depicting its realization. Furthermore, we want to present a variety of domain-specific applications of sketching within 3D virtual environments based on our framework instead of implementing yet another application for sketching.

Bimber, O., Encarnacao, L.M. and Stork, A. A Multi-Layered Architecture for Sketch-based Interaction within Three-dimensional Virtual Environments. Computers and Graphics - The international Journal of Systems and Applications in Computer Graphics, vol. 24, no. 6, pp. 851-867, 2000

application/pdfManuscript (303Bytes)