Page Areas:

Current Submenu:

Position Indication:



The Institute of Computer Graphics carries out research in a modern field that has been coined "visual computing". Our core disciplines cover the imaging, processing, visualization and displaying of visual data. These are enabled by new fields and technologies, such as light fields, projector-camera systems, responsive optics, mobile computing, and visual analytics.

Please select one of the following topics or years for detailed information.


Subobject Detection through Spatial Relationships on Mobile Phone

We present a novel image classification technique for detecting multiple objects (called subobjects) in a single image. In addition to image classifiers, we apply spatial relationships among the subobjects to verify and to predict locations of detected and undetected subobjects, respectively. By continuously refining the spatial relationships throughout the detection process, even locations of completely occluded exhibits can be determined. Finally, all detected subobjects are labeled and the user can select the object of interest for retrieving corresponding multimedia information. This approach is applied in the context of PhoneGuide, an adaptive museum guidance system for camera-equipped mobile phones.
We show that the recognition of subobjects using spatial relationships is up to 68% faster than related approaches without spatial relationships. Results of a field experiment in a local museum illustrate that unexperienced users reach an average recognition rate for subobjects of 85.6% under realistic conditions.

Brombach, B., Bruns, E. and Bimber, O., Subobject Detection through Spatial Relationships on Mobile Phones, In proceedings of International Conference of Intelligent User Interfaces (IUI2009), pp. 267-276, 2009

application/pdfManuscript (9.3 MB)


Phone-to-Phone Communication for Adaptive Image Classification

We present a novel technique for adapting local image classifiers that are applied for object recognition on mobile phones through ad-hoc network communication between the devices. By continuously accumulating and exchanging collected user feedback among devices that are located within signal range, we show that our approach improves the overall classification rate and adapts to dynamic changes quickly. This technique is applied in the context of our PhoneGuide system -- a mobile phone based museum guidance framework that combines pervasive tracking and local object recognition for identifying a large number of objects in uncontrolled museum environments. We explain a technique that distributes the user feedback information during runtime through ad-hoc network connections between local devices. By doing so, we enforce cooperative classification improvements during the actual stay of the visitors. The general functionality of our technique has been tested with a small number of real devices in a museum. For proving its scalability, however, we have developed a simulator that evaluates our method for many hundred devices under several conditions. The simulation parameters have all been gathered in a museum, and are therefore realistic. We will show that ad-hoc phone-to-phone synchronization not only leads to higher overall classification rates, but also to quicker adaptations to dynamic changes during runtime.

Bruns, E. and Bimber, O., Phone-to-Phone Communication for Adaptive Image Classification, In proceedings of International Conference on Advances in Mobile Computing & Multimedia (MoMM2008), pp. 276-281, 2008

application/pdfManuscript (3.3 MB)

VirtualStudio2Go: Digital Video Composition for Real Environments

We synchronize film cameras and LED lighting with off-the-shelf video projectors. Radiometric compensation allows displaying keying patterns and other spatial codes on arbitrary real world surfaces. A fast temporal multiplexing of coded projection and flash illumination enables professional keying, environment matting, displaying moderator information, scene reconstruction, and camera tracking for non-studio film sets without being limited to the constraints of a virtual studio. The reconstruction of the scene geometry allows special composition effects, such as shadow casts, occlusions and reflections. This makes digital video composition more flexible, since static studio equipment, such as blue screens, teleprompters, or tracking devices, is not required. Authentic film locations can be supported with our portable system without causing a lot of installation effort. We propose a concept that combines all of these techniques into one single compact system that is fully compatible with common digital video composition pipelines, and offers an immediate plug-and-play applicability.

Grundhoefer, A. and Bimber, O., VirtualStudio2Go: Digital Videocomposition for Real Environments, In proceedings of ACM Siggraph Asia (ACM Transactions on Graphics), vo. 27, no. 5, article 151, 2008

Superimposing Dynamic Range

We present a simple and cost-efficient way of extending contrast, perceived tonal resolution, and color space of reflective media, such as paper prints, hardcopy photographs, or electronic paper displays. A calibrated projector-camera system is applied for automatic registration, radiometric scanning and superimposition. A second modulation of the projected light on the surface of such media results in a high dynamic range visualization. This holds application potential for a variety of domains, such as radiology, astronomy, optical microscopy, conservation and restoration of historic art, modern art and entertainment installations. In our experiments, we achieved contrast ratios of up of 45,000-60,000:1 with a peak luminance of more than 2,750 cd/m^2, could technically re-produce more than 620 perceptually distinguishable tonal values. Furthermore, we attained color space extensions of up to a factor of 1.4 (compared to a regular projection on white screens) or factor of 3.3 (compared to regular paper prints under environment light). Thereby, the hardcopy resolution can be several thousand DPI or several hundred LPI, while luminance and chrominance are modulated with a registration error of less than 0.3 mm. Thus, compared with most existing interactive HDR displays, we support near distance viewing at a contrast frequency of up to 7 cpd (given our current registration precision and assuming a viewing distance of 50 cm).

Bimber, O. and Iwai, D., Superimposing Dynamic Range, In proceedings of ACM Siggraph Asia (ACM Transactions on Graphics), vo. 27, no. 5, article 150, 2008

Bimber, O., and Iwai, D., Superimposing Dynamic Range, Siggraph'08 (NewTech Demo Abstract), 2008

Grundhoefer, A. and Bimber, O., Superimposing Dynamic Range, Siggraph'08 (Poster+Talk), 2008

Source Code  (Neues Fenster)

Adaptive Training of Video Sets for Image Recognition on Mobile Phones

We present an enhancement towards adaptive video training for PhoneGuide, a digital museum guidance system for ordinary camera–equipped mobile phones. It enables museum visitors to identify exhibits by capturing photos of them. In this article, a combined solution of object recognition and pervasive tracking is extended to a client–server–system for improving data acquisition and for supporting scale–invariant object recognition. A static as well as a dynamic training technique are presented that preprocess the collected object data differently and apply two types of neural networks for classification. Furthermore, the system enables a temporal adaptation for ensuring a continuous data acquisition to improve the recognition rate over time. A formal field experiment reveals current recognition rates and indicates the practicability of both methods under realistic conditions in a museum.

Bruns, E. and Bimber, O., Adaptive Training of Video Sets for Image Recognition on Mobile Phones. In Journal of Pervasive and Ubiquitous Computing, vol. 13, no. 2, pp. 165-178, 2009 (submitted: June 2007, accepted: February 2008)

application/pdfManuscript (5.3 MB)


Mobile Phone Enabled Museum Guidance with Adaptive Classification

We present a new adaptive classification system for museum guidance tasks. It uses camera-equipped mobile phones for on-device object recognition in ad-hoc sensor networks and provides location and object aware multimedia content to museum visitors. Our approach is invariant against perspective, distance and illumination. It supports the scalable identification of single objects and multiple sub-objects, pervasive tracking, phone-to-sensor and phone-to-phone communication. It adapts to user behaviour and environmental conditions over time and achieves high recognition rates under realistic conditions. Our decentralized classification approach makes the system highly scalable to an arbitrarily large number of users since the heavy-weight training process is carried out off-line on the server while the lower-weight classification task is performed individually and in parallel by each mobile phone.

Bruns, E., Brombach, B. and Bimber, O., Mobile Phone Enabled Museum Guidance with Adaptive Classification. In IEEE Computer Graphics and Applications, vol. 28, no. 4, pp. 98-102, 2008 (submitted: November 2007, accepted: March 2008)

application/pdfManuscript (1.0 MB)

Unsynchronized 4D Barcodes

We present a novel technique for optical data transfer between public displays and mobile devices based on unsynchronized 4D barcodes. We assume that no direct (electromagnetic or other) connection between the devices can exist. Time-multiplexed, 2D color barcodes are displayed on screens and recorded with camera equipped mobile phones. This allows to transmit information optically between both devices. Our approach maximizes the data throughput and the robustness of the barcode recognition, while no immediate synchronization exists. Although the transfer rate is much smaller than it can be achieved with electromagnetic techniques (e.g., Bluetooth or WiFi), we envision to apply such a technique wherever no direct connection is available. 4D barcodes can, for instance, be integrated into public web-pages, movie sequences, advertisement presentations or information displays, and they encode and transmit more information than possible with single 2D or 3D barcodes.

Langlotz, T. and Bimber, O., Unsynchronized 4D Barcodes. In proceedings of International Symposium on Visual Computing (ISVC'07), pp. 363-374, 2007

application/pdfManuscript (7.1 MB)

Radiometric Compensation Through Inverse Light Transport

Radiometric compensation techniques allow seamless projections onto complex everyday surfaces. Implemented with projector-camera systems they support the presentation of visual content in situations where projection-optimized screens are not available or not desired - as in museums, historic sites, air-plane cabins, or stage performances. We propose a novel approach that employs the full light transport between projectors and a camera to account for many illumination aspects, such as interreflections, refractions, shadows, and defocus. Pre-computing the inverse light transport in combination with an efficient implementation on the GPU makes the real-time compensation of captured local and global light modulations possible.

Wetzstein, G. and Bimber, O., Radiometric Compensation through Inverse Light Transport. In proceedings of Pacific Graphics, pp. 391-399, 2007

Wetzstein, G. and Bimber, O., Radiometric Compensation of Global Illumination Effects with Projector-Camera Systems. ACM Siggraph`06 (Poster), 2006

Imperceptible Calibration for Radiometric Compensation

We present a novel multi-step technique for imperceptible geometry and radiometry calibration of projector-camera systems. Our approach can be used to display geometry and color corrected images on non-optimized surfaces at interactive rates while simultaneously performing a series of invisible structured light projections during runtime. It supports disjoint projector-camera configurations, fast and progressive improvements, as well as real-time correction rates of arbitrary graphical content. The calibration is automatically triggered when mis-registrations between camera, projector and surface are detected.

Zollmann, S. and Bimber, O., Imperceptible Calibration for Radiometric Compensation. EUROGRAPHICS (short), pp. 61-64, 2007

application/pdfManuscript (9.1 MB)

Dynamic Adaptation of Projected Imperceptible Codes

We present a novel adaptive imperceptible pattern projection technique that considers parameters of human visual perception. A coded image that is invisible for human observers is temporally integrated into the projected image, but can be reconstructed by a synchronized camera. The embedded code is dynamically adjusted on the fly to guarantee its non-perceivability and to adapt it to the current camera pose. Linked with real-time flash keying, for instance, this enables in-shot optical tracking using a dynamic multi-resolution marker technique. A sample prototype is realized that demonstrates the application of our method in the context of augmentations in television studios.

Grundhoefer, A., Seeger, M., Haentsch, F., and Bimber, O., Dynamic Adaptation of Projected Imperceptible Codes. In proceedings of IEEE International Symposium on Mixed and Augmented Reality (ISMAR'07), pp. 181-190, 2007

application/pdfManuscript (2.8 MB)

Laser Pointer Tracking in Projector-Augmented Architectural Environments

We present a system that applies a custom-built pan-tilt-zoom camera for laser-pointer tracking in arbitrary real environments. Once placed in a building environment, it carries out a fully automatic self-registration, registrations of projectors, and sampling of surface parameters, such as geometry and reflectivity. After these steps, it can be used for tracking a laser spot on the surface as well as an LED marker in 3D space, using inter-playing fisheye context and controllable detail cameras. The captured surface information can be used for masking out areas that are critical to laser-pointer tracking, and for guiding geometric and radiometric image correction techniques that enable a projector-based augmentation on arbitrary surfaces. We described a distributed software framework that couples laser-pointer tracking for interaction, projector-based AR as well as video see-through AR for visualizations with the domain specific functionality of existing desktop tools for architectural planning, simulation and building surveying.

Kurz, D., Haentsch, F., Grosse, M., Schiewe, A., and Bimber, O., Laser Pointer Tracking in Projector-Augmented Architectural Environments, In proceedings of IEEE International Symposium on Mixed and Augmented Reality (ISMAR'07), pp. 19-26, 2007

application/pdfManuscript (1.0 MB)


Passive-Active Geometric Calibration for View-Dependent Projections onto Arbitrary Surfaces

Projecting images onto surfaces that are not optimized for projections becomes more and more popular. Such approaches will enable the presentation of graphical, image or video content on arbitrary surfaces. Virtual reality visualizations may become possible in everyday environments - without specialized screen material or static screen configurations. Upcoming pocket projectors will enable truly mobile presentations on all available surfaces of furniture or papered walls. The playback of multimedia content will be supported on natural stonewalls of historic sites without destroying their ambience through the installations of artificial projection screens. We present a hybrid technique for correcting distortions that appear when projecting images onto geometrically complex, colored and textured surfaces. It analyzes the optical flow that results from perspective distortions during motions of the observer and tries to use this information for computing the correct image warping. If this fails due to an unreliable optical flow, an accurate –but slower and visible– structured light projection is automatically triggered. Together with an appropriate radiometric compensation, view-dependent content can be projected onto arbitrary everyday surfaces. An implementation mainly on the GPU ensures fast frame rates.

Zollmann, S., Langlotz, T. and Bimber, O., Passive-Active Geometric Calibration for View-Dependent Projections onto Arbitrary Surfaces, Journal of Virtual Reality and Broadcasting, vol. 4, no. 6, 2007 (re-print from Workshop on Virtual and Augmented Reality of the GI-Fachgruppe AR/VR 2006)

Zollmann, S., Langlotz, T. and Bimber, O., Passive-Active Geometric Calibration for View-Dependent Projections onto Arbitrary Surfaces, Workshop on Virtual and Augmented Reality of the GI-Fachgruppe AR/VR, pp. 181-191, 2006

Real-Time Adaptive Radiometric Compensation

Our new radiometric compensation algorithm considers the human visual perception properties to reduce visible artefacts resulting from the limited dynamic range and brightness of projectors. It preserves a maximum of luminance and contrast and is implemented entirely on the GPU. Real-time frame rates are achieved for supporting animated and interactive content. Initially our algorithm performs an off-line analysis of the projection surface’s geomety and reflectance. The image content is then analyzed to determine the average luminance values, the amount of high spatial frequencies, and a luminance threshold map. The threshold map stores information about the maximum non-perceivable luminance differences for each pixel. The radiometric compensation is carried out in two passes: In the first pass the intensity values are translated and scaled globally depending on the surface reflectance and the image content itself. The result is analyzed for clipping errors. These errors are then blurred with a Gaussian kernel. The applied sigma is inverse proportional to the amount of high spatial frequencies in the local image areas. In the final pass the image intensities are translated and scaled globally, but the luminance values are also adjusted locally depending on the defocused clipping errors. Time dependent adaptation factors are used for global and local transformations to avoid popping artifacts in animated and interactive content.

Grundhoefer, A. and Bimber, O., Real-Time Adaptive Radiometric Compensation. In IEEE Transactions on Visualization and Computer Graphics (TVCG), vol. 14, no. 1, pp. 97-108, 2008, (submitted: August 2006, accepted: February 2007, electronic version published: March 2007)

Grundhoefer, A. and Bimber, O. Real-Time Adaptive Radiometric Compensation. ACM Siggraph`06 (Poster), 2006

Augmenting Large Scale Optical Holograms

Large scale optical holograms require large scale display technology for combining them with interactive graphical elements. Shuttered projection screens (SPS) can be used to sequentially display stereoscopic graphics (in the diffuse mode) and to reconstruct the holographic content (in the transparent mode). While the SPS is shuttered with 50Hz, the stereo pairs are synchronized and time-modulated at approximately 100Hz. Depth information of the holographic content are required to create consistent occlusion and illumination effects with the graphical content. A two-lens stereo camera system can be used for scanning the hologram partially. The different point clouds have to merge into a common coordinate system to form the whole surface. Due to the limited resolution of the range sensor, small gaps appear between the actual surface points. Instead of triangulating the points into a mesh of triangle primitives, the points remain unconnected. They are rendered as point primitives (splatted) with appropriate radii to fill the gaps of missing surface information. The splat size and resolution is adapted dynamically with respect to the observer’s position to ensure interactive frame rates.

Bimber, O., Merging Graphics and Holograms. Journal of Holography and Speckle, vol.3, no. 2, pp. 73-79, 2006

Bimber, O., Augmenting Holograms. IEEE Computer Graphics and Applications (CG&A), vol. 26, no. 5, pp. 12-17, 2006


Enabling Mobile Phones To Support Large-Scale Museum Guidance

Mobile phones have the potential of becoming a future platform for personal museum guidance. They enable full multimedia presentations and –assuming that the visitors are using their own devices– will significantly reduce acquisition and maintenance cost for museum operators. However, several technological challenges have to be mastered before this concept can be successful. One of them is the question of how individual museum objects can be intuitively identified before presenting corresponding information. We have developed an enhanced museum guidance system called PhoneGuide that uses widespread camera equipped mobile phones for on-device object recognition in combination with pervasive tracking. It provides additional location- and object-aware multimedia content to museum visitors, and is scalable to cover a large number of museum objects. In a field survey our system was able to identify 155 real museum exhibits from multiple perspectives with a recognition rate of 95% and a classification speed of less than one second per object. A coarse grid of only eight low-cost Bluetooth emitters distributed over two museum floors was used to achieve these results. Once an object has been recognized, related multimedia presentations such as videos, audio , text, computer graphics and images are displayed on the phone.
Special thanks to the City Museum of Weimar and to CellIQ for their support.

Bruns, E., Brombach, B., and Zeidler, T., Bimber, O., Enabling Mobile Phones To Support Large-Scale Museum Guidance. In IEEE MultiMedia, vol. 14, no. 2, pp. 16-25, 2007 (submitted: October 2005, accepted: July 2006)

application/pdfManuscript (4.2 MB)

Compensating Indirect Scattering for Immersive and Semi-Immersive Projection Displays

Concavely shaped projection screens, such as CAVEs, two-sided workbenches, domes, or cylinders scatter a fraction of light to other screen portions. The amount of indirect illumination adds to the directly projected image and causes the displayed content to appear partially inconsistent and washed out. We have developed a reverse radiosity method that compensates first-level and higher-level secondary scattering effects in real-time. The images appear more brilliant and uniform when reducing the scattering contribution. A numerical solution is approximated with Jacobi iteration for a sparse-matrix linear equation system on the GPU. Efficient data structures allow packing the required data into textures which are processed by pixel shaders. Frame-buffer objects are used for a fast exchange of intermediate iteration results, and enable computations with floating point precision. Our algorithm’s result can be optimized for quality or performance.

Bimber, O., Grundhoefer, A., Zeidler, T., Danch, D., and Kapakos, P. Compensating Indirect Scattering for Immersive and Semi-Immersive Projection Displays. In proceedings of IEEE Virtual Reality (IEEE VR'06), pp. 151-158, 2006

application/pdfManuscript (3.7 MB)

Multi-Focal Projection

Many multi-projector rendering techniques exist that aim at creating a high consistency of image geometry, intensity and color. We proposed a concept and a solution for considering and optimizing a fourth image property – its focus. We describe a novel multi-focal projection concept that applies conventional video projectors and camera feedback. Multiple projectors with differently adjusted focal planes, but overlapping image areas are used. They can be either arbitrarily positioned in the environment, or can be integrated into a single projection unit. During an automatic one-time calibration process, structured light projection together with camera feedback allows to measure the relative focus value of every projector pixel on an arbitrary diffuse surface. Thereby, the focus values are geometrically and radiometrically corrected. If this is known, a final image with minimal defocus can be composed in real-time from individual pixel contributions of all projectors. Our technique is independent of the surfaces’ geometry, color and texture, of the environment light, as well as of the projectors’ parameters (i.e., position, orientation, luminance and chrominance).

Bimber, O. and Emmerling, A., Multi-Focal Projection: A Multi-Projector Technique for Increasing Focal Depth. In IEEE Transactions on Visualization and Computer Graphics (TVCG), vol. 12, no. 4, pp. 658-667, 2006

Bimber, O., Wetzstein, G., Emmerling, A., and Nitschke, C. Enabling View-Dependent Stereoscopic Projection in Real Environments. In proceedings of International Symposium on Mixed and Augmented Reality (ISMAR'05), pp. 14-23, 2005

Enabling View-Dependent Stereoscopic Projection in Real Environments

With this work we take a first step towards an ad-hoc stereoscopic projection within real environments. We show how view-dependent image-based and geometric warping, radiometric compensation, and multi-focal projection enable a view-dependent visualization on ordinary (geometric complex, colored and textured) surfaces within everyday environments. All these techniques are accomplished at interactive rates and on a per-pixel basis for multiple interplaying projectors. Special display configurations for immersive or semi-immersive VR/AR applications that require permanent and artificial projection canvases might become unnecessary. Such an approach does not only offer new possibilities for augmented reality and virtual reality, but also allows merging both technologies. This potentially gives some application domains – like architecture – the possibility to benefit from the conceptual overlaps of AR and VR.
Special thanks to the Faculty of Architecture, Bauhaus-University Weimar for their support.

Bimber, O., Wetzstein, G., Emmerling, A., and Nitschke, C. Enabling View-Dependent Stereoscopic Projection in Real Environments. In proceedings of International Symposium on Mixed and Augmented Reality (ISMAR'05), pp. 14-23, 2005

PhoneGuide: Museum Guidance Supported by On-Device Object Recognition on Mobile Phones

We present PhoneGuide – an enhanced museum guidance approach that uses camera-equipped mobile phones and on-device object recognition. Our main technical achievement is a simple and light-weight object recognition approach that is realized with single-layer perceptron neuronal networks. In contrast to related systems which perform computational intensive image processing tasks on remote servers, our intention is to carry out all computations directly on the phone. This ensures little or even no network connectivity and consequently decreases cost for online times. Our laboratory experiments and field surveys have shown that photographed museum exhibits can be recognized with a probability of over 90%. We have evaluated different feature sets to optimize the recognition rate and performance. Our experiments reviled that normalized color features are most effective for our method. Choosing such a feature set allows recognizing an object below one second on up-to-date phones. The amount of data that is required for differentiating 50 objects from multiple perspectives is les than 6KBytes.
Special thanks to the Senckenberg Museum of Natural History, Frankfurt, to the Museum für Ur- und Frühgeschichte, Weimar, and to CellIQ for their support.

Foeckler, P., Zeidler, T., Brombach, B., Bruns, E., and Bimber, O. PhoneGuide: Museum Guidance Supported by On-Device Object Recognition on Mobile Phones. In proceedings of International Conference on Mobile and Ubiquitous Computing (MUM'05), pp. 3-10, 2005


Interacting with Augmented Holograms

Holography and computer graphics are being used as tools to solve individual research, engineering, and presentation problems within several domains. Up until today, however, these tools have been applied separately. Our intention is to combine both technologies to create a powerful tool for science, industry and education. We are currently investigating the possibility of integrating computer generated graphics and holograms. We presents several applications of interaction techniques to graphically enhanced holograms and give a first glance on a novel method that reconstructs depth from optical holograms.
Special thanks to Deutsche Forschungsgemeinschaft (DFG) for their support.

Bimber, O., Augmenting Holograms. IEEE Computer Graphics and Applications (CG&A), vol. 26, no. 5, pp. 12-17, 2006

Bimber, O., Zeidler, T., Grundhoefer, A., Wetzstein, G., Moehring, M., Knoedel, S., and Hahne, U. Interacting with Augmented Holograms. In proceedings of SPIE Conference on Practical Holography XIX: Materials and Applications, 2005

Embedded Entertainment with Smart Projectors

Video projectors will play a major role in future home entertainment and edutainment applications – ranging from movies and television, over computer games, to multimedia presentations. With video projectors images can be displayed that are larger than the devices themselves. However, we have to give up living space and ambience to set up artificial canvases that have to be as large as the desired image. Smart projectors are able to display correct images onto arbitrary existing screen surfaces, like wallpapered walls or window curtains. Thus it can function without an artificial canvas and consequently leaves a bit more freedom to us in the decision on how to arrange our living space. Our smart projectors combine camera feedback with structured light projection to gain information about the screen surface and the environment. The calibration of such a device is fast, fully automatic and robust, and the correction of video signals can be achieved in real-time. Neither geometry information nor projector or camera parameters need to be known. Instead, the entire calibration and correction (geometry and color) is done on a per-pixel level – supported by modern pixel shader hardware. Such devices might make it possible to convert your bookshelf into a TV screen, or your kid’s closet into an interactive virtual playground.
Special thanks to the Bennert Group for their support.

Bimber, O., Emmerling, A., and Klemmer, T. Embedded Entertainment with Smart Projectors. IEEE Computer, vol. 38, no. 1, pp. 56-63, 2005

Video See-Through AR and Optical Tracking with Consumer Cell Phones

To enable mobile devices, such as head-mounted displays and PDAs, to support video see-through augmented reality is a popular research topic. However, such technology is not widely-spread outside the research community today. It has been estimated that by the end of the year 2005 approximately 50% of all cell phones will be equipped with digital cameras. Consequently, using cell phones as platform for video see-through AR has the potential of addressing a brought group of end users and applications. Compared to high-end PDAs and HMDs together with personal computers, the implementation of video see-through AR on the current fun- and smart-phone generations is a challenging task: Ultra-low video-stream resolutions, little graphics and memory capabilities, as well as slow processors set technological limitations. We have realized a prototype solution for video see-through AR on consumer cell phones. It supports optical tracking of passive paper markers and the correct integration of 2D/3D graphics into the live video-stream at interactive rates. We aim at applications, such as interactive tour guiding for museums and tourism, as well as at mobile games.
Special thanks to Nokia Research for their support.

Moehring, M., Lessig, C. and Bimber, O. Video See-Through AR on Consumer Cell Phones. In proceedings of IEEE/ACM International Symposium on Augmented and Mixed Reality (ISMAR'04), pp. 252-253, 2004

Moehring, M., Lessig, C. and Bimber, O. Optical Tracking and Video See-Through AR on Consumer Cell Phones. In proceedings of Workshop on Virtual and Augmented Reality of the GI-Fachgruppe AR/VR, pp. 193-204, 2004


Superimposing Pictorial Artwork with Projected Imagery

Pictorial artwork, such as paintings and drawings, can tell interesting stories. The capabilities of museums to communicate such and other information, however, are clearly limited. Text legends and audio guides can mediate facts, but offer little potential for presenting visual content, such as embedded illustrations, pictures, animations, and interactive elements. Also due to the difficult economic situation, edutainment is becoming an important factor for museums. By applying new media technologies – such as computer graphics, virtual reality and augmented reality– exhibit oriented information might be communicated more effectively, but certainly in a more exciting way. We describe a novel technological approach, a mathematical model, a real-time rendering algorithm and examples of presentation techniques for integrating almost any kind of visual information directly into pictorial artwork. It allows displaying such information while keeping the observers’ attention on the original artifact, and does not require additional screens.
Special thanks to the Museum het Rembrandthuis, Amsterdam for their support.

Bimber, O., Coriand, F., Kleppe, A., Bruns, E., Zollmann, S., and Langlotz, T. Superimposing Pictorial Artwork with Projected Imagery. IEEE MultiMedia, vol. 12, no.1, pp. 16-26, 2005

The HoloStation

The Holographic Workstation (short: HoloStation) is a desktop display that allows the integration of interactive stereoscopic or auto-stereoscopic graphical elements into high-quality optical holograms.
Archaeologists, for instance, use optical holograms to archive and investigate ancient artefacts. The distribution of copy holograms enables scientists to perform their research without having access to the original artefacts or inaccurate replicas. Combining these static holograms with interactive computer graphics will allow them to integrate real-time simulation data or to perform experiments that require a direct user interaction, such as packing reconstructed soft-tissue into a hologram of a fossilized dinosaur skull. In addition, specialized interaction devices can simulate haptic feedback of the holographic and the graphical content while performing such interactive tasks. An entire collection of artefacts will fit into a single album of holographic recordings, while a light-box-like display can be used for visualization and interaction.
The current HoloStation prototype can be used in combination with many different holograms, such as transmission and reflection types. It is a compact and flexible display that provides several forms of interaction via integrated touch-screen and real-time range scanner.
Special thanks to Deutsche Forschungsgemeinschaft (DFG) for their support.

Bimber, O., Augmenting Holograms. IEEE Computer Graphics and Applications (CG&A), vol. 26, no. 5, pp. 12-17, 2006

Bimber, O. Combining Optical Holograms with Interactive Computer Graphics. In IEEE Computer, vol. 37, no. 1, pp. 85-91, January 2004

Bimber, O. HoloGraphics: Combining holograms with interactive computer graphics. The Holographer (online Journal), April 2004

Bimber, O., Combining Optical Holograms with Interactive Computer Graphics. SPIE International Technical Group Newsletter, vol. 16, no. 1, pp, 1+9, June 2005

Combining Optical Holograms with Interactive Computer Graphics

A hologram is a photometric emulsion that records interference patterns of coherent light. The recording itself stores amplitude, wavelength and phase information of light waves. In contrast to simple photographs (which can record only amplitude and wavelength information), holograms have the ability to reconstruct complete optical wavefronts. This results in a three-dimensional appearance of the captured scenery which is observable from different perspectives.
Today, many applications for optical holograms exist. Examples include interferometry, copy protections, data storage and holographic optical elements. Optical holograms, however, are static and lack in interactivity. Electro-holography aims at the computer-based generation and display of holograms in real time. Since a massive amount of data has to be processed, transmitted and stored to create holograms, today’s computer technology still sets the limits of electro-holography. To overcome some of the performance issues, advanced reduction and compression methods have been developed. This results in electro-holograms that are interactive, but small, low-resolution and pure in color.
A novel approach has been proposed that combines holography, autostereoscopy and projection technology to integrate interactive graphical elements into high-quality optical holograms. Beside the conceptual idea and the optical solutions, efficient rendering techniques and a proof-of-concept prototype are presented. Furthermore, potential application areas, such as museums and scientific visualization are outlined.
Special thanks to Deutsche Forschungsgemeinschaft (DFG) for their support.

Bimber, O., Augmenting Holograms. IEEE Computer Graphics and Applications (CG&A), vol. 26, no. 5, pp. 12-17, 2006

Bimber, O. Combining Optical Holograms with Interactive Computer Graphics. In IEEE Computer, vol. 37, no. 1, pp. 85-91, January 2004

Bimber, O. HoloGraphics: Combining holograms with interactive computer graphics. The Holographer (online Journal), April 2004

Bimber, O., Combining Optical Holograms with Interactive Computer Graphics. SPIE International Technical Group Newsletter, vol. 16, no. 1, pp, 1+9, June 2005

Consistent Illumination within Augmented Environments

To achieve a consistent lighting situation between real and virtual environments is important for convincing augmented reality (AR) applications.
A rich pallet of algorithms and techniques have been developed that match illumination for video- or image-based augmented reality. However, very little work has been done in this area for optical see-through AR. We believe that the optical see-through concept is currently the most advanced technological approach to provide an acceptable level of realism and interactivity.
Methods have been developed which create a consistent illumination between real and virtual components within an optical see-through environment – such as the Virtual Showcase. Combinations of video projectors and cameras are applied to capture reflectance information from diffuse real objects and to illuminate them under new synthetic lighting conditions. For diffuse objects, the capturing process can also benefit from hardware acceleration – supporting dynamic update rates. To handle indirect lighting effects (like color bleeding) an off-line radiosity procedure is outlined that consists of multiple rendering passes. For direct lighting effects (such as simple shading, shadows and reflections) hardware accelerated techniques are described which allow to achieve interactive frame rates. The reflectance information is used in addition to solve a main problem of a previously introduced technique which creates consistent occlusion effects for multiple users within such environments.

Bimber, O., Grundhoefer, A., Wetzstein, G., and Knoedel S. Consistent Illumination within Optical See-Through Augmented Environments. In proceedings of IEEE/ACM International Symposium on Augmented and Mixed Reality (ISMAR'03), pp. 198-207, 2003

application/pdfManuscript (285Bytes)

Augmented Reality Digital Storytelling

Interactive digital storytelling techniques are recently being applied in combination with new media forms, such as virtual reality (VR) and augmented reality (AR). Thereby the technological progress that is being made within these areas allows shifting interactive digital storytelling more and more into the third dimension and into the physical world. One of the main advantages of this transition is the possibility to communicate information more effectively with digital means by telling stories that can be experienced directly within a real environment or in combination with physical objects. The user experience is thus transformed from relating different pieces of information to one another to ‘living through’ the narrative. The perceptual quality and the unique aura of a real environment (e.g., a historical site) or object (e.g., an ancient artifact) cannot be simulated by today’s technology. Thus it is not possible to substitute them with virtual or electronic copies without them losing their flair of originality. This circumstance can be a crucial reason for using augmented reality as a technological basis for interactive digital storytelling. Several research groups apply personal displays, such as head-mounted displays (HMDs) to realize AR-based new media experiences. These all-purpose display types, however, have to face technological problems that up until today have not been sufficiently solved1. These shortcomings cause a substantial credibility gap if a certain level of realism is required.
We want to discuss how an application-customized type of augmented reality display –the Virtual Showcase– overcomes most of these technological shortcomings, and how it can be used as a new platform for digital storytelling.
To underline our statements we discuss and evaluate a case study that focuses on using AR digital storytelling to communicate scientific information to a novice audience in a museum context. We have introduced the application that led to our case study in a previous publication. In this paper, we want to describe the technical components that have been developed to realize this application.
In addition, we present a first user feedback and illustrate our current efforts of improvement.

Bimber, O., Encarnacao. L.M., and Schmalstieg, D. The Virtual Showcase as a new Platform for Augmented Reality Digital Storytelling. In proceedings of Eurographics Workshop on Virtual Environemnts IPT/EGVE'03, pp. 87-95, 2003

application/pdfManuscript (284Bytes)

Real-Time Image Warping for Curved Mirror Displays

Displays which require a non-linear pre-distortion to present undistorted gfx on non-planar surfaces, or to neutralize optical distortion usually apply multi-pass rendering techniques. Projective textures or uniform grids are used to deform the image generated during the first pass before it is displayed as a texture map during the final pass. However, these approaches do not consider the error that is generated from a piecewise linear texture interpolation to adapt the underlying geometry.
Curved mirror displays, for instance, that stereoscopically produce three-dimensional gfx generally don't pre-distort the graphics before they are displayed. Yet, some systems apply additional optics (such as lenses) to stretch or undistort the reflected image. But these devices constrain the observer to a single point of view or to very restricted viewing zones. For the Virtual Showcase mirror display, however, a view-dependent rendering is required to support freely moving observers. The existing image warping algorithms for curved Virtual Showcases predistort a uniform image grid and consequently do not consider the local error that is generated from a piecewise linear texture interpolation. These algorithms can only produce an acceptable image quality within a significantly large amount of rendering time.
The algorithm that is presented improves the uniform image warping techniques for curved Virtual Showcases by defining an appropriate error metric and implementing a quadtree-based selective refinement method that generates adapted local levels of detail. It approaches to produce a predefined image quality within a minimum amount of rendering time. While display specific details of our algorithm are explained based on the Virtual Showcase display, its general functionality is valid for other non-linear displays -such as for curved projection-based systems.

Bimber, O., Froehlich, B., Schmalstieg, D., and Encarnacao, L.M. Real-Time View-Dependent Image Warping to correct Non-Linear Distortion for Curved Virtual Showcase Displays. Computers and Graphics - The international Journal of Systems and Applications in Computer Graphics, vol. 27, no.4, pp. 512-528, 2003

application/pdfManuscript (294Bytes)


Augmented Paleontology

Paleontology is filled with mysteries about the plants and animals that lived thousands, millions, even billions of years before the first humans walked the earth. To answer questions about these organisms, paleontologists rely on the excavation, analysis, and interpretation of fossils. Embedded and preserved in the earth’s crust, fossils are the remains or traces of ancient life forms, including bones, teeth, shells, leaf imprints, nests, and footprints.Fossils can disclose how organisms evolved over time and their relationship to one another. While they reveal much, such as the general shape and size of ancient living things, fossils keep us guessing about these organisms’ color, sound, and—most significantly—their behavior. For several years, modern paleontologists have used 3D computer graphics to help reconstruct these pieces of the past. State-of-the-art scanning technology produces 3D fossil replicas that scientists can process and study without physical constraints. Paleontologists typically generate volumetric data sets for analysis, such as magnetic resonance imaging or computed axial tomography scans, and they use surface models for digital preservation and reproduction. To study ontogeny—an organism’s growth and form—paleontologists apply mathematical models for simulation and visualization. Likewise, computer animations help study dinosaur locomotion. Beyond building knowledge of our world, the results of this work influence how dinosaurs appear in museums, illustrations, and movies, and as toys. In the past 40 years, technological advances have continued to blur the boundary between real and computer-generated worlds. Augmented reality leverages this technology to provide an interface that enhances the real world with synthetic supplements. Paleontologists can use AR to present virtual data, such as 3D computer graphics, directly within a real environment rather than on a flat monitor. We coined the term augmented paleontology to refer to the application of AR to paleontology. AP seeks to support paleontologists in their research, and communicate the results of paleontology to museum visitors in an exciting and effective way. An interdisciplinary team of paleontologists, graphics designers, and computer scientists has already applied the AP interface to soft-tissue reconstruction and the study of dinosaur locomotion.

Bimber, O., Gatesy, S.M., Witmer, L.M., Raskar, R. and Encarnacao, L.M. Merging Fossil Specimens with Computer-Generated Information. IEEE Computer, vol. 35, no.1 , pp. 45-50, 2002

Occlusion Shadows

Projection-based augmented reality systems, such as the Virtual Showcase, share many positive properties of projection-based virtual environments. These displays provide high resolution, improved consistency of eye accommodation and convergence, little motion sickness potential, and the possibility of an integration into common working environments. One of the main challenges for projection-based AR systems as well as for head-mounted optical see-through displays is the generation of correct occlusion effects between virtual and real objects. Additionally shadows of virtual objects cast onto real ones and consistent illumination of the real and virtual scenery are often difficult to achieve. We introduce projector-based illumination techniques for view-dependent optical seethrough AR displays. This approach has the potential to solve all of the above mentioned problems. Here, we focus on using projector-based illumination for creating correct occlusion effects for mixed reality configurations.
We have implemented and tested such a system in the context of the Virtual Showcase, which consists of a horizontal projection screen and a convex half-silvered mirror assembly. Virtual and real objects can be displayed in the same space inside the showcase. The original Virtual Showcase used a standard light bulb to illuminate real objects. This setup does not provide very much control over the lighting situation. By using a computer-controlled video-projector as a replacement for the simple light bulb, we are able to fully control the lighting situation inside the showcase on a perpixel basis. Our main contribution is a solution to the problem of correct occlusion for mixed reality scenarios with viewdependent optical see-through displays. Our method produces correct occlusion effects between virtual and real objects by projecting shadows onto real objects located behind virtual ones using projector-based illumination.

Bimber, O. and Froehlich, B. Occlusion Shadows: Using Projected Light to Generate Realistic Occlusion Effects for View-Dependent Optical See-Through Displays. In proceedings of International Symposium on Mixed and Augmented Reality (ISMAR’02), pp. 186-195, 2002

application/pdfManuscript (295Bytes)


The Virtual Showcase

Intuitive access to information in habitual realworld environments is a challenge for information technology. An important question is how can we enhance established and well-functioning everyday environments rather than replace them by virtual environments (VEs)? Augmented reality (AR) technology has a lot of potential in this respect because it augments realworld environments with computer-generated imagery.
Today, most AR systems use see-through head-mounted displays, which share most of the disadvantages of other head-attached display devices. We introduce a new projection-based AR display system—the Virtual Showcase. The Virtual Showcase has the same form factor as a real showcase, making it compatible with traditional museum displays. Real scientific and cultural artifacts are placed inside the Virtual Showcase allowing their 3D graphical augmentation. Inside the Virtual Showcase, virtual representations and real artifacts share the same space providing new ways of merging and exploring real and virtual content. Solely virtual exhibits may also be displayed. The virtual part of the showcase can react in various ways to a visitor, which provides the possibility for intuitive interaction with the displayed content. Another interesting aspect of our system is its support for two to four simultaneously tracked users looking at the Virtual Showcase from different sides. This feature allows the collaborative exploration of artifacts shown in the Virtual Showcase. These interactive showcases contribute to ambient intelligent landscapes, where the computer acts as an intelligent server in the background and visitors can focus on exploring the exhibited content rather than on operating computers. The Virtual Showcase consists of two main parts: a convex assembly of halfsilvered mirrors and a graphics display. So far, we’ve built Virtual Showcases with two different mirror configurations. Our first prototype consists of four halfsilvered mirrors assembled as a truncated pyramid. Our second prototype uses a single mirror sheet to form a truncated cone. We placed these mirror assemblies on top of a projection screen. Users can see real objects inside the showcase through the half-silvered mirrors merged with the graphics displayed on the projection screen. We illuminated the showcases’ contents with a controllable light source while presenting view-dependent stereoscopic graphics to the observer. For our current prototypes, we use standard shutter glasses controlled by infrared emitters. Head tracking is accomplished using an electromagnetic tracking device. Our pyramid shaped prototype supports up to four viewers simultaneously looking at the showcase from four different sides. Our cone-shaped prototype provides a seamless surround view onto the displayed artifact.

Bimber, O., Froehlich, B., Schmalstieg, D., and Encarnacao, L.M. The Virtual Showcase. IEEE Computer Graphics & Applications, vol. 21, no.6, pp. 48-55, 2001

The Extended Virtual Table

We describe a prototype of an optical extension for table-like rear-projection systems -the Extended Virtual Table. A large half-silvered mirror is used as the optical combiner to unify a virtual and a real workbench, whereby the shortcomings that are related to the Reflective Pad and Transflective Pad (i.e., indirect line-of-sight, limited field of view, and tracking distortion) are reduced. The virtual workbench has been enabled to display computer graphics beyond its projection boundaries and to combine virtual environments with the adjacent real world. A variety of techniques is described that allow indirect interaction with virtual objects through the mirror.
Systems, such as the Extended Virtual Table, approach a conceptual and technical extension of traditional Virtual Reality by means of Augmented Reality (xVR), and a seamless integration of such technology into habitual work environments. Furthermore, optical distortions that are caused by the half-silvered mirror combiner and the projector, as well as non-optical distortions caused by the tracking device are analyzed and appropriate compensation methods are described.

Bimber, O., Encarnacao, L.M. and Branco, P. The Extended Virtual Table: An Optical Extension for Table-Like Projection Systems. Presence: Teleoperators and Virtual Environments, vol.10, no. 6, pp. 613-631, 2001

application/pdfManuscript (302Bytes)

Projection-Based Augmented Engineering

Traditional Augmented Reality (AR) displays, such as head-mounted devices entail a number of technological and ergonomic drawbacks. Their limited resolution, the restricted field-of-view, the visual perception issues that are due to the fixed focal length caused by a constant and head-attached image plane, the increased incidence of discomfort provoked by simulation sickness and their cumbersomeness prevent their usage in a number of application areas. To overcome some of these drawbacks, but also to open new application areas for AR, we propose a projectionbased AR concept that combines spatially aligned optical see-through elements (essentially half-silvered mirror beam-splitters) with off-the-shelf projection-based Virtual Reality displays. This concept offers possibilities to combine the advantages of both technologies: the well established projection-based Virtual Reality with the potentials of Augmented Reality. We describe the early stages of a new proof of-concept prototype that has been developed to extend the scope of projection-based AR towards the engineering domain.

Bimber, O., Stork, A., and Branco, P. Projection-based Augmented Engineering. In proceedings of Human-Computer Interaction International (HCII’01), vol. 1, pp. 787-791, 2001

application/pdfManuscript (290Bytes)

Seamless Integration of Virtual Reality into habitual Workplaces

Employing virtual reality (VR) technology, computer-based work environments are clearly evolving into a major component of the next-generation workplace. Over the past few years, VR has become a practical reality for many applications, thanks to a number of technology inventions. Among these has been the development of large-screen display systems that make use of stereoscopic projection. Systems such as the Virtual Table and Responsive Workbench both employ a horizontal projection philosophy, whereas other devices, such as the Powerwall and surround-screen projection systems (SSPS) like the CAVE , also employ vertical back-projection. Due to their effective application, these devices are gaining greater acceptance by the research and industrial user communities. Such devices offer a large field of view, brilliant image quality, high resolution and (especially the table-like devices) a promising compatibility with traditional workspaces. To support this integration, however, ergonomic and human-centered interaction techniques must be developed.
We introduce ideas, proposed technologies, and initial results for a seamless integration of virtual reality in habitual workspaces, thus successively transforming them into efficient, high-tech work environments.

Bimber, O., Encarnacao, L.M. and Stork, A. Seamless integration of virtual reality in habitual workplaces. Journal for Industrial Science, Munich University of Technology, vol.55, no.2, pp. 103-112, 2001

application/pdfManuscript (299Bytes)


Transflective Props

We describe the use of a hand-held semi-transparent mirror to support augmented reality tasks with back-projection systems. This setup overcomes the problem of occlusion of virtual objects by real ones linked with such display systems. The presented approach allows an intuitive and effective application of immersive or semi-immersive virtual reality tasks and interaction techniques to an augmented surrounding space. Thereby, we use the tracked mirror as an interactive image-plane that merges the reflected graphics, which are displayed on the projection plane, with the transmitted image of the real environment. In our implementation, we also address traditional augmented reality problems, such as real-object registration and virtual-object occlusion. The presentation is complemented by a hypothesis of conceivable further setups that apply transflective surfaces to support a mixed reality (i.e., combined AR and VR) environment.

Bimber, O., Encarnacao, L.M., and Schmalstieg, D. Augmented Reality with Back-Projection Systems using Transflective Surfaces. Computer Graphics Forum (proceedings of Eurographics 2000 - EG’2000), vol. 19, no. 3, pp. 161-168, 2000

application/pdfManuscript (300Bytes)

Reflective Props

We introduce the idea of using real mirrors in combination with rear-projection systems for the purpose of interacting with and navigating through the displayed information. Subsequently a derived application is described. For this, we use a hand-held planar mirror and address two fundamental problems of applying head tracking with rear-projection planes: the limited viewing volume of these environments and their incapability of simultaneously supporting multiple observers. Furthermore, we describe the possibility of combining a reflective pad with a transparent one, thus introducing a complementary tool for interaction and navigation.

Bimber, O., Encarnacao, L.M., and Schmalstieg, D. Real Mirrors Reflecting Virtual Worlds. In proceedings of IEEE Virtual Reality (IEEE VR’00), pp. 21-28, 2000

application/pdfManuscript (313Bytes)

Sketch-Based and Multi-Modal Interaction

Leonardo da Vinci's drawings of machines and other objects illustrate one of the most fundamental purposes of sketches: the ability to communicate design and functionality to others. Nowadays, it is widely accepted that sketching is a form of critical, reflective dialog that handles communication on one or more different levels of abstraction simultaneously. Various approaches have been taken to support this kind of dialog between humans and computers, and to build human-computer interfaces that are able to interpret such freehand sketches for different purposes.
In this context, the creation or reconstruction of 3D objects from 2D sketches is of major concern in many application areas. This so-called 'pencil-and-paper' approach is used for rapidly designing approximate threedimensional scenes. While some systems analyze the orthographic or perspective projections to reconstruct 3D shapes that, based on psychological assumptions, are most plausible to the human observer, others interpret 2D gestures while the objects are sketched. Within the last decade, the conceptual design phase has been increasingly supported by sketch systems that allow the expression of ideas on a computer-aided, but still human-centered basis. However, putting an emphasis on sketching, most of these systems are sealed off from real-world applications rather than being generally applicable as components.
We introduce a framework for sketch-based interaction within three-dimensional virtual environments. We describe each layer of the framework using illustrative examples depicting its realization. Furthermore, we want to present a variety of domain-specific applications of sketching within 3D virtual environments based on our framework instead of implementing yet another application for sketching.

Bimber, O., Encarnacao, L.M. and Stork, A. A Multi-Layered Architecture for Sketch-based Interaction within Three-dimensional Virtual Environments. Computers and Graphics - The international Journal of Systems and Applications in Computer Graphics, vol. 24, no. 6, pp. 851-867, 2000

application/pdfManuscript (303Bytes)