Mobile Augmented Reality Goes Way Beyond Markers

The dust from CES 2009 has barely settled over many shiny new devices, and new advancements in Handheld Augmented Reality software are already emerging from Vienna.

Daniel Wagner and his team at Graz University have come up with new and improved capabilities.

High Speed Natural Feature Tracking on a mobile phone

We saw an early implementation of Studierstube ES at ISMAR 08, so I asked Daniel what’s new about this capability, besides being faster and more robust.

Daniel: We can now track multiple images and switch arbitrarily. I believe it is now at a level that it can really be used in practice.

Games alfresco: Looks great. Based on the video it seems that it runs on Windows Mobile 6 (ASUS p552w, iPAQ 614c). What about other platforms?

Daniel: Not bad! It is written in C/C++, [but] since this is pure math code, it could be ported easily to any platform. Our AR framework is still Windows Mobile only, although we now also have Linux support (desktop only since we lack a Linux phone). MacOS and Symbian are in the making and should be available in a month or so.

Tracking of Business Cards on a mobile phone

Daniel: On January 20th we have the official opening of our “Christian Doppler Lab” (founded by the Christian Doppler agency). For that purpose I created a small demo for tracking business cards. In the future we’ll replace the 3D content with something more useful…

I can’t talk about Daniel without mentioning WARM ’09; He is the main organizer of this Winter Augmented Reality Event on 12th-13th February 2009 at Graz University, Austria. Registration is over, but if you really want to go and have something cool to present – you may be able to convince Daniel to let you in.

Should you get your hands on this powerful technology (assuming Imagination makes it available for licensing soon) what would YOU do with it?

Live from ISMAR ’08 : Perfecting Augmented Reality

The last session of ISMAR ’08 is about to begin, and it concentrates on perfecting Rendering and Scene Acquisition in augmented reality and making it even more realistic.

First on stage is Yusaku Nishin with a challenging talk attempting Photometric registration by adaptive high dynamic range image generation for augmented reality.

His goal : development of photorealistic augmented reality with a High Dynamic Range (HDR) image.

Estimating the lighting environment of virtual objects is difficult because of low dynamic range cameras. In order to overcome this problem, they propose a method that estimates the lighting environment from an HDR image and renders virtual objects using an HDR environment map. Virtual objects are overlaid in real-time by adjusting the dynamic range of the rendered image with tone mapping according to the exposure time of the camera. The HDR image is generated from multiple images captured with various exposure times.

Now you are ready to watch the resulted effect. Incredible.

[youtuve=http://www.youtube.com/v/M53Tqqdk9w0]

~~~

Next on stage is the soon-to-be-hero-of-the-show Georg Klein (more on that later…) Compositing for Small Cameras

Blending virtual items on real scenes. It can work with small cameras. Video from such cameras tend to be imperfect (blurring, over saturation, radial distortion, etc) so when you impose a virtual item it tend to stick out in a bad way. Since we can’t improve the live video – we will try to adapt the virtual item to match the video at hand. Simply put, Georg samples the background and applies it to the image which matches blur, radial distortion, rotation, color saturation, etc) and he does it in 5 millisecond on a desktop… For details check the pdf paper; take a look for yourself and tell me if it works on Kartman:

Done! Georg is already working on the next challenge.

~~~

Following is Pished Bunnun introduces his work: OutlinAR: an assisted interactive model building system with reduced computational effort

Building 3D models interactively and in place (in-situ), using a single camera, and low computational effort – with a makeshift joystick (Button and wheels.)

In this case the video does a better job at explaining the concept than any number of words would…

Pished demonstrates it’s fast and pretty robust. You judge for yourself.

If you absolutely need more words about this – start here.

The team’s next challenge: make curved lines…

~~~

In the very last talk of the event Jason Wither courageously takes on another challenge to perfecting augmented reality, with his talk: Fast Annotation and Automatic Model Construction with a Single-Point Laser Range Finder

Jason is using a laser finder typically used by hunters (though he will not be shooting anything or anybody), mounted on the head or handheld, in conjunction with a parallel camera. First he wants to create an annotation. that’s totally trivial. But you can then orient the annotation according to a building for example.

Next, he is going to correct occlusion of virtual objects by real objects for improved augmented realism. Just click before and after the object and pronto:

Finally he will create a 3D model of an urban environment semi-automatically, by creating a depth map courtesy of the laser. To achieve that he’s using a fusion process. You got to see that video; the laser’s red line advancing on buildings reminds me the blob swallowing the city in that quirky Steve McQueen movie.

In conclusion this is a really low cost and fast approach for modeling and annotation of urban environments and objects. That capability would become extremely handy once Augmented Reality 2.0 picks up and anyone would want to annotate the environment (aka draw graffiti without breaking the law).

Next is the event wrap up and the results of the Tracking Competition. Stay tuned.

====================

From the ISMAR ’08 program:

Rendering and Scene Acquisition

  • Photometric registration by adaptive high dynamic range image generation for augmented reality
    Yusaku Nishina, Bunyo Okumura, Masayuki Kanbara, Naokazu Yokoya
  • Compositing for Small Cameras (pdf paper)
    Georg Klein, David Murray
  • OutlinAR: an assisted interactive model building system with reduced computational effort
    Pished Bunnun, Walterio Mayol-Cuevas
  • Fast Annotation and Automatic Model Construction with a Single-Point Laser Range Finder
    Jason Wither, Chris Coffin, Jonathan Ventura, Tobias Hollerer

Live From ISMAR 08: Augmented Reality Sensors and Sensor Fusion

The last day of ISMAR ’08 is upon us, and the day opens by stimulating our senses with a session about sensors.

Gabriele Bleser starts this session with a talk about Using the marginalised particle filter for real-time visual-inertial sensor fusion

She starts by showing a short clip with an erratic camera motion that makes everyone dizzie…it actually proves an important capability that she studied which creates less jitter and less requirements imposed on the camera.

She explains the basics of particle filter and the use of inertial measurement.  In the past researchers studied standard particle filter. This is the first study using the a marginalised particle filter.

Testing using the new technique (non linear state space model with linear Gaussian substructure for real time visual inertial pose estimation) with 100 particles resulted in increased robustness against rapid motions.

To prove: Gabriele shows the rapid camera movements once again…

Well, we have to suffer now so that in the future users won’t have to. Kudos Gabriele.

~~~

Next is Daniel Pustka with Dynamic Gyroscope Fusion in Ubiquitous Tracking Environments. This is part of Gudrun Klinker’s journey towards Ubi-AR.

What you need for ubiquitous tracking is automatic discovery of tracking infrastructure, and shield applications from tracking details.

Gyroscopes are very interesting to use (low latency, high update rate, always available), but they have drawbacks (drift, only  for rotation) and are only usable when fused with other sensors.

Daniel and team have proved that the ubiquitous tracking tool set consisting of spatial relationship graphs and patterns is very useful to analyze tracking setups including gyroscopes. It allows a Ubitrack system to automatically infer occasions for gyroscope fusion in dynamically changing tracking situations.

~~~

Jeroen Hol presents Relative Pose Calibration of a Spherical Camera and an IMU

This study builds on the idea that by combining vision and inertial sensors  you get accurate real time position and orientation in a robust and fast motion, and this is very suitable for AR applications. However, calibration is the essential point for this to work.

An easy to use algorithm has been developed and yields results with real data.

Ron Azuma asks: When the image is captured in high motion does it create blur?

Jeroen answers that it can be addressed by changing some parameters.

~~~

Last for this session is Wee Teck Fong from NUS to discuss A Differential GPS Carrier Phase Technique for Precision Outdoor AR Tracking.

The solution that Fong presents provides good accuracy with low jitter, drift and low computational load – and no resolution ambiguities. It works well for outdoor AR apps. With just one GPS you get an accuracy of about 10 meters plus you get high jitter of the tracking. Differential GPS using 2 GPS receivers (low cost 25mm sized) improves the accuracy of tracking. Fong and team have taken it a steps further with an advanced computational model that delivers higher precision for outdoor AR tracking. Fong claims that with a more expensive receiver he can achieve a less than 1mm accuracy, but you can’t use this technique anywhere. An infrastructure of stationary GPS stations transmitting wirelessly could provide a wide constant coverage for this technique.

Fong concludes with a positive note regarding the upcoming European update to the GPS system dubbed Galileo (in 5 years) were things will get significantly better.

===============

From ISMAR ’08 Program

  • Using the marginalised particle filter for real-time visual-inertial sensor fusion
    Gabriele Bleser, Didier Stricker
  • Dynamic Gyroscope Fusion in Ubiquitous Tracking Environments
    Daniel Pustka, Gudrun Klinker
  • Relative Pose Calibration of a Spherical Camera and an IMU
    Jeroen Hol, Thomas Schoen, Fredrik Gustafsson
  • A Differential GPS Carrier Phase Technique for Precision Outdoor AR Tracking
    Wee Teck Fong, S. K. Ong, A. Y. C. Nee

Live from ISMAR ’08: Tracking – Latest and Greatest in Augmented Reality

After a quick liquid adjustment, and a coffee fix – we are back with the next session of ISMAR ’08, tackling a major topic in augmented reality: Tracking.

Youngmin Park is first on stage with Multiple 3D Object Tracking. His first demonstration is mind blowing. He shows an application that tracks multiple 3D objects, which have never been done before – and is quite essential for an AR application.

The approach combines the benefits of multiple approaches while avoiding their drawbacks:

  • Match input image against only a subset of keyframes
  • Track features lying on the visible objects over consecutive frames
  • Two sets of matches are combined to estimate the object 3d poses by propagating errors

Conclusion: Multiple objects are tracked in interactive frame rate and is not affected by the number of objects.

Don’t miss the demo.

~~~

Next two talks with Daniel Wagner from Graz university about his favorite topic Robust and Unobtrusive Marker Tracking on Mobile Phones.

Why AR on cell phones? there are more than a billion phones out there and everyone knows how to use them (which is unusual for new hardware).

A key argument, Daniel is making: Marker tracking and natural feature tracking are complementary. But we need a more robust tracking for phones, and create less obtrusive markers.

The goal: Less obtrusive markers. Here are 3 new marker designs:

The frame markers (the frame provides the marker while the inner area is used to present human readable information.

The split marker (somewhat inspired by Sony’s by the eye of judgment) we use barcode split, with a similar thinking to the frame marker.

A third marker is a Dot marker. It covers only 1% of the overall area (assuming it’s uniquely textured – such as a map).

Incremental tracking using optical flow:

These requirements are driven from industrial needs: “more beautiful markers” and of course making them more robust.

~~~

Daniel continues with the next discussion about Natural feature tracking on mobile phones.

Compared with marker tracking, natural feature tracking is less robust, more knowledge about the scene, more memory, better cameras, more computational load…

To make things worse, mobile phones have less memory, with less processing power (and no floating point computation), and a low camera resolution…

The result is that a high end cell phone runs x10 slower than a PC, and it’s not going to improve soon, because the battery power is limiting the advancement of this capabilities.

So what to do?

We looked at two approaches:

  • SIFT (one of the best object recognition engines – though slow) and –
  • Ferns (state of the art for fast pose tracking – but is very memory intensive)

So both approaches wont work for cell phones…

The solution: combine the best of both worlds into what they call: PhonySift (Modified SIFT for phones). And then complementing it with PhonyFern – detecting dominant orientation and predicting where the feature will be in the next frame.

Conclusion: both approaches did eventually work on mobile phones in an acceptable fashion. The combined strength made it work, and now both Fern and Sift work at similar speeds and memory usages.

================

From ISMAR ’08 Program:

  • Multiple 3D Object Tracking for Augmented Reality
    Youngmin Park, Vincent Lepetit, Woontack Woo
  • Robust and Unobtrusive Marker Tracking on Mobile Phones
    Daniel Wagner, Tobias Langlotz, Dieter Schmalstieg
  • Pose Tracking from Natural Features on Mobile Phones
    Daniel Wagner, Gerhard Reitmayr, Alessandro Mulloni, Tom Drummond, Dieter Schmalstieg

Exclusive! HitLab NZ Releases an Augmented Reality Authoring Tool for Non Programmers

I am excited. I have in my hands a flier I just received from Mark Billinghurst (one of the AR gods at ISMAR ’08)

This flier includes the URL for a totally new augmented reality authoring tool developed by HITLab New Zealand. What’s really new about this too is that it targets non programmers (as in you and me).

BuildAR is a software application that enables you to create simple augmented reality scenes on your desktop.

BuildAR provides a graphical user interface that simplifies the process of authoring AR scenes, allowing you to experience augmented reality first hand on your desktop computer. All you need is a computer, a webcam and some printed patterns.

Mark says I am the first one to receive the flier – hence the exclusive news.

Without further ado (I haven’t even tried it myself yet…), here is the URL: http://www.hitlabnz.org/wiki/BuildAR

I promised Mark that by tonight (as clocked in Honolulu) the entire world will have tried it.

Don’t let me be wrong…

Tell us, does it work? do you like it? want more of these?

Live from ISMAR ’08: Augmented Reality – What Users Are Saying

Everyone is back from lunch and the afternoon session is on: User studies in augmented reality.

First on stage is Benjamin Avery to talk (with an animated Australian accent) about User Evaluation of See-Through Vision for Mobile Outdoor Augmented Reality. This

The study took users outdoors in various scenarios to test the performance of AR vision using see through displays. Then, they compared it with a second group that watched the video through a desktop computer

[link to paper, videos, images to come]

The result demonstrates complex trade-off between AR and desktop visualizations. AR system provided increased accuracy in locating specific points in the scene. AR visualization was quite simple, beating the desktop in tracking and in better visualization.

Stay tuned for the demo (which was hauled all the way from Australia to Cambridge)!

~~~

Next on stage is Cindy Robertson from Georgia Tech (Honorable mention in ISMAR 2007) and she discusses An Evaluation of Graphical Context in Registered AR, Non-Registered AR, and Heads-Up Displays.

How are users affected when there are many registration errors  or in other words when when tracking is not perfect? Can the user handle it better if a graphics context is provided?

They tested it with a set of tasks encompassing placing virtual Lego blocks with groups using Registered AR, Non-Registered AR, and Heads-Up Displays.

Following an exhaustive analysis of the resulted data they uncovered the following insights:

  • Head movement and memorization increased performance
  • Head movement affected perceived mental workload and frustration
  • When you have graphics obstructing your view, and switching between it and real world is frustrating
  • HUD-visible case was surprisingly faster than the other cases. But people hated it…

Final conclusion: Registered outperformed both the non-registered AR and graphics displayed on a HUD. Non-registered AR does not offer any significant improvement.

Future plans are to test home-like scenarios and impose more complex tasks.

~~~

On stage Mark Livingston is getting ready to talk about The Effect of Registration Error on Tracking Distant Augmented Objects.

A basic assumption is that registration errors limits performance of users in AR. “We wanted to measure the sources (such errors are noise, latency, position and orientation error) and see the affect on the user – and then be able to write requirements for future systems.”

For this study, they used the nVisorST.

The tasks were trying to measure the users ability to understand behaviors and situational awareness in the AR application: following a target (car) when buildings stand in between.

Conclusions are straight forward though somewhat surprising:

  • Latency has significant effect on performance and response time – was the worse.
  • Noise was disliked but did not have significant impact on performance
  • Orientation error fifn’t have significant effect
  • Weather had significant impact on results: darker weather delivered improved performances. Brightness was a major distraction.

===============

From the ISMAR Program

User Studies (from ISMAR ’08 program)

  • User Evaluation of See-Through Vision for Mobile Outdoor Augmented Reality
    Benjamin Avery, Bruce H. Thomas, Wayne Piekarski
  • An Evaluation of Graphical Context in Registered AR, Non-Registered AR, and Heads-Up Displays
    Cindy Robertson, Blair MacIntyre, Bruce Walker
  • The Effect of Registration Error on Tracking Distant Augmented Objects
    Mark A. Livingston, Zhuming Ai

Augmented Reality Made Simple in Flash

Make Me Pulse uncovered a Japanese innovation (from Saqoosha) making augmented reality a tad more accessible for the casual user.

This is as close to do-it-yourself-augmented-reality that I have seen so far: draw it yourself; no installation needed. Runs in flash. Very cool.

Check this movie, then try it yourself here.

More details on libraries and translations at MakeMePulse.

Thanks guys!

A new (media) power in the race for augmented reality supremacy

Media Power announced today a donation of $5M to the GVU research center at Georgia Tech – for the advancement of Mobile Augmented Reality (http://www.cc.gatech.edu/news/media-power-donates-5m-to-gvu-center).
It’s intriguing that Media Power’s founder is none other than the controvertial Carl Freer, the executive from Gizmondo – a mobile game device that went belly up “under a cloud” after demostrating huge potential in 2005. Although it made it to the #1 position of “The 10 Worst-Selling Handhelds of All Time” on gamepro it was pretty popular among mobile augmented reality research(Demo).

So now Carl will not only resurrect Gizmondo, but will also establish a new division – Magitech – “centered around the very promising field of Augmented Reality”.
The objective of the joint initiative between Magitech and Georgia Tech is to “envision, prototype and evaluate the next generation of mobile AR games and entertainment applications and positions the company as a leader in AR.”

This initiative looks promising mostly thanks to its ability to attract worldwide top talent in the field of augmented reality (many of them regular contributors to this blog – games alfresco):
Dr. Leonard Kleinrock (Professor, University of California at Los Angeles), Blair McIntyre (Professor, Georgia Tech), Mark Billinghurst (Professor, University of Canterbury), Daniel Wagner (Professor, University of Graz, Vienna), Dr. Michael Gervautz (CEO Imagination, Vienna)

Now, what would you do with $5M and that kind of caliber to advance augmented reality games?

***Update***

A couple of months later Media Power made another major investment. This time the sum was $2.7M and the benefactor –  Mark Billinghurst’s HIT Lab in New Zealand.

Top 10 Augmented Reality engines that will ignite a new era of video games

Welcome back!

So far we have looked at demos of augmented reality gameplay and investigated preferred hardware devices. That was just the warm up. Now we’re getting into the real beef: who offers the best software engine that can augment our reality (and our spirits)? Who will spearhead the creation of a totally new gaming experience?

Games alfresco brings you the unofficial roundup of the top 10 best augmented reality engines.

We will look at one engine at a time (enough with these looong posts!) and at the end of the round up, will rank them based on the feedback from you, the community. This is collaborative work in progress of group of passionate contributors on facebook. If you want to take part of the fun, join us at: augmented reality games.

Here is a sneak peak at some of the engines that we may or may not review:

Discontinued:

Stay tuned…or better yet – tell us what you think!

This just in: updates to "10 best AR devices…"

Hours following the publication of my latest post: “10 best AR Devices…” multiple announcements hit the air supporting my predictions:
1) Enkin: handheld navigation concept for Google Android phone
2) Apple’s patented head mounted display for iPhone
3) Gaming appearing recession proof, with close to a million handheld game devices sold in March 2008.