Open Letter to Apple: Let us Augment Reality with the iPhone!

A letter sent to Apple Developer Relations.

Dear Apple,

We are a collection of augmented reality (AR) enthusiasts and professionals (from business and academia), who have been working on a multitude of AR apps for the iPhone. These apps are poised to change the way people interact with the real world.

But here is the rub: we are currently unable to publish these apps on the app store because the iPhone SDK lacks public APIs for manipulating live video.

We are asking Apple to provide a public API to access live video in real time, on the iPhone.
We will be happy to offer additional technical details.

The impact of augmented reality (AR) on our lives could be as significant as the introduction of the PC.
In 10 years, we believe augmented reality will change the way everyone experiences travel, design, training, personal productivity, health care, entertainment, games, art, and advertising (videos).

Looking back just a few years, AR pioneers had to hack a slew of components into ridiculously large backpacks and HUDs, and be confined to rigged environments. Nowadays, it comes in friendly, affordable packages and the iPhone is one of the first devices to have it all – except for a public API.

The battle to determine the winning device has already begun; a public API to access live video will give the iPhone a lucrative ticket to compete.
We believe Apple has a window of opportunity of about 3 months before developers start looking elsewhere. If Apple decides to publish the API in that time frame – in the next 10 years, everyone might be using the iPhone as the preferred device to interact with the real world.

Here is how augmented reality could open up new opportunities for the iPhone this year:

Arf (Georgia Tech)

a virtual pet you take anywhere

ARghhhh (Georgia Tech)

first person table-top action game

Sekai Camera (Tonchidot)

AirTag the real world

Kweekies (int13)

a portal to creatures in a parallel world

Layar (SPRXmobile)

Browse the world with an AR browserDetails

Artoolkit for the iPhone (Artoolworks)

the most popular AR kit now on the iPhone

StudierStube ES (Imagination, Graz TU)

the only AR engine designed for mobile devices, now on iPhoneDetails

PTAM on the iPhone (Oxford University)

next generation AR tracking with no markers or images

Wikitude (Mobilizy)

a travel guide that “tells you what you see”

Virtual Santa (Metaio)

interactive Christmas application using the augmented reality

Augmented Reality Sightseeing (Fraunhofer IGD)

Historic photographs overlaid on your field of view while strolling in a street

These are apps that are practically ready to go. There is a whole bunch of apps and games that are just waiting for the API to be available.

…And Apple, we know you can’t share your plans…so please surprise us soon!

Many many thanks for your consideration –

Michael Gervautz – Managing Director Imagination GesmbH
Robert Rice – CEO Neogence
Georg Klein – PhD PTAM creator from Oxford University
Stephane Cocquereaumont –  President & Lead Developer Int13 (Kweekies)
Maarten Lens-FitzGerald – Founder & Partner SPRXmobile, developer of Layar
Ori Inbar – Author of and CEO and founder – Ogmento (formerly Pookatak Games)
Philippe Breuss – Lead developer, Mobilizy
Philip R. Lamb – CTO, Artoolworks
Noora Guldemond – Metaio
Takahito Iguchi – CEO, Tonchidot
Blair MacIntyre – Associate Professor, Georgia Institute of Technology

Bruno Uzzan – CEO, Total Immersion
Michael Zoellner
Fraunhofer IGD
Andrea Carignano – CEO,  Seac02

If you are developing an AR app for the iPhone and wish to join this effort – just let us know.

Wrap up 2008: Your Greatest Augmented Reality Moments

Top 10 AR milestones in 2008 was one of the most popular posts this year. What came out of it was even more gratifying: a multitude of reflections, impressions, and thoughts I received about your own AR moments, including some last minute finds.

Here is an anecdotal collection of your greatest AR moments in 2008:

1. The Most fundamental AR milestone in 2008

Oriel Bergig: During 2008 we have seen some major advances in the field of Augmented Reality. Porting AR technology to mobile devices and especially cellular phones creates an opportunity to reach millions of users. For several years, the biggest AR labs and companies have made huge steps in this direction. In 2008 these efforts have started to show results. Pose estimation has been upgraded with the StbTracker release in the end of 2007. Research focusing on better user experience, and in particular on making mobile AR technology accessible to people with no special training, is being conducted by the best minds of the HitLabNZ. During one of the top covered events of the year, CES2008, Intel’s CEO Paul Otellini demonstrated Total Immersion’s technology enabling mobile AR experiences such as urban guidance. To wrap-up, the 2008 most fundamental milestone would be: AR technology is closing up fast on the mass user market.

Charles Woodward: The greatest milestone? Commercial breakthroughs by Metaio and Total  Immersion.

Thomas Wrobel: Wikitude I think. Seems the first released, useful, AR software. Runner up to the AR Geisha doll

2) The best AR device of the year

Oriel: Since 2008 would most be remembered for its advances in mobile AR technology, the AR device of the year is the mobile phone. Nokia has released the Navigator phone that includes a GPS and an accelerometer, which make a valuable addition. The N95 has been demonstrated as well in many more contexts as a good choice for AR applications.  The next AR device of the year would be the Nokia N97 and of course the iPhone with its huge global success. iPhone feets very well AR applications and a successful attempt to port ARToolKit to iPhone has already been made by ARToolWorks. Appealing applications are next to come but only after the iPhone OS has better support for real time video acquirement.

Charles: Best device? iPhone, and/or Nokia 6210…

Thomas: hmz…tricky. I personally think hardware is still rather lackluster, and I have had little experience with some of the most recently released stuff.
I guess probably the iPhone + G1 devices…while far from ideal, they are at least getting location-aware services, and “barcode scanning” style product information into public hands.

Eric Rice shares what gets him excited about a video comparing between PS2 Eyetoy and PS3 Eye.

Vodpod videos no longer available.

3) Best AR Demo

Oriel: The best demo of 2008 is the demo that will be remembered by most people a decade from now. The demo that reached most of the people in the world is most likely Intel’s CEO Paul Otellini keynote talk during CES2008.

Charles: Haunted Book, Cherrer et al at ISMAR2008 – just beatiful!
(click Interaction on the left menu bar and then Haunted House.)

Thomas: LevelHead [by Julian Oliver], I think. Although this pet demo [ARf] is also nice;
(that may be because I want my own desuke though :p)

Vodpod videos no longer available.

4) Person of the AR year

Charles: Georg Klein – leads the way in anything he touches [single handedly won ISMAR 2008 tracking competition]

Thomas: There’s been so much development by so many individuals and companies I don’t know one specific person.

5) The most significant AR deal of 2008

Charles: Beijing Olympics fake fireworks. About the viewers of the Olympics openning ceremony:”What they did not realise was that what they were watching was in fact computer graphics, meticulously created over a period of months and inserted into the coverage electronically at exactly the right moment. ”

Thomas: Not sure about AR deals as such, but Total Immersion getting offices in the US is a good sign for the company and AR in general.

6) A [Predictable?] disappointment

Gizmondo won’t be coming out this year after all…The Nordik Link has the scoop.

7. Last minute find: A Surprising Simplicity in AR

Anyone can build 3D models with Google’s Sketchup. With the AR Media plugin from Inglobe – anyone can bring it into an augmented reality scene. ArchDaily tried it here.

Thanks for contributors and especially: Charles Woodward, Oriel Bergig, and the always there AR enthusiast: Thomas Wrobel

Bonus: Blair MacIntyre shares his greatest AR moments in 2009 in his blog.

Live from ISMAR ’08 : Perfecting Augmented Reality

The last session of ISMAR ’08 is about to begin, and it concentrates on perfecting Rendering and Scene Acquisition in augmented reality and making it even more realistic.

First on stage is Yusaku Nishin with a challenging talk attempting Photometric registration by adaptive high dynamic range image generation for augmented reality.

His goal : development of photorealistic augmented reality with a High Dynamic Range (HDR) image.

Estimating the lighting environment of virtual objects is difficult because of low dynamic range cameras. In order to overcome this problem, they propose a method that estimates the lighting environment from an HDR image and renders virtual objects using an HDR environment map. Virtual objects are overlaid in real-time by adjusting the dynamic range of the rendered image with tone mapping according to the exposure time of the camera. The HDR image is generated from multiple images captured with various exposure times.

Now you are ready to watch the resulted effect. Incredible.



Next on stage is the soon-to-be-hero-of-the-show Georg Klein (more on that later…) Compositing for Small Cameras

Blending virtual items on real scenes. It can work with small cameras. Video from such cameras tend to be imperfect (blurring, over saturation, radial distortion, etc) so when you impose a virtual item it tend to stick out in a bad way. Since we can’t improve the live video – we will try to adapt the virtual item to match the video at hand. Simply put, Georg samples the background and applies it to the image which matches blur, radial distortion, rotation, color saturation, etc) and he does it in 5 millisecond on a desktop… For details check the pdf paper; take a look for yourself and tell me if it works on Kartman:

Done! Georg is already working on the next challenge.


Following is Pished Bunnun introduces his work: OutlinAR: an assisted interactive model building system with reduced computational effort

Building 3D models interactively and in place (in-situ), using a single camera, and low computational effort – with a makeshift joystick (Button and wheels.)

In this case the video does a better job at explaining the concept than any number of words would…

Pished demonstrates it’s fast and pretty robust. You judge for yourself.

If you absolutely need more words about this – start here.

The team’s next challenge: make curved lines…


In the very last talk of the event Jason Wither courageously takes on another challenge to perfecting augmented reality, with his talk: Fast Annotation and Automatic Model Construction with a Single-Point Laser Range Finder

Jason is using a laser finder typically used by hunters (though he will not be shooting anything or anybody), mounted on the head or handheld, in conjunction with a parallel camera. First he wants to create an annotation. that’s totally trivial. But you can then orient the annotation according to a building for example.

Next, he is going to correct occlusion of virtual objects by real objects for improved augmented realism. Just click before and after the object and pronto:

Finally he will create a 3D model of an urban environment semi-automatically, by creating a depth map courtesy of the laser. To achieve that he’s using a fusion process. You got to see that video; the laser’s red line advancing on buildings reminds me the blob swallowing the city in that quirky Steve McQueen movie.

In conclusion this is a really low cost and fast approach for modeling and annotation of urban environments and objects. That capability would become extremely handy once Augmented Reality 2.0 picks up and anyone would want to annotate the environment (aka draw graffiti without breaking the law).

Next is the event wrap up and the results of the Tracking Competition. Stay tuned.


From the ISMAR ’08 program:

Rendering and Scene Acquisition

  • Photometric registration by adaptive high dynamic range image generation for augmented reality
    Yusaku Nishina, Bunyo Okumura, Masayuki Kanbara, Naokazu Yokoya
  • Compositing for Small Cameras (pdf paper)
    Georg Klein, David Murray
  • OutlinAR: an assisted interactive model building system with reduced computational effort
    Pished Bunnun, Walterio Mayol-Cuevas
  • Fast Annotation and Automatic Model Construction with a Single-Point Laser Range Finder
    Jason Wither, Chris Coffin, Jonathan Ventura, Tobias Hollerer