Live from ISMAR ’08: Near-Eye Displays – a Look into the Christmas Ball

The third day of ISMAR ’08, the world’s best augmented reality event, is unfolding with what we expect to be an eye popping keynote (pun intended) by Rolf R. Hainich, author of The End of Hardware.

He is introduced as an independent research and started to work on AR in the early ’90s – so he could be considered as a pioneer…

A question on everyone’s mind is: Why Christmas ball and not a Crystal ball?

Rolf jumps on stage and starts with a quick answer: Christmas balls can help produce concave mirrors – useful for near eye displays.

First near eye display was created in 1968 by Ivan Sutherland; in 1993 an HMD for out of cockpit view was built in a Tornado simulator. In 2008, we see multiple products such as NVIS, Zeiss HOE glasses, Lumus, Microvision, but Rolf doesn’t consider them as true products for consumers.

Rolf ,defined the requirements for a near eye display back in 1994. It included: Eye tracker, camera based position sensing, dynamic image generator, registration, mask display, holographic optics. And don’t forget no screws, handles, straps ,etc…

He then presents several visions of the future of human machine interaction which he dubs 3D operating system.Then he briefly touches on the importance of sound, economy and ecology – and how near eye displays could save so much hardware, power, and help protect the environment.

But it requires significant investment. This investment will come from home and office applications (because of economies of scale- other markets such as military, medical, etc – will remain niche markets.

The next argument relates to the technology: Rolf gives examples of products such as memory, displays, cell phones, cameras which experienced dramatic improvements and miniaturization over the last years. And here is the plug for his famous joke: Today, I could tape cell phones on my eyes and they would be lighter than the glasses I use to wear 10 years ago…

Now, he schemes through different optional optical designs with mirrors, deflectors, scanners, eye tracker chips, etc (which you can review in his book The End of Hardware) These design could support a potential killer app – eye operated cell phone…

Microvision website is promoting such a concept (not a product), mostly to get the attention of phone manufacturers, according to Rolf.

Rolf, then tackles mask displays, a thorny issue for AR engineers and suggests it can achieve greater results than you would expect.

Eye Tracking is necessary to adjust the display based on where the eye is pointing. It’s once thing that AR didn’t inherit from VR. But help could come from a different disciplines – computer mouse which have become pretty good at tracking motion.

Other considerations such as Aperture, focus adjustment (should be mechanical), eye controller, are all solvable in Rolf’s book.

Squint and Touch – we usually look where we want to touch, so by following the eye we could simplify the user interface significantly.

Confused? Rolf is just getting started and dives effortlessly into lasers, describing what exists and what needs to be done. It should be pretty simple to use. And if it’s not enough, holographic displays could do the job. Rolf has the formulas. It’s just a matter of building it.

he now takes a step back and looking at the social impact of this new technology: when everybody “wears” anybody can be observed. The big brother raises its ugly head. Privacy is undermined, Copyright issues get out of control. But…resistance is futile.

Rolf wraps up with a quick rewind and fast forward describing the technology ages: PC emerged in the 80’s, AR in the 2020’s, and chip implants (Matrix style) will rule in the 2050.

Question: It didn’t look like the end of hardware…

Rolf: it’s the end of the conventional hardware – we will still have hardware but it could be 1000 times lighter.

Tom Drummond (from the audience): there is still quite a lot of work to get these displays done and there is still some consumer resistance to put on these head up displays…

Rolf: People wear glasses even for the disco – it’s a matter of fashion and of making it light – with the right functionality.


From the ISMAR ’08 Program:

Speaker: Rolf R. Hainich, Hainich&Partner, Berlin

We first have a look at the development of AR in the recent 15 years and its current state. Given recent advances in computing and micro system technologies, it is hardly conceivable why AR technology should not finally be entering into mass market applications, the only way to amortize the development of such a complex technology. Nevertheless, achieving a ‘critical mass’ of working detail solutions for a complete product will still be a paramount effort, especially concerning hardware. Addressing this central issue, the current status of hardware technologies is reviewed, including micro systems, micro mechanics and special optics, the requirements and components needed for a complete system, and possible solutions providing successful applications that could catalyze the evolution towards full fledged, imperceptible, private near eye display and sensorial interface systems, allowing for the everyday use of virtual objects and devices greatly exceeding the capabilities of any physical archetypes.

Live from ISMAR ’08: Latest and Greatest on Augmented Reality Displays

Welcome back to ISMAR ’08; This is the second day and we are getting to the meaty topics.

Ozan Cakmakci is on stage and kicks off with walking through his paper: Optical free form surfaces in Off-Axis Head-Worn Display Design.

Ozan zooms through a quick history of optics and switches to a set of graphs and functions which you can review in his paper.

The conclusion is pretty clear though: Free form surfaces are useful in optical design to maximize performance in pupil size or field of view.

Questions such as who’s going to build it, when or how much it will cost – are left for guessing…


Next on stage is Sheng Liu from University of Arizona with the topic: An Optical See-Through Head Mounted Display with Addressable Focal Planes

Sheng talks about the stress on the eye in an AR situation where the eye has to accommodate real and virtual object and adjust the focus accordingly, and could cause headache to the viewer.

The solution is a variable-focal plane in a liquid lens.

Vari-focal with liquid lens for AR

Subjective tests result in a pretty good response from the participants. With the vari-focal plane in liquid lens, the human eye can accommodate change in focus from infinity to near focus and can be used for AR applications. This would even be improved in the future with upcoming improvements in liquid lens.

One of the members of the audience asks why not do this in software vs. hardware? Wouldn’t it be less expensive?

– Sheng claims the results are more accurate with the hardware approach.

To learn more about this, check out their website, the paper [link will be posted here], or contact sliu[at]


In the third leg of the “Displays” session Ernst Kruijff will speak about Vesp’R: design and evaluation of a handheld AR device.

UMPCs are a good starting point for AR displays – but tend to get bulky…

VAIO used for outdoor AR tracking at Oxford University

VAIO used for outdoor AR tracking at Oxford University

[I have analyzed this and other devices in my post: Top 10 AR devices]

Ernst will present an alternative design. The motivation for the research and the resulting paper was the lack of published knowledge  on this topic.

The team looked at a wide range of AR apps (such as Vidente an AR app for field workers) on different platforms and observed the common needs.

The need is simple: a lightweight device, with options for more controls, for long duration of use – indoors and outdoors.

UMPCs such as Vaio could are pretty heavy and become very tiring, especially when you hold it high.

Here is the result:

A solid case; velvety grip; controls are built into handles.
How good is it?
Based on a user attitude study – the new design is reasonable but not ideal…
When comparing with existing devices -some aspects were better and others not.

The conclusion is that although Vesp’R doubles the weight of a usual UMPC, it still provides improved ergonomics. But there is room for more research and improvements in this domain.

A member of the audience dares to ask: what if you used a much lighter device (such as a cell phone), would the results still be the same…?

Ernst is positive; just try to hold your hands straight ahead with no device at all – and you’ll feel the pain in a few minutes…

Stay tuned for the outdoor demo on Wednesday!


From the ISMAR ’08 Program

  • Optical Free-Form Surfaces in Off-Axis Head-Worn Display Design
    Ozan Cakmakci, Sophie Vo, Simon Vogl, Rupert Spindelbalker, Alois Ferscha, Jannick Rolland
  • An Optical See-Through Head Mounted Display with Addressable Focal Planes
    Sheng Liu, Dewen Cheng, Hong Hua
  • Vesp’R: design and evaluation of a handheld AR device
    Eduardo Veas, Ernst Kruijff

Unveiling Tonchidot: A Cool Parallel World – on the iPhone

Yesterday, at the TechCrunch 50 conference in San Francisco, a small start up from Tokyo stole the show; the name: Technidot, the cool product they unveil: Sekai Camera (World Camera) running on the iPhone.

In their own words, the Sekai Camera is:

…a real-world interface for the iPhone that connects real and virtual worlds, allowing anybody to create, experience and participate in both.

While at your favorite mall, point your iPhone camera at things around you such as food, toys, art, transit maps, and get detailed information about it. More fun for you, more business for retailers.

AirFilter takes care of the customized search to bring you the additional information, based on a database generated by advertizers.

And you’re not on your own; the Sekai Camera is also a social networking environment: leave messages to your friends – in space – so they can see it when they pass by.

Isn’t that a killer augmented reality application?

And Tonchidot’s take on Evolution could become the symbol of the AR revolution…

Here is the coverage of the demo by Anthony Ha on VentureBeat : including enthusiasts and skeptics.

This just in! More skeptics!

Blair MacIntyre finds it hard to believe it’s real. The TechCrunch panelists were amused and confused but blamed it on the language barrier.

What do you think? Is it real? or really augmented?

Filip, many thanks for the tip!

I want my iPhone augmented too!

The buzz continues.

As the coolest gadget around, the iphone is raising a lot of interest in the augmented reality community. This was once again confirmed in an interview with Stéphane Cocquereaumont, President and Lead developer at Int13, a mobile games developer:

“The iPhone is in fact our main target, the next demo we’ll publish will be on this device, this is probably the best device to do mobile AR today, even if its camera is far from perfect.”

Stéphane conversed with me following a post of Int13’s demo on the augmented reality games facebook group titled: “Augmented Reality on mobile devices that just works” 

He added:

“Our AR library should work easily on devices such as Intel’s MID [which is coming out this summer – games alfresco]…but we’re more interested in Smartphones…”

Games alfresco: How central do you see augmented reality in your company’s future?

Stéphane: “We plan to continue working on our AR technology and improve it, we’re especially interested in markerless tracking.”

And here is what I really liked in this interview:

“But our main objective is to create cool games and sell them”

Int13 is among the first pure game developers to dive into the AR space.

“if our current AR projects reveal themselves to be commercially successful then AR could become central for us.”

Cautious optimism, always a good trait for a game developer.

Thanks Stéphane and France from Int13 for sharing your experiences and plans!


Don’t you want your iPhone augmented too?

How do you spell Augmented Reality in iPhonese?

Doesn’t it feel like every other day, a new augmented reality demo for the iPhone hits YouTube?


Here is a play by play of the sequence of events that led the iphone to the top of the AR devices list:

First, on June 20th, “Augmented Reality Navigation” on the iphone hits youtube…


Then the Holotext hoax (on the iPhone) confuses the already confused youtube audience…


Furthermore, on July 18th, Thom York shows up singing as a [tricked] hologram on the iphone…


Nonwistawnding, artoolworks surprises, on July 21st, with the release of arttoolkit 4.4 for iPhone… 


What will they do next?

The End Of Museums As We Know It

As the light at the end of the (summer vacation) tunnel is almost insight, let me ask you a reflective question:

Where do kids prefer to be on a summer day:

(a) Museums

(b) Theme parks

(c) Staying at home and playing video games

Any volunteers for (a) ?…

What if you could combine all three into one?

What if you could transform learning about cultures, art, science, history – into a fun experience for kids? What if museums were as much fun as outdoor adventures and video games combined?

This fantasy is becoming a reality thanks to efforts by pioneers around the world.

Here are my picks of the 4 5 best augmented reality tours that are reinventing museums:

1. Lifeplus in Pompei

Pompei Ladies in an afternoon promenade in front of your eyes

Breathing new life into the ruined streets of Pompei

Visitors stroll in the real streets of Pompei, while watching thru their glasses, virtual scenes of city natives living their lives as if it’s 79 AD, minutes before the eruption of Vesuvius.

This EU funded project was lead by MIRALab – university of Geneva in 2004. See more at Lifeplus.

2. DNP-Louvre Museum lab

A behind the scenes look at exhibits

A mobile device with live video, shows on the display virtual objects such as a balloon that guides visitors through the exhibits. The climax of this tour arrives (1:37) when shards of an antique Islamic platter are virtually reconstructed to create the real platter.

Kudos to Metaio who developed the experience for DNP-Louvre Museum Lab in Tokyo, though they should try trimming the bulky device…

See more at DNP-Louvre Museum lab in Tokyo

3. Mobile Augmented Reality Quest (MARQ) –         Expedition Schatzsuche

Treasure hunt in a museum (in Austria)

A team oriented game where museum visitors play the role of investigators required to solve 3d virtual puzzles surrounding exhibits. Successful completion of puzzles reveal further steps of the story.

Beyond the new type of interaction with museum exhibits, MARQ introduces multi user collaboration: collected virtual items can be shared between groups, and “guided tour replays” can be viewed at any time – on the Gizmondo (RIP) gaming device.

This novice approach to experiencing Museum exhibits was developed by the Graz University team in Vienna, led by Daniel Wagner and Dieter Schmalstieg. It was shot at the Kärner Landesmuseum in Klagenfurt/Carinthia.

See more about MARQ at Studierstube

And here is an older project from the same team: Enigma Rally at the Vienna Technical Museum.

4. Rome Reborn, now Augmented

Undo the barbarians

Rome Reborn is the largest computer simulation of an ancient city. Cool. But what Fraunhofer (Institute for Computer Graphics Research) has done with it is way cool: walk among the ruins of the Roman Forum and point your Vaio UMPC anywhere to see buildings being reconstructed.

See more about the technology used at InstantReality

5. Voices of Oakland

If a cemetery were a museum

Blair MacIntyre and his team at GA Tech have done the impossible: they have turned the Oakland cemetery in Atlanta to a visitor magnet – all thanks to an augmented reality tour which can be experienced on a cell phone. You have to see it to believe it.

Unfortunately, the cemetery was destroyed last year by a hurricane. So the students of subsequent years wont be able to keep playing with it.

6. Science Museum in Paris

Navigate Museums with AirTags

A new implementation by Tonchidot for La Villette Museum in Paris where visitors use AirTags provided by the Museum or by users to enrich their museum experience.
In the creators own words:

Sekai camera turns a museum into a “living” internet environment…The real world becomes “clickable”

7. Digital Binocular Station for Cultural Museums

A stationary Augmented Reality device developed by Mind Space Solutions. Because it is fixed to a single location, it allows the use photorealistic, cinema-quality visuals, and compensate for the lack of parallax by presenting everything in stereoscopic 3D.


What are the 10 ingredients to augment a museum tour?

  1. A practical augmented reality device (avoid backpacks and bulky displays) with visual tracking software
  2. High quality 3D models of exhibits, and how they looked and behaved in the past (and future?)
  3. A combination of learning and fun with a really really good story
  4. Breath life into inanimate objects
  5. Sprinkle some sound when necessary
  6. Incinerate verbose plaques; say what you have to say in 2 to 5
  7. Indoor tours are great; outdoor tours are even better
  8. Multi user interaction and collaboration
  9. Location based services, including (fun) navigation instructions
  10. Did I mention it has to be fun?
…and don’t forget to send the kids home with a souvenir DVD: “my augmented tour at the museum”

Didn’t make the list…

The following efforts didn’t make the list, mostly because they forgot ingredient #10…

Have you seen other augmented museum tours? Share your experiences!

10 best augmented reality DEVICES that will reinvent mobile video games

In my last post, I proposed a countdown of the top 10 augmented reality demos that are poised to revolutionize video games. That collection focused on concepts that inspire a totally new kind of gameplay. Novice readers testified it was a great introduction to augmented reality, while AR savvy insiders found some newly discovered ideas quite inspiring. Thank you all for the great feedback.

Now, let’s talk hardware: how in the world are we going to play these games?

Courtesy of “Window to The Future” By Steve Kosareff

AR game devices run the gamut from 20 lbs-backpack-and-head-mounted-display-systems to tiny handheld cell phones.

Once again, Marshall McLuhan’s legacy (“the medium is the message”) guides the discussion: The evolution of AR hardware devices changes the game mechanics and opens the door for a revolution in game experience.
Scholars have studied it at length (sidebar); I propose a simplified view spanning 3 generations:

The Past: Generation “Kit Bag”

Tinmith epitomizes generation “kit bag”

* took place over the last 10+ years
* custom built backpack with laptop, accessories
* head mounted display
* used exclusively in research
* groundbreaking experience yet –
* heavy, complex, and expensive

The Present: Generation “Hand Bag”

Generation “hand bag” took off with The Invisible Train
* started in 2003 with a horizon of 2010
* mass production: banking on the ubiquity of mobile devices
* aspiring for larger screens with more powerful devices
* easy to carry, ergonomic, affordable, yet –
* occupies hands, limits immersion

The Future: Generation “No Bag”

Leonard Low’s concept eyewear for augmented reality

* next 3-15 years
* eyewear: glasses and later contact lens
* handsfree; immersive
* in short: nirvana…(also means in Hindu – the end of suffering)

So, what’s the perfect device for mobile augmented reality games?

Experts argue that it depends on the type of game.
I buy that. Here’s the revised challenge: if game developers want to build a game like Roku’s Reward – what handheld device should they zero in on?

Looking at present and future generations, here is my countdown of the top 10 AR mobile devices with which developers will reinvent video games:

10. PDAs (Personal Digital Assistant)

MIT’s Environmental Detectives

PDAs knocked out the “kit bag” generation and signaled the dawn of the “hand bag” generation. They delivered mobility and extensibility (it’s easy to add a camera, GPS or other accessories) and it offered reasonable processing power.
However, with the convergence of devices such as cell phones, cameras, and the miniaturization of computers – PDAs are becoming obsolete.
Notable PDAs for AR include: iPAQ, Dell Axim, Fujitsu-Siemens, and Asus

9. UMPC (Ultra Mobile PC)

Outdoor tracking with a UMPC at the University of Cambridge

UMPCs are the most powerful handheld devices out there. Its Windows operating system makes it a familiar and practical platform for development.
Sony Vaio UX is used often by researchers; Raon Digital Everun is another candidate.
UMPCs would have been at a higher position on this list – had it not introduced 2 barriers for adoption as an AR game device: price (upwards of $1500) and size (needs a sizable bag). After all, the UMPC is designed for business – not games.

8. Smart Phones

C-Lab’s KickReal runs on a Siemens phone

If you’re after the masses, smart phones are your best bet thanks to their ubiquity (out of 1.14 billion cell phones sold in 2007 – 10% were smart phones according to IDC) . In two or three years, it will be nearly impossible to buy a conventional cell phone (as agreed at the CTIA wireless show 2008.)
So, almost everyone has them, cameras are getting better, they’re adding accelerometers, and positioning capabilities and some games have proven to work on this type of devices.
Popular mobile phones (sorted by OS popularity according to canalys) include Symbian, Microsoft, RIM, iPhone, Palm (Acquired by Access), and Linux.
Google’s Android phone has ambitions to carve its niche, but has still a long way to go (too slow, not stable). It may emerge as a viable option in 2009.
One AR company suggests the HTC P6500 as “a good approach [for augmented reality games] with faster and easier development tools.”
But because of its small screen, low end processors, and major market fragmentation – smart phones cant leap beyond the 8th spot.

7. iPhone

The iPhone could surpass other game devices (Roughly Drafted Magazine )

Though considered a smart phone – owing to its unmatched user experience – the iPhone deserves its own category.
Its form factor, touch screen, accelerometer, more modern technology, and now with an open SDK – it’s irresistible.
Although Apple is not known as a great gaming company there are currently 373 games listed under iPhone webapps.
Apple’s total control of the whole product compared with Gphone’s patchwork of multiple companies will result in a better experience.

6. Handheld Game Devices (PSP, Gizmondo, Nintendo DS)

Gizmondo used for the Caleb project at Graz University

Handheld game devices are leading the growth in the game hardware market. Some people believe they will cut into the console market. Devices such as Sony’s PSP, Nintendo DS (and Gizmondo assuming it will indeed reemerge) have a great form factor – and the fact that they are designed specifically for games, gives them an edge over general purpose devices.
But here’s the caveat: PSP and the DS need to be complemented with accessories such as camera, as well as accelerometers, positioning and ubiquitous connectivity capabilities – to be able to play in this game.

5. Nokia Phones (N Series)

MARA project at Nokia

Haven’t we already covered smart phones at #8?. Well, Nokia isn’t just another smart phone company. Through extensive research, it pushes the envelop of augmented reality experiments on phones more than any other manufacturer.
Notable devices include N-81, N95, N-810.
The feature packed multimedia heavy N95 stands out as the most popular among researchers despite some limiting factors:
* runs on symbian – not great for heavy programming
* small 2.6” screen with 240 x 320 pixels only
* Expensive (above $500) and not really for mass market

4. MID (Mobile Internet Devices)

Intel announced these MIDs to be released in 2008

Intel is putting its muscle behind the MID. The MID is designed for multimedia consumption on the go. It’s extremely portable with an ideal screen size (4-5”); it’s fully connected and armed with a strong (low power consumption) processor. The fact that it runs on a scaled down Linux makes it accessible and fast. But, what really separates it from the UMPC is its sub $500 target price.
Before year end ,Intel hardware partners are planning to release these products: LG Xnote, BenQ, Aigo, Lenovo MID.
Although experts have no real world experience with the MID – it’s shaping up as the preferred choice; if you plan to develop an augmented reality game this year – I would single out the MID as your #1 device.

When the flood of MIDs comes upon us, here is the criteria I suggest to separate the wheat from the chaff:
* screen size between 3.5″-5.5″ (4″-5″ ideal)
* min 65K colors, 800×480 pixels
* strong processor – min 400MHz
* with CGI acceleration (e.g. NVidia APX2500, TI OMAP 3)
* ubiquitous connectivity with cell (3G) and wifi/wimax
* high quality video camera: resolution (640×480), 30 fps
* positioning capabilities (GPS or software based locationing)
* inertial sensors, accelerometers, digital compass
* blue tooth for adding essential accessories
* touch screen – nice too have
* price – below $500; $200 would be ideal for the masses

Now let’s delve into the future; here devices become more conceptual, which means – not proven, yet stimulating. Prepare to be audaciously hopeful.

3. Looking glass (design concept)

In its designer Mac Funamizu’s own words:

“This is what I wish the Internet search will be able to do with a mobile device in the NEAR future. Touch screen, built in camera, scanner, WiFi, Google map (hopefully Google earth), Google search, image search… all in one device. Like this way, when you can see a building through it, it gives you the image search result right on the spot.”

Will anyone ever build it? I don’t know – but it sure looks cool.

2. Glasses

Mirage Innovations is one of the contenders for the future generation

Nokia dubbed mobile phones as the 4th screen (following cinema, TV, PC). According to this count, glasses will be the 5th screen. It spearheads the next generation of augmented reality: the “no bag generation”. This is when we become liberated. Handsfree. The interactive overlays on top of reality surround us anywhere we look. Can we start developing games for glasses? not quite yet. They still lack built in cameras, eye tracking, and tend to cause dizziness. Should we start thinking about it? Absolutely. Rolf Hainich has dedicated a book to this concept and aptly called it the end of hardware. Ben Averch writes about this market in his future-looking blog.

Companies to watch:

Microvision is working on lightweight eyewear for augmented reality

Lumus Optical has just launched their new and slick eyewear

Vuzix from military to consumer video eyewear

Here is the criteria I recommend to use to evaluate the progress of contending eyewear companies:
* looks and weighs like regular sun glasses
* see thru, high resolution display
* integrated camera
* eye tracking
* position and movement sensors
* integrated sound

1. Contact Lens

Darpa project for the creation of micro- and nano-scale display technologies

Apart from topping the list of “things you wouldn’t borrow from a friend”, contact lens leap to the top spot of AR devices. Normally, the mere thought would be scientifically dismissed as “too far off”, but this Darpa project makes it slightly more conceivable. In Vernor Vinge’s Rainbows End near future novel – everybody (except retreads) wears them. It becomes a natural extension of human intelligence, with user interface that can be summed as squint and gesture. It gives itself away only due to the occasional blank stare which is typically confronted with: “are you googling again!?”.
The story is taking place in 2025. Not too far off.

Back to the present: Scientists at the University of Washington have already created a proof of concept – see video below.

Didn’t make the list
* Laptops – laptops were used extensively during the “kit bag” generation as a key ingredient of the back pack, but during the hand bag generation – no one wants to the roam the streets tethered to a laptop. First implementation of Plundr ran on a laptop – but gamers soon clamored for the Nintendo DS version.
* Zune – should it be?…
* Brainchips – beyond the horizon of contact lens lies the world of brainchips . I am not touching this one with a 12 foot pole…I consider myself a technology enthusiast, but this makes me feel uneasy. I want to always have the choice to de-augment my reality when I’ve had enough. Hard connections into my brain may deprive me from that right.

Many thanks to the expert contributors: Daniel Wagner, Charles Woodward, Blair McIntyre, Peter Wojtowicz, and Jose Carlos dos Santos Danado (YDreams).
What’s your take? I really want to hear from you.