As soon as the exhibition floor opened today at the Game Developer Conference (GDC) in San Francisco, I rushed to visit the Vuzix-Metaio booth to witness the first consumer-oriented augmented reality booth ever to exhibit at GDC.
I was not disappointed. Just watch this clip.
Stefan Misslinger (Metaio) did a fantastic job acting it out, don’t you think?
Vuzix also had on the display the upcoming-anticipated-long-awaited-soon-to-be-released 920AV AR glasses. It wasn’t in working condition, but Vuzix promised it will be released this fall.
When the event kicked off, I offered to serve as your eyes and ears; Peter Milford took the offer and expressed his curiosity about the latest from Vuzix. This one is for you Peter: yours truly posing with the goods.
Now the key question is:
how would you rank the look of these specs on a dork-to-cool scale?
This week at TED, on the very stage where Bill Gates unleashed mosquitos into the audience to make a point about the need to cure Malaria, an MIT researcher, Patty Maes, unveiled a “six sense” device.
Maes demonstrated a portable device constructed out of commercial of the shelf products such as a web camera, pocket projector, and a cell phone.
What kind of “six sense” feats can it achieve?
Yahoo Tech captured the new ways to interact with the world made possible with this device:
turn any surface into a touch-screen for computing, controlled by simple hand gestures
take photographs by framing a scene with your hands
project a watch face by creating a circle on your wrist with your finger
recognize items on store shelves and provide personalized recommendations
look at an airplane ticket and know whether the flight is on time
project information about a book while browsing at a store
recognize articles in newspapers, retrieve the latest related stories or video from the Internet and play them on pages
Augmented reality enthusiasts would immediately recognize these fantastic ideas. Whether you use cell phones, goggles, or a projectors to view the added information – it’s a whole new way to interact with the world.
Now we have to wait patiently until TED uploads the video. ***update*** see videos below.
Last year cellphones took center stage in spearheading augmented reality into the main stream. Out of the blue comes this spatial augmented reality cobbled-together-device and takes the spot light.
Welcome to the climax of ISMAR ’08. On stage the 9 “gods” of the augmented reality community. And they are siting in a panel to muse about the next 10 years of augmented reality.
Dieter Schmalstieg took on the unenviable job of moderating this crowd of big wigs. See if he can curb them down to 3 minutes each.
Here is a blow-by-blow coverage of their thoughts.
Ron Azuma (HRL)
The only way for AR to succeed is when we insert AR into our daily lives – it has to be available all the time (like Thad Starner from GA Tech which always wears his computer)
Ron asks – What if we succeed? what are the social ramifications? those who have thought about it are science fiction writers…such as Vernor Vinge (have you read Rainbows End and Synthetic Serendipity.)
Reinhold Behringer (Leeds)
AR is at the threshold of broad applications.
Cameras, GPS, bandwidth have improved immensely – split into lo-fi AR, approximate registration, low end hardware. and also hi end AR, live see through displays, etc.
What’s missing is APIs, common frameworks, ARML descriptor (standardization)
Mark Billinghurst (HitLab NZ)
Mobility (now) – It took 10 years to go from backpack to palm
Ubiquity (5+ years) – how will AR devices work with other devices (TV, home theater, …),
Sociability – it took us 10 years to go from 2 to 4 to 8 users . When will we have massive scale?
Next is AR 2.0 with massive user generated content and a major shift from technology to user interaction
Steve Feiner – Columbia
AR means “The world = your user interface”
What will it take to make this possible?
Backpacks are ridiculous; handheld devices will look ridiculous 5 years from now – so don’t write off eyewear.
A big one is dynamic global databases for identification/tracking of real world objects. Tracking could be viewed as “just” search (granted a new kind of search.)
There is more to AR than registration; AR presentations need to be designed (AR layouts).
Gudrun Klinker – TU Munchen
|ntegrating AR with ubiquitous. We are interfacing with reality, with our senses and others are mental. We need those lenses to connect to our “senses” (not just visually – it could also be sound, etc). Combining the virtual with the real – where is the information? and can we see it? How do we communicate with the stationary world? We need to connect with the room we are in and hear the “story”. The devices at least need to talk to each other.
We also need to think about “augmented” building, they do not evolve as fast as cell phones. Another aspect is how are we going to survive “this thing”. We need much more usability studies and connect it with real world applications. The ultimate test (I challenge you to show it in next year’s competition) is a navigation system for runners. It’s easy to do it for cars – but may be harder for people.
Nassir Navab – TU Munchen
Medical augmented reality – showing fascinating videos of medical overlays [add videos]
The simplest idea is getting into the operation room – combining X Ray and optics as part of the common operating workflow.
Next is fusion of pre/intra operative functional and anatomical imaging; patient motion tracking and deformable registration; adaptive, intuitive and interactive visualization; Integration into surgical workflow
Finally we need to focus on changing the culture of surgeons (e.g. training with AR simulation).
Haruo Takemura – Osaka University
Showing a table comparing the pros and cons of hardware platforms: e.g. mobile have potential benefits vs HMD (but also drawbacks – such as processing power); desktop is cheap and powerful but not mobile (tethered).
Cell phones have another issue – they are tied to the carriers which is problematic for developers.
Bruce Thomas – UniSA
We are extremely interdisciplinary – and should keep it up.
However with so many of these it’s hard to develop and evaluate. And by the way innovation is difficult to articulate.
We are in a “Neat vs. Scruffy” situation – the bottom line is that a smaller self-contained pieces of research is easier to get in front of the community – and get results.
Questions floating:
is high end or low end AR the goal?
is ubiquity in AR realistic or wishful thinking?
are we innovative/.
Does augmented reality need to make more money to survive?
Platforms: Don’t write off eyewear?
Social: what if we succeed with AR?
What is the position of ISMAR in the scientific community?
A controvertial question from the audience to the panel: How many of you have subject matter expert working in your office on a daily basis? (few hands) How many of you have artists working a daily basis? (even fewer hands) How many of your research have reached the real world? (once again – few hands)
A question from the audience about the future of HMD. Mark takes the mic and asks the audience:
How many of you would wear a head mounted display? (5 hands)
How many of you would wear a head mounted display that looks like a normal glasses? (75% of the audience raise hands)
Dieter asks the panel members to conclude with one sentence each (no semi columns…)
Ron: I want to refer to the comment that the cell phone is too seductive. We should make it indispensable so users won’t want to give it up – just like a cell phone.
Mark: We need to make sure that children, grandparents, in Africa and everywhere – could use AR
Steve: You ain’t seen nothing yet; look at the progress we have made in the last 10 years! No one can predict what will happen.
Gudrun: We have to be visionary and on the other hand. We need to be realistic and make sure RA doesn’t end up like AI…don’t build hopes in areas where people shouldn’t have them…don’t let AR get burned…
Nassir: Next event we should include designers and experts from other disciplines; and create solutions that go beyond the fashion
Haruo: Maybe combining information like Googles with devices
Bruce: I want you to have fun and be passionate about what you do! We can change the world!
A beatiful 4 min concept film about how augmented reality will change the way we interact with information.
It’s a bit geeky in the file managment piece , and too emotional around the edges – but other than that – awe-inspiring. Nuff said, watch Stewart Morgan in action.
To see the man behind the images, here is an interview with Stewart Morgan.
In my last post, I proposed a countdown of the top 10 augmented reality demos that are poised to revolutionize video games. That collection focused on concepts that inspire a totally new kind of gameplay. Novice readers testified it was a great introduction to augmented reality, while AR savvy insiders found some newly discovered ideas quite inspiring. Thank you all for the great feedback.
Now, let’s talk hardware: how in the world are we going to play these games? Courtesy of “Window to The Future” By Steve Kosareff
AR game devices run the gamut from 20 lbs-backpack-and-head-mounted-display-systems to tiny handheld cell phones.
Once again, Marshall McLuhan’s legacy (“the medium is the message”) guides the discussion: The evolution of AR hardware devices changes the game mechanics and opens the door for a revolution in game experience.
Scholars have studied it at length (sidebar); I propose a simplified view spanning 3 generations:
The Past: Generation “Kit Bag” Tinmith epitomizes generation “kit bag”
* took place over the last 10+ years
* custom built backpack with laptop, accessories
* head mounted display
* used exclusively in research
* groundbreaking experience yet –
* heavy, complex, and expensive
* started in 2003 with a horizon of 2010
* mass production: banking on the ubiquity of mobile devices
* aspiring for larger screens with more powerful devices
* easy to carry, ergonomic, affordable, yet –
* occupies hands, limits immersion
The Future: Generation “No Bag” Leonard Low’s concept eyewear for augmented reality
* next 3-15 years
* eyewear: glasses and later contact lens
* handsfree; immersive
* in short: nirvana…(also means in Hindu – the end of suffering)
So, what’s the perfect device for mobile augmented reality games?
Experts argue that it depends on the type of game.
I buy that. Here’s the revised challenge: if game developers want to build a game like Roku’s Reward – what handheld device should they zero in on?
Looking at present and future generations, here is my countdown of the top 10 AR mobile devices with which developers will reinvent video games:
10. PDAs (Personal Digital Assistant) MIT’s Environmental Detectives PDAs knocked out the “kit bag” generation and signaled the dawn of the “hand bag” generation. They delivered mobility and extensibility (it’s easy to add a camera, GPS or other accessories) and it offered reasonable processing power.
However, with the convergence of devices such as cell phones, cameras, and the miniaturization of computers – PDAs are becoming obsolete.
Notable PDAs for AR include: iPAQ, Dell Axim, Fujitsu-Siemens, and Asus
9. UMPC (Ultra Mobile PC) Outdoor tracking with a UMPC at the University of Cambridge UMPCs are the most powerful handheld devices out there. Its Windows operating system makes it a familiar and practical platform for development.
Sony Vaio UX is used often by researchers; Raon Digital Everun is another candidate.
UMPCs would have been at a higher position on this list – had it not introduced 2 barriers for adoption as an AR game device: price (upwards of $1500) and size (needs a sizable bag). After all, the UMPC is designed for business – not games.
8. Smart Phones C-Lab’s KickReal runs on a Siemens phone
If you’re after the masses, smart phones are your best bet thanks to their ubiquity (out of 1.14 billion cell phones sold in 2007 – 10% were smart phones according to IDC) . In two or three years, it will be nearly impossible to buy a conventional cell phone (as agreed at the CTIA wireless show 2008.)
So, almost everyone has them, cameras are getting better, they’re adding accelerometers, and positioning capabilities and some games have proven to work on this type of devices.
Popular mobile phones (sorted by OS popularity according to canalys) include Symbian, Microsoft, RIM, iPhone, Palm (Acquired by Access), and Linux.
Google’s Android phone has ambitions to carve its niche, but has still a long way to go (too slow, not stable). It may emerge as a viable option in 2009.
One AR company suggests the HTC P6500 as “a good approach [for augmented reality games] with faster and easier development tools.”
But because of its small screen, low end processors, and major market fragmentation – smart phones cant leap beyond the 8th spot.
Though considered a smart phone – owing to its unmatched user experience – the iPhone deserves its own category.
Its form factor, touch screen, accelerometer, more modern technology, and now with an open SDK – it’s irresistible.
Although Apple is not known as a great gaming company there are currently 373 games listed under iPhone webapps.
Apple’s total control of the whole product compared with Gphone’s patchwork of multiple companies will result in a better experience.
6. Handheld Game Devices (PSP, Gizmondo, Nintendo DS) Gizmondo used for the Caleb project at Graz University
Handheld game devices are leading the growth in the game hardware market. Some people believe they will cut into the console market. Devices such as Sony’s PSP, Nintendo DS (and Gizmondo assuming it will indeed reemerge) have a great form factor – and the fact that they are designed specifically for games, gives them an edge over general purpose devices. But here’s the caveat:PSP and the DS need to be complemented with accessories such as camera, as well as accelerometers, positioning and ubiquitous connectivity capabilities – to be able to play in this game.
Haven’t we already covered smart phones at #8?. Well, Nokia isn’t just another smart phone company. Through extensive research, it pushes the envelop of augmented reality experiments on phones more than any other manufacturer.
Notable devices include N-81, N95, N-810.
The feature packed multimedia heavy N95 stands out as the most popular among researchers despite some limiting factors:
* runs on symbian – not great for heavy programming
* small 2.6” screen with 240 x 320 pixels only
* Expensive (above $500) and not really for mass market
4. MID (Mobile Internet Devices) Intel announced these MIDs to be released in 2008
Intel is putting its muscle behind the MID. The MID is designed for multimedia consumption on the go. It’s extremely portable with an ideal screen size (4-5”); it’s fully connected and armed with a strong (low power consumption) processor. The fact that it runs on a scaled down Linux makes it accessible and fast. But, what really separates it from the UMPC is its sub $500 target price.
Before year end ,Intel hardware partners are planning to release these products: LG Xnote, BenQ, Aigo, Lenovo MID.
Although experts have no real world experience with the MID – it’s shaping up as the preferred choice; if you plan to develop an augmented reality game this year – I would single out the MID as your #1 device.
When the flood of MIDs comes upon us, here is the criteria I suggest to separate the wheat from the chaff:
* screen size between 3.5″-5.5″ (4″-5″ ideal)
* min 65K colors, 800×480 pixels
* strong processor – min 400MHz
* with CGI acceleration (e.g. NVidia APX2500, TI OMAP 3)
* ubiquitous connectivity with cell (3G) and wifi/wimax
* high quality video camera: resolution (640×480), 30 fps
* positioning capabilities (GPS or software based locationing)
* inertial sensors, accelerometers, digital compass
* blue tooth for adding essential accessories
* touch screen – nice too have
* price – below $500; $200 would be ideal for the masses
Now let’s delve into the future; here devices become more conceptual, which means – not proven, yet stimulating. Prepare to be audaciously hopeful.
3. Looking glass (design concept)
In its designer Mac Funamizu’s own words:
“This is what I wish the Internet search will be able to do with a mobile device in the NEAR future. Touch screen, built in camera, scanner, WiFi, Google map (hopefully Google earth), Google search, image search… all in one device. Like this way, when you can see a building through it, it gives you the image search result right on the spot.”
Will anyone ever build it? I don’t know – but it sure looks cool.
2. Glasses Mirage Innovations is one of the contenders for the future generation
Nokia dubbed mobile phones as the 4th screen (following cinema, TV, PC). According to this count, glasses will be the 5th screen. It spearheads the next generation of augmented reality: the “no bag generation”. This is when we become liberated. Handsfree. The interactive overlays on top of reality surround us anywhere we look. Can we start developing games for glasses? not quite yet. They still lack built in cameras, eye tracking, and tend to cause dizziness. Should we start thinking about it? Absolutely. Rolf Hainich has dedicated a book to this concept and aptly called it the end of hardware. Ben Averch writes about this market in his future-looking blog.
Companies to watch:
Microvision is working on lightweight eyewear for augmented reality
Lumus Optical has just launched their new and slick eyewear
Here is the criteria I recommend to use to evaluate the progress of contending eyewear companies:
* looks and weighs like regular sun glasses
* see thru, high resolution display
* integrated camera
* eye tracking
* position and movement sensors
* integrated sound
1. Contact Lens Darpa project for the creation of micro- and nano-scale display technologies
Apart from topping the list of “things you wouldn’t borrow from a friend”, contact lens leap to the top spot of AR devices. Normally, the mere thought would be scientifically dismissed as “too far off”, but this Darpa project makes it slightly more conceivable. In Vernor Vinge’s Rainbows End near future novel – everybody (except retreads) wears them. It becomes a natural extension of human intelligence, with user interface that can be summed as squint and gesture. It gives itself away only due to the occasional blank stare which is typically confronted with: “are you googling again!?”.
The story is taking place in 2025. Not too far off.
Back to the present: Scientists at the University of Washington have already created a proof of concept – see video below.
Didn’t make the list * Laptops – laptops were used extensively during the “kit bag” generation as a key ingredient of the back pack, but during the hand bag generation – no one wants to the roam the streets tethered to a laptop. First implementation of Plundr ran on a laptop – but gamers soon clamored for the Nintendo DS version.
* Zune – should it be?…
* Brainchips – beyond the horizon of contact lens lies the world of brainchips . I am not touching this one with a 12 foot pole…I consider myself a technology enthusiast, but this makes me feel uneasy. I want to always have the choice to de-augment my reality when I’ve had enough. Hard connections into my brain may deprive me from that right.
Here is my countdown of the top 10 best AR demos poised to revolutionize video games:
10. Human Pac Man
When Dr. Adrian David Cheok (NUS) wanted to create an exciting augmented reality game, he chose to remake the first video game to ever introduce a character – the legendary Pac Man. Cheok literally stepped into Pacman’s shoes in this first-person-shooter-like real world game.
9. Come Out & Play
In 2006 the Come Out & Play Festival turned New York City into a playground for a weekend, then did the same for the city of Amsterdam in 2007. Hundreds of players gathered to play dozens different Big Games across each city.
None of the games played in the festival made it to my top ten list – individually. They fell short of stretching the interactivity between real and virtual. But as a group – they reminded us that playing games outdoors can be fun, and technology can make it even funner.
8. Second Life Avatar Enters the Real World
Tobias Lang and Blair MacIntyre (GA Tech) give us an extraordinary glance at what happens when the virtual world “leaks” into the real world. No, they didn’t use a green screen as you can see behind the scenes. Is it a hint for some of us to ditch Second Life and augment our First…?
7. WIFI ARMY FPS
The world is the battlefield, your phone is your weapon. Players organize in 2 teams armed with cell phones with the goal to locate and take pictures of their opponents. The phone compares captured pictures against a database of player faces and awards points for correct hits. Peter Whatanitch of W2Pi (creator of the game) explains how it will work. It could become quite an experience as staged in the Lumix Battle commercial.
6. Shadow Monsters
Phil Worthington is an artist that injects coolness into augmented reality.
“Magical monsters appear from shadows cast by the hands of participants, reacting to gestures with sound and animation. Wolf like creatures, birds and a rastafarian are among the characters that speak and squeek as imaginary mouths open and close. Shadow Monsters is an intuitive and magical experience for young and old alike to play with body posture for creating crazy narratives.”
5. AR Grafitti by DAIM
Artist DAIM creates virtual 3D graffiti floating in space in his latest art project “Tagged in Motion” of NextWall .
What if everyone could overlay reality with their own virtual creation?
4. CARCADE: In-Car Video Game
Andreas Nicolas Fischer and team suggest to take advantage of the fast changing scenery experienced by car passengers – and turn it into a video game (remember Gondry’s Star Guitar Video Clip?). Could be a great way to pass time. I’d call it: “Are we there yet..?”
3. Total Immersion at Demo ’07
Probably the most commercially successful company that specializes in augmented reality, Total Immersion delivers the best live demos in this domain – with a French accent. They have also featured in CES 2008 keynote by Intel’s CEO Ottelini.
2. Roku’s Reward
This concept* video shows the potential of augmented reality utilizing today’s technology:
A handheld camera device,
live video overlaid with 3D graphics,
computer recognition – identifying real life objects,
positioning and acceleration sensors,
virtual objects interacting with reality,
…wrapped up in a fantasy narrative make up this yet-to-be-developed game.
*According to Phil Stenton (HP Labs, UK): “Roku remains a demo film. We haven’t signed up with anyone to create the game yet.”
Though delivered in words only, this “demo” paints the farthest vision for augmented reality so far. Everything else you’ll see on this topic will look like excuses: “sorry, we don’t have that technology yet…”
Yes, but do you have the imagination?
“There was something familiar about this prey. It was young and clever looking … a newborn from Juan’s own design! And that meant its Mommy would be nearby. Juan said, “You know, I don’t think –”
“The Problem Is, None Of You Think Nearly Enough.” The sound was premium external, like sticking your head inside an old-time boom box. Too late, they saw that the tree trunks behind them grew from yard-long claws. Mommy. Drool fell in ten-inch blobs from high above.
This was Juan’s design scaled up to the max.”
* You may ask how come I left out another classic remake: AR Quake. Well, this remake is indeed played outdoors, but let me ask you this: does it add much to the original experience of that first person shooter?
* How come I am ignoring what some call “the future of books” such as EyeMagic Book or WizQubes? Is it really that different than manipulating 3D characters on the computer screen?
* Daniel Wagner, a major force behind recent AR innovation, may be disappointed that his pioneering handheld AR game the Invisible Train didn’t make it. No hurt feelings; it’s a great proof of concept, but isn’t it more fun to play with the *visible* train?
* Other educational oriented games such as MIT’s Environmental Detective or U-WIS’s MadCity Mystery certainly beat learning in a classroom – but aren’t they indistinguishable from traditional low tech scavenger hunts?
Games Alfresco 