Pencil and Paper are not Dead: Augmented Reality Sketching Games at VR 2010

Tomorrow, I’ll be at the IEEE VR 2010 conference in Boston. Monday is dedicated to a series of augmented reality presentations.

One of the most interesting one is:

In-Place Sketching for Content Authoring in Augmented Reality Games

By the all star team from Ben Gurion University (Israel) and HIT Lab (New Zealand):

  • Nate Hagbi
  • Raphaël Grasset
  • Oriel Bergig
  • Mark Billinghurst
  • Jihad El-Sana

When it comes to AR games – we are all still searching for “Pong” a simple game that will captivate millions of players and kickoff this new genre.

One of the challenges in many AR games, is the reliance on printouts of ugly markers.

Plus many games use the markers as controllers which is a bit awkward (especially to a bystander).

Sketching offers an alternative for a more natural user interface.

Sketching is more natural than drawing with a mouse on a PC, even more intuitive that a touch screen. That’s still the first thing that kids are taught in school.

It’s not necessarily a better interface – but it’s an alternative that offers a very intuitive interaction, and enriched the player’s experience. I believe it could create a whole new genre of games.

In place sketching in AR games has huge potential in gaming – but many questions arise:

  • What’s the design space for such a game?
  • What are the tools to be used?
  • How do you understand what the player meant in a sketch?
  • What’s the flow of interaction?
  • How do you track it?

What’s “In-place AR”?  It’s when the augmented content is extracted from the real world (an illustration, an image , a sketch, or a real life object)

Here is the sequence of research efforts leading to this:

Here are 2 game prototypes the team created called AR Gardener and Sketch-Chaser. It is played on a regular white board.

AR Gardener

Draw symbols on the white board and 3D content is pulled from a database of objects to appeas in an Augmented Reality (AR) scene.

The sketch determines what object to create, its location, scale, and rotation.

The outer line sketched here defines the game anchor and is served for tracking; in this game it becomes a brown ridge.

Simple symbols drawn generate a couple of benches, a cabin, and in the spirit of the playground theme – rockers, and swings.

Virtual elements could also be created based on a real life object such as a leaf; here it is used to create a patch of grass using the color and shape of the leaf (and no, it can’t recognize that’s a leaf, or 3D object whatsoever)

The color of the marker could define the type of virtual object created: For example, blue represents water. Other objects that are put in it will sink.


In the second game you basically create an obstacle course for a car chase.

It’s a “catch the flag” or tag game. The winner is whoever has the flag for the most time.

First you draw, then play.

Once again, the continuous brown line represents a ridge and bounds the game.

A small circle with a dot in it represents the starting point for the cars.

A flag becomes the flag to capture. A simple square creates a building, etc.

The player adds more ridges to make it more challenging. Adds blue to generate a little pond  (which also indicates a different physical trait to this area)

Then – graphics are generated, the players grab their beloved controllers and the battle begins!

This research represents an opportunity for a whole new kind of game experience that could make kids play more in the real world.

Many questions still remain, such as how do you recognize in a sketch what the player really means without requiring her to be an artist or an architect. Or where does the sketch fit in the game play? Before, after or during?

Now, it’s up to game designers to figure out what sketching techniques work best, what’s fun, what’s interesting, and what’s just a doodle.

Who want’s to design an sketch-based augmented reality a game?

2008 Wrap Up: Top 10 Milestones in Augmented Reality

2008 was a great year for augmented reality.

This emerging technology is on a 15-plus-year-long journey from the lab and into the mainstream. With too many events to list, 2008 marks an important year in that quest.

Here is the countdown of the top 10 most important AR milestones of 2008:

10) Otellini’s CES keynote showcases AR technology

The year started with the largest consumer electronics show CES. Total Immersion’s demo had the attention of the entire consumer electronics community during Otellini’s (Intel CEO) 2008 keynote.

9) Video game gurus recognize AR as the future of gaming

My personal idol in the game industry, Will Wright, delivered the best augmented reality quote of the year. When describing AR as the way of the future for games, he explained:

Games could increase our awareness of our immediate environment, rather than distract us from it”.

Futurist Bruce Sterling made controversial statements about games in 2043 in front of a developer only audience in the Austin Game Developer Conference . Here is one statement that stood above all:

“What do the games of 2043 look like? “I think you would call [them]  ‘augmented reality’”

8) ISMAR 2008

The world’s most important augmented reality event, ISMAR 2008, was more significant than ever. It  demonstrated, above all, the level of maturity AR research has reached.

7) High end mobile AR devices hit the market

2008 gave rise to a flood of new AR-worthy mobile devices: iPhone 3G (see below), Android based G1, Itelco’s IDOL, HTC’s Touch HD, Blackberry’s Storm, Nokia N97… It also signaled the dawn of hands free AR with prototype glasses and contact lens (see Top 10 devices.)


6) iPhone and App store

It might not be the best AR device but it certainly has the most buzz. In 2008, the iPhone was highly sought after by game developers and researchers. The App store which amassed 10,000 apps in half a year, offers an instant distribution model for AR games. In 2008, the iPhone was the gadget to beat.

5) Native mobile tracking engines released

AR engines and tools did not stay behind in 2008 and delivered new tracking engines for the hottest platforms du jour. Some notable examples include ARToolkit for iPhone, AR in Flash, and Studierstube ES for mobile phones. See the entire collection of engines and tools.

4) AR apps win acclaim in major competitions

AR made headlines in 2008 with 2 applications that dribbled into top 50 lists: Tonchidot made the Crunch50 and Wikitude made the Android 50 finalists. Total Immersion won another award at NVISION ’08 for Best Application of Visual Computing.


3) AR games win awards

Ghostwire may or may not be the first AR game to win a game award. But, 2008 was certainly the first year were 6 out 0f top 10 games were selected as finalists in game awards such as Nokia’s Mobile Innovation Challenge.

One AR game did not win any awards this year, but was certainly an audience favorite (the most hits): Cyber Figure Alice – the first adult only AR game.

2) Major investments in AR research

Media Power made significant investments in augmented reality research organizations around the world. If the $5M for GA Tech GVU donnation and $2.7M for NZ HIT Lab donnation don’t speak for itself, the results are already showing:

1) Record commercial deals

The AR market picked up steam in 2008 with high visibility deals such as Total Immersion’s with Six Flags (“magically superimpose clown masks on riders” as their waiting for the Dark Night ride) and Metaio (book deals with publishers ArsEdition and Knowledge Media). Metaio also scored a lucrative advertising contract to promote the MINI.

What, in your opinion, was the most significant augmented reality milestone in 2008?

Whatever you do, don’t miss “If the Augmented Reality Industry Got a Report Card”

Subscribe (top right) to get the complete 2008 wrap up series.