Here is a crazy thought – is the industry mature enough for an AR benchmark? Specifically I thought such a benchmark should cover the following two topics:
- Comparing the performance of AR frameworks (such as Qualcomm’s AR SDK, Samsung SARI, etc.) using a given set of tests.
- Comparing the performance of devices using a given set of AR tests.
Which leads to the following questions, I’ll be glad hearing your thoughts about them (in the comments, on twitter, or via email, whatever!):
- Will such a benchmark be beneficial to any one? Interesting?
- What tests should the benchmark include? The main problem here is that test should be easily recreated and yet meaningful. We can test “tracking success” by tracking an on screen marker that is transformed by code (to account for change in lights, motion blur, noise and other effects, which can be hard to accurately recreate using a real physical marker).
- Would any one of you volunteer to run those tests? I don’t have access to lots of AR capable devices, so unless you would like to sponsor me (i.e. buy me an iPad2), I count on the volunteers to run the actual tests and report back.
- How about benchmarking AR applications? Does it make sense?
Filed under: Uncategorized |
Games Alfresco 
Keep me posted on how you progress… we even might be able to help out.
I was recently thinking along the same lines. It does seem there are enough options out in the marketplace that it would warrant a bake-off comparison. We’ve used a good cross section of what’s available (metaio, Qualcomm, ARToolworks, imagination, and String) in various projects and prototypes, and would be willing to spend some amount of time helping with the effort.
Such a benchmark would be great, but it would be very hard (if possible at all) to create an objective evaluation.
There aren’t that many commercial AR SDKs (especially when focusing on mobile devices as you seem to) and they vary heavily in the features and the platforms they support.
For commercial SDKs it would also be interesting to look at features that make it easy to develop apps (rather than just tracking robustness): Number of platforms/devices supported, developer tools supported, content creation pipeline, developer documentation, developer support, etc.
BTW: What is Samsung SARI? I haven’t heard that one yet.
Benchmarking specific area’s makes sense, but its so broad as to be hard to benchmark the lot.
It would be nice, however, to know what phones have the most accurate compass’s, or GPS. (Id LOVE that information)
I think focusing on specific areas should be possible.
Won’t be buying anyone an iPad2 though, not after Apples patent spamming of AR and transparent screens used for AR. For someone relatively late to the game they are a bit cheaky to claim ownership.
suggestions on how to measure GPS and compass accuracy?