Message Boards Message Boards


Augmented Reality

Posted 9 years ago
5 Replies
16 Total Likes
So I have been programming augmented reality apps in Mathematica and I am now trying to find neat applications for augmented reality.  If you have any ideas for things that you think would be a cool augmented reality app, then post them here.

5 Replies
Posted 9 years ago
Hi Chris,

This is a wide-open area where a lot of things are possible. One way to think about this is categorically.  Some of the easier channels along which you can gather input: images, coordinates, and sound, as well as messages and records left electronically in some kind of geotagging scheme.  In the environment, I can think of three general subjects of interest:  people, places, and things. 

Visual identification of people  who does this person look like?  Have I seen them before?  What did we last talk about?
Visual/coordinate-based identification of places  have I been here before?  What were my last experiences here?  Is this generally a place where good things happen? 
Visual identification of things  what are these objects?  How much do they cost?  Where did they come from?  Some of this doesn't have to be too sophisticated:  I can imagine an application that counts the number of eyeglasses people are wearing in a room and gives a rough estimate of their combined value.
Combinations or person and place  where am I likely to run across this person?

Sound identification of people  what are they saying?  (Automatic closed captioning)
Sound identification of places  is this library or this coffee shop actually a quieter place to study?
Sound identification of things  what was that noise? 

The other question is what happens when you can allow people to leave behind information you then retrieve later. Location-linked information is one scheme for this based on common technologies.

I think if I was to pick any of these, I might pick something that combines sound-level analysis with left messages.  When is a place a good place to chat, versus study, versus hear music?

(Added Feb. 4:  If you have access to a temperature sensor, you might consider help making a thermal map of a house;  combined with heating bills and some domain knowledge about the cost of home improvements, you might be able to do a cost/benefit analysis of making various changes in windows and insulation.  There is an example of home thermal modeling for SystemModeler that would be interesting to combine with real data.  The advantage to such an application that even the most basic heat map would be interesting, but you could iterate with increasing sophistication.)   

All the best and have fun,
1.  Law enforcement app.  Scan of license plates surrounding a police officer.  Database interrogated and the officer learns whether any vehicles are owned by people with outstnading warrants or criminal records.  Might enhance officer safety.  Obviously there are privacy issues and a potential for misuse.

2. Party/Conference/Business Meeting/Dating app.  People wear QR codes on their clothes.  OK, it is a bit dorky but little more so than name tags.  You look at the QR codes and your glasses display a lot of quick information about the person.   

Good luck with the project.  I did my best with a few spare moments but I have some confidence you will come up with better ideas than these.
This is a very interesting topic that I often ponder about. I gave code for a simple fun application in this post: Masking faces in a webcam stream.  A general application areas are listed in the linked Wikipedia article:
StringSplit[Import["", "Data"][[1, 1, 2, 2]], " "][[All, 2]]

Out[]= {"Archaeology", "Architecture", "Art", "Commerce", "Education", "Everyday", "Industrial", "Medical", "Military",
"Navigation", "Office", "Sports", "Task", "Tourism", "Translation"}
As I come up with some possible cases I add them to the list below. I am thinking about something we can probably quickly tackle with Mathematica.
 - Recognizing sign language and decorating live images of hands with corresponding text
 - Recognizing manufactured parts (like color code of electric resistors) and decorating live images with technical values
 - Annotating moving objects with their speed and distance values – maybe good with Kinect type devices
 - Annotating pets in live video with their animal class – can come handy for future home robotics  Imagine Google glasses or something similar that in real time lets you see augmented motion.

There is a FABULOUS augmented reality app lurking here.  And the MIT source code is apparently freely available, although it is written in Matlab, and there is apparently some patent app pending.  Boo!  Check out
Here is a somewhat exotic suggestion, different than Google glasses, which I admit might not fit your program.

Basically, science is augmented reality. Every time Mathematica is used to produce some model of reality, it contributes to augmented reality. So what could be done ?

Schematically, science begins with observations or experiments, then continues with assumptions and reasonings that lead to models or theories. That is where Mathematica commonly intervenes by computing representations of reality that scientists compare with reality with the purpose to adjust and improve their models or theories. Then, when the models are relevant, they are used to steer the design or creation of technical objects or possibly to drive instruments or processes.

From this unusual viewpoint about augmented reality, Mathematica could be involved in the experimental process by some capacities to connect to scientific instruments (or the files they generate) and directly get the data. It could also be involved in the process of producing models or theories, typically by including tools like system modeler. Then, the models can be superposed on the data, either to test whether they fit them or to merge the data with the models, which constitutes augmented reality.

There is a symmetric application in which Mathematica might be involved too, when the computer is used to reconstitute a more or less realistic simulation of reality from some model, which is commonly called virtual reality. Mathematica might be more involved in this domain thanks to, beyond its current solvers, future finite element solvers or other kinds of solvers, a discrete-time version of Animate in view of stepwise simulations (a stream approach, as opposed to ListAnimate) and possibly with sorts of high level dynamic graphic primitives able to implement scenarios.

I believe there are three complementary domains in which Mathematica might also contribute to augmented reality.

The first one regards the modeling process, which involves not only computations but also the writing, reading and analysis of scientific publications. A future version of Mathematica could manage warehouses of "hyperlinked" publications and data, probably with the help of CDF and WolframAlpha plus a Google approach to the Mathematica notebooks put on the web; a sort of Mathematica web, better structured than the current awkward html-xml-php-javascript-sql web.

The second one regards the Mathematica language which is transformational, while the scientific language is more relational, so a logical extension of Mathematica would be welcome. Actually, Roman Meader developed such a logical extension on top of Mathematica, a sort of little Prolog designed in the frame of the Mathematica syntax. Unfortunately, this package no longer works (since version 6); an updated version would be welcome. Besides, I believe there could be interesting couplings between logical programming, graph algorithms and database design, but this is probably a topic for another discussion thread…

The third one regards the design of objects or the driving of processes, for which an extended version of Controller* might enable these sorts of operations directly from within Mathematica. 3D printing is a kind of first step in this direction but the link should be thought not only with devices but also with specialized pieces of software.

In all cases, there should be a theory of that, I mean a formulation able to describe such features and above all to discuss and reason about them.
Reply to this discussion
Community posts can be styled and formatted using the Markdown syntax.
Reply Preview
or Discard

Group Abstract Group Abstract