Message Boards Message Boards

[WSS18] Augmented & Virtual Reality implementation in Wolfram Language

Augmented and Virtual reality functions in Mathematica

Overture

The goal of my project at Wolfram Summer School '18 was to implement VR and AR visualization features in the Wolfram Language. Conceptually, the underlying idea was to create two simple functions through which deploy in a VR or AR environment any 3D object created from Mathematica, spanning from a random solid (e.g. created using the ConvexHullMesh function):

ConvexHullMesh[RandomReal[5, {50, 3}]]

to a detailed view of 3D elevation data for a specific region:

ListPlot3D[GeoElevationData[GeoDisk[Entity["City", {"Tivoli", "Lazio", "Italy"}], 4 Quantity[1, "Kilometers"]], GeoZoomLevel -> 12]]

How

The two functions ARDeploy and VRDeploy work in conjunction with the WebVR BoilerPlate platform and AR.js library, which provide the required JavaScript tools to manage the VR/AR environments. The objects representation is translated from Mathematica into a js compatible format by Three.js and Aframe.js libraries. In particular, VRDeploy enables to investigate abstract mathematical (and any 3D Mathematica models) representations in a new environment leading to a clearer and easier understanding of them. In combination with mobile devices, equipped with a headset, enables the user full immersion experience. ARDeploy, on the other side, enables the "on-screen" exploration of the same objects compatible in VRDeploy, attaching it to a marker which could be visualized in the Mathematica notebook. The great difference between the two functions relies on the way the 3D object is explored. The VRDeploy offers a static full immersive point of view while ARDeploy offers a dynamic exploration.

These two functions (in form of Mathematica package) and all the required files can be downloaded at https://github.com/DomenicoRomano/ARVRDeploy.

VRDeploy Example Code

VRDeploy (as ARDeploy) works with any 3D surfaces from Plot3D or Example Data. In the example code below the 3D surface from Plot3D is translated in a VR object, VRDeploy returns a link to a webpage in which there is the WebVR environment with the Plot3D object visualised:

VRDeploy[Plot3D[0.5 *Sin[0.4*x y], {x, -4, 4}, {y, -4, 4}, MaxRecursion -> 4]]

Below a screenshot view of the webpage returned by VRDeploy as seen from mobile devices, this provides the full immersion experience. It is also possible to switch to normal view and then navigate the VR scene using the mobile device orientation hardware.

Mobile device view of the full immersive experience

Below a view of the same webpage as above returned by VRDeploy as seen from a laptop/desktop computer:

Web page visualizing the output of the above VRDeploy line of code

Another example is the VR visualization of the stars distribution around the Earth, using the StarData function. The following code extracts the position of the 35000 closest stars to our Sun, then converts the distances to parsec and VRDeploy them:

VRDeploy[Graphics3D[
  Sphere[QuantityMagnitude[
    DeleteMissing[StarData[EntityClass["Star", 
     List[Rule[EntityProperty["Star", "DistanceFromEarth"], 
       TakeSmallest[35000]]]], "HelioCoordinates"]]/3.26],
   .05], {
  Axes -> {True, True, True}, 
   AxesLabel -> {"pc", "pc", "pc"}, ImageSize -> Large}]]

Resulting in the picture below:

A full immersive view of the closest 35000 stars as seen from Earth, created with VRDeploy and the StarData function.


ARDeploy example

A simple example of ARDeploy function presented here use the Viking Lander model from the 3D Example data of Wolfram Mathematica, following is the code:

ARDeploy[ExampleData[{"Geometry3D", "VikingLander"}]]

The returned CloudObject is a link to a webpage, which can be easily converted to a QR code and scanned by mobile devices.

BarcodeImage[
 First[ARDeploy[ExampleData[{"Geometry3D", "VikingLander"}]]], "QR"]

The webpage opened in the mobile device will ask "to allow the application to access the camera". After granting the access to the camera the mobile device will recognize the marker, plotting on it the ARDeployed object:

The Viking Lander 3D model, taken from Mathematica Example data, visualized attached to a marker on a laptop screen.

Future Directions

Next steps require to enrich the visualization experience with interactive capabilities, displaying specific information or changing the object properties. Another required step is the marker independent placement of the object, using absolute GPS coordinates. The required mobile hardware improvements will be soon available. It will be also interesting to offer a virtual reality Mathematica notebook in which the user could experience the advantages to learn in a full-immersive three-dimensional virtual world.

POSTED BY: Romano Domenico
6 Replies

Awesome, i would like to see this kind of projects in videos, i would be great have all this ideas in youtube

POSTED BY: Oscar Rodriguez

These examples would be perfect if only I could get them to work! I wish there were more detailed instructions or images of how to get a Graphics3D object to be viewed on an Oculus Go for example. I tried uploading such a graphic to the Wolfram Cloud but no success! No 3D images... only a grid of lines that extended to the horizon.

POSTED BY: Donald Barnhart

Hi Donald, that's strange, I'm still using it and it works. Could I ask few questions?

  • Which browser did you use?
  • Did you get any errors somewhere in the pipeline?
  • Which kind of graphics did you use?
  • Which version of VRDeploy was used? the classic branch in github (which has OBJ/mtl conversion) or the master branch (which has .wrl file export)?

Thanks for your feedback!

POSTED BY: Romano Domenico

enter image description here - Congratulations! This post is now a Staff Pick as distinguished by a badge on your profile! Thank you, keep it coming!

POSTED BY: Moderation Team

Very nice topic, great work!

POSTED BY: Viviana Letizia

This has great potential for education and entertainment at home and on the go. I hope to see more tech progress and applications in future!

POSTED BY: Larry Temlock
Reply to this discussion
Community posts can be styled and formatted using the Markdown syntax.
Reply Preview
Attachments
Remove
or Discard

Group Abstract Group Abstract