Message Boards Message Boards

How to make 360 degree videos

GROUPS:

Several years ago, all of a sudden 3D videos became popular. Just last months (year?) 360 degree videos became popular. Facebook and Youtube both have support for 360 degree videos. So it was of course only a matter of time I wanted to try this out. In addition, I will show you how to do it. In order to create such a video, one needs videos that are 'flattened out', which basically means that instead of x and y coordinates we will have θ and φ spherical coordinates (or equivalently longitude and latitude). Here is a snapshot of my trial video; you're inside a torus!

enter image description here

Note the amazing distortion at the top (the top line of pixels is actually a point!) This is caused by 'unwrapping' a sphere, and this is what causes cartographers nightmares, one can not un-wrap a sphere without stretching part of the surface! This is also why Greenland has a similar size to Africa (on Google maps), but in reality it is much smaller. Ok enough about cartography, How did I create such an image?

The basic idea is that we have a scene (a torus in this case) and we put our camera at a certain point and take snapshots in the different directions. So Imagine we are inside some polyhedron and look at each of the faces of it and take a snapshot. Depending on how accurate you want to do this you can choose to take 4 very wide snapshots in a tetrahedral configuration, or 6 in a cubic arrangement, 12 in a dodecahedronal arrangement, or 20 in an icosahedronal configuration. I opted for the last option:

enter image description here

ClearAll[RefineSphere]
RefineSphere[{fi_, vc_}] := Module[{vcc, max = Max[fi], nfi, avgs, midpoints, newtriangles, newvc},
  vcc = DeleteDuplicates[Sort /@ Flatten[Subsets[#, {2}] & /@ fi, 1]];
  nfi = MapThread[Rule, {vcc, max + Range[Length[vcc]]}];
  avgs = Mean[vc[[#]]] & /@ vcc;
  newvc = Normalize /@ (vc~Join~avgs);
  midpoints = Partition[#, 2, 1, 1] & /@ fi;
  newtriangles = MapThread[Append[Flatten[#, 1] & /@ ({#2, Partition[Sort /@ RotateRight[#1], 2, 1, 1]}\[Transpose]), Sort /@ #1] &, {midpoints, fi}];
  newtriangles = Flatten[Replace[newtriangles, nfi, {3}], 1];
  {newtriangles, newvc}
  ]
refine = 0 (* integer <= 5.... 20*4^refine triangles *)
sphereFI = PolyhedronData["Icosahedron", "FaceIndices"];
sphereVC = N@PolyhedronData["Icosahedron", "VertexCoordinates"];
{sphereFI, sphereVC} = Nest[RefineSphere, {sphereFI, sphereVC}, refine];
sphereVC *= 40;
Graphics3D[GraphicsComplex[sphereVC, Polygon[sphereFI]], Lighting -> "Neutral"]

To start, I just used a normal icosahedron, without refining (20 faces):

enter image description here

Now imagine we put the camera at the center of this shape and we look at the centers of each of the triangles. We have to choose an appropriate viewing angle such as to fully see the triangle. After a bit of playing around I found out that a good viewing angle is 80 degrees. Now with that viewing angle we see a certain square-section in 3D, which is determined by two vectors, one going up, and one perpendicular to the vector going up and the viewing-vector:

size = 600;  (* each view will be rendered this size squared *)
vangle = 80 \[Degree]; (* view angle is 80\[Degree] *)

(* calculate the various viewing angles *)

viewvectorup = viewvectors = Normalize[Mean[Part[sphereVC, #]]] & /@ sphereFI;
crosslen = (Norm /@ viewvectors) Tan[vangle/2];
viewvectorup = Normalize[{0, 0, 1} - ({0, 0, 1}.Normalize[#]) Normalize[#]] & /@ viewvectorup;
viewvectorup *= crosslen;
viewvectorright = MapThread[Normalize@*Cross, {viewvectors, viewvectorup}];
viewvectorright *= crosslen;

(* make black-white masks for each view, note that we use 0.96 viewing angle here such that each of the views overlaps a bit *)
masks = MapThread[Rasterize[Graphics3D[{EdgeForm[], White, 
       GraphicsComplex[sphereVC, Polygon[#1]]}, Boxed -> False, 
      Lighting -> "Neutral", ViewVertical -> {0, 0, 1}, 
      ViewVector -> {{0, 0, 0}, #2}, ViewAngle -> (0.96 vangle), 
      ImageSize -> {size, size}, Background -> None], "Image", 
     Background -> None] &, {sphereFI, viewvectors}];

(* this is what all the viewingvectors look like (blue), green is the pointing-up vector, and in red is the vector to the right *)
gr = 
 Graphics3D[MapThread[{Blue, Arrow[Tube[{{0, 0, 0}, #1}]], Green, 
     Arrow[Tube[{#1, #1 + #2}]], Red, 
     Arrow[Tube[{#1, #1 + #3}]]} &, {viewvectors, viewvectorup, viewvectorright}]
]

enter image description here

The red vectors are pointing to the right (from the camera perspective), green pointing up, and blue is the direction of view. As you notice, this is getting quite complicated already... I also calculate the masks that we need later. I will render each of the face, and need the masks for the triangles. I purposely made the masks slightly bigger such that the image of each face overlaps a tiny bit with the adjacent faces.

The masks look like this:

RemoveAlphaChannel /@ masks

enter image description here

Now that we have the masks, we need a scene: Here is what I made:

ClearAll[MakeScene]
MakeScene[t_]:=Module[{rot,p1,p2},
    rot=2\[Pi] t;
    p1=ParametricPlot3D[{-8.2+(8+Cos[v]) Sin[u+t],(8+Cos[v]) Cos[u+t],Sin[v]+0.75},{u,0,2Pi},{v,0,2Pi},ViewVector->{{0,0,0},{0,1,0}},ViewAngle->80\[Degree],PlotStyle->Directive[Green,Opacity[1]],MeshShading->{{Red,Blue},{Blue,Red}},MeshFunctions->{#4&,#5&},PlotPoints->80,Axes->False,Mesh->{51,10},Lighting->{{"Ambient",White}},ViewVertical->{0, 0, 1}];
    p2=Graphics3D[{Orange,Sphere[{-8.2-8Sin[2rot],8Cos[2rot],0.45},0.2]},Lighting->{{"Ambient",White}}];
    Show[{p1,p2}]
]

A simple torus that rotates around as a function of time, it looks like something like this:

enter image description here

Now for each of the viewing vectors, we create an image (this is inside the torus!):

views = MapThread[Rasterize[Show[MakeScene[0.0], ViewVector -> {{0, 0, 0}, #1}, 
 ViewVertical -> {0, 0, 1}, ViewAngle -> vangle, Boxed -> False, 
 ImageSize -> {size, size}, Background -> White], 
"Image"] &, {viewvectors}]

enter image description here

Now we masks each of the views with its corresponding masks to get:

antetransform = MapThread[ImageMultiply, {masks, views}];

enter image description here

Now we have to transform each of the images to θ - φ space, or equally longitude-latitude. There are now 2 ways to do this: One: for each pixel in the above images I calculate the θ and φ coordinates, and then color in that pixel in θ-φ space. Or the other way around: For each point in θ-φ space i calculate the x-y coordinates of the pixel in the images. Once I transformed all the images to θ-φ space I can stack them all up to get my final image. While both can be done, it is in this case easier to go from θ-φ space to x-y coordinates in the images. This has to with interpolation when you go the other way; it is confused by the spherical coordinates which are periodic in one direction and data for -179 and 179 degrees will interpolate in between -179 to 179 degrees, rather than wrapping around. (if you don't understand what I'm saying, don't worry, we'll use the other method).

For speed reasons I calculate some normalized vectors and the length of vectors before:

vvv = {viewvectors, viewvectorright, viewvectorup}\[Transpose];
normvvv = Map[Normalize, vvv, {2}];
nvvv = Map[Norm, vvv, {2}];
svv = vvv/nvvv^2;
ClearAll[invtransfunc];
invtransfunc[{\[Phi]_, \[Theta]_}, n_] := Module[{vp},
   vp = {Cos[\[Phi]] Sin[\[Theta]], Sin[\[Theta]] Sin[\[Phi]], 
     Cos[\[Theta]]};
   If[vp.vvv[[n, 1]] <= 0,
    vp = (vp nvvv[[n, 1]]/(vp.normvvv[[n, 1]]));
    {vp.svv[[n, 2]], vp.svv[[n, 3]]}
    ,
    {-2, -2}
    ]
   ];

This function converts θ-φ coordinates for the nth view vector to x-y coordinates of that image of that view. The if statement is necessary to make sure that if a θ-φ pair is requested that is opposite to the camera-view (i.e. behind the camera) that it will not project these angles wrongly. I give it the {-2,2} coordinates in that case, which corresponds to a point outside the image, so it will be transparent.

So we perform now this transformation for each image:

posttransform=Table[ImageTransformation[antetransform[[n]],invtransfunc[#,n]&,DataRange->{{-1,1},{-1,1}},PlotRange->{{-\[Pi],\[Pi]},{0,\[Pi]}},Padding->Transparent],{n,Length[antetransform]}];

There is lot going on in that simple line, and it take a while to compute:

RemoveAlphaChannel /@ posttransform

enter image description here

Once we have these images, we can combine them using ImageCompose (and some fiddling around with the masks):

\[Alpha]cs = AlphaChannel /@ posttransform;
posttransform = RemoveAlphaChannel /@ posttransform;
\[Alpha]cs = Binarize[#, 0.95] & /@ \[Alpha]cs;
posttransform = MapThread[SetAlphaChannel, {posttransform, \[Alpha]cs}];
imgout = ImageCompose[First[posttransform], Rest[posttransform]];
imgout = RemoveAlphaChannel[imgout];

To finally get this:

enter image description here

Now if we do that for a bunch of frames and create a movie of this we get this flat video. If we no post this again but with some metadata injected, we can now go through a tunnel and look around in this 360 degree video on YouTube!

POSTED BY: Sander Huisman
Answer
1 year ago

All the code:

Viewing polyhedron

SetDirectory[NotebookDirectory[]];
$HistoryLength = 2;
ClearAll[RefineSphere]
RefineSphere[{fi_, vc_}] := 
 Module[{vcc, max = Max[fi], nfi, avgs, midpoints, newtriangles, 
   newvc},
  vcc = DeleteDuplicates[
    Sort /@ Flatten[Subsets[#, {2}] & /@ fi, 1]];
  nfi = MapThread[Rule, {vcc, max + Range[Length[vcc]]}];
  avgs = Mean[vc[[#]]] & /@ vcc;
  newvc = Normalize /@ (vc~Join~avgs);
  midpoints = Partition[#, 2, 1, 1] & /@ fi;
  newtriangles = 
   MapThread[
    Append[Flatten[#, 
         1] & /@ ({#2, 
          Partition[Sort /@ RotateRight[#1], 2, 1, 1]}\[Transpose]), 
      Sort /@ #1] &, {midpoints, fi}];
  newtriangles = Flatten[Replace[newtriangles, nfi, {3}], 1];
  {newtriangles, newvc}
  ]
refine = 0
sphereFI = PolyhedronData["Icosahedron", "FaceIndices"];
sphereVC = N@PolyhedronData["Icosahedron", "VertexCoordinates"];
{sphereFI, sphereVC} = 
  Nest[RefineSphere, {sphereFI, sphereVC}, refine];
sphereVC *= 40;
Graphics3D[GraphicsComplex[sphereVC, Polygon[sphereFI]], 
 Lighting -> "Neutral"]

Vectors

size = 600;  (* each view will be rendered this size squared *)

vangle = 80 \[Degree]; (* view angle is 80\[Degree] *)

(* calculate the various viewing angles *)

viewvectorup = 
  viewvectors = Normalize[Mean[Part[sphereVC, #]]] & /@ sphereFI;
crosslen = (Norm /@ viewvectors) Tan[vangle/2];
viewvectorup = 
  Normalize[{0, 0, 1} - ({0, 0, 1}.Normalize[#]) Normalize[#]] & /@ 
   viewvectorup;
viewvectorup *= crosslen;
viewvectorright = 
  MapThread[Normalize@*Cross, {viewvectors, viewvectorup}];
viewvectorright *= crosslen;

(* make black-white masks for each view, note that we use 0.96 \
viewing angle here such that each of the views overlaps a bit *)

masks = MapThread[
   Rasterize[
     Graphics3D[{EdgeForm[], White, 
       GraphicsComplex[sphereVC, Polygon[#1]]}, Boxed -> False, 
      Lighting -> "Neutral", ViewVertical -> {0, 0, 1}, 
      ViewVector -> {{0, 0, 0}, #2}, ViewAngle -> (0.96 vangle), 
      ImageSize -> {size, size}, Background -> None], "Image", 
     Background -> None] &, {sphereFI, viewvectors}];

(* this is what all the viewingvectors look like (blue), green is the \
pointing-up vector, and in red is the vector to the right *)
gr = 
 Graphics3D[
  MapThread[{Blue, Arrow[Tube[{{0, 0, 0}, #1}]], Green, 
     Arrow[Tube[{#1, #1 + #2}]], Red, 
     Arrow[Tube[{#1, #1 + #3}]]} &, {viewvectors, viewvectorup, 
    viewvectorright}]]

Functions

vvv = {viewvectors, viewvectorright, viewvectorup}\[Transpose];
normvvv = Map[Normalize, vvv, {2}];
nvvv = Map[Norm, vvv, {2}];
svv = vvv/nvvv^2;
ClearAll[invtransfunc];
invtransfunc[{\[Phi]_, \[Theta]_}, n_] := Module[{vp},
   vp = {Cos[\[Phi]] Sin[\[Theta]], Sin[\[Theta]] Sin[\[Phi]], 
     Cos[\[Theta]]};
   If[vp.vvv[[n, 1]] <= 0,
    vp = (vp nvvv[[n, 1]]/(vp.normvvv[[n, 1]]));
    {vp.svv[[n, 2]], vp.svv[[n, 3]]}
    ,
    {-2, -2}
    ]
   ];

ClearAll[MakeScene]
MakeScene[t_] := Module[{rot, p1, p2},
  rot = 2 \[Pi] t;
  p1 = ParametricPlot3D[{-8.2 + (8 + Cos[v]) Sin[u + t], (8 + 
        Cos[v]) Cos[u + t], Sin[v] + 0.75}, {u, 0, 2 Pi}, {v, 0, 
     2 Pi}, ViewVector -> {{0, 0, 0}, {0, 1, 0}}, 
    ViewAngle -> 80 \[Degree], 
    PlotStyle -> Directive[Green, Opacity[1]], 
    MeshShading -> {{Red, Blue}, {Blue, Red}}, 
    MeshFunctions -> {#4 &, #5 &}, PlotPoints -> 80, Axes -> False, 
    Mesh -> {51, 10}, Lighting -> {{"Ambient", White}}, 
    ViewVertical -> {0, 0, 1}];
  p2 = Graphics3D[{Orange, 
     Sphere[{-8.2 - 8 Sin[2 rot], 8 Cos[2 rot], 0.45}, 0.2]}, 
    Lighting -> {{"Ambient", White}}];
  Show[{p1, p2}]
  ]
ClearAll[Make360]
Make360[scenef_, t_] := 
 Module[{scn, views, antetransform, posttransform, \[Alpha]cs, imgout},
  \[Beta]++;
  scn = scenef[t];
  views = MapThread[
    Rasterize[
      Show[scn, ViewVector -> {{0, 0, 0}, #1}, 
       ViewVertical -> {0, 0, 1}, ViewAngle -> vangle, Boxed -> False,
        ImageSize -> {size, size}, Background -> White], 
      "Image"] &, {viewvectors}];
  antetransform = MapThread[ImageMultiply, {masks, views}];
  posttransform = 
   Table[ImageTransformation[antetransform[[n]], invtransfunc[#, n] &,
      DataRange -> {{-1, 1}, {-1, 1}}, 
     PlotRange -> {{-\[Pi], \[Pi]}, {0, \[Pi]}}, 
     Padding -> Transparent]
    ,
    {n, Length[antetransform]}
    ];
  \[Alpha]cs = AlphaChannel /@ posttransform;
  posttransform = RemoveAlphaChannel /@ posttransform;
  \[Alpha]cs = Binarize[#, 0.95] & /@ \[Alpha]cs;
  posttransform = 
   MapThread[SetAlphaChannel, {posttransform, \[Alpha]cs}];
  imgout = ImageCompose[First[posttransform], Rest[posttransform]];
  imgout = RemoveAlphaChannel[imgout];
  imgout
  ]

Calculation/Rendering/Export

SetDirectory[NotebookDirectory[]];
Dynamic[\[Beta]]
\[Beta] = 0;
n = 150;
t = Most[Subdivide[0, 1, n]];
fns = "out" <> ToString[#] <> ".png" & /@ Range[Length[t]];

CloseKernels[];
LaunchKernels[4];
DistributeDefinitions[Make360, MakeScene, n, t, fns, i, vvv, normvvv, 
  svv, nvvv, invtransfunc, masks, size, vangle, viewvectors, 
  viewvectorright, viewvectorup, sphereFI, sphereVC];
SetSharedVariable[\[Beta]];
ParallelEvaluate[$HistoryLength = 2];


ParallelDo[
 If[! FileExistsQ[fns[[i]]],
  out = Make360[MakeScene, t[[i]]];
  Export[fns[[i]], out];
  ]
 ,
 {i, 1, Length[fns]}
 ,
 Method -> "FinestGrained"
 ]
Attachments:
POSTED BY: Sander Huisman
Answer
1 year ago

A final note: the rendering took a LOT of time (order of 10 hours). The spherical to x-y coordinates calculation can probably be sped up using compile. Furthermore, instead of rendering 20 images, it would be nice to make a function that for each θ-φ pixel chooses the 'right' view and get that specific pixel, that would speed it up ~20x.

And a Python script was used to 'inject' some metadata for YouTube to recognize it as 360 degree video: Have a look here: 360 metadata youtube

The scripts have changed slightly since they posted that video and you have to call (also through the Terminal) gui.py, this will show you a small interface in which you can select your video and have it injected with some metadata. Then simply uploading it to YouTube and they do the rest.

Creating the sphere can now be done much easier in version 10, by using regions:

DiscretizeRegion[Sphere[], MaxCellMeasure -> 1]
DiscretizeRegion[Sphere[], MaxCellMeasure -> 0.1]
DiscretizeRegion[Sphere[], MaxCellMeasure -> 0.01]

This will also create the polygons that are more and more refined, it looks like it is using the same algorithm to refine!

POSTED BY: Sander Huisman
Answer
1 year ago

Awesome, simply awesome, @Sander Huisman, I've just read about these yesterday and you just happen to run in with this beautiful post! Thanks so much for sharing, I'll pass this along to my various connections.

POSTED BY: Vitaliy Kaurov
Answer
1 year ago

Thanks! Glad you like it! I'm not yet sure if I will use it for something useful, but now I have some idea how to do it! It, however, take a LOT of time! So if you want to do higher resolution and more frames it will take considerable amount of time! Some renderers (like 3DS max) can render immediately in 360 degree format, the Wolfram language not yet! But that does make it more fun ;)

POSTED BY: Sander Huisman
Answer
1 year ago

Wow! Clap! Clap! Clap! This can only be the start of something big. How complicated is the python script that you need to upload to youtube?

POSTED BY: Matthias Odisio
Answer
1 year ago

Hi Matthias,

The python script is VERY simple (at least on Mac). But should be equally simple on Linux, windows: not so sure. Just download it here: https://github.com/google/spatial-media and follow the video to change the permissions chmod ....... and the run it ./gui.py in the window you can browse your file and resave it. DONE

POSTED BY: Sander Huisman
Answer
1 year ago

Thanks for the pointer!

POSTED BY: Matthias Odisio
Answer
1 year ago

This is what it should look like, and that is about all you need to do:

enter image description here

POSTED BY: Sander Huisman
Answer
1 year ago

Sander, I tried the above code but looks like my outputs are always coming out zero. No errors are there so not sure what is missing. Any tips?

Best, Vijay

POSTED BY: Vijay Sharma
Answer
1 year ago

Hi Vijay,

If you follow the code in the first post above, then the image is stored inside imgout. in general, if the code ends on a semi-colon (;) it will not show output unless an explicit Print command is somewhere in the code.

If you run code from the second post, then the images are stored as png in the same folder as the notebook.

Note that the code is not very fast, so be prepared to wait a bit...

POSTED BY: Sander Huisman
Answer
1 year ago

enter image description here - another post of yours has been selected for the Staff Picks group, congratulations !

We are happy to see you at the tops of the "Featured Contributor" board. Thank you for your wonderful contributions, and please keep them coming!

POSTED BY: Moderation Team
Answer
1 year ago

Thanks!

POSTED BY: Sander Huisman
Answer
1 year ago

Group Abstract Group Abstract