I completely agree. This would be most appropriate if the Graphics were somewhat modular, and you could move and manipulate subsections/layers.
I too have a Vive and can "easily" see implementation within Mathematica.
