I'm unsure i'm new to 11.0 myself
obviously, turn off automatic things you didn't above (Clipping, Axes, anything that might get evaluated while rotating)
this works pretty smoothly:
Graphics3D[
Table[Sphere[{Random[], Random[], Random[]}, 0.01], {i, 0, 1000}]]
but it's 1,000 objects - each object has many "texels" in 3d lingo (each ball has a whole plane of vertices). video games typically have high vertex count but low(er) object count, and objects at a distance are specially obstructed / blurred because: they aren't fully being rendered.
50,000 objects is allot. I think there is some thing here where your drawing 50,000 objects not a single mesh model with 50,000 vertices (say, a mesh of some object like and airplane) and expecting 50,000 rendered shaded objects to be the same speed as 50,000 vertexes object who's surface is shaded 1x. rotating the two would not be the same "job".
i see the same thing you do but usure if that is "blazing fast" or "slow and un-optomized" (if it re-evaluates all per degree of rotation to avoid storing sets of points in memory?). i am pretty sure that any slowness is like mm adding things like Axes or other extras who's viewing quality are preffered to flat out speed.
interesting question