Dear @Martijn, I agree with Ahmed -- it would be lovely to see your work published on Community -- looks very impressive.

Yes!! that solved all the issues I had. And somehow it also fixed the issue with the wrong colors with grouped primitives.

So in summary, I now us the following:

Graphics3D[{CapForm["Square"] ,JoinForm["Miter"] ,Tube[...], Method -> {"TubePoints" -> {6, 2}}]

With the grouped primitives it allows for very smooth images and image manipulation even up to 5000 tracts and tubes simultaneously.

Here are some nice images as a thank you.

Other than the CapForm and JoinForm setting which makes memory consumption difference, the "TubePoints" setting {5, 1} is an invalid specification which causes data cache invalidation all the time. Use {5, 2} instead will improve the rotation performance greatly.