This is the form of your input (where each vec is a possibly different 3-vector):
{{vec, vec}, {vec, vec}}
This is an example of the form that is needed for the special case of only two by two vector values:
Table[vec, {x, 0, 1, 1}, {y, 0, 1, 1}, {z, 0, 1, 1}]
gives
{{{vec, vec}, {vec, vec}}, {{vec, vec}, {vec, vec}}}
Note that above each {{vec,vec},{vec,vec}} is a 2x2 planar set of 3-vectors (the first in the x-y plane at z=0the second in parallel to the x-y plane at z=1.
Note that your example will not work if we just put an additional set of list brackets around it since is would only comprise vector field values in a single plane and so the interpolation will have nothing to work with in the 3rd dimension. But this contrived example does work (it replicates your data twice in the z-direction):
ListVectorPlot3D[{{{{1, 1, 1}, {2, 3, 5}}, {{2, 3, 1}, {6, 3, 1}}}, {{{1, 1,
1}, {2, 3, 5}}, {{2, 3, 1}, {6, 3, 1}}}}]