Part of the problem is that the graphics data contain polygons with holes, which are not supported for vector export:
Export["holes.pdf", Graphics[Polygon[{{0, 0}, {1, 0}, {1, 1}, {0, 1}} -> {{1/2, 1/2}, {3/4, 1/2}, {3/4, 3/4}}]]]
The short answer is, they cannot. While the result from evaluation contains graphics primitives (instead of a raster), the renderer does not support writing them out as such to PDF and SVG. So instead they are rasterized. This is similar to what happens when you use HatchFilling and friends in 2D (which is implemented as a pixel shader).
HatchFilling
