A PlotRange specification will not clip the graphics as you already found out. Unfortunately there is no 'clipping' primitive as of yet, though I have requested it. In this case you can solve it by using a 3rd argument in Circle. This will not work in general for any primitive unfortunately...
Graphics[{Yellow, Rectangle[{-1, -1}, {1, 1}], Black, Thick,
Circle[{0, 0}, 0.9, {0, Pi/2}]}, PlotRange -> {{-1, 1}, {-1, 1}}]
Sometimes you can get away by using Overlay, or rasterizing your Graphics and then using ImageOverlay. Or using RegionUnion and (discretised) versions of your primitives along with your a rectangular viewing range region. To sum up: you have to be creative sometimes!