I don't have much experience with images but before someone competent answers we can implement what you just said:

pic = ExampleData[{"TestImage", "Mandrill"}];

nf = Nearest[List @@@ {Red, Green, Blue, Yellow}];

ImageApply[First@*nf, pic]

I don't quite think so, but I could also just be confused. I'd like to have a (short) list of RGB colors and somehow tell Export to only use those colors. But maybe there's a way to do this with ColorFunction that I just don't understand?

When exporting to GIF, Export obviously has to discretize the colors in the frames somehow. I would just like to be able to give it guidance: to tell it something like "convert every color in the Graphics object I'm giving you to the nearest match from this list of colors".

Now, I realize I could do this myself by Rasterizeing and manipulating the list of pixels, but I'm wondering if there's something built in.