The method option "VectorMapRemoveLabels" is an experimental/undocumented option, used internally, and it may change at any time (with a nice documented syntax). I talked a bit about that in my presentation at the WTC 2021.
I see, even when all that data is somehow in the map, the vector format has as main goal to "display" things, more than being able to compute with them. About the elements in the map, even when they are not meant for computing anything (the main goal of the vector format is the nice display), you can extract them from the GeoGraphics output by using the pattern Annotation[primimitives_, {type_, name_, layer_, opts___}, "WolframGeoGraphics"], so you can extract them and play with the ones you want. Something like:
Cases[GeoGraphics[...],
Annotation[primimitives_, {type_, name_, layer_, opts___}, "WolframGeoGraphics"] :>
({type, name, layer} -> primitives), Infinity]
The primitives use the projected space coordinates, and you can convert them using GeoGridPosition and GeoPosition if needed.
I also explained some of this internal structure in my talk at WTC 2020 (which is already published: https://www.wolfram.com/broadcast/video.php?c=104&p=3&v=3233, https://notebookarchive.org/geodata-by-francisco-javier-rodriquez-arias--2020-10-4nd9yhx/ ). Again, all this is part of the internal structure of GeoGraphics, and may get updates or changes in future versions.
