The FontFamily seems to work on both axes, I'm not detecting differences between
ListLinePlot[Prime[Range[25]], Filling -> Axis, PlotStyle -> LightGray, BaseStyle -> {FontFamily -> "Consolas", FontSize -> 40}]
and
ListLinePlot[Prime[Range[25]], Filling -> Axis, PlotStyle -> LightGray, LabelStyle -> {FontFamily -> "Consolas", FontSize -> 40}]
especially the 20 looks on y-axis the same like on x-axis, doesn't it?