Thought I'd post something I was struggling with in order to get Mathematica (and any sub-menus it might launch) to display correctly on high resolution monitors in Linux.
The default ScreenResolution is 72dpi in a newly installed Mathematica. My monitor (and almost all modern monitors) use a higher resolution. In my case 96dpi. This means that everything rendered is tiny (it renders to 72/96 of what it was intended to run at). Controlling some fonts fails to control sizes in menus, icons, and tool bars in general. Bits and pieces of Mathematica can be enlarged through font or ScreenResolution changes, but it is difficult at best to get even part of what is too small back to a normal size. So here is how to fix this with a single step...
Setting environment variables like this will fix the issue...note that 96/72, the dpi of actual monitor versus what seems to sometimes be hard coded into Mathematica, is about 1.33. This will correct the issue if starting on command line:
QT_AUTO_SCREEN_SCALE_FACTOR=1 QT_SCALE_FACTOR=1.33 Mathematica &
Starting Mathematica like this will allow the palettes, icons, buttons, and window frames to enlarge uniformly.