You can control the tick lengths if you specify them yourself using the FrameTicks option. In this case, the design of the ticks is specified by the syntax given by Ticks. (See both in the help system.) Unfortunately, unlike Thickness, there does not appear to be a style option for length. When you use FrameTicks, you have to design the ticks yourself -- by specifying the position, label, and length of every tick mark. I don't see a way to make rounded tick marks, other than through primitive graphics elements.